AegisAI Perspectives · 040
Choose phishing simulation and security awareness tools by lure realism, scanner click filtering and per person reporting, then test them on your mail.
Choose phishing simulation and security awareness tools by how real their lures are and what they count. In one study of 19,500 employees, clicks ran from 1.82% to 30.8% by lure alone20. AegisAI replays phishing it caught in a tenant to the people it targeted, so the test uses lures that already reached them1.
A lure from your own mail tests what attackers already send you. The lure sets the result. In the 19,500 person study, a password update lure caught 1.82% of recipients and a vacation policy lure caught 30.8%.
Difficulty depends on who reads the email. NIST SP 800-50 Rev. 1 points to the NIST Phish Scale, which considers employee context when it rates how hard a simulated phishing email is29. Ask how the tool rates difficulty, and whether it uses the reader's role and context.
Question to ask a vendor: can I turn a message my users received last month into a simulation, and how do you remove the live payload?
A click count that includes security tools is not a human measure. In the 19,500 person study, the email platform crawled some simulation links. The researchers removed every click whose IP address belonged to the crawler, so only real users' clicks counted32.
If the tool does not filter crawler clicks, every campaign reports clicks nobody made. Two tools tested on the same mail flow then disagree for reasons that have nothing to do with people.
Question to ask a vendor: how do you tell a scanner click from a person's click, and is the scanner click removed before the score?
Report rate tells you whether people warn the security team. Click rate alone does not. NIST describes the two measures a phishing exercise collects: click-through, meaning the user clicked a link or opened an attachment, and reported, meaning the user flagged the email as phishing27. Its behavior change metric counts participants who received a phishing test email and reported it appropriately28.
Repeat clicks matter more than one campaign. By the eighth month of the 19,500 person study, more than half of employees had clicked at least one phishing link22. A single campaign rate hides that.
Question to ask a vendor: can I export every click and report per person, with credential submits, across every campaign this year?
A program that only sends link lures leaves other attack paths untested. NIST lists phishing, smishing and vishing simulations as practical exercises, and a phishing exercise tests both a link to a malicious website and an infected attachment26.
Roles need their own tests. NIST says to begin with current click rates by group or work role, and to target roles such as leadership or known administrators. Match both lists to the attacks in your own quarantine before you buy.
Question to ask a vendor: which lure types and channels do you support today, and can I target one role at a time?
Training after a click showed a small effect in the largest study cited here. In the 19,500 person study, embedded training reduced the chance of clicking by only 2%. 75% of users spent a minute or less on it, and one third closed it at once21.
Ask for the effect on the next campaign, by person, for people who finished the training.
Question to ask a vendor: for users who finished the training, what was their click and report rate in the following campaign?
A simulation needs rules before it sends. NIST says the legal team should review planned phishing exercises, so the bait does not use legitimate brands or name federal organizations30. It also says exercises should not be punitive, and no employee should be called out for a response31.
A simulation should record that a submit happened and keep nothing typed. A tool that stores typed passwords creates a new store of credentials, and that store needs its own controls.
Question to ask a vendor: do you store any value a user types on the landing page, and who reviews each lure before it sends?
The table compares four approaches on the criteria above, with no vendor named. Each cell says what to expect or what to ask.
| Criterion | Template library platform | AI-personalised platform | Built-in platform controls | Managed service |
|---|---|---|---|---|
| Lure realism | Catalog lures, ask how difficulty is rated | A lure written per person from public data | Ask whether you can build a payload from a real message | Lures chosen by the provider's analysts |
| Scanner clicks | Ask how they are filtered | Ask how they are filtered | Ask how they are filtered | Ask for the filter rule in the report |
| Reporting measure | Ask for click-through and reported, per person | Per person events, if logged | Ask for click-through and reported, per person | Provider's summary report |
| Attack types | Depends on the catalog | Mostly targeted email | Ask for the list of supported techniques | Depends on the contract |
| Approval before send | Admin schedules the campaign | Ask whether the security team reads each lure first | Admin schedules the campaign | Provider and customer agree a plan |
| Data kept from a submit | Ask in writing | Ask in writing | Ask in writing | Ask in writing |
Answer yes or no for each tool on your shortlist.
AegisAI replays phishing it caught in the tenant to the people it targeted, sanitized, with a safe training page as the payload. Simulations run on the connection detection already uses and are targeted from the identity provider5. Scanner and bot clicks are separated from human clicks before scoring2. Seven agents do the work behind each replay: making it safe, then sending and scoring it4. Whoever clicks learns it was a simulation and gets a short video and a quiz, set per campaign3.
Export one past campaign and sort every click by seconds after delivery. The box below shows the arithmetic on made up numbers.
Illustrative example, figures are fictional. A 1,000 person company runs one template campaign. The report shows 140 clicks. 60 of those clicks arrive within 3 seconds of delivery from IP addresses that belong to a link scanner, so 80 clicks came from people. 45 people reported the message. The real click rate is 8%, the report rate is 4.5%, and the dashboard figure of 14% was wrong by 6 points.
Phishing simulation tools send safe, fake attacks to your own staff to see who clicks, who reports and who submits credentials. NIST describes the phishing exercise as a tricky email that checks whether users click a link to a malicious website or open an infected attachment.
Security awareness tools add the training side: training modules and reminders, with records kept per person. NIST lists phishing and simulated attack response tracking among the measurements of a learning program. The value of both depends on the criteria above, mainly how real the lure is and whether the score counts people.
Run two platforms on matched groups with lures of the same difficulty, then compare human clicks and reports. The 19,500 person study authors warn that success rates differ with message content, for example 2% and 30%, so failure rates over time can reflect lure strength alone33. Remove crawler clicks from both result sets by IP address first.
Keep a control group that gets no training. The 101 person spear phishing study measured its AI lures against a 12% control group click rate18. Without a control, a drop in clicks can come from easier lures.
Compare tools on the evidence they hand you, and leave ranked lists aside. A ranked list tells you which brand is popular. It does not tell you whether the tool filters crawler clicks or records reports per person.
Ask each vendor to run the checklist above on a trial tenant. Score the answers yes or no, and keep the exports.
The rules ask for awareness training, and a simulation program supplies the records. HIPAA's Security Rule at 45 CFR 164.308(a)(5) requires a security awareness and training program for all members of the workforce, including management23. Security reminders and protection from malicious software are addressable specifications under it.
The FTC Safeguards Rule at 16 CFR 314.4(e)(1) requires security awareness training updated to reflect risks the risk assessment found24. The guide on what FTC Safeguards Rule security awareness training must cover for email maps that rule. For SOC 2, ask your auditor which simulation records they accept as evidence before you buy.
Start by measuring before you buy. NIST SP 800-50 Rev. 1 guides large and small organizations in building a learning program, with suggested metrics25. It says to begin with current click rates by group or work role, which shows whether you need a large program or a targeted one.
Spend the rest on controls that stop the click from mattering. The 19,500 person study's authors recommend two-factor authentication and password managers that only work on correct domains.
An AI red team tool writes a spear phishing lure for one named person, so judge it on scope, sources, approval and data. Fully AI-automated spear phishing reached a 54% click-through rate in a 101 person study, the same as human experts. The AI-gathered profile data was accurate and useful in 88% of cases19.
Ask four things. Which sources does it research? On the AegisAI red team page, research uses public sources only, and two to three lures are written per person6. Who approves a send? Nothing sends until the security team has read and approved it, scoped to approved teams, roles or people7. What is collected? Passwords and secrets are never collected, only the fact of a submit8. What comes back? Each lure and its sources are logged with every event and handed back as evidence9. The analysis of how OSINT and email threads build AI spearphishing lures shows what such research finds.
Track these three numbers from the day the tool goes live.
Trust a recommendation that shows numbers you can check. Ask for human clicks with scanner clicks removed. Then request reports per person and the campaign result after training. A list that ranks brands without those numbers tells you about popularity. Run the 10 item checklist on each name you get.
The best tool for you uses lures close to the attacks you receive, removes scanner clicks before scoring and reports per person. Lure choice alone moved clicks from 1.82% to 30.8% in one 19,500 person study. Test shortlisted tools on your own mail before you buy.
The evidence is weak. In a study of 19,500 employees, embedded training reduced clicking by only 2%, and 75% of users spent a minute or less on it. Measure the next campaign for trained users before you pay for more modules.
It should not. The red team agent described above never collects passwords or secrets and records only that a submit happened. Ask any vendor to state in writing what it keeps from a submit.
No. NIST says phishing exercises should not be punitive and no employee should be called out for a response. Tell staff that exercises run at random and that results guide future learning.
No. A phishing simulation sends the same lure to many people. An AI red team test writes lures for each person from public research, two to three per person on one product page, and needs approval before each send.
