AegisAI Lab · Lab Note 005
Detection accuracy is two error rates. Measure false negatives and false positives on labelled mail, and bound a zero miss result with 3/n.
Detection accuracy is two error rates, the false negative rate and the false positive rate, and a result of zero misses in n attacks only shows the first is below 3/n at 95% confidence. To measure both, replay labelled attacks and legitimate mail through the product and count every miss and every false alarm.
Test labs publish how they chose their samples. The AMTSO standard asks every test plan to state where samples came from and how they were curated. Ask any vendor for the same two answers before you read its catch rate.
Fix the formulas before you collect mail. Detection rate (recall) is TP/(TP+FN) and the false positive rate is FP/(FP+TN). A false negative is an unwanted message the product treated as legitimate.
Build the attack set from mail you have already confirmed as malicious. In Microsoft 365, admins report false negatives from the Submissions page of the Defender portal. In Google Workspace, admins search Gmail log events in the security investigation tool. Label each message with a technique from MITRE ATT&CK T1566, which has four sub-techniques.
Sample legitimate mail from live mailboxes and label each message safe. Include bulk mail and automated notices, because the false positive rate divides by all safe mail.
Keep the delivery path honest. If you allow-list the sender of your test mail, you switch off the filter you are testing. In Microsoft 365, matched messages get no action from filters, zero-hour auto purge, Safe Links blocking or Safe Attachments detonation. In Gmail, mail from an allowlisted IP is not put in spam.
Run the product beside your current stack, in monitor mode, on the same live mail. An API connection makes that possible without an MX record change, which is how AegisAI connects.
Record two verdicts per message: one at delivery and one after the 48 hour auto purge window closes. Zero-hour auto purge searches the last 48 hours of delivered mail and does not notify users. Its removals change counts in Threat Explorer but not in the Mailflow status report. Log a miss that auto purge later removes as a late catch.
Compute the rates for each category on its own. Divide misses by attacks sent for the false negative rate, and false alarms by legitimate messages for the false positive rate. Do not trust accuracy alone: at a 1% attack rate, a filter that flags nothing scores 99% accuracy. Small samples swing hard: with four safe messages, one false alarm gives a 25% false positive rate.
Put a bound on the result. Zero misses in n attacks gives a 95% upper bound of 3/n. With one or a few misses, use the exact binomial interval, because NIST notes that normal approximations are not always accurate enough for very small counts.
Categories carry different costs. The FBI's IC3 recorded 24,768 business email compromise complaints and $3,046,598,558 in reported losses for 2025. It recorded 191,561 phishing and spoofing complaints in the same year. A miss in the first group and a miss in the second are different events, so a blended rate hides the difference.
Labs follow the same logic. SE Labs reports four targeted attack categories separately: business email compromise, phishing, social engineering and malware. Its total accuracy score combines protection and false positives, because a product that blocked all legitimate mail would catch every attack.
Decide the scoring table before the first message is sent. SE Labs awards 10 points for stopping a threat and 5 for the Junk folder. Quarantine earns 6 or 10, depending on whether the user or only the administrator can recover the message. A malicious message in the inbox loses 10 points. Legitimate mail takes the same amounts as deductions when it is junked or quarantined.
Write your own table and give it to every vendor. Then score every product with it. AMTSO tells testers to treat all test subjects fairly and equally, and calls an advance look at the samples for some vendors but not others unfair.
| Measure | Formula | What it tells you | What it hides |
|---|---|---|---|
| Detection rate (recall) | TP / (TP + FN) | How many attacks the product caught | Good mail it blocked on the way |
| False negative rate | FN / (TP + FN) | How many attacks got through | Needs a labelled attack set that is large enough |
| False positive rate | FP / (FP + TN) | How much good mail was flagged | Swings hard on small samples |
| Accuracy | (TP + TN) / all messages | Share of verdicts that were right | Looks high when attacks are rare |
Illustrative example, figures are fictional. A team replays 400 labelled attacks and 4,000 legitimate messages through one product. It misses 6 attacks and flags 20 legitimate messages. The false negative rate is 6 of 400, or 1.5%. The false positive rate is 20 of 4,000, or 0.5%. A second product misses 0 of 100 attacks in a smaller test, which shows only that its rate is below 3/100 at 95% confidence.
Ask for these in writing:
A figure that arrives without these four items cannot be set beside your own test.
Track three numbers each week. Each one comes from a report you already have.
For the deployment questions around a trial, read the vendor evaluation framework. For tuning after a false alarm, read why false positives happen and how to fix the root cause.
It is the share of malicious messages a product treated as legitimate. Virus Bulletin's email test methodology defines a false negative as an unwanted test case identified as legitimate, and a false positive as a legitimate test case identified as unwanted.
Enough to make the bound useful. With zero misses in n attacks the 95% upper bound is 3/n, so it shrinks as n grows. With a few misses, compute the exact binomial interval instead.
Ask for the test plan first. AMTSO expects the test plan to show sample provenance and curation, and vendors to disclose any variance between the tested product and the shipped one. A catch rate without a false positive rate is half a result.
Not to test filters. In Microsoft 365, matched simulation mail gets no action from filters or auto purge, and automated investigation ignores it. A simulation measures people. Use real attack samples to measure the product.
Search delivered mail and label what you find. Microsoft 365 admins report false negatives from the Submissions page. Google Workspace admins search Gmail log events and mark messages as phishing.
Report it on its own line. IC3 logged 24,768 such complaints and $3,046,598,558 in reported losses for 2025. Weighting is your policy, so state the weights before the test and apply them to the category rates.
