AegisAI Perspectives · 041
Track pre-delivery catch, post-delivery removals, time to detect, time to respond and user report rate, each read from a report your mail platform keeps.
Track email security metrics in four groups: what gets through; how fast it is found and removed; how users respond; what it costs. Business email compromise reports to the FBI listed $3,046,598,558 in 2025 losses2. Count removals after delivery as well as blocks: AegisAI, for example, reads each delivered email for what it is trying to do and removes the ones that fail from every mailbox they reached. Mean time to detect (MTTD) and mean time to respond (MTTR) run on separate clocks, and each one needs its own start and stop timestamp.
Build the program from the reports your mail platform already keeps, then add one clock per question. Each step names the report or standard that holds the number.
Split the funnel into two lines. Count malicious mail stopped before delivery in the Threat protection status report, which also counts messages hit by zero-hour auto purge (ZAP) and filters up to 90 days10. Count mail removed after delivery from the ZAP removed figure in the Mailflow view of the Mailflow status report.
Start the detection clock at delivery. ZAP only searches the last 48 hours of delivered email13, so every message it removes sat in a mailbox first. Pull each one from Threat Explorer by filtering the Additional action column for ZAP.
Stop the response clock at the remediation record. The Action center History tab lists each remediation with its creation date and status, and keeps 30 days15. Export it before the window closes. Record which action ran: move to junk, soft delete or hard delete.
Do the same in Google Workspace. The security investigation tool reads Gmail log data and lets an admin find and erase malicious emails21, so the erase event is your stop timestamp.
Count what users report. The User reported messages report lists messages reported with the built-in Report button in Outlook, and it needs audit logging turned on. Report rate is reported messages divided by malicious messages delivered.
Measure people with simulations. Actual compromise rate is users compromised divided by users who received the simulation17. Time to report is the Phishing Reported On time minus SuccessfullyDeliveredEmail_TimeStamp in the exported CSV.
Track identity and domain outcomes. The Compromised users report shows accounts marked Suspicious or Restricted in the last 7 days. For your own domain, record the DMARC policy (none, quarantine or reject) defined in RFC 99897, and keep a rua address so receivers send aggregate reports.
AegisAI reads mail through the Microsoft 365 or Google Workspace API after delivery, with no change to MX records22. It investigates every inbound email for what it is trying to do and removes the ones that fail from every mailbox they reached, forwards included. Every verdict comes with its reason in plain language, so the team can check it and share it. It runs in monitoring mode first, and removal needs write access that an admin grants and can revoke.
Keep eight metrics, each tied to the decision it supports. A metric with no decision behind it is a count, and counts belong in an appendix.
| Group | Metric | Formula | Where the data lives | Decision it supports |
|---|---|---|---|---|
| What gets through | Pre-delivery catch | Malicious messages stopped before delivery | Threat protection status report | Tune or keep filtering policy |
| What gets through | Post-delivery share | Messages removed after delivery divided by all malicious messages | Mailflow view, ZAP removed | Decide if post-delivery removal needs more coverage |
| Speed | Time to detect | Verdict time minus delivery time | Threat Explorer, ZAP filter | Find the attack types that sit longest |
| Speed | Time to respond | Removal time minus verdict time | Action center History | Change approval steps or automation |
| People | Report rate | Reported messages divided by malicious messages delivered | User reported messages report | Fix the Report button rollout |
| People | Compromise rate | Users compromised divided by users who received a simulation | Attack simulation report | Target training to groups that fail |
| Identity | Flagged accounts | Accounts marked Suspicious or Restricted | Compromised users report | Start account takeover response |
| Domain | DMARC policy | none, quarantine or reject | DNS record and aggregate reports | Move to enforcement |
Blocked counts alone hide the post-delivery line. A program that only reports total messages blocked cannot show how long a missed attack stayed open, which is the number the response team controls. For the wider list of what security leaders track when AI reads email, see what to measure with AI in email security.
The example below walks one tenant through the eight rows. Every figure in it is invented to show the arithmetic.
Illustrative example, figures are fictional. A 2,000 mailbox tenant runs the scorecard for one month. The Threat protection status report shows 1,400 malicious messages stopped before delivery. The Mailflow view shows 60 messages removed after delivery, so the post-delivery share is 60 of 1,460, or 4.1%. Threat Explorer puts the median delivery to verdict time at 2 hours 10 minutes. The Action center shows a median of 50 minutes from verdict to removal, because each removal waits for an approver. Users reported 18 of the 60 delivered messages, a report rate of 30%. A simulation sent to 400 users compromised 14, a compromise rate of 3.5%, and 3 users are repeat offenders. The team's decision: automate approval for one message type, which targets the 50 minute response clock first.
Most programs report activity counts, such as messages blocked and courses completed. NIST SP 800-55 frames measures as a way to judge the adequacy of the controls already in place4. A blocked count does not do that, because it rises with attack volume and says nothing about what was missed. Training completion has the same gap, because it records attendance and says nothing about behaviour. Pair every number with the decision it changes, as the table above does, and drop the ones that change nothing.
Yes. NIST SP 800-61 Rev. 3 treats detection and response as separate activities to improve5. For email, MTTD runs from delivery to the first malicious verdict. MTTR runs from that verdict to the last copy removed from the last mailbox. One blended number hides where the time goes. A slow MTTD points at detection. A slow MTTR points at approvals, since a remediation waits until someone with the Search and Purge role approves it16. How detection works after delivery is covered in how to cut SOC time spent on phishing triage.
None of the standards cited here sets a target, so set your own from a baseline. Run two simulations a quarter apart and compare report rate and time to report from the exported CSV. Watch repeat offenders: a user compromised by consecutive simulations, with a default threshold of two19. Check which button users press. Reports sent through non-Microsoft tools are not captured in attack simulation reports, so a third-party report add-in leaves those reports out of your measured rate.
Count the attacks that were stopped and how long the missed ones stayed open. A board reads two lines: messages stopped before delivery, and messages removed after delivery with their exposure time. Give the loss context once. IC3 logged 24,768 business email compromise complaints in 20253, out of 1,008,597 complaints across all crime types. Then show your own trend over 90 days, which the console reports already hold. For the questions to put to a vendor about those numbers, see how to evaluate email security vendors.
Pick three numbers and read them from the same reports every week.
Track domain posture beside these. NIST SP 800-177 recommends SPF, DKIM and DMARC for sending domains6, so a DMARC policy still at none is an open item. Passing DMARC does not make a message safe, as why phishing that passes DMARC still lands explains.
Track four groups: what gets through; how fast it is found and removed; how users respond; what it costs. In practice that means eight metrics: pre-delivery catch, post-delivery share, MTTD, MTTR, report rate, compromise rate, flagged accounts plus DMARC policy, each read from a named report.
MTTD is mean time to detect: the average time from delivery of a malicious message to the first verdict that flags it. In Microsoft 365, filter Threat Explorer for ZAP actions to list the messages detected after delivery, then time each one.
A metric is any number you record, such as messages removed after delivery. A KPI is a metric tied to a target and a decision, such as median removal time under one hour this quarter. NIST SP 800-55 treats measures as a test of whether controls are adequate.
Each report keeps its own window. The Compromised users report shows 90 days in aggregate and 30 days in detail. The Action center History tab keeps 30 days of remediation actions. Export monthly so your trend survives the window.
Yes, as a posture metric. Record the policy value, none, quarantine or reject, and the share of your mail passing in aggregate reports. Without a rua address, receivers send no aggregate reports.
Start with three: post-delivery share, median delivery to removal time and report rate. All three come from reports the platform already keeps, so they need no new tool.
