Wouldn't my current vendor catch what you catch?
How to run a fair head-to-head email security evaluation in your own tenant, and how to tell a real miss from noise when two tools run side by side.
AegisAI Answers · Part 3
How to run a fair head-to-head email security evaluation
A sharp CIO put it to us directly: "I'm sure you'll find things my current tool misses. But if the roles were reversed, wouldn't it find things you miss? Doesn't it all average out?" It's the right question. The answer is that it doesn't have to be a matter of opinion, because it can be measured in your own tenant.
Questions in this article
- Doesn't every vendor find something the other one missed?
- If you run alongside my current API-based tool, how do you know a catch is a real miss?
- How do I compare the two fairly in the other direction?
- How long does a test take?
- What does the data say across customers?
Doesn't every vendor find something the other one missed?
Sometimes. That's why a claim from any vendor, including us, is worth less than a test. The useful question is not whether misses exist, but what kind of attacks each tool misses and what they would cost you.
In our evaluations, the attacks that slip past established tools are disproportionately the hard ones: messages from trusted vendors whose accounts have been taken over, and multi-stage links hidden behind redirects and CAPTCHAs. Those are the attacks an evidence-based investigation is built to catch.
If you run alongside my current API-based tool, how do you know a catch is a real miss?
API-based tools receive a notification of each new message at the same moment, so no one has a head start. If your existing tool is installed and an AegisAgent finds a phishing email still sitting in a user's inbox, that message is, by definition, one your current stack let through.
How do I compare the two fairly in the other direction?
Existing tools usually move what they catch into a quarantine folder inside your tenant. AegisAI can read that folder, which gives you an apples-to-apples view of overlap even without a direct integration to the other vendor.
Two more ways to test it:
- Export threat logs from both tools for the same period and compare them line by line.
- Once AegisAI has proven itself on live mail, release a sample from your existing tool's quarantine and see whether AegisAI catches it too.
How long does a test take?
AegisAI connects to Microsoft 365 or Google Workspace through the API in read-only mode, with no change to mail flow. Customers typically see their first findings in under a week.
What does the data say across customers?
Across our benchmark evaluations, AegisAI delivered 22% more detections with 90% fewer false positives. Every customer's mail is different, which is why we'd rather prove it on yours.
More from AegisAI Answers
- Part 1: What happens when the attacker hides the payload? (coming soon)
- Part 2: Can I trust AI agents inside my inbox? (coming soon)
- Part 3: Wouldn't my current vendor catch what you catch? (this article)
- Part 4: What happens to my mail flow if you go down? (coming soon)
- Part 5: What happens to my data? (coming soon)
- Part 6: Will outbound DLP drown my team in false positives? (coming soon)
- Part 7: What if the attacker is already inside a trusted account? (coming soon)
- Part 8: How realistic are your simulations, and what about attacks outside the inbox? (coming soon)
Related posts
More from the AegisAI blog.