AegisAI Perspectives · 091
Score email security vendors in an RFP with weights fixed before demos, a 1 to 5 scale with written meanings, and a live-mail trial on your own tenant.
Score email security vendors on a weighted sheet you fix before the first demo, with each criterion rated 1 to 5 against meanings written in advance9. Pass or fail gates come first: one failed mandatory requirement disqualifies a bid18. On AegisAI, you connect a tenant, run it in monitoring mode and compare it with what your current stack delivered22. The criteria themselves are covered in how to evaluate email security vendors and the 12-point checklist for choosing an email security solution.
Write every rule down before any vendor presents, so no pitch shapes the scorecard.
List the mandatory requirements as pass or fail: support for your platform (Google Workspace, Microsoft 365 or both), written data handling terms, and a reason shown with every verdict. A bid that fails one gate is out.
Agree four pass targets before any demo: detection, false positives, remediation time, admin effort per week3.
One published split gives 35% to detection, 20% to remediation and reporting, and 15% each to three more: integration and identity; outbound and authentication; operations and data1. Adjust the split to your own requirements instead of copying it2.
Write what each score means for each criterion before scoring starts. On the common 1 to 5 scale, 5 is Exceptional and 1 is Insufficient.
Give each shortlisted option two to four weeks on live mail covering finance, executives, HR and a sample of general users, with the platform's own protection unchanged4.
Each evaluator scores independently first, a step that reduces bias and early favorites19. Discuss any score that differs by 2 or more points11.
Weighted score is weight times score: a requirement weighted 3 and scored 4 yields 1210. Record the evidence beside every number.
Use the sheet below as the RFP's scoring table: five criteria, starting weights from one published split, and the evidence that earns each score.
| Criterion | Weight | What a 1 looks like | What a 3 looks like | What a 5 looks like | Evidence to ask for |
|---|---|---|---|---|---|
| Detection on your own mail | 35 | Flags little beyond what your current filter already caught | Flags some delivered threats, misses AI-written lures or requests with no link | Flags the confirmed threats your current stack delivered, AI-written samples included | Trial report listing each flagged message with its verdict |
| Remediation and reporting | 20 | An admin removes each message by hand, one mailbox at a time | Removes messages in bulk, reports only by export | Removes every copy across mailboxes and logs each action | Removal log from the trial, with timestamps |
| Integration and identity | 15 | Needs a mail routing change you cannot reverse quickly | Connects, but permissions are not written down in advance | Connects through the platform API with written, revocable permissions | List of API permissions and the admin consent screen |
| Outbound and authentication | 15 | Ignores the authentication results (SPF, DKIM, DMARC) | Reads authentication results without explaining them | Reads every authentication result (SPF, DKIM, DMARC) and shows it in the verdict | A sample verdict with its authentication results |
| Operations and data | 15 | Verdicts carry no reason, data terms are unclear | Some verdicts carry a reason, data terms on request | A plain reason with every verdict; retention and training terms in writing | Data processing terms, retention period, sample verdicts |
Move points toward the gaps your own incidents show. The authentication row rests on SPF, DKIM and DMARC, which NIST SP 800-177 describes as mechanisms for authenticating a sending domain, and which it calls useful for small and medium sized organizations21.
Score the trial only on mail your current stack delivered, since a threat the existing filter already stopped proves nothing new. Run each option in monitor mode on the same mailboxes for the same weeks. Count a detection when an option flags a delivered message that your team confirms as malicious.
Count a false positive when an option flags a message your team confirms as legitimate, such as a real invoice. Log the admin minutes each option needed per week, the third pass target from step 2. Then apply one rule to every row:
Add them, because AI-written lures now draw clicks at the rate expert-written ones do. In a study of 101 participants, fully AI-automated phishing emails reached a 54% click-through rate, against 12% for the control group15. Emails written by human experts also reached 54%, and AI-written phishing with a human in the loop reached 56%16.
The same study found a language model scored well above 90% at detecting those phishing emails, with low false positive rates17. Get written sign-off from your mail admin, then send a labelled set from a test domain you control to the roles named in step 5.
Include requests with no link and no attachment, such as a payment change. Send the same set to every option in the same hour. Score it as its own line under detection, so a tie can be broken on it later.
Break a tie on the criterion with the most weight, then on the strength of the written evidence. Write the tie rule before scoring, as you did the weights. Reconcile any evaluator gap of 2 or more points first, because a tie built on an unreconciled score is not a real tie.
Then compare the detection row, the 35 point criterion. If those match, compare the AI-written sample line. Decide in advance whether price only breaks ties or carries its own weight. Federal solicitations must state whether all non-cost factors combined are significantly more important than, about equal to, or significantly less important than cost or price13.
Illustrative example, figures are fictional. A 300-mailbox company on Google Workspace scores three unnamed options, A, B and C, with the starting weights from the sheet above. On integration they give Option C a 2 and a 4, discuss the gap, and settle on 4 after reading C's list of API permissions.
Option A scores 4, 3, 5, 4 and 3 on the five rows: 140 + 60 + 75 + 60 + 45 = 380 of 500. Option B scores 5, 4, 3, 3 and 4: 175 + 80 + 45 + 45 + 60 = 405. Option C scores 4, 5, 4, 3 and 4: 140 + 100 + 60 + 45 + 60 = 405.
The tie rule sends it to detection: B scored 5 and C scored 4, so B ranks first.
Email security vendor scoring is a written method that turns each RFP answer and trial result into one number per criterion, weighted by how much that criterion matters to you. Federal rules make the same demand on quality: FAR 15.304 says the quality of the product or service shall be addressed in every source selection through non-cost factors14.
Choose the baseline that lets you trial every option without risk to mail flow, and score all of them against it. A monitor-mode trial runs on live mail with the platform's own protection unchanged, so pick that shape wherever an option supports it. If a shortlisted secure email gateway needs a routing change for its trial, write the rollback plan into the integration row. Give no points for architecture type; score the outcomes in the sheet.
The core criteria are the five weighted rows of the sheet, with mandatory gates in front of them. For a lean team with no SOC, keep a reason with every verdict as a gate, so each quarantine can be checked without a separate investigation. Keep admin effort per week inside operations as a target agreed before demos.
Put criteria in rows and options in columns, multiply each score by its weight, and sum each column. Two weighting styles work: percentages that total 100, as in the sheet above, or multipliers where critical requirements get 3x, important ones 2x and nice to haves 1x. Pick one before the RFP goes out and keep it for every option. Store each evaluator's raw scores beside the reconciled ones, so the matrix shows where consensus moved a number.
Treat each framework your company answers to as a gate, and keep the weighted points for performance. List which of SOC 2, HIPAA or PCI-DSS applies to you, ask each option for the documents your auditor will want, and mark the gate pass or fail.
Keep four records: the weights and gates, dated before the RFP went out; each evaluator's independent scores; the reconciliation notes; the trial evidence behind each score. Federal practice sets the floor: all factors and significant subfactors that affect award, and their relative importance, must be stated in the solicitation, while the rating method need not be disclosed12. A private buyer is not bound by FAR.
AegisAI connects to Microsoft 365 or Google Workspace through the API in read-only mode, with no change to mail flow23. Every verdict comes with its reason in plain language, so your team can check it and share it25. See your results in five days24.
Track three numbers from the first day of the trial, each kept where an auditor can see it.
A monitoring-mode trial of AegisAI starts from the trial request page.
Fix the gates and weights before any demo, along with what 1, 3 and 5 mean. Score each option alone, reconcile gaps of 2 or more points, then multiply weight by score and total. Run the leaders on your own mail in monitor mode for two to four weeks and score the trial on the same sheet.
Two to four weeks on live mail, in monitor mode, with your platform's own protection unchanged. Cover finance, executives, HR and a sample of general users, so the trial sees the mail those roles receive.
In a 101-person study, fully AI-automated phishing drew a 54% click-through rate, against 12% for the control group. Send a labelled set from a test domain you control and score it as its own line under detection.
One published split gives detection 35% of the total. Treat that as a starting point and move points toward the gaps your own incidents show. Whatever you choose, fix it before the first demo.
Federal buyers must state every factor and significant subfactor that affects award, and their relative importance, while the rating method need not be disclosed. That rule comes from FAR 15.304 and binds federal procurement only.
Break the tie with the rule you wrote before scoring. Reconcile any evaluator gap of 2 or more points first, then compare the highest-weighted criterion, then the AI-written sample results. Decide in advance whether price breaks ties or carries its own weight.
