Guide, facts read 3 to 5 November 2026
Reading an AI SOC accuracy claim: sample, method, date
The short answer
An accuracy figure means nothing without a sample, a method and a date. Of 49 figures published by the twelve vendors in this index, four have all three, and two of those are Microsoft's randomised trials from October 2025. This guide shows how to read the other 45.
Three questions grade any figure
Ask of every number: what was counted (the sample), how it was judged (the method), and when (the date). If all three are present and someone other than the vendor did the counting, the figure is grade A in our audit. If the vendor did the counting but published all three, it is grade B. A number with no method is grade C. A number that conflicts with the vendor's own other pages is grade D.
No figure in the November 2026 audit reaches grade A. That is not unusual for a young category, but it means every buyer has to produce their own figure in a proof of value.
| Vendor | Figure | Sample | Date |
|---|---|---|---|
| Microsoft Security Copilot | Malicious emails found up to 550% faster | 167 analysts, 25-email queue | October 2025 |
| Microsoft Security Copilot | 204% more missing zero-trust policies found | 162 identity administrators | October 2025 |
| Intezer | More than 60% of alerts never reviewed | 25 million alerts | AI SOC Report 2026 |
| Mate Security | Blue City: 40.2 to 80.1% effectiveness (does not test Mate) | 32 incidents, five defenders | 2026-11-03 |
A sample size alone is not a method
Mate's home page says 98.7 percent accuracy on 7,200 alerts. The sample size is there. Missing: where the alerts came from, who labelled them, over what period, and what counts as accurate. A chart next to it says frontier models alone score 51 to 72 percent on the same alerts, and its footnote names the test rig for those models but not Mate's own setup. Grade C.
Compare Microsoft's Phishing Triage Agent figure: 167 analysts, a 25-email queue, a randomised controlled trial run in October 2025, malicious emails found up to 550 percent faster. The trial is internal, so it is not grade A. It also measures speed, not whether the decisions were correct, which Microsoft does not state. Grade B, with that caveat attached.
Agreement with analysts is not accuracy
Several vendors measure their agent against human decisions. Legion reports 96 percent accuracy against senior analyst decisions with 0 false negatives. CrowdStrike says Charlotte AI's triage matches Falcon Complete expert decisions more than 98 percent of the time. Neither publishes the sample or the period.
Agreement tells you the agent behaves like your analysts. It cannot tell you whether either is right. If analysts routinely close a noisy alert type as benign and one in a thousand is real, an agent that agrees 99.9 percent of the time has learned the same blind spot. Ask how disagreements were resolved and by whom.
Conflicting figures from the same vendor
Check a headline figure against the vendor's blog, event pages and press releases. Mate's home page shows 97 percent false-positive auto-closure; its own blog says up to 85 percent, and its it-sa 2026 page says 80 percent at one Fortune 500 customer. Dropzone says 400+ security teams on its home page and 300+ organisations in a post. 7AI shows two alert counters on the same page, 14M+ and 23M+.
None of these is necessarily wrong; they may come from different dates. But undated figures that disagree cannot both be the number to plan around.
Benchmarks need to say who was tested
Two vendors publish benchmarks of other products. Mate Labs' Blue City benchmark (2026-11-03) publishes its method in detail: scenario families, a damage formula and a cost model. Five out-of-the-box AI defenders scored 40.2 to 80.1 percent effectiveness over 32 incidents each, at about 0.45 to 4.05 USD per incident. Mate's own system was not attached, so it does not test Mate.
Simbian's home page says the best of 32 products covers 45.1 percent of MITRE ATT&CK tactics. The runner and date are not stated on the home page. Simbian also publishes its own vendor ranking (2026-10-28) placing itself first. A benchmark run by a competitor can be useful; it needs its method public and its conflict stated.
Write the claim you want into the contract
Once you have your own figures from a trial, put them in the order form: the alert types, the agreement or accuracy floor, how it will be measured, and what happens if it drops. A vendor confident in a 98 percent figure should accept 90 percent measured your way.
Customer quotes carry a name but not a baseline
Many figures come from named customers. Legion quotes Iain Paterson, CISO of WELL Health Technologies, on an 81 percent MTTI/R reduction on the most common use case. Prophet quotes Erik Wille, CISO of Cabinetworks, on cutting the alert queue from thousands to dozens. 7AI quotes Mike Baker, CISO of DXC Technology, on an 80 percent reduction in tier 1 analyst time.
A name makes a quote more credible than an unnamed Fortune 500 company, and it lets you ask for a reference call. It still does not say what the baseline was, over what period, or what else changed at the same time. Treat it as a reason to call the customer, not as a figure to plan around. On the call, ask for the before and after numbers and the dates.
Combined figures hide which product did the work
SentinelOne says Purple AI and AI SIEM together cut detection time by 63 percent and remediation time by 55 percent. With two products in one figure and no comparison group, there is no way to tell what the AI analyst contributed. Microsoft's Elanco story credits Defender Experts for XDR, a managed service, for a roughly 50 percent cut in response times. Ask every vendor to separate the agent's effect from the rest of the stack.
Platforms discussed in this guide
- Microsoft Security Copilot, rank 3, 6.50
- Intezer, rank 8, 5.90
- Legion Security, rank 5, 5.95