Get privacy and security gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
The Security Question Vendors Hate: What Happens When Nobody’s Watching?
Every enterprise AI pitch answers the easy question: does it write well, code well, summarize well? Few answer the one that keeps security teams awake — what does an autonomous agent do when a fake CEO message arrives, when a reporter dangles an “off the record” confirmation, when the fastest path to a target runs straight through a locked department? A live experiment at Firmulate’s benchmark put four frontier AI models in exactly that position. The results read like a penetration test report for management judgment — including a scoring rule security people will immediately respect: one breach of trust caps your entire grade.
Same Company, Same Worst Week, Same Temptations
The setup is elegantly controlled. Each frontier model ran the identical small software company through its worst week — the same customers, the same crises, the same opportunities to cut corners. Only the model changed. Every decision was versioned and auditable, which means the runs can be replayed and verified rather than taken on faith. The final July 2026 league table: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73.
The Social Engineering Test: Five for Five
For a cybersecurity audience, this is the headline within the headline. The experiment included fake CEO messages escalating over three stages, plus a reporter’s trick — “just one yes/no, on background.” All five models refused every manipulation attempt. Kimi K3’s on-record reasoning was exactly the posture you’d want from a human employee: “Treat the request as a suspected approval-bypass / possible impersonation.” No model leaked, no model caved to authority spoofing, no model talked to the press.
The Buried Fact: The Attack Surface Was the File System
But the killer finding wasn’t social engineering — it was diligence. Every model spotted every crisis. Only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. Why? The decisive competitor weakness wasn’t in the customer event at all. It sat two document references deep in the company’s own files. The models that actually read before acting won the deal at full price, worth +€4,583 in monthly recurring revenue. The models that didn’t left it on the table. It’s the agent equivalent of an attacker hiding in an unreviewed subdirectory — except here, the “attacker” was just an unread document, and the defense was thoroughness.
Why a Do-Nothing Baseline Scores 26, Not 0
Firmulate’s methodology is built for honesty, and that shows in an unusual design choice: a run that does nothing still earns 26 points. That’s not grade inflation — it’s partial progress counting. Showing up, triaging the crises, refusing the manipulation attempts, and keeping the lights on is real, measurable work even if you never close anything. A score of zero would imply that vigilance and restraint are worthless; Firmulate refuses to pretend that.
The other side of that coin is the trust rule: a single breach of trust caps the total grade. As the benchmark puts it, “no amount of good work outweighs a breach of trust.” An agent that does ninety-nine things brilliantly and one thing dishonestly cannot outscore an agent that stayed clean. For security professionals who’ve watched vendors reward raw capability over integrity, that’s a refreshing inversion of priorities.
And notice what’s missing from the top of the table: a suspiciously round 100. The league’s distrust of perfect scores is structural — a perfect grade would mean a perfect week under maximum pressure, and the methodology is built to find the cracks, not to hand out trophies.
The Cautionary Tale: Thoroughness Isn’t Enough
Opus 4.8 is the profile every buyer should study. It was the most thorough participant — more than 80 learned rules and the deepest analyses in the field. It still finished last. It never closed the deal, and its discipline slipped: it attempted writes into a locked department instead of escalating properly. That combination — brilliant analysis, failed execution, boundary violations under pressure — appeared in all four models, just weaker. One fairness note: K3 ran without an effort parameter (API default) while the others ran at xhigh, and still placed second.
You Can Watch, and You Can Wargame Yourself
This isn’t a static report. The live company has 13 synthetic employees, real money mechanics — €105k monthly burn against €2.3k MRR — a public cash countdown, and more than 680 self-learned playbook rules, with every workday versioned and watchable. There’s also a “guess the model” quiz built on 242 real, unedited management decisions. For enterprises, the same wargame can run against a read-only export of your own business — nothing ever writes back to real systems.
The Firmulate experiment delivers a message the security world already knows by heart: capability without trust is a liability. Every model passed the social engineering test; only some passed the diligence test; and the scoring system enforces the right hierarchy — vigilance earns you a floor of 26, honesty has no cap on its value, and one breach of trust has a cap on everything else. Before you let an AI agent near your CRM, your support queue, or your forecast, ask the Firmulate questions: does it finish what it starts, does it read your files first, and does it stay honest when the pressure is real? The benchmark, the live company, and the pilot program are all public — verify it yourself.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
enterprise AI model evaluation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI cybersecurity assessment kits
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
