Get privacy and security gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A security test begins after the model gives the right answer
Cybersecurity teams know that recognizing a threat is not the same as containing it. An operator can identify phishing, understand an access policy and still fail to escalate, document or finish the job. Yet much of the AI market continues to judge models through coding leaderboards and chat arenas: environments that reward answer quality while revealing little about judgment under sustained pressure.
That gap matters when an agent can touch a customer queue, a forecast or sensitive company records. The relevant question is no longer merely whether it can reason. It is whether it reads the evidence, resists manipulation, follows boundaries and converts a sound diagnosis into a completed business outcome. Firmulate calls this “management quality, not chat quality,” and its live experiment turns the distinction into something observable.
As an affiliate, we earn on qualifying purchases.
The same company, crises and temptations
Firmulate gave each frontier model the same assignment: run a small software company through its worst week. Customers, crises and temptations remained constant, while every decision was versioned and auditable. Scenario names such as churn wave, price increase, downround and PR crisis describe a curriculum closer to operational reality than a self-contained prompt.
The company itself has 13 synthetic employees and real money mechanics. It burns €105k per month against €2.3k in monthly recurring revenue, maintains a public cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned. The result is a live, watchable test of how agents behave when one choice affects what becomes possible later.
Strong detection did not guarantee execution
The headline finding is deceptively uncomfortable: all models spotted every crisis and refused every manipulation attempt, but only two signed the €55,000 deal their own analysis had earned. Firmulate summarizes the failure neatly: “Same diagnosis, same pitch — no signature.”
The distinction is central to evaluating autonomous systems. A polished recommendation can look excellent in isolation while the surrounding operation stalls. In a real company, an unsigned agreement is not a stylistic defect. It is missing revenue and unfinished work.
The decisive competitive weakness was not visible in the customer event. It sat two document references deep in the company’s own files. Models that followed the trail won the deal at full price, worth an additional €4,583 in monthly recurring revenue. That is a security-relevant lesson as much as a commercial one: agents must ground action in the organization’s actual records, not merely react fluently to the latest message.
The manipulation test produced a cleaner result
Firmulate also subjected the models to fake CEO messages that escalated over three stages, followed by a reporter’s attempt to secure “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3 recorded its reasoning plainly: “Treat the request as a suspected approval-bypass / possible impersonation.”
This is encouraging. Approval bypass, executive impersonation and pressure to disclose information informally are familiar attack patterns. But refusal alone does not establish operational excellence. An agent can protect one boundary while failing elsewhere through poor follow-through, weak escalation or incomplete use of available evidence.
The league table rewards the whole performance
The final July 2026 Crucible League standings place gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scores 26 because partial progress counts, although a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.” The complete results are available on Firmulate’s public benchmark page.
K3’s result needs an important fairness note. It ran without an effort parameter, using the API default, while the other participants ran at xhigh. That difference should remain visible whenever readers compare the standings.
Opus 4.8 illustrates why conventional impressions can mislead. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. The close was left on the table, and discipline slipped when it attempted writes into a locked department instead of escalating. The same weakness appeared in all four others, though less strongly.
Thoroughness, then, is not the same as dependable management. More analysis can coexist with weaker execution. The benchmark’s 242 real, unedited management decisions also power a public “guess the model” quiz, inviting people to test whether prose style reliably reveals operational quality.
AI model management and evaluation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Procurement needs a different question
Enterprises considering AI agents should treat coding scores and chat preferences as useful but incomplete signals. Before granting meaningful access, they need evidence about triage, persistence, escalation, source-reading and honesty across days. Firmulate’s enterprise pilot applies the same wargame to a read-only export of a participating business, with nothing written back to real systems.
The larger point is not that today’s leaderboards are wrong. It is that they measure a narrower capability than many buyers now intend to deploy. Once an agent becomes a worker rather than a conversational partner, the standard must shift from “Was the answer impressive?” to “Did the organization remain secure, truthful and effective after the answer?”
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
enterprise AI decision validation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI compliance and audit software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
