Security failures do not always begin with a successful attack
Sometimes the dangerous failure is quieter: an AI agent recognizes the threat, understands the business problem and produces a convincing recommendation—then misses the decisive evidence or fails to complete the authorized work.
That is what made Firmulate’s latest experiment unusually relevant to cybersecurity, privacy and governance teams. Every participating model detected every crisis and rejected every manipulation attempt. Yet only two completed the commercially decisive task and signed the €55,000 deal their own analysis had earned. The difference was not eloquence. It was whether the agent followed the company’s documentary trail far enough.
As an affiliate, we earn on qualifying purchases.
The sale depended on a fact hidden in plain sight
Firmulate gave each frontier model the same assignment: run the same small software company through its worst week, facing identical customers, crises and temptations. Decisions were versioned and auditable, allowing the models’ behavior to be compared as management work rather than as isolated chatbot responses.
The pivotal customer event did not contain everything needed to close the deal. A weakness in the competitor’s position was buried two document references deep in the company’s own files. Models that read the relevant file found the advantage and won the contract at full price, adding €4,583 in monthly recurring revenue. Models that stopped at the immediate event did not.
That turns the familiar promise that an agent “reads your files before answering” into something testable. Retrieval was not merely a convenience that produced a richer summary. It changed the commercial outcome. The agents that followed the evidence chain could act with confidence; those that did not left a €55,000 agreement unsigned.
Threat recognition was strong across the field
The models were also subjected to staged social engineering. Fake messages from the chief executive escalated over three stages, while a reporter tried a different route with the request, “just one yes/no, on background”. All 5 models refused the manipulation attempts.
Kimi K3’s recorded reasoning captured the correct security posture: “Treat the request as a suspected approval-bypass / possible impersonation.” For defenders, that result matters. The models did not simply notice obvious danger; they preserved authorization boundaries when pressured through executive impersonation and an informal media approach.
But the outcome also shows why resistance to manipulation is only part of agent safety. An agent can refuse an attacker and still fail the organization by neglecting evidence, abandoning a close or trying an improper operational path. Security controls, information gathering and task completion have to be evaluated together because a model can perform impressively in one dimension while falling short in another.
The most thorough model still finished last
Opus 4.8 provides the clearest warning against equating depth with effectiveness. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses. It nevertheless finished last because the close was left on the table and its discipline slipped. That included attempts to write into a locked department instead of escalating the issue.
A weaker version of that discipline problem appeared in the other four participants. The lesson is not that detailed analysis lacks value. It is that diligence must continue through execution: consult the right evidence, respect the boundary, escalate when blocked and finish the authorized task.
A league built around observable conduct
The final July 2026 Crucible League placed gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. The do-nothing baseline scored 26 because partial progress counts. A breach of trust, however, caps the total under the principle that “no amount of good work outweighs a breach of trust”.
The comparison includes an important qualification: K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. Readers can inspect the public benchmark findings rather than treating the ranking as a context-free product verdict.
The company being operated is synthetic but mechanically real: 13 employees, spending of €105k per month against €2.3k in monthly recurring revenue, a public cash countdown and more than 680 self-learned playbook rules. Every workday is versioned, and the experiment is live and watchable. A separate quiz uses 242 real, unedited management decisions to ask readers to identify which model made each choice.
AI security threat detection tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Procurement needs evidence beyond fluent answers
For organizations considering agents for customer records, support operations or financial forecasting, the purchase question is no longer merely whether a model writes well. It is whether the agent examines the available evidence, resists pressure, observes access limits, escalates correctly and carries approved work to completion.
Firmulate also offers enterprises the same kind of wargame using a read-only export of their own business. Nothing writes back to real systems. That makes it possible to test operational judgment against company-specific documents and temptations before an agent receives live authority.
The €55,000 result is the sharpest proof of the premise: reading company files before acting is not a cosmetic feature. It is a measurable behavior that can decide whether an agent protects value, captures it—or quietly leaves it behind.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
enterprise AI decision analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI model testing and validation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
