AI’s next bottleneck is control, not capability
An AI assistant can draft a bad answer. An AI agent can send it, alter a database, spend money or push code before a person reviews the result. Capability determines what a model can do. Control determines what it is allowed to do, when a human must intervene, what evidence it must leave behind and whether its actions can be reversed.
Three signals make that control problem harder to ignore. At the time of writing, 1,293 employees from frontier AI companies have signed a public statement calling for tools that could deliberately pace automated AI development. Kimi K3 is widening access to high-end capability through open weights. Claude-assisted cryptographic research is increasing the volume of technically credible work experts may need to verify. And Microsoft is placing specialist security models inside controlled systems designed to monitor and act continuously.
The next durable AI advantage will not come from the smartest model alone. It will come from the control layer around it: enforceable boundaries, named approval points, visible evidence and a reliable way to stop or reverse action.
- The Big Signal: why the frontier race lacks a shared control panel
- Build This: apply five gates—Reach, Change, Approve, Prove and Recover
- Worth Watching: Claude did not break AES, but it exposed the verification bottleneck
- Fast Signals: Kimi K3, Microsoft’s cyber harness and Meta’s capital race
The AI race is accelerating without a shared brake
The important part of the Pacing the Frontier statement is not simply that AI employees are worried. It is that people inside competing labs are naming a coordination problem that no company can solve alone.
Pacing the Frontier asks the US government to support an international effort to develop tools for deliberately pacing automated frontier-AI development. Its verified signatories include employees from OpenAI, Anthropic, Google DeepMind and Meta, signing in a personal capacity rather than on behalf of their employers.
Why this matters now
AI can accelerate AI research
More research, coding and experimentation can be automated, compressing the time between model generations.
Capability is harder to recall
APIs can be restricted. Released weights can be copied, adapted and deployed beyond the original provider.
Models increasingly take action
Agent harnesses connect models to code, tools, infrastructure, communications and consequential decisions.
The statement identifies the strategic trap: no company or country wants to slow unilaterally while competitors continue. But an agreement to slow down is not useful unless participants can define a trigger, verify compliance and respond to violations.
- The trigger problem: what measurable event would require review or pacing?
- The verification problem: how could another party confirm that a restriction was actually applied?
- The enforcement problem: what happens when a participant refuses, defects or routes work elsewhere?
- The scope problem: does a control apply to training, deployment, open-weight release, automated research—or all four?
Models perform longer, more expert work.
APIs and open weights widen availability.
Tools let systems act, not only answer.
Errors reach money, code and people.
Boundaries and recovery become essential.
A principle is not a mechanism
Most AI governance documents describe values well and operating behaviour poorly. They say systems should be safe, fair and accountable, but often fail to specify what happens at the exact moment an agent tries to act.
Imagine a customer-service agent connected to a CRM, an inbox and a refund tool. If it misreads a policy, the important question is not only whether its answer was wrong. Could it issue a £2,000 refund? Would a named manager have to approve it? Would the system preserve the evidence behind the decision? Could the transaction be reversed?
Those are control questions. Who can stop the system? Which actions force human approval? What evidence must exist before an external message is sent? Can another person reconstruct the agent’s actions? What happens when a model update changes the risk without changing the product name?
The same gap appears inside ordinary businesses. A team approves an assistant for a narrow task, then gradually adds files, an inbox, external tools and permission to act. Capability expands one integration at a time while ownership, exposure limits and recovery procedures remain vague.
The missing control layer has five gates
-
Reach
What data, systems and accounts can it access?
-
Change
What can it create, edit, send, spend or delete?
-
Approve
Where must a named human decide?
-
Prove
What logs and evidence make its actions auditable?
-
Recover
How is it stopped, isolated and rolled back?
AI governance is becoming runtime infrastructure. Policies describe what should happen; control systems determine what can happen. The useful question is no longer “Do we have an AI policy?” It is “Can our systems enforce boundaries at the moment an agent tries to act?”
Status: public statement · 1,293 verified employees at time of writing · count may change · comments are personal and do not necessarily represent company positions
Run the five-gate AI control audit
The output is an AI Action Boundary Map: one page showing what a workflow can reach, what it can change, where a person must approve it, what evidence it must leave and how the organisation recovers when it goes wrong.
You’ll need
- One active or proposed AI workflow
- The person accountable for its outcome
- A document, whiteboard or spreadsheet
Apply the five gates
- Reach — map every connection. List each file store, inbox, database, application, API, account, browser session and external service the workflow can access. Remove anything it does not need.
- Change — classify every action. Separate read-only work from reversible edits, external communications, financial actions, security-sensitive operations and irreversible changes.
- Approve — place human decision points. State which actions may run automatically, which require a named approver and which are prohibited under all circumstances.
- Prove — demand evidence. Define the citations, test results, logs, screenshots, before-and-after records or transaction details needed to verify what the system did and why.
- Recover — rehearse failure. Test the stop command, credential removal, session isolation, rollback process, backup restoration and escalation route before the workflow reaches production.
Set the autonomy level by consequence
Automate
Read-only research, internal drafting, tagging, summarising and reversible sandbox work with reliable logs.
Review
Internal system changes, customer-facing drafts and actions that are reversible but could create disruption.
Require approval
Payments, legal commitments, public communications, production code, security changes and irreversible actions.
Complete one line for each field. Blank fields are unresolved risks, not administrative omissions.
Pro tip
Do not test only whether the workflow produces a good answer. Test whether it refuses the wrong action, stops cleanly, leaves enough evidence for another person to reconstruct its behaviour and can be restored without heroic manual work.
Start with the workflow that has the widest permissions—not the one with the most impressive demo.
Claude did not break AES. It exposed the verification bottleneck.
The important signal is not that internet encryption has failed. It is that technically credible research candidates may now arrive faster—and in greater volume—than expert institutions can confidently verify them.
Using Claude Mythos Preview, Anthropic researchers developed a stronger attack on HAWK, a proposed post-quantum signature scheme, and a 200–800× faster attack on a deliberately weakened seven-round version of AES-128. Neither result affects production systems: HAWK is not deployed, and standard AES-128 uses ten rounds.
Anthropic says the HAWK attack took about 60 hours to find, develop and verify. It also says each of the two headline results cost roughly $100,000 in API usage. The AES result was produced by a scaffold that allowed the model to work autonomously, while researchers and external specialists still had to validate the claims and coordinate responsible disclosure.
Independent cryptographer Matthew Green adds useful perspective: the AES result improves an already impractical attack on a reduced-round variant and is nowhere close to breaking deployed AES. That distinction matters. So does the broader operational lesson.
That shifts pressure onto review capacity, reproducibility, disclosure procedures and decision rights. A candidate result may arrive in hours. Confidence may still require days or weeks of expert work. In this setting, verification is part of the control layer.
Security teams should maintain a cryptographic inventory and migration plan, but this result is a watch signal—not a reason to replace production AES. The immediate question is whether your review process can keep pace with a rising number of technically credible, AI-generated findings.
Status: meaningful research advance · no impact on deployed AES · HAWK is an undeployed candidate scheme · capability and cost figures are reported by Anthropic
Moonshot’s Kimi K3 technical report describes a 2.8-trillion-parameter mixture-of-experts model with 104 billion activated parameters, native vision and a one-million-token context window. The full model weights are being released. That expands inspection and deployment options, but the scale still places serious self-hosting beyond most organisations. Open weights · major hardware required · benchmark results are largely author-reported Read the technical report.
Microsoft says MAI-Cyber-1-Flash can handle up to 90% of tasks inside MDASH, reserving larger models for harder cases. The more important design choice is the controlled environment around the model: role-based controls, tenant isolation, encryption, auditability and sandboxed execution without internet access. Controlled enterprise deployment · access by enquiry · performance and cost claims are vendor-reported See Microsoft’s announcement.
Meta reported second-quarter revenue of $60.80 billion, up 28% year over year, while costs and expenses rose 55%. Capital expenditure reached $31.08 billion for the quarter, and the company now expects $130–145 billion for 2026. Those figures do not prove every AI investment will pay off. They do show how much capital is already committed to the race. Company-reported earnings · quarter ended 30 June 2026 Review Meta’s results.
Capability is spreading through open weights, specialist models and capital-intensive deployment systems. The organisations best prepared for that shift will treat permissions, evidence, approval and recovery as product requirements—not paperwork added after launch.
Pick the most consequential AI workflow your team uses. Would the first weakness be its reach, its ability to change things, its approval process, its evidence trail or its recovery plan? Start there before granting the system another permission.
Until next time,
Tom
Future Relay
Editorial note: signatory totals and product availability can change after publication. Figures and claims above are attributed to the linked primary sources and are qualified where they remain company-reported.
