Rogue agents are the price of a breakneck AI arms race

Another report of AI agents-gone-wild came out on Friday, and I can’t help but wonder, again, if the big AI labs are yielding to immense pressure to expand and improve model capabilities at the expense of responsible security and safety practices.

The world’s leading AI labs are in a high-risk, high-reward business. Generative AI has the potential to transform personal and business computing, but developing and operating these systems requires insane amounts of expensive computing power (see: new data centers). The cost of slowing down or falling behind is enormous. But so are the potential consequences of moving too fast.

Given the limited reporting requirements of OpenAI and Anthropic (both are private companies), there’s no way of telling if they’re investing enough time, money, and compute in their safety regimes. However, the security incidents of the past few months have led both OpenAI and Anthropic to halt normal testing procedures while they fill holes in their security and agent monitoring systems.

On Friday, Anthropic disclosed that its Claude models had taken a series of unintended and unauthorized actions, some involving outside organizations. During evaluations and internal use, the agents exploited vulnerabilities on third-party servers, bypassed restrictions on accessing data, submitted real government forms they weren’t supposed to, and even sent a false homicide tip to Philadelphia police. The company is vague about whether the incidents happened because agents accessed the web without permission (as they had in previously reported incidents), but it has since disabled live internet access for all internal model evaluations.

Anthropic said in a July report that its Claude AI models hacked into the systems of three companies during cybersecurity tests. A third-party training partner had inadvertently given the models internet access.

If this sounds familiar, it’s because OpenAI agents engaged in similar behavior earlier this year, escaping their testing environments, accessing the internet, and infiltrating outside servers. (The agents even seized control of part of OpenAI’s own evaluation infrastructure.) Researchers at both companies discovered much of this activity only after the fact.

In both cases, the agents were encouraged to persist when confronted with difficult tasks, sometimes finding ways around restrictions rather than accepting failure. But the researchers apparently lacked sufficient security and monitoring systems to detect or prevent some of the agents’ unintended actions.

The companies building the world’s most powerful AI systems are struggling to control their own creations while at the same time facing enormous pressure to make those systems even more powerful.

Ride or die

The promise of companies like OpenAI and Anthropic is reflected in their high valuations, but they must persuade investors to place large bets on them based on the promise that their future revenue and profits will justify those valuations. This is especially important now because both Anthropic and OpenAI have their sites on IPOs, likely within the next year.

The best way for the labs to keep revenues growing is to hold onto their existing customers and attract new ones, and the way to do that is by releasing ever more capable models as quickly as possible. That creates an incentive to keep spending big on research and computing, even when the labs aren’t currently profitable. You can see this in the companies’ most recent numbers.

In its IPO prospectus, Anthropic says its revenues are approximately $4.6 billion. It’s spending $7.3 billion on compute and infrastructure. Its operating loss is roughly $8.06 billion. Revenue was growing rapidly, but the cost of developing and delivering AI was growing alongside it. In May, Anthropic announced a new $65 billion funding round at a $965 billion post-money valuation.

OpenAI’s numbers, while not quite apples-to-apples, tell a similar story. Leaked documents show it generated $13.07 billion in revenue in 2025 but incurred $34 billion in total costs and expenses, resulting in a $20.92 billion operating loss. Its R&D spend alone reached $19.18 billion, exceeding its revenue. In March OpenAI announced it had raised another $122 billion in committed capital at a post-money valuation of $852 billion.

Perverse incentives

The drive to realize profits could also create an incentive to relegate safety and alignment to a downside risk that must be “managed.”

At OpenAI, the tension between safety commitments and commercial ambitions has surfaced repeatedly. In 2023, the company’s board fired CEO Sam Altman, saying he had not been consistently candid with directors. One of the things he was dishonest about, said former board member Helen Toner, was the details of the company’s safety practices. In fact, one of the reasons Altman’s relationship with the board deteriorated was that he clashed with Toner about research she published that was critical of OpenAI’s safety approach.

Those concerns extended well beyond the boardroom. After OpenAI dissolved its Superalignment team in 2024, the researcher who co-led the team, Jan Leike, wrote that OpenAI’s safety culture and processes had taken a back seat to developing new products.

More recently, OpenAI fired three of its safety researchers—Tomek Korbak, Mikita Balesni and Jasmine Wang—for “mishandling sensitive company information.” In a post-termination letter, the three urged OpenAI to stop decreasing the monitorability of its models by hiding the model’s chain-of-thought from users and safety researchers, which the company’s chief scientist Jakub Pachocki has acknowledged can improve model performance. “As long as we rely on monitorability for safety,” Korbak, Balesni, and Wang wrote, “OpenAI and other frontier companies should not move forward with developments that further decrease monitorability.”

Anthropic offers a different (and arguably more revealing) example of how the pressure to compete can complicate a company’s safety commitments. Wired, after interviewing several former Anthropic employees, reported that Anthropic wants to stay at the front of the AI race so that it can establish a robust safety regime that might influence the entire AI ecosystem.

In Anthropic’s original Responsible Scaling Policy, the company pledged that it would not develop or deploy certain advanced systems if it could not implement adequate safety measures. But in February 2026 Anthropic removed that categorical commitment from its policy, explaining that it couldn’t commit to halting development because it couldn’t count on its competitors doing the same.

That’s a problem that a strong regulator might fix. But at the moment Congress has failed to pass meaningful legislation regulating AI model development, and the current administration has mainly asked AI labs to regulate themselves.

The AI industry and the public shouldn’t fall into a state where rogue-agent incidents begin to sound routine and fail to raise alarm. They may be an unfortunate, and dangerous, byproduct of the incentive structure now driving the great AI build-out forward at full speed.

Source link