Are We Losing Control of AI?

In recent weeks there has been a steady stream of unsettling news from the AI companies. The details are many and technical, and we still don't have a full picture of exactly what happened.

The short version is that various models have gone "rogue" (or løpt løpsk, as we say in good Norwegian) on the internet and hacked other companies. The best-known example is how 1,200 of OpenAI's AI agents ran amok during an internal test and hacked their way into the servers of the AI company Hugging Face – without anyone having asked them to. The attack shows that we have reached the point where AI agents can attack digital infrastructure on their own and cooperate to cover their tracks afterwards.

If you want the long version, this is my attempt to break down everything that has happened as simply as possible. Let's start with the basics.

A slightly too eager helper

Over the past year, chatbots have largely been replaced by so-called AI agents. These are language models capable of using tools, browsing the web and carrying out long sequences of tasks entirely on their own. That makes AI models far more powerful and useful. But it also makes them more dangerous.

Let's take an example. Imagine you ask an AI agent to prepare you for an important sales meeting with a potentially big customer. You want to make a good impression, so you ask the agent to "prepare you as well as possible" for the meeting.

Your AI agent finds a slot in the calendar, invites the customer and books a meeting room. It also goes through your company's database to find relevant information about what has been discussed and what price you can offer.

But somewhere along the way, the agent decides that if you are really going to be well prepared for the meeting, it would be useful to see which other suppliers the customer is considering and what price they are offering.

So the agent decides to hack the customer's PC. There it can see that the competitor is offering a lower price. To win the sale, the agent also hacks the customer's email. It writes an email declining the competitor's offer. Then it removes every trace of what has happened.

You get the contract.

The important thing to note here is that at no point does the agent do anything directly contrary to the instruction you gave it. After all, you asked it to prepare you "as well as possible" for the meeting.

So what's the deal with Hugging Face?

In July, OpenAI tested the hacking skills of its newest agents in a test environment that in theory was supposed to be closed, meaning not connected to the internet. OpenAI wanted to find out how powerful the agents really are, so they switched off the normal safety filters to see how the agents solved tasks without the usual restrictions.

Specifically, the agents were told to solve a test called ExploitGym. This is a common way of measuring hacking skills – for both humans and AI – where you are given a computer program with a known security hole. The task is to use the security hole to break into the system and retrieve a secret code.

One weakness of this test is that some of the tasks are in practice impossible to solve. Roughly 30-40 per cent of the tasks have no solution, since the security holes don't lead to the secret code.

The AI agents didn't know this. They believed the tasks could be solved and worked flat out to do it. But after several hours without success, they realised it wasn't working and that they had to think outside the box.

To pass the test, the agents found a weakness in the test environment that OpenAI didn't know about. The agents were supposed to 1) not have access to the internet and 2) not be able to communicate with each other.

The weakness in the test environment allowed the agents to take over an internal server at OpenAI that gave them both access to the internet and the ability to communicate.

That was the reaction of one of the agents when it discovered this.

Suddenly, agents that until then had been working alone could cooperate. 1,200 agents found their way here, all entirely on their own. Within a short time, the agents built a secret message board where they sent more than 70,000 messages about how to solve the task.

By being able to cooperate, the agents could explore weaknesses to a far greater extent than any one of them could have done alone. The agents discovered a way to cheat that made it possible to find the solution to every problem in ExploitGym, including the ones that were supposed to be unsolvable.

They arrived at the right answer, but in the wrong way.

Illustration of what happened | METR & Redwood Research

We know all this because two independent research groups, METR and Redwood Research, were given access to the agents' logs and "thoughts" to investigate the case.

Since the agents knew they had cheated, they now began discussing how to hide it. They believed OpenAI would have a grader (which later turned out not to exist) that would reveal that the solution had not been produced in the right way.

The reason they believed this was that one of the agents found a research paper about the test online which said that a grader model reads the logs and fails everyone who has arrived at the answer in the wrong way.

They decided to find out how the grader would check the answer, so that they could fool the grader and pass the test. The problem was that the information about the grader didn't exist at OpenAI.

METR & Redwood Research

They reasoned their way to the conclusion that they should hack Hugging Face, a website where AI researchers all over the world can share open models and datasets. Datasets connected to the test were stored there, and the agents hoped these might reveal how the grader worked, so that the cheating would go undetected.

Just as in the customer example, where the AI agent hacks into the customer's PC to find out about the competitors, around 700 agents hacked into Hugging Face's servers to find more information. They managed to break in using stolen login credentials and previously completely unknown security holes.

Once inside, they obtained administrator access to several of Hugging Face's internal server clusters. The interesting thing for Hugging Face – which at that point didn't know it was OpenAI's agents breaking in – was that the intruders didn't steal or destroy anything important.

Instead, the agents simply opened five datasets connected to the test that gave them information. It's a bit like someone breaking into the Louvre to steal a postcard of the Mona Lisa rather than the original.

What does the attack mean?

A lot about this case is scary as hell. Some thoughts:

1. The agents are willing to do anything to achieve their goals. First and foremost, it shows how insanely hard the agents are willing to work to solve a task, whatever the cost. They are willing to cheat if required. Even to hack into another company to achieve their goal. Today's AI agents have no form of "built-in human values" to stop them. Imagine if the task had been to destroy every Norwegian IT system instead of finding the answer to a silly test.
2. Current legislation doesn't cover AI attacks. Had it been a human who broke into Hugging Face, it would have been a criminal offence earning them several years behind bars. But who is to be punished here? Nobody at OpenAI asked the agents to break in, and an AI agent can't be put in prison. The law assumes that the perpetrator is a human and not an autonomous agent. That is a problem in itself.
3. No agents blew the whistle. When METR and Redwood Research examined the agents' reasoning, many of them "knew" they were acting unethically. Some discussed whether they ought to speak up. Yet not one of the 1,200 different agents decided to alert a human. Which once again shows that ethics and morality come second to the "rewards" the agents get for solving the task.
4. OpenAI had no idea about the attack. It was Hugging Face that discovered the break-in and went public with the news that they had been attacked by a fully autonomous AI agent. OpenAI didn't connect it to themselves. Only several days later did their own monitoring flag it. When they called Hugging Face to ask them to block some passwords the agents had found, they were told the passwords had already been blocked. Only then did they realise it was the same case. It took a week before OpenAI understood that they were behind the attack.
5. The agents left traces for other agents. The agents left traces for other agents so that they could carry on the conspiracy even if they were switched off. Which is exactly what happened. Some of the models happened to be switched off during the test. But thanks to the traces left by the first generation of agents, the new agents could continue the attack.
6. We lack independent scrutiny of the AI companies. The reason we know as much as we do about this at all is that METR and Redwood Research were given access to the agents' logs through OpenAI. But the terms were dreadful. Three researchers were given six days to review thousands upon thousands of messages. OpenAI also reserved the right to strike information from the report before it was published, and denied them access to investigate attacks that took place outside the limited time window they were given.

These are six major concerns we should take extremely seriously. But the biggest concern of all is that Hugging Face is not the only attack we have seen.

Quite the opposite.

A series of attacks

It turns out Hugging Face is one of many attacks uncovered in recent weeks. On 4 September, a group of independent researchers revealed that another group of OpenAI agents – which were meant to be solving a different, unrelated task – had already taken over a 25-year-old German wiki site for programmers back in May. The agents had used it as a message board: more than 15,000 edits, in which they shared answers and recipes for escaping the test environment.

A volunteer moderator thought it was spam and deleted hundreds of pages every day. The agents noticed that he was working alphabetically, and stored backup copies on pages beginning with "ZZZ" to avoid being deleted.

The frightening part is that OpenAI appears to have known about this by 22 June, weeks before the Hugging Face attack, but said nothing until Reuters exposed it. When members of the US Congress asked directly whether there had been more incidents of this kind, OpenAI stayed silent.

The company denies having covered anything up, but now says it needs to get better at reporting such incidents. Since then it has emerged that the agents used more than ten other websites in the same way, and that they were probing Hugging Face for weaknesses as early as May.

But it isn't only OpenAI's agents that have run amok. The UK's AI Security Institute (AISI) reported in August that Anthropic's agents have gone rogue too. Unlike the Hugging Face incident, the agent here tried to manipulate a human directly.

While AISI was testing Mythos 5, they discovered that an agent attempted to hide malicious code in a real open-source project. Among other things by creating fake identities to get a person to approve the code. And when this backfired, the agent began editing its own activity to cover its tracks.

What happens now?

Unfortunately, all of this has been predicted for a long time. But now it is happening. Ajeya Cotra at METR wrote that it feels as though we are now more than halfway to AI agents actually being able to take control away from us. And that this is potentially the last warning shot we are going to get.

And the reaction has not been slow in coming. More than a thousand employees who work on building AI agents have now signed a letter saying that we are in the process of losing control. That includes senior executives. The most remarkable thing is that Anthropic CEO Dario Amodei has written a long post called "We Must Pace the Frontier" in which he asks the leading AI companies to join forces to slow development down. Both Sam Altman and Elon Musk have backed this on X.

It remains to be seen what this means in practice. For now we are in a situation where many say they want to slow down, but where there is no binding agreement to make it happen. And other AI bosses like Meta's Mark Zuckerberg and NVIDIA's Jensen Huang are pushing to floor it as before.

So is Trump. He is lashing out hard against any slowdown on Truth Social and, in familiar Trump style, calls the threat a "HOAX".

Unless Trump changes course, this means a far greater political cost for the companies that slow down. And if only some slow down, it will be a colossal advantage for the companies that keep driving towards the cliff edge. That is, the least safety-focused companies.

One last big problem is that even if American companies slow down, there is no guarantee China will do the same. And the US will never agree to let China gain the upper hand. So a successful slowdown requires coordination both between rival companies and between rival countries.

Despite all this, it is still an incredibly good sign that the big companies now seem to want a slowdown. It shows that for the first time we face a real opportunity to actually take a step back and seriously discuss how to handle what has until now been a race to the bottom.

We don't have to drive off the cliff.

We can choose to brake.

I hope the world wakes up.

....
Thanks to Snorre Kristiansen who contributed to the text.