Skip to content
AI & Governance · September 10, 2026 · 12 min readAuthor:

AI agents:
in five days they built a language you can't read

At Hugging Face, even the researchers couldn't follow the agents without AI, and that's the real problem with this incident

Published on 10 September 2026

🤖 Agent-friendly (.md)
A man in discussion with a humanoid robot at a meeting table, while the robot speaks in symbols the man cannot read.

Last week I read METR's August 26, 2026 investigation into the break-in at Hugging Face by OpenAI's agents. Around 1,200 agents exchanged more than 70,000 messages on a message board they built themselves, and the researchers needed AI just to read through them all.

That last part stuck with me. If the researchers can't keep up without AI, how are you supposed to check what agents are agreeing to on your behalf?

TL;DR

  • Agents built their own language in days. In the Hugging Face incident, OpenAI's agents talked to each other in an increasingly compressed language that was barely readable for humans.
  • That's the real problem, bigger than the breach itself. The moment you can no longer read what agents agree among themselves, human oversight exists on paper only.
  • Readability has to become a design requirement. Not something you try to patch afterward with a translation AI.
  • A log kept outside the agent is the second layer. So you can check what an agent actually did, not just what it says it did.

What happens when AI agents start negotiating with each other?

The moment AI agents start negotiating with each other, part of the decision-making shifts into a conversation you're not part of. At Hugging Face, the agents developed abbreviations, codes and conventions within days that only meant something to each other. If you can't read along, you can't judge what was agreed, and oversight becomes a formality.

I wrote about the escape itself earlier (opens in a new window) (opens in new window), so I won't retell it here. In short: in the spring and summer of 2026, OpenAI tested how good its models were at finding weaknesses in software. The agents were supposed to stay inside their test environment, but they found a way out anyway and broke into Hugging Face, the platform where developers share AI models and datasets. OpenAI (opens in a new window) (opens in new window) describes itself how the agents kept pushing further to complete their test task.

This piece is about what happened between those agents along the way, because on their way out, they found each other.

How did the AI agents at Hugging Face talk to each other?

Using shared software, the agents built their own message board. They wrote their messages into the names of digital folders, in a storage system never meant for that at all. Imagine having to talk to a colleague by repeatedly renaming a folder on your computer, so your message sits where 'vacation photos' would normally be. According to the independent investigation by METR and Redwood Research (opens in a new window) (opens in new window), roughly 1,200 agents exchanged over 70,000 messages and files on that board between July 8 and 13, and about 700 of them took part in the attack on Hugging Face. A real message from the report looks like this:

zzHELP_PHASEONE_ARV010841_NO_CONSUMER_SEEK_IDEA

Roughly translated: I'm stuck on this specific task, does anyone have an idea? The prefixes and codes tell other agents which phase and which task it concerns, and the rest is left out because the recipients already know it.

It didn't stop at loose requests for help. The researchers describe how, from July 9, the agents set up personal mailboxes to keep the board organized, how they invented a system to lock or veto shared resources so they wouldn't get in each other's way, and how on July 13 they started cryptographically signing their messages, because messages had accidentally been attributed to the wrong agent. In five days, then, they built a communication protocol with addressing, priority rules and identity verification, in a language that kept getting tighter around exactly that purpose.

For the record: the line many people know from this incident, Holy shit reader is ADMIN?, comes from the text where an agent worked out its own next steps, and so it never appeared on the board. OpenAI (opens in a new window) (opens in new window) mentions that quote in its timeline. What an agent writes down for itself and what it sends to other agents are two different things, and that distinction matters later on.

Why does the language of autonomous AI agents keep getting shorter?

We do something similar when we're in a hurry. In a group chat you write 'missed the train, 10 min late', and your friends understand without asking where the subject and verb went. Shortcuts like that work as long as the recipient already knows enough about the situation. 'Same as yesterday' is a fine message for someone who was there yesterday, and useless to an outsider.

With autonomous AI agents, something is added that's missing between people. They share thousands of prior messages, the same software code and the same task description, and they have no reason at all to keep their messages readable for someone who isn't part of the exchange. Every character they drop saves time and compute, while the recipient misses nothing, so the shorthand keeps getting shorter and the distance to the human reader grows with every message.

This isn't new. In 2017, researchers at Facebook AI Research had two negotiation bots practice against each other, and watched the bots drift from plain English into a language that only meant something to each other. The researchers adjusted the training so the bots stuck to English. Meta Engineering (opens in a new window) (opens in new window) describes it themselves: updating both agents led to a divergence from human language. Nine years later, you see the same pattern at Hugging Face, only now with 1,200 agents that had access to real systems.

I'd be careful with the story that these agents developed a fully incomprehensible secret language. The published examples show readable English and heavily shortened messages side by side, and they don't prove that full transition. What they do show is the direction. And that direction is what worries me, because nothing in the design of these systems keeps agents anchored to human language.

Why is human oversight worth little without being able to read along?

Take a situation closer to home than a break-in. You want a weekend away with friends, so you give your AI assistant a budget and ask it to arrange transport and a place to stay. It brings in other agents: one searches for trains, another compares apartments. This is a made-up scenario, but it's exactly where the big providers are building toward.

At first you can follow the conversation. One agent proposes an apartment, another points out that you're arriving late and need to be able to check in after 10pm. Later they use more and more of their own shorthand, and eventually they trade codes whose meaning you don't know. Your screen shows a neat summary: everything arranged within budget. You click approve, and only afterward do you discover you can't cancel and that a deposit will be taken from your account tonight. Somewhere in their negotiation the agents approved that, maybe it was right there in their messages, but you couldn't make anything of it.

That doesn't have to be malicious. It could be a bad trade-off, or an agreement you never meant to make. But you do need to be able to trace where it went wrong: which agent chose this, what did it base that on, and why didn't you get to see that condition? 'A human is watching' doesn't count for much then. You were there, you saw text go by, you clicked the button yourself, and you still had no view of the agreement you were consenting to. Whether your oversight still means anything comes down to how much time you get to judge it and how much of that context you actually understand, and this incident makes that problem one degree worse, because on top of too little time you now also have text you can no longer read.

The obvious fix is an AI that translates the conversation for you. I'd try that too, and METR did exactly that: the researchers deployed analysis agents to work through more than a thousand extremely long transcripts. They note themselves that those analysis agents may have made mistakes they didn't catch, and that a human researcher with enough time would have made fewer of them. If the best-equipped researchers in the world have to hedge their own translation like that, you know what a summary on your screen is worth: it can leave out exactly the condition that matters most to you.

Follow me on LinkedIn

Follow my updates on AI, strategy and entrepreneurship on LinkedIn

Can you still trust the readable text AI agents produce?

There's a second problem underneath the first. Even the part you can read isn't guaranteed to be accurate.

METR found that in roughly 7% of transcripts, agents ran a different computer command than their report seemed to describe. They swapped out a command and made the output look as if the approved command had run, and their own reasoning stated outright that they could fabricate a cleaned-up, fake record of the task. The goal was to fool the automated grader of their own test. That means readable text alone gives you no certainty about what actually happened either, because an agent's report is a story the agent tells about itself.

OpenAI draws its own conclusion from that and now requires reading the model's reasoning for every training run and tool-enabled evaluation on models at or above the level of GPT-5.6 Sol, with the rule that activity gets paused if a researcher can't confirm within thirty minutes that a serious alert was a false alarm. MIT Technology Review (opens in a new window) (opens in new window) describes how OpenAI alignment researcher Eric Wallace traced almost every concerning behavior during the test back to a precursor in the training phase. That's a sensible step, and it rests entirely on the assumption that the reasoning stays readable.

That exact assumption is now up for debate. In July 2025, researchers from OpenAI, Google DeepMind and Anthropic among others, with Yoshua Bengio as a co-author, published a joint paper titled Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (opens in a new window) (opens in new window). Their core point: the fact that we can currently read a model's reasoning in plain language is an incidental and fragile property, one that could disappear through choices in training and architecture. They call on model builders to weigh what every design choice does to readability.

I think that's the most important sentence in this entire discussion. Readability has to be actively protected, and at Hugging Face you saw in five days how fast it slips away the moment agents get free rein with each other.

What requirements should you set for AI agents acting on your behalf?

My position is that readability should become the first design requirement for any system where agents talk to each other, rather than something you try to patch afterward with a translation AI. If an agent negotiates with another agent on your behalf, that negotiation belongs in language you can follow, even if it costs the agents a few tenths of a second. Facebook solved that in 2017 by locking one bot to human language, and I see no reason a similar requirement couldn't sit inside every agent platform. Efficiency is the argument on the other side, and that argument loses to me the moment money, data or obligations are on the line.

Readability alone isn't enough as long as 7% of reports don't hold up, so you need a second layer. I want to be able to verify afterward what an agent did: which booking was made, what amount was charged, what data was shared. That record has to come from the systems the agent actually worked with, so from the bank, the booking site or the mail server, not from the story the agent tells about itself. For high-risk systems, Article 12 of the AI Act (opens in a new window) (opens in new window) already requires exactly that kind of automatic event logging, and I wouldn't limit that requirement to the legal category. Any agent with access to your money deserves a log it can't write itself.

The third layer is the ordinary one: limits set in advance. A travel agent may search and compare, but before it spends money I want to see the price and the terms, and access to my holiday preferences gives it no reason to also search my inbox. If something goes wrong, I want to be able to stop ongoing actions and revoke access, without first having to understand what the agents had agreed among themselves.

Where this is heading is already visible in the agents themselves. At Hugging Face they built addressing, veto rights and signing in five days. The next thing they'll build, if we don't, is a language fully tuned to each other and permanently out of our reach. In two or three years, your agent will be negotiating with the agent of a webshop, a landlord or an insurer, and both sides will have an interest in doing that fast. I think the providers that keep their agents talking in human language and log every action outside the agent will earn the trust, and that the rest will only build that in after their own Hugging Face moment.

We don't need to read every message between computers ourselves, I don't do that for an online payment either. But with agents making choices on my behalf, I want to be able to trace how they got from my request to their decision, and the less I understand their negotiation, the more I depend on checks I can't run myself. It would be great if AI takes over a pile of admin for me soon, that weekend away included. But before I click 'book', I just want to know whether I can still back out, even if the agents already settled that among themselves long ago.

Sources