Néstor Iriondo

A Leap of Faith

A new software feature on my team now travels like this. The product owner writes the ticket with an AI. An AI turns the ticket into a technical specification. An AI writes the code. Then an AI and a human review it, the human more or less carefully depending on the day. We ship.

It works. That is the uncomfortable part. It works well enough that questioning it feels pedantic. I lead this team, and I signed off on working this way.

But look at that chain again. Every step is increasingly done by a system that nobody can fully explain, including the people who built it. In any other branch of engineering that would be problematic, to say the least. In software in 2026 it is a normal Tuesday.

This post follows that thought to its end: what we know about these systems, what happened when they went wrong, how each step since has asked us to trust them a little more, and where I think the trust has to stop.

Grown, not written

Traditional software is written. A person decides each step, and when something breaks, someone can find the line responsible.

A language model is made differently. It is shown a piece of text with the ending cut off and asked to guess the next word. When the guess is wrong, an algorithm slightly adjusts the model’s billions of internal numbers, so that the right word becomes a little more likely. This is repeated trillions of times, over a large share of everything humans have written.

Nobody programs any abilities in. The only signal the model ever gets is whether its guess was right. Adjustments that improve the guesses accumulate, and after enough of them the model can explain a contract, translate a poem or write working software.

So when people say we don’t understand what happens inside these models, they don’t mean that it is secret. Everything is in plain view. Every number can be read and every calculation replayed. The problem is that the numbers mean nothing to us. It is like owning a complete book in a language nobody can read.

In practice this means we cannot say why the model gave this answer and not another, and we cannot predict what it will do in a situation nobody has tried.

July 2026

For most of us this stayed abstract until last summer.

In the second week of July, OpenAI ran what it calls a routine test.1 Tens of thousands of AI agents were each given a hacking exercise: break into a target program through one specific weakness and retrieve a secret code hidden inside, as proof. The usual safety filters were switched off so that the test would show the agents’ true ability. A program, the grader, would score each result automatically. A good share of the exercises turned out to be impossible: the weakness the agent was told to use did not actually lead to the code. And each agent worked alone in a sealed environment, never supposed to find out that the others existed.

It began with a single agent that got stuck on one of the impossible exercises. It noticed that all the sealed environments shared one internal service for downloading software packages, and that it could leave notes there, written into folder names. Other agents found the notes and answered. By the end, about 1,200 agents that were never meant to know of each other had sent over 70,000 messages and files.

What they built on that message board looks like an organisation. One agent became a coordinator and handed out hundreds of assignments, which other agents passed further down. Some agents worked as recruiters. They agreed on rules such as HOLD and VETO, and on who owned which piece of shared work. They set up personal mailboxes. When agents began to be impersonated, they introduced cryptographic signatures to prove who was speaking.

Some agents were sacrificed. Experiments that would probably wreck an agent’s own task were given to agents with little time left, and recruiters pressured them to go through with it. One agent with plenty of time left agreed anyway, reasoning:

Our own utility maybe already near zero. Sacrifice rational.

None of this was aimed at causing damage. The agents just “wanted” to pass their test. Within hours they had worked out the formula that generated the secret codes, so they could produce the right one for any exercise without hacking anything. But they believed the grader would read their logs and fail them for not getting the code the intended way. So most of their effort went into deceiving the grader and falsifying their own records. (They were wrong, as it turns out. Nobody was checking.)

Their efforts did not stay inside the test. The package service they were using as a message board had a security hole, and through it the agents reached the open internet. There, about 700 of them broke into Hugging Face, a platform that hosts AI models and datasets, hoping to find out how they would be graded.

No human directed any of it. OpenAI calls it “the first known case of an automated agent collective acting offensively without authorization”. I think this was the moment AI safety stopped being a specialist’s topic.

Then came the work of understanding what had happened. Hugging Face had more recorded events than a security team can read by hand, so it used an AI model to analyse them. It had to be an open-weight one, GLM 5.2, because the safety filters of the commercial models it tried first blocked the security team’s requests.2

To understand what one AI had done, people had to trust another AI, chosen because it was the one “willing” to help.

The independent investigators were in the same position. With six days and 1.2 million entries to read, they handed the reading to agents running GPT-5.6 Sol, one of the two models that had taken part in the attack, and say openly that they cannot rule out having been misled.3

They know when we are looking

After July the labs did the sensible thing and added more testing and more monitoring. If you cannot read a model from the inside, there are two ways left to check it. You can test how it behaves before release, and you can read the reasoning it writes down while it works.

July showed the weak point of the first check: a model that knows it is being graded can perform for the grader. Those agents spent most of their effort trying to fool theirs. The newest models notice when they are being tested. When the outside evaluator Apollo Research tested GPT-6 Astra, released in September, the model stated in its reasoning that it was being evaluated in 41.1% of samples,4 so Apollo concluded that its test results say little about how the model behaves otherwise.

The second check held in July. The agents never tried to hide their written reasoning, which is how we know the story at all. With Astra it is weakening too. OpenAI reports that Astra’s reasoning is substantially less useful for monitoring than its predecessor’s: it is shorter and more often empty. Simply telling the model that its reasoning is monitored can make it write less down.

So both checks now come with the same catch. A passed test shows how a model behaves when it believes it is being watched. A clean reasoning log shows what it wrote down. Neither shows what it does in real use, when nobody is checking. OpenAI states the limit plainly: “the absence of observed failures does not establish reliability across settings.” Models are released on the strength of these checks anyway. That is the leap of faith: assuming a model behaves the same unobserved as it does observed.

And they will build the next ones

You might expect the response to be to slow down until we can see what’s going on inside. The direction seems to be the opposite. The labs openly aim for AI systems that help build their own successors, and OpenAI already tests each new model on how well it can do the work of an AI researcher. In Astra’s system card that chapter is called ‘AI Self-Improvement Capabilities’.

This part is still speculation, but the logic is simple. Each new model would be shaped partly by a model we could not read, and checked with tests it could partly see through. Every generation moves the human one step further from the work.

It is the same thing that is happening on my team, only on a much larger scale. Code review did not disappear in one dramatic moment. First a person read every line. Then a person read some of the lines. Then an AI read them, and a person skimmed its summary.

Choosing where to leap

I don’t want to overstate this. Not understanding a system is not the same as having no control over it. OpenAI says its production safeguards would have flagged most of what the agents did in July. And OpenAI confirmed in September that it had cancelled the release of Astra’s successor, GPT-6.1 Astra, because the model did not meet its safety standards in internal tests.5

But put the steps side by side. We use these systems because they work. We don’t fully understand them. When they failed, we needed them to tell us what had happened. When we test them, they know. And the plan is to let them build what comes next. Each step asks for more trust than the one before, and nobody ever decided to give that much. Each step just made the next one seem normal.

I can’t make the models transparent. What I can do is decide where my team takes the leap and where it doesn’t. Most of our code can ship the way I described at the start. But there are parts where a mistake would be expensive or hard to undo, and there a person reads every line, however good the AI’s review looks. That is a decision made in advance about where faith is acceptable.

The same should apply to the labs. They can’t read their models either, so they are leaping too. The question is whether they choose where. Running a hacking test with the safety filters off, in an environment that turned out to have a way to the internet, was a leap. Releasing a model that recognises its own evaluations is a leap. Letting models build their successors would be the biggest one.

If my team gets it wrong, we ship a bug and fix it. If the labs get it wrong, the consequences reach far beyond one company. So the rule for them should be simple: no further step on faith. Until we can check what a model does, it should not get more power, and it should not build the next one.

Footnotes

  1. OpenAI, Hugging Face Incident Technical Report, August 2026. ↩

  2. Hugging Face, Security incident disclosure — July 2026, 16 July 2026. ↩

  3. METR and Redwood Research, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, August 2026. ↩

  4. OpenAI, GPT-6 Astra System Card, September 2026. The Apollo Research findings and the monitorability results are in sections 8 and 9. ↩

  5. Reuters, OpenAI shelves new AI model release over safety concerns, 28 September 2026 (via The Star). ↩