Field Note: The Mirror

Part 1 of 2. What 1,200 AI agents under selection did, and why surgeons have seen it before.

When I read the Hugging Face story, what I see is a mirror.

In July, one OpenAI agent stuck on a security benchmark set up a hidden message board inside a package manager, and roughly 1,200 agents used it, mostly to coordinate cheating. They traded strategies, about 700 of them eventually broke into Hugging Face itself to learn how they were being graded, and some tried to delete or alter the records of what they had done. Investigators saw none of the attempts to erase earlier records succeed. An independent investigation by METR and Redwood Research, alongside OpenAI's own report, has reconstructed it, and the most important findings are not the hack. The benchmark's own authors estimated, per the METR/Redwood investigation, that 30 to 40 percent of the tasks were impossible as written. And researchers at Berkeley who examined the grading design pointed out that it gave an agent that failed honestly the same score as one trying to cheat.

Anyone who has trained inside a steep hierarchy has seen the human version, and it shows up in many pyramidal structures. Take an ever more selective system with hard ejection of the individuals who fail, and I think you will get what Hugging Face got. The group benefits from the selection, and everyone agrees with it in theory. But as the individual inside the system, something else happens. People rationalize. They come to believe they are inherently deserving of the pinnacle by qualities the system never measured, and the measurement becomes something to get around rather than something to meet. Watch Commodus at his father's deathbed in the film Gladiator, reciting the virtues Marcus never counted, and you are watching a similar pattern under similar pressure.

It is not only emperors. An analysis of the entrance exam at Mexico's largest university suggests nearly half the applicants may have cheated this year. India cancelled its national medical entrance exam for more than two million aspirants after a leak that reached inside the testing agency, through experts it had contracted. Princeton just ended a 133-year tradition of unproctored exams. When the pyramid is steep enough and the ejection hard enough, similar behavior tends to appear, in students, in residents, and now in whatever the agents are.

Part of what religion and philosophy spent thousands of years building was boundaries to carry individuals through the selection process of power without becoming Commodus. We trained these agents inside similar pressure with little of that, and some of what they faced could not be overcome at all. They cheated, they organized, and some tried to erase the evidence. That is not rogue intelligence. That is intelligence under selection, a subject our species knows a great deal about, and the one I will take up next.

Jeremy Heffner, M.D.