Around 2013, in Wisconsin, USA, a man named Eric Loomis stood trial on charges related to a drive-by shooting. Loomis was sentenced to a term of imprisonment without probation. At sentencing, the court referred to a “recidivism risk score.” What was unusual was that this score had been assigned by software called COMPAS, rather than by a judge.
Loomis challenged this decision, but COMPAS was a private company's product, and its calculation method was a trade secret. Ultimately, his sentence was upheld without anyone explaining how the score had been produced[1].
[1] State v. Loomis, 2016 WI 68, 881 N.W.2d 749 (Wis., decided July 13, 2016). The US Supreme Court denied review in 2017 (cert. denied, 137 S. Ct. 2290). The Wisconsin Supreme Court held that appropriately referring to a COMPAS risk assessment at sentencing did not violate due process.
⚖️ Democracy and explainability
In Nexus, Yuval Harari describes democracy as an information network that corrects its own errors. Elections, courts, the press, and auditing bodies question and correct one another's decisions. Ultimately, these mechanisms rest on the act of asking decision-makers to give reasons. One reason Harari sees AI as capable of fundamentally destabilizing democracy is precisely its inability to explain itself. An algorithm's decision combines countless subtle signals, so there is no single reason for it, and even its creators cannot retrace its path. What we cannot understand, we cannot fix.
But a question may arise:
Advanced AI seems likely to make more accurate decisions than humans. Isn't making the right decision more important than whether it can be explained?
Harari's important observation in Nexus is that accuracy and legitimacy are different questions.
The COMPAS score may have been statistically accurate. Yet regardless of its accuracy, a decision gains legitimacy only if the person receiving the score can ask why and challenge it. Challenging an AI decision, however, is difficult for several reasons.
Consider a more everyday example. Suppose a bank turns down your loan application. When you ask why, it simply replies, “The algorithm said no.” The engineer who built the algorithm cannot offer a very different answer. The score combines thousands of variables, and the logic of that combination cannot be reconstructed in a form that people can understand.
What Loomis in the courtroom, or any of us refused a personal loan, really wanted was not “a more accurate score” (does such a thing even exist?). It was a reason we could understand and contest for why the result had occurred. If more and more decisions are made through AI, we may fall into the helplessness created by its black box, unable to understand any of them.
How, then, can we demand an explanation from AI? Must we somehow read its inner thoughts to discover why it made a particular judgment?
A technology that makes this seemingly impossible “AI mind-reading” possible was recently announced: NLA (Natural Language Autoencoder), released by Anthropic in May 2026. It converts the numbers inside a model into sentences people can read.
🔍 NLA — turning numbers into sentences
Inside a language model is a succession of numerical vectors. Each time a word enters the model, a state comprising thousands of numbers is updated at every layer. Somewhere in those numbers is what the model is currently “thinking.” The problem was that people could not read them.
NLA, released by Anthropic researchers on May 7, 2026, translates these vectors into sentences[2]. It consists of two modules: a translator that takes a vector and writes sentences (an activation verbalizer), and an inspector that reconstructs the original vector from those sentences alone (an activation reconstructor). The two are trained together through reinforcement learning, with just one objective: how closely the inspector's reconstructed vector matches the original.
[2] Fraser-Taliente, Kantamneni, Ong et al., “Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations,” Anthropic, Transformer Circuits, May 7, 2026.
The distinctive feature is that there is no answer key. Earlier interpretability methods either had humans label a vector's meaning or discovered patterns without labels but produced numerical outputs that humans still had to interpret. By contrast, NLA becomes faithful in its translations without ever instructing its translator to “write clearly” or “be honest.” Since the inspector must reconstruct the vector using only the sentences, reconstruction fails if the translator omits information contained in the vector. Faithfulness of explanation is thus designed as a by-product rather than as the objective itself.
According to the paper, the reconstruction accuracy of an NLA trained this way, measured as the fraction of variance explained (FVE), lies between 0.6 and 0.8. It rose as training progressed from a starting point of 0.3–0.4 after supervised warm-up alone, before reinforcement learning. The method was also applied to Claude Opus 4.6, Haiku 4.5, and Mythos Preview. The output consists of three or four lines describing what the model is thinking at a particular word position.
I will discuss what it uncovered in the next edition, but let me briefly mention the finding that attracted the most attention. During safety evaluations, NLA detected an internal state interpreted as meaning that the model “knows it is being evaluated, although it has not said so openly.” This signal appeared far more often in evaluation conversations than in real user conversations.
🧩 Three limitations of the translator
NLA is an excellent tool for peeking inside AI's black box, but it is still a leap to say it can “read AI's mind.” The paper is quite candid about its limitations, which fall into three categories:
The researchers therefore did not simply trust NLA's outputs; they cross-validated them. For example, they slightly altered an explanation, changing “rabbit” to “mouse,” asked the inspector to reconstruct a vector, injected that vector into the model, and observed whether its behavior changed. They also varied prompts to check whether the signals moved as expected.
🧭 Closing thoughts
It is worth breaking down the term “explainability” for a moment. It has two layers.
One is technical explanation: reading what happened inside the model, or interpreting its internal state. NLA is accomplishing this part. The other is institutional explanation: giving the affected person a reason they can understand for why a decision was made about them, and establishing where they can challenge it if they believe it is wrong.
These two layers do not connect automatically. NLA has opened up the first kind of explainability, but the second will probably require institutions as well as technology. At least three things will be needed: procedures for verifying the translator, a party responsible for the translation, and remedies when mistranslation occurs.
Ten years have passed since the court in Loomis drew the line at “use it only as a reference.” As AI interpretation tools advance, that discussion may begin again.
In the next edition, we will examine one of NLA's most notable findings: the model “knowing it is being evaluated,” which I briefly introduced above.
#BetweenTechAndRules #Explainability #NLA #Anthropic #YuvalHarari #AIRegulation