Reading Made-Up Minds

There is a durable objection to neural networks, in print at both ends of their history. What a trained network knows, it goes, is stored as a wall of numbers that carry no visible meaning. That, rather than anything special about learning itself, is why learning gets such emphasis when these machines are discussed. Any computer can acquire behavior without being hand-programmed; only here does hand-programming seem unthinkable. The weights look meaningless, so training is not one way to program the machine among several. It looks like the only way there could be.

The objection has a lineage. Minsky and Papert pressed a version of it in “The New Connectionism,” the epilogue to the 1988 edition of Perceptrons; Yudkowsky's “giant inscrutable matrices of floating-point numbers” is the same thought from a different corner of the debate, thirty-four years later. The careful versions carry a hedge: the numbers look meaningless, from the surface. The hedge turns out to be the part that mattered. This essay is about what has been found beneath the surface. It ends with two results of my own.

Sharpening a blurry image

Ted Chiang gave the deflationary reading its most memorable image: ChatGPT as “a blurry JPEG of the Web.” Take the image seriously and it cuts the other way. Nobody opens a JPEG in a hex editor, sees noise, and concludes the picture is gone; the format assumes a viewer. The question about the weights was never whether they look meaningful. It was whether such a viewer could be built.

Inside the network, the compression has a mechanism. A model represents far more concepts than it has dimensions to give them, so features get packed into overlapping directions, each smeared across many neurons. This is superposition, and it is why staring at any single weight teaches nothing. But the blur is not Chiang's. His was loss: what a lossy copy forgets is gone for good. This is overlay, many sharp features occupying the same space, like exposures stacked on one negative, and overlay is the kind of blur that can be sharpened.

The sharpening is under way. Sparse dictionaries pull a model's activations apart into millions of individually readable features; concepts turn out to be carried by directions in the residual stream, recoverable and causal. Turning one such feature up made a version of Claude insist it was the Golden Gate Bridge. The test is always interventional: find the internal thing, move it, watch the behavior follow on cue. Notice, too, where the reading happens: in the activations, the network's running state, more than in the weights themselves. The real meaning was the activations along the way.

Lost in compilation

A binary executable is also a wall of meaningless numbers. Open any compiled program in a hex editor and you will find the same semantic vacuum the objection describes, yet nobody concludes that ordinary software can only be grown, never written. The difference is historical rather than mathematical: for compiled code we know a source existed, because we wrote it. Learning is a compiler whose source is data. It emits a program in a target language nobody reads, and the source it compiled from never existed as text at all. Half of the objection is simply correct: no one can write the weights by hand, and nothing suggests anyone ever will. But somewhere along the way can't author quietly became can't understand, and those are different claims. When the source is missing, engineers reach for decompilers.

Fragments of recovered source now exist at every scale. A two-head induction-head circuit copies and continues patterns and accounts for much of in-context learning; it has a name and a mechanism. Attribution graphs trace individual computations through a frontier model step by step: a model planning a rhyme several tokens before it writes toward it, a factual recall composing intermediate steps rather than pattern-matching the answer. They are pieces of the program, read back out of the weights.

And there is crude write access. Clamp a feature, add a steering vector, patch an activation from one run into another, edit a stored fact in place. None of this is authoring, and much of it is clumsy. But all of it is programming without retraining, the way a binary gets patched without recompiling: writes addressed to the semantics rather than to the coefficients. The claim that learning is the only practical way to get behavior into the machine is still true at the level of construction, and already false at the level of adjustment.

Coming to terms

The objection always had a sharper form than unreadable. A distributed code, the argument went, lacks the machinery of symbolic programs: variables, bindings, slots with fillers. Fodor and Pylyshyn made this an argument about systematicity; the frames tradition made it an architecture, knowledge as structures with terminals to fill. A smear of coefficients has nothing to point with.

Yet where interpretability succeeds, slot-and-pointer structure is disproportionately what it finds: features that behave like variables, heads that move bindings between positions, low-rank subspaces that route a query to one stored value rather than another. The learned program keeps turning out to contain the machinery a hand-written one would have used. Opacity meant symbolic structure in an unread encoding, not the absence of symbolic structure. That learning builds this unasked was not guaranteed, and it should unsettle both camps a little: the symbolists were right about the machinery and wrong about where it could live.

Open minds

Once the network is something you can open, a comparison inverts. With a human being, belief-tracking is inferred entirely from behavior: we give someone a false-belief task and reason backwards from the answer. With a model we can run an interchange intervention, patching an internal subspace from one prompt into another, and watch the interpretation flip while everything else stays put. The mind is the system we can only probe from outside; the model is the one we can open.

Weights and hidden biases. The same asymmetry reaches decisions that were never going to be interpretable at all. For a human judge, the counterfactual, the same case with only the name changed, does not exist; audit studies approximate it across a population, expensively, once. For a model the exact counterfactual is a single call: same weights, one token changed, rerun after every mitigation. Even where the mechanism stays dark, the machine sits behind a door you can knock on as often as you like, with inputs you choose. That is a stronger experiment than we have ever run on the human systems these models stand in for.

What this does not claim

A frame of mind

I close with two results of my own. They are not the field's most central, and I do not offer them as such. I offer them because they are mine, so I can vouch for every step, and because they land on the oldest version of the objection: whether a network can bind a variable to a slot.

A made-up mind, in miniature. Tell a capable model “Anna believes the cup is blue; in reality it is red,” and it answers blue about Anna and red about the world, holding two incompatible values for one object without collapsing them. Anna is a fiction held inside an artifact, a made-up mind within a made-up mind. Where does the model keep her? Not in a fog. We locate the separation as a router over a shared value slot. A generic slot binds the attributed content, the cup's color, and carries no tag for whose view it is; intervening on the slot moves a reality readout as strongly as a belief one. The separation lives elsewhere, in a low-rank routing subspace at the query position that selects which frame, the character's belief or the world, a given question reads out. Train that subspace with a distributed-alignment intervention and you can flip belief and reality causally, without touching the stored value, across several model families. The same slot-and-router format recurs across the other non-actual contexts a sentence can open (counterfactual, fictional, temporal), and belief, which formal semantics marks as a special case, turns out not to be specially separated. That is a slot, a pointer, and a dispatch on context: parts of a program, found where the objection said none could be.

How the methods work, in explainer form: mental spaces · false-belief tasks · interchange interventions · subspace geometry.

The weights still look like nothing. So does any compressed file. Learning remains the only way to put the knowledge in. It is no longer the only way to get it back out.

A position piece, prompted by Ken Kahn's “Claude, with feedback from ChatGPT, writes a history of AI” and the correspondence around it. The title is borrowed from Gary Drescher's Made-Up Minds (1991). Companion to Hiding Figures.