Causally Active, Not Decorative
The best place to disrupt a concept inside a transformer isn't where the concept is easiest to see. Take the layer where a concept--say, negation, or sentiment, or deception--separates most cleanly from its opposite, and knock it out. Suppression is 3.59x stronger than knocking out a layer just outside that peak (p = 1.58x10^-141), and 606.6x stronger than knocking out a random direction of the same size. So whatever's sitting at that peak is real, not noise. But the peak isn't even the best place to hit it.
I have two methods for this work. One, which I call the Concept Allocation Zone (CAZ), finds that peak--the region of the network where a concept is most clearly represented. The other, Geometric Evolution Maps (GEM), tracks how the concept's internal representation rotates as it forms, and marks the layer where that rotation stops--the "handoff" layer, where the model has committed to a direction instead of still assembling one. Move one layer past where CAZ's zone ends, to the point GEM says the concept has settled, and the damage gets worse, not better--in 304 of 442 model-concept pairs, 68.8% overall. One example: knocking out the peak for negation in GPT-2-XL leaves 59% of the original separation intact. Knocking out the settled point one layer later leaves 31%.
CAZ finds the neighborhood. GEM finds the address. And the address turns out to matter more than the neighborhood did.
What It's Checking
I published both methods in May, meant to work as a pair--CAZ for where, GEM for exactly where. But locating something and proving it matters are different claims. A zone that lights up when a model is thinking about deception might just be correlated with deception, the way a smoke detector's LED is correlated with fire without being the thing that's burning. Neither original paper, on its own, proved the model's behavior actually depends on what's sitting there.
So the study I just finished--internally I've been calling it P3--goes back with a fresh 28-model corpus across 8 architecture families and 17 concepts, and does the thing a location claim alone can't: reach in and disable what CAZ and GEM say is there, together, and check whether the model's behavior actually changes.
What Held
The handoff advantage isn't uniform, and I'd rather show you that than hide it. Across the full 26-model extension, GEM's settled point beats CAZ's peak in 73.7% of MHA model-concept pairs, 63.9% of GQA pairs, and just 44.1% of Gemma pairs--Gemma turns out to localize its concepts differently--at its final layer of global attention rather than at the peak CAZ identifies at all. The advantage is real and broad. It is not the same size everywhere, and I'm not going to pretend otherwise.
The gentle-CAZe result is the one I trust most, because it's the one I was least sure about going in. The original CAZ paper flagged "gentle CAZes"--allocation regions with a weak geometric signature, easy for a detector to miss entirely. P3 hits them with the same activation-patching protocol used on the obvious zones, and they hold up: 63.2 percentage points of mean suppression, nearly matching the major zones despite looking like nothing on a separation curve. The part of the theory that looked the most like wishful thinking is the part that turned out to be the most causally real.
The multimodal-zone claim held harder than expected, too. 76.3% of concept-model pairs assemble across more than one region of the network--not a fixed layer, a genuine zone, exactly as the original paper argued, now on a corpus built specifically to break it.
The concept-ordering claim held, in a more honest shape. Syntactic concepts still emerge earliest in a model (21-37% depth) and behavioral concepts still localize deepest (68-81%), with a statistically robust correlation across 27 of the 28 models (Wilcoxon p = 1.49x10^-8). The one exception, GPT-2-medium, isn't a counterexample so much as a shrug--its ranking comes out an exact tie, not a reversal. Gemma is a separate story: its models show the ordering too, but the underlying directions are unstable enough under resampling that I don't trust the measurement at this sample size, so I'm reporting it as unreliable rather than confirmed. P3 downgrades the whole claim from what the original paper implied to what the recomputation actually supports: a moderate-strength tendency, not a clean law. The ordering is real. It's just not as tidy as I first made it sound, and for one architecture family I'm not confident I measured it well enough to say.
What Didn't
The GEM paper's original claim was that MHA architectures show a meaningfully stronger handoff effect than GQA architectures--78.3% versus 47.1% of trials. On the 28-model corpus, that gap is gone. Mean separation reduction is 0.588 for MHA, 0.625 for GQA. Comparable, not different. That finding, along with the ordering downgrade above, is what fed a v2 correction to both original papers. I'd rather find out my own validation study broke my own claim than have someone else find it first.
I'd also pre-registered seven predictions before running any of this, specifically so I couldn't talk myself into a good result afterward. None of the seven came back simply confirmed. I was wrong about the details more often than I was right. I was not wrong about the thing that actually matters: whether the zones are doing anything.
What's Actually Solid Now
Two papers is a claim. A third study built specifically to try to break the first two, that keeps the mechanism and drops the two claims that didn't survive contact--that's closer to settled. Concepts assemble across zones, not layers. Those zones, including the faint ones, are causally load-bearing, not decorative. The architecture-specific story about which model families show it more is not something I actually know yet, and I've stopped claiming otherwise.
It's up at research.henrynet.ca, data and code on Zenodo. arXiv put it on a moderation hold with no paper-specific reason given--my guess is the disclosed AI assistance tripped a filter aimed at LLM-generated survey mills, though I can't confirm that.
Related: Enter the Concept Allocation Zone, Using CAZ to See What the Models Are Thinking, Weights and Measures.
James Henry is a Senior Security Consultant and independent AI/ML interpretability researcher, he saw Rush last week and keeps thinking that "All this machinery making modern music can still be open hearted." very much applies to the modern AI discussion. He publishes at waypoint.henrynet.ca.
Discussion