The Weights Already Know

August 25, 2026 — James Henry

Last week's post on the ARC White-Box Estimation Challenge--predict what a neural net would output, from its weights alone, without ever running it--ended on a throwaway line about the organizers' own write-up: their "toward non-random networks" section names the thing the whole challenge doesn't have yet--a population of trained networks with shared statistics, to diff against the random-init null everyone's been computing all year. They call it future work and move on.

I'd been sitting on the other half of that same gap for months. This spring's post introduced the Concept Allocation Zone and Geometric Evolution Maps--work that never once had a matched population to check itself against: same architecture, same init, one group trained and one left alone. Without that comparison, every "look, structure!" finding is unfalsifiable. Maybe it's learned. Maybe it's just what this architecture looks like before you touch it.

Two blog posts, two blind spots, one obvious missing object sitting between them. So I built it.

1,569 bias-free ReLU MLPs at ARC's exact Phase-1 spec--depth 32, width 256, He-Gaussian init, no skip connections, no normalization. 69 families spanning widths 64 to 512, depths 8 to 64, three training budgets, a weight decay sweep across three orders of magnitude, real classification tasks and whitened MNIST. Every net trained with full provenance--task, hyperparameters, seed, outcome versus the exact Bayes-optimal accuracy. The corpus and paper are public now, archived on Zenodo: Task Rank Is Imprinted in the Input Layer.

The question the whole thing answers: given a trained network's weights, before you ever run it, what do you already know about where its learned structure will show up?

Some of the seven findings are worth a paragraph and a receipt. Four of them are worth the rest of this post. Quickly, the other three:

An exact law, not a trend. On a C-class task, every converged net's first layer carries exactly C-1 significant dimensions--the rank of the simplex the class centers span--with zero exceptions across 240 nets. Real MNIST hugs the same line.
Two seeds, same code, different coat. Train the same task on two different seeds and their raw activation directions look like strangers--chance-level overlap. Fit an orthogonal Procrustes rotation between the two spaces on held-out data--pure rotation, no scaling, no shear, the fit gets zero freedom to fake a match--and the overlap jumps to 0.99 in early layers. Every claim about models "converging to the same representation" needs that caveat now.
A law I pre-registered and got wrong. I predicted C₅₀ ≈ 2.7 × ceiling(width) for where a fixed-size network stops converging. It broke at width 64. The real wall isn't a hard capacity limit--it's a compute frontier that grows slowly with width and training budget. A width-512 network that flatly couldn't learn 64 classes at the original budget gets most of the way there at ten times the steps.

All three are worth more than a paragraph--the Procrustes result especially is probably a post of its own. Today they're Sir Not Appearing In This Film.

Predicting the network without running it

This is where the ARC connection stops being background and becomes the method. ARC's challenge is entirely about propagating a distribution through a network's weights analytically--no sampling, no forward passes, just formulas--and reading off what comes out. I pointed that same machinery at these 1,569 trained nets instead of random ones and asked what it predicts about a network I haven't run yet.

Push a standard-normal input distribution through the trained weights, purely on paper, and track how spread out the resulting representation is at each layer. I call it the q-clock--q is the effective dimension of that layer's representation divided by the width, so it winds down from fully spread (1.0) to fully collapsed (near 0) as you go deeper. Random networks all wind down to the same fixed point, set entirely by depth. Trained networks don't. They stop wherever the task's own difficulty says they should--more spread for a 20-class problem, less for an easy 2-class one.

That prediction, computed entirely from the weights file with no data and no execution, matches what you measure later by actually running the network and looking at its activations. Two completely different instruments--one that never touches data, one that only touches data--arrive at the same answer. A model can be checked against its own architecture's null straight from a weights file, before deployment, without anyone choosing an input distribution or being allowed to run it at all.

Weights carry the where, input sharpens the what

Push genuinely structureless noise through a trained network--not the task, just N(0, I) vectors matched to the task's mean and variance--and the activations already show the same rank structure real input would produce. The actual input sharpens the picture a little. It doesn't create it.

The interesting boundary isn't "does the model see real data." It's "was this weight matrix trained." That's a stronger claim than the usual mechanistic-interpretability framing, which tends to assume the interesting thing only happens once you feed a model real, meaningful input.

Worth being precise about what "input" means here, because in an LLM it isn't the training data--it's the context window. Whatever's sitting in there when the model runs, put there by a human typing a prompt or an agent stuffing in tool output, is the input this finding is about. If the geometry is mostly set by training and only sharpened by input, then a lot of what a context window is doing is pointing at structure the weights already have, not building new structure from scratch.

Most of what training touches doesn't matter

Sweep weight decay from zero up to a value strong enough to shrink the network's random, task-irrelevant weight mass down to a thousandth of its starting scale--three full orders of magnitude of deletion--and accuracy moves less than 0.008. The network computes exactly the same function at exactly the same performance while almost everything a naive census would call "structure" gets erased.

A lot of what interpretability tooling flags as interesting weight-space structure is, functionally, noise the optimizer never bothered to clean up--present, measurable, and irrelevant to what the network does. A census that scores that bulk as meaningful is measuring optimizer hygiene, not learning.

Borrowing back my own tool

This is the one that ties the loop shut. The analytic weight-propagation machinery from predicting the network without running it is accurate on random networks--a few percent error, all the way through 32 layers. Point the same machinery at trained weights and the error blows past 80% by the mid-network.

The formulas assume the weights behave like a fresh random draw--no structure between entries, nothing correlated. Training is the process of building exactly that structure. The C-1 law, the task-tracking q-clock, all of it is correlation the formulas were never built to expect, so they degrade fast once it shows up.

The fix is the trajectory-fitted stabilizer from the ARC challenge itself, refit on this trained population instead of ARC's random one. It repairs the error 7 to 12x in the middle layers. It also fails cleanly on a held-out set of half-trained networks it never saw--which is the honest result you want: the correction is real, and it's bound to the population it was fit on, not a universal fix.

The lens, not the number

If there's one sentence that's the point of all seven findings, it's this: a trained network's activation geometry is a property of its weights that inference reveals rather than creates. Where structure lives, how much of it there is, how long it persists--all of that is fixed before a single input is ever run. What inference adds is sharpening, and a coordinate system to read the answer in.

The corpus, the eight audited instruments, and the full paper are public now--research.henrynet.ca/model-zoo-cartography, archived on Zenodo. It's still a draft (v0.12 as I write this), one architecture family, GMM tasks that are linearly separable by construction. The obvious next step mirrors the method that got me here: pre-register what a census should read on some wild, AdamW-trained model given its claimed training--then go check, the same way I checked myself on the class-count wall and got it wrong in a way worth publishing.

James is a security engineer who came in 133rd in a math competition and, apparently unsatisfied, went and built the missing control group himself. He publishes at waypoint.henrynet.ca.