Neural Network Compression Reveals Geometric Patterns in Superposition
A tiny model's pentagon formation demonstrates how AI packs more features than dimensions
1Unexpected geometric patterns
When compressing five equally important features into two dimensions, the neural network arranged them into a near-perfect pentagon. This geometric pattern emerged naturally through gradient descent, without any explicit symmetry constraints in the loss function. The angles between feature vectors measured 71.5-72.5 degrees apart, demonstrating optimal packing to minimize interference between features during reconstruction.
2Polysemanticity as strategy
The experiment confirms that polysemanticity (neurons responding to multiple concepts) isn't noise but a deliberate compression strategy. When features are sparse (rarely active simultaneously), networks can share dimensions by packing features at angles. This superposition allows representing more features than available dimensions, with interference only occurring when rarely co-active features fire together.
3Hand-derived gradients
The researcher implemented backpropagation manually using NumPy without autograd libraries, deriving matrix gradients for tied weights. This approach revealed insights often obscured by abstraction layers - particularly how weight sharing affects gradient flow during both compression and reconstruction phases of the autoencoder-like architecture.
4Importance-based feature selection
With 20 features compressed into 5 dimensions and geometrically decaying importance weights, the network developed a sharp cutoff - accurately reconstructing top-priority features while effectively ignoring less important ones. This emergent behavior occurred despite the loss function only specifying continuous importance weights, not discrete selection criteria.
5Phase transition in sparsity
Experiments varying sparsity (feature activation frequency) revealed a clear phase transition around 0.8 sparsity. Below this threshold, superposition caused more harm than good through interference. Above it, the network fully committed to the superposition strategy, demonstrating a binary rather than gradual adoption of this compression technique.
6Measuring feature interference
The W⊤W matrix quantitatively measured how reconstructing one feature affected others, with off-diagonal entries showing interference levels. This confirmed the same feature importance cutoff observed visually, validating that the geometric arrangement directly corresponds to measurable interference patterns in the reconstruction process.
7Tiny model, big implications
The minimal setup (30 synthetic features, 5 hidden dimensions) successfully reproduced phenomena central to interpreting large language models. This demonstrates how fundamental compression strategies scale across model sizes, explaining why polysemanticity persists even in billion-parameter networks where manual inspection is impossible.
8Practical interpretability consequences
The findings explain why mechanistic interpretability requires specialized tools like sparse autoencoders to unpack superimposed features. When thousands of concepts are compressed into fewer neurons, individual neuron analysis becomes inadequate - the geometric packing strategy fundamentally changes how we must approach model interpretation.
9Beyond theoretical claims
The hands-on implementation validated key claims from Anthropic's 2022 paper: networks do represent more features than dimensions, they exploit sparsity, and their geometric arrangements aren't arbitrary but mathematically optimal. Seeing these phenomena emerge from actual code provides concrete understanding beyond theoretical descriptions.
10Future research directions
This work suggests promising avenues for developing better interpretability tools that account for inherent feature superposition. Understanding these compression dynamics could lead to more effective methods for probing neural representations and potentially designing architectures that maintain interpretability without sacrificing efficiency.