Mechanistic interpretability: First study on disentangling convolutional neurons
Novel techniques for single-neuron level feature disentanglement in neural networks
1Single-neuron analysis
The researcher began by examining a single 1x1 convolutional neuron in the InceptionV1 model. Using Hadamard product, they revealed the patterns the neuron 'sees'. This method offers a new technique for understanding which visual features trigger neuron responses.
2Discovery of clean clusters
The method revealed clean, monosemantic clusters of known activations like cars, cats and dogs. But it also detected unexpected patterns with lower activation values such as letters and human faces, demonstrating the complexity of neuron behavior.
3Analysis of hidden patterns
In low-activation clusters (like letters), dependent neurons were observed firing on the same concept. The balanced distribution of positive and negative weights among these suggests deliberate behavior by gradient descent.
4Visualization support
The study includes Distill.pub-style rich visualizations. These graphics make it easier to understand the patterns detected by neurons and concretize the method's effectiveness.
5Convolutional challenges
The researcher considers starting with convolutional neurons a mistake due to limited interest in this area. However, plans to apply the techniques to language models demonstrate the work's potential breadth.
6AI transparency
This study represents a significant step in illuminating the black box nature of neural networks. Neuron-level feature disentanglement could open new horizons in model interpretability and pave the way for more reliable AI systems.