Transformers Exhibit Relativity-Like Positional Encoding Effects
New research reveals how paragraph structure influences transformer attention mechanisms
1Position encoding analogy
The research draws parallels between general relativity's coordinate systems and transformer positional encoding. Just as GR's metric depends on position rather than just coordinates, transformers may need more sophisticated position representations. The author designed experiments to test whether standard positional encodings fully capture hierarchical text structure like paragraphs and sentences.
2Hierarchical text structure
Human readers perceive text as a hierarchy (document > paragraph > sentence > token), not just a linear sequence. The same sequence of tokens can have 2^(n-1) possible paragraph segmentations. Standard positional encodings like RoPE and ALiBi only capture linear position, potentially losing important structural information that affects how humans understand text.
3hRoPE design
The author created hierarchical Rotary Positional Encoding (hRoPE) that gives paragraph, sentence and token indices their own independent channels. This allows changing paragraph coordinates while holding other position information constant. The design maintains separability - each level rotates its own block of dimensions without interaction between levels.
4Experimental controls
Three controlled variants were created: rand_axial (random paragraph labels), period_axial (mechanical grid), and manipulations where real paragraph boundaries were hidden (fake-merge) or fake boundaries inserted (fake-split). These isolate whether effects come from real paragraph structure versus just having extra coordinates or patterns correlated with token distance.
5Attention pattern changes
Experiments showed removing real boundaries lowers cross-paragraph attention, while inserting fake boundaries raises it. Random paragraph labels also affected attention, but differently than real structure. This proves paragraphs influence attention beyond just token distance, though some effect comes merely from having extra coordinates.
6Paragraph displacement effects
The research measured how attention suppression grows with paragraph displacement (number of boundaries between tokens). Real models showed a 'well' pattern where suppression increases up to 2-5 paragraphs then decreases. The depth of this well (U*) varied meaningfully across corpora for real models but not controls.
7Shuffling test results
Shuffling paragraph order affected real models' attention depth in all corpora (WikiText-2, OpenWebText, Python code), while random-label controls showed no change. This demonstrates depth reflects genuine paragraph structure rather than being an artifact of the encoding channel's density.
8Corpus-dependent patterns
The hierarchical model showed corpus-sensitive depth variations (range 0.46) while the control's depths clustered narrowly (range 0.11). Attempts to explain depth using corpus statistics like lexical persistence or paragraph length were partially successful but incomplete, leaving room for further research.
9OpenWebText anomaly
Unlike other corpora, OpenWebText showed no depth difference between real and random models, though the shuffle test still distinguished them. This may reflect either a true null effect or limited statistical power - an important open question for future investigation.
10Relativity analogy confirmed
The research concludes transformer attention exhibits coordinate-dependent response fields over hierarchical position, similar to how GR separates coordinates from metric. Standard positional encodings provide coordinates, but attention computes over a more complex 'distance' that depends on text structure.