GPT-2's Vocabulary Visualized in Hyperbolic Space
32,070 tokens in an explorable Poincaré ball
1Language Model in Hyperbolic Space
GPT-2's 32,070-token vocabulary has been modeled in hyperbolic space to reflect its natural tree structure. Using a Poincaré ball, this visualization presents word relationships in three dimensions. Users can explore this structure through a touch interface on their devices.
2The Technology Behind It
The project is based on the idea that hierarchical relationships in language models are better represented in hyperbolic rather than Euclidean space. Hyperbolic geometry's exponentially increasing area from the center naturally accommodates complex word relationships, using an exact mathematical layout without optimization.
3User Experience
The interactive model works on mobile devices, offering a touch-based navigation interface. When a token is tapped, a Möbius transformation reorganizes the entire space. Users can explore word relationships by tapping neighboring tokens in this immersive visualization.
4Structural Analysis
The visualization reveals one giant tree with 2,300 tokens, hundreds of smaller word families, and 6,700 isolated tokens. This distribution showcases the hierarchical nature of word relationships in the language model, with hyperbolic space naturally containing these complex tree structures.
5Technical Details
The project uses only raw token embeddings from GPT-2-small model. Instead of automatic embedding algorithms, it employs mathematically precise hyperbolic space arrangements. This approach preserves the natural similarity relationships between words while enabling effective visualization.
6New Perspective on Language Models
This work provides an innovative tool for understanding the inner workings of AI language models. The use of hyperbolic geometry may set new standards for visualizing complex word relationships. It serves as a valuable resource for researchers and language enthusiasts alike.
Original article: r/MachineLearning — GPT-2'nin Kelime Dağarcığı Hiperbolik Uzayda Görselleştirildi