Analyzing 'Trump' Embedding Geometry in GPT-2 Small
Comparison of Discrete and Continuous approaches.
1Token Embeddings in GPT-2 Small
This analysis examines the 'Trump' token in GPT-2 Small's static embedding table. The model's learned token embeddings are considered before attention or context is applied.
2Visualization with t-SNE Projection
32,070 alphabetic tokens, each with at least two characters, are visualized using t-SNE projection. This method makes high-dimensional data understandable in two dimensions.
3Discretized Approach
In the discretized approach, each coordinate is thresholded before neighbors are calculated. This method reveals general political terms like Mitt, Hillary, Pelosi, and Blair.
4Continuous Approach
In the continuous approach, original coordinates are retained to calculate neighbors. This method reveals a more specific group including Obama, Clinton, Bush, and Eisenhower.
5Family Members and Rivals
The continuous approach reveals more specific relationships, such as Trump's family members, staff, and rivals. This shows how detailed the embeddings learned by the model are.
6Model Understanding and Insights
This analysis provides important insights into how embeddings learned by models like GPT-2 Small work. Understanding the inner workings of models can help develop better models.
Original article: r/MachineLearning — GPT-2 Small'da 'Trump' Embedding Geometrisi Analizi