GPT-2 is a language model OpenAI released in 2019. It turns every token of text into a list of 768 numbers, looked up from a table it learned during training. This map is that table, flattened so you can see it: each dot is a word, and words the model treats as similar sit close together.
GPT-2 was trained to do one thing: read some text and guess the next token. It practised on about 40 GB of web pages, and every time it guessed, all of its numbers, including this table, were nudged towards a better guess. Words that fit into the same sentences end up with similar numbers. That's why unhappy sits right next to happy. The meanings are opposite, but the words turn up in the same places.
To predict the next word, GPT-2 looks up each token's row, passes them through 12 layers that mix in context from the rest of the sentence, and then scores the result against every row of this same table. The closest rows become the likeliest next words. So a word's neighbours here are roughly the alternatives the model weighs up whenever it picks that word. They also help it generalise: what it learns about glad partly carries over to pleased.
Closeness is measured by cosine similarity, which checks whether two lists point in the same direction. It's the percentage shown next to each neighbour. There's only one row per word, so bank has a single position, and its neighbours are all about money.
UMAP and t-SNE squash 768 dimensions into two while trying to keep neighbours together. Words that are close on the map are probably close in the model, but distances between clusters mean little, and some neighbours get pulled apart. The colours come from the Louvain algorithm, which grouped words by their neighbours in the full 768 dimensions. Nobody labelled them; each label is just the three words nearest the middle of its group.
Switching between UMAP and t-SNE is a good check, because groupings that appear in both are more likely to be real. The 3D view is a separate UMAP run in three dimensions. GPT-2's full vocabulary has 50,257 tokens, and we kept the ones that are 3 to 20 plain letters long. Some are still word fragments (inding, aceutical), because GPT-2 splits rarer words into pieces.
| Model | Year | Parameters | Numbers per token |
|---|---|---|---|
| GPT-1 | 2018 | 117M | 768 |
| GPT-2 small (this map) | 2019 | 124M | 768 |
| GPT-2 XL | 2019 | 1.5B | 1,600 |
| GPT-3 | 2020 | 175B | 12,288 |
| Llama 3.1 405B | 2024 | 405B | 16,384 |
OpenAI hasn't published sizes for its models since GPT-3. Newer models have wider tables and larger vocabularies (Llama 3 has about 128,000 tokens), and are thousands of times bigger overall. The basic idea hasn't changed, though: each token still gets one learned list of numbers. GPT-2 small is just small enough to fit on one screen.
These are the numbers GPT-2 starts with, before it has read any context. Its later layers are what tell a riverbank from a savings bank, so this is closer to the model's dictionary than to its reasoning. It's also a picture of one 2019 model trained on web pages, not of English in general.
Hover or tap a dot to see its neighbours. Click to pin it, and Esc clears it. Selected words go into the page address, so you can share them.
Built with D3.js and Three.js. Inspired by WEIGHTS, which printed every parameter of GPT-1 as a set of books.
Model weights ↗
·
Source code ↗