From vectors to the Transformer architecture that changed everything.
Machines don't read letters. They split text into tokens — chunks that might be whole words, parts of words, or single characters. Each token gets a unique number (an ID). The model only ever sees these numbers.
If every word were a token, you'd need millions of tokens to cover all languages, typos, code, and made-up words. The vocabulary would be enormous and most tokens would rarely appear, making them impossible to learn well.
BPE solves this by learning subword units. Common words stay whole. Rare words decompose into known pieces. The model can handle any input — even words it's never seen — by combining subword tokens it has seen.
Example: "ChatGPT" might tokenize as ["Chat", "G", "PT"]. The model knows "Chat" from other contexts, "G" as a letter, and "PT" from abbreviations. It can reason about the whole word from its parts.
Each token ID maps to a vector — a list of numbers (typically 768 to 12,288 of them). Think of it as coordinates in a high-dimensional space. Here we project them down to 3D so you can see them. Drag to rotate. Scroll to zoom.
A vector like [0.2, -0.5, 0.8, ...] doesn't have one "meaning dimension." Instead, meaning emerges from relative positions. "King" and "queen" are close together. "King" and "banana" are far apart. The distance and direction between vectors IS the meaning.
This is called a distributed representation. No single number tells you the word is an animal. But the pattern across all 768 numbers, taken together, encodes everything the model knows about that word.
The embedding matrix is literally a lookup table: token ID 4521 → row 4521 of a giant matrix. That row is the vector. The matrix starts random and is learned during training.
The most famous discovery about word vectors: king − man + woman ≈ queen. Vector arithmetic captures analogies. The direction from "man" to "woman" is the same direction as "king" to "queen." Meaning is geometry.
Mikolov et al. (2013) at Google discovered this with Word2Vec. The idea: train a small neural network to predict a word from its neighbors (or vice versa). The hidden layer weights become the word vectors. The prediction task forces the network to encode meaning into the vector positions.
Why does arithmetic work? If "king" and "queen" appear in similar contexts except for gender-related words, the network learns to represent them identically except along the "gender direction." Subtracting "man" and adding "woman" moves along that direction.
Modern LLMs use contextual embeddings (each word's vector changes based on the sentence it's in), but the geometric intuition still holds. The embedding space is where meaning lives.
How do you measure if two vectors point in the same direction? The dot product. Multiply corresponding numbers and add them up. If it's large and positive, the vectors are similar. Near zero means unrelated. Negative means opposite.
The dot product scales with vector length: longer vectors have larger dot products regardless of direction. To isolate the direction (meaning similarity, not magnitude), we normalize: divide by both vector lengths. This gives cosine similarity: $\cos\theta = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}||\vec{b}|}$
Cosine similarity ranges from -1 (opposite) through 0 (orthogonal) to +1 (identical direction). In embedding space, "cat" and "kitten" might have cosine similarity 0.85. "Cat" and "democracy" might be 0.05. The model uses this to decide which words should influence each other.
This is the insight from Google's 2017 paper "Attention Is All You Need" (Vaswani et al.) that changed everything. Each word asks a question (Query), advertises what it contains (Key), and holds its information (Value). Attention computes: for each Query, how well does it match each Key? Then it mixes Values weighted by those match scores.
Before 2017, the dominant approach for language was recurrent neural networks (RNNs/LSTMs). They processed words one at a time, left to right, carrying a hidden state. Two fatal problems:
1. Sequential bottleneck. You can't parallelize. Word 100 has to wait for words 1-99 to finish. Training was painfully slow.
2. Long-range dependencies. By word 100, the hidden state has been overwritten so many times that information about word 1 is mostly lost. LSTMs helped but didn't solve this.
The Transformer fixes both. Attention connects every word directly to every other word, regardless of distance. Word 100 can attend to word 1 with a single matrix multiplication. And the entire sequence processes in parallel — massive GPU speedups.
The name "Transformer" comes from the idea that the network transforms representations through layers of attention. The paper was authored by Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin at Google Brain and Google Research.
Think of a database lookup. You have a query ("find me articles about cats"). The database has keys (article tags). Each key is associated with a value (the article content). You match your query against all keys and retrieve the matching values.
Attention does the same thing, but soft. Instead of exact match (returns 1 article), it returns a weighted blend of ALL values, where the weights are determined by how well each key matches the query. A good match gets weight 0.5, a mediocre match gets 0.05, a bad match gets 0.001.
The Query, Key, and Value vectors are created by multiplying the input embedding by three different learned weight matrices: $Q = XW_Q$, $K = XW_K$, $V = XW_V$. The model learns what to ask (Q), what to advertise (K), and what to send (V) during training.
A Transformer is a stack of identical blocks. Each block has two stages: multi-head attention (tokens talk to each other) and a feed-forward network (each token thinks independently). Residual connections and layer normalization keep the signal stable through dozens of layers.
Attention treats input as a SET, not a sequence. "The cat ate the fish" and "The fish ate the cat" would look identical without position information. The Transformer adds a positional encoding to each embedding so the model knows word order.
The original paper used sinusoidal functions: $PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d})$ and $PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d})$. Each position gets a unique pattern of sines and cosines at different frequencies. The model can learn to read relative positions from the interference patterns.
Modern models (GPT, Claude, Llama) use learned positional embeddings or Rotary Position Embeddings (RoPE) instead. RoPE encodes position by rotating the query and key vectors, which elegantly makes the dot product depend only on relative position, not absolute.
Imagine a 96-layer network. Each layer is a function $f_i$. Without residuals, the output is $f_{96}(f_{95}(...f_1(x)))$. The gradient must backpropagate through ALL 96 functions. If any function's gradient is slightly less than 1, the signal decays exponentially. This is the vanishing gradient problem.
With residuals, each layer computes $x + f_i(x)$ instead of $f_i(x)$. The gradient gets a direct path: the "+ x" creates a shortcut that carries the gradient straight through, undiminished. The layer only needs to learn the residual — what to add — not the entire transformation. This idea came from He et al. (2015, ResNet) and is why we can build networks with hundreds of layers.
An LLM generates text by predicting the next token, then feeding that prediction back as input, repeating until it decides to stop. At each step, the model outputs a probability distribution over the entire vocabulary. Temperature controls how random the choice is.
Greedy decoding (always pick the top token) produces repetitive, boring text. It gets stuck in loops because the most likely next token after "The cat is" might always be "a" and then "cat" again.
Top-k sampling: Only sample from the top k most likely tokens. If k=50, tokens ranked 51+ get zero probability. This prevents very unlikely (often nonsensical) tokens from being chosen.
Top-p (nucleus) sampling: Instead of a fixed count, include tokens until their cumulative probability exceeds p. If p=0.9, you include enough tokens to cover 90% of the probability mass. More adaptive than top-k: sometimes 5 tokens cover 90%, sometimes 200 do.
In practice, production LLMs use a combination: top-p with a moderate temperature. Claude, GPT, and others tune these per-task for the right balance of coherence and creativity.
The remarkable finding of the last 5 years: making the model bigger, with more data, reliably makes it smarter. This is the "scaling law" (Kaplan et al. 2020). Loss decreases as a power law with model size, dataset size, and compute.
GPT-2 (2019): 1.5B parameters. Could write paragraphs. GPT-3 (2020): 175B parameters. Could write essays and do few-shot learning. GPT-4 (2023): rumored ~1.8T parameters (MoE). Can reason, code, pass exams.
Nobody fully understands WHY this works. The Transformer architecture seems to be unusually efficient at extracting and storing patterns from text. More parameters = more capacity to store patterns = more things the model "knows." But whether there are fundamental limits, or whether scaling will continue to produce qualitative leaps, is an open research question.
Built for Autodidactive — learn by doing, not by reading.
The math is real. The visualizations are simplified projections of high-dimensional operations.
Paper: Vaswani et al., "Attention Is All You Need" (2017)