The Architecture and Evolution of Large Language Models — From Transformer to ChatGPT, an Interactive Deep Dive

Introduction

Since the advent of ChatGPT, Large Language Models (LLMs) have transformed society. Writing code, summarizing papers, answering complex questions — all of this emerges from a remarkably simple principle: predicting the next token.

This article explains the complete picture of LLMs through six pillars:

1. History and genealogy — Evolution from Word2Vec to GPT-4
2. Tokenization — How text is converted to numbers
3. Transformer architecture — The engine of LLMs
4. Self-attention — The core algorithm of Transformers
5. Text generation — Temperature, top-k, and sampling strategies
6. Training pipeline — From pre-training to RLHF

The LLM Genealogy — A Decade of Evolution

Let's trace the evolution of language models.

The Word2Vec Era (2013)

Mikolov et al. (2013) proposed Word2Vec, which converts words into fixed-dimensional vectors (embeddings). The famous example king - man + woman ≈ queen demonstrated semantic relationships. However, Word2Vec produced static embeddings that didn't consider context.

The Transformer Revolution (2017)

Vaswani et al. (2017) changed everything with "Attention Is All You Need." By replacing RNN's sequential processing with self-attention, they enabled parallel computation. This innovation became the foundation for all modern LLMs.

The Pre-training Era (2018–)

Three architecture families evolved from the Transformer base:

Encoder-only (BERT family)

BERT (2018) is pre-trained with Masked Language Modeling (MLM), predicting masked tokens. By capturing bidirectional context, it excels at understanding tasks like classification, NER, and similarity computation.

Encoder-Decoder (T5 family)

T5 (2019) unifies all tasks as "text-to-text." Translation, summarization, and QA all share the same format.

Decoder-only (GPT family)

The GPT series uses autoregressive language models, generating one token at a time from left to right. As parameter counts increased, capabilities improved dramatically — GPT-3 (175B parameters) demonstrated few-shot learning. Today's ChatGPT, Claude, and Gemini all descend from this lineage.

Scaling Laws

Kaplan et al. (2020) discovered that language model loss follows a power law with respect to parameters $N$, data $D$, and compute $C$:

Furthermore, Hoffmann et al. (2022) (Chinchilla paper) showed that parameters and tokens should be scaled equally. For example, a 70B parameter model needs approximately 1.4 trillion training tokens.

Tokenization — Converting Text to Numbers

LLMs don't understand text directly — they process tokens, numerical sequences representing text.

BPE (Byte Pair Encoding)

The most widely used tokenization method in modern LLMs is BPE (Sennrich et al., 2016).

Algorithm:

1. Split text into character (or byte) units
2. Find the most frequently adjacent pair in the corpus
3. Merge that pair into a new token
4. Repeat until vocabulary reaches target size

# Simplified BPE pseudocode
vocab = set(all_characters)
for i in range(num_merges):
    # Find the most frequent pair
    pair = most_frequent_pair(corpus)
    # Merge into new token
    new_token = pair[0] + pair[1]
    vocab.add(new_token)
    corpus = merge_in_corpus(corpus, pair, new_token)

GPT-4 has a vocabulary of approximately 100,000 tokens. For languages like Japanese with complex morphology, a single character often maps to more than one token.

Why Subwords?

• No OOV problem: Unknown words can be represented as combinations of subwords
• Vocabulary control: A middle ground between character-level (small vocab, long sequences) and word-level (large vocab, OOV issues)
• Multilingual support: The same BPE tokenizer can handle multiple languages

Transformer Architecture

Let's examine the structure of the Transformer (decoder variant), the engine of LLMs.

A Transformer block consists of 5 core components. Let's explore each in detail.

1. Input Embedding + Positional Encoding

Converts token IDs into high-dimensional vectors (e.g., 4096 dimensions). Unlike RNNs, Transformers process inputs in parallel, so positional information must be explicitly injected via positional encoding.

Sinusoidal Positional Encoding (Original)

Vaswani et al. (2017)'s Transformer used sinusoidal positional encoding:

The heatmap below shows sinusoidal PE patterns. Lower dimensions (left) form high-frequency waves, while higher dimensions (right) form low-frequency waves.

Why sinusoids? For any fixed offset $k$, $\text{PE}(pos + k)$ can be expressed as a linear function of $\text{PE}(pos)$, making it easy for the model to learn relative positions.

RoPE (Rotary Position Embedding)

Modern LLMs (LLaMA, Gemma, Mistral, etc.) use RoPE. RoPE pairs vector dimensions and applies 2D rotations at position-dependent angles.

RoPE's key property: the dot product $q_m \cdot k_n$ depends only on the relative position $(m - n)$. This naturally captures relative positional relationships without explicitly encoding absolute positions.

RoPE also enables position extrapolation (handling sequences longer than training), forming the basis for context length extensions like NTK-aware scaling and YaRN.

2. Multi-Head Self-Attention

The heart of the Transformer. We'll cover the mechanics in detail in the next section, but here let's understand its role in the overall architecture.

A single attention head can only capture one type of relationship pattern. Multiple heads learn different patterns in parallel — subject-verb agreement, adjective-noun modification, coreference, etc.

where $\text{head}_i = \text{Attention}(Q W_Q^{(i)}, K W_K^{(i)}, V W_V^{(i)})$. GPT-3 uses 96 heads; LLaMA 2 70B uses 64 heads.

MQA / GQA — Optimizing Inference Efficiency

Standard multi-head attention (MHA) gives each head independent K, V projections. To address the massive KV cache during inference:

GQA (Grouped-Query Attention) groups Q heads and shares K,V within each group. LLaMA 2 70B adopted GQA, achieving near-MHA quality with significantly faster inference.

3. Residual Connections & Layer Normalization

Each sublayer (self-attention and FFN) output is added to its input (residual connection), followed by normalization.

Why Residual Connections Are Essential

Without residual connections, backpropagation gradients exponentially decay (or explode) passing through $N$ layers. Residual connections provide a gradient shortcut path, allowing gradients to reach deep layers directly.

This enables training networks with 100+ layers (GPT-3 has 96 layers, LLaMA 2 70B has 80 layers).

Post-LN vs Pre-LN

RMSNorm (Root Mean Square Layer Normalization) is a simplified LayerNorm that skips mean subtraction:

It's computationally cheaper while matching LayerNorm's performance, making it standard in post-LLaMA models.

4. Feed-Forward Network (FFN)

A fully-connected network applied independently to each token. While attention captures "relationships between tokens," FFN performs "feature transformation of each token." FFN parameters comprise roughly 2/3 of a Transformer's total parameters — a massive "knowledge store."

Standard FFN

The original Transformer (Vaswani et al., 2017) used ReLU; GPT-2 onward switched to GELU:

The intermediate dimension $d_{\text{ff}}$ is typically $4 \times d_{\text{model}}$ (e.g., $d = 4096$ → $d_{\text{ff}} = 16384$).

SwiGLU — The Modern Standard

Shazeer (2020) proposed SwiGLU, combining Gated Linear Units with Swish activation. Adopted by LLaMA, PaLM, Gemma, and others.

Why SwiGLU is superior:

• Gating mechanism: $x W_3$ acts as a gate, controlling selective information flow
• Swish activation: $\text{Swish}(x) = x \cdot \sigma(x)$ is smooth unlike ReLU and passes small negative values, improving gradient flow
• SwiGLU uses 3 matrices ($W_1, W_2, W_3$), so $d_{\text{ff}} = \frac{8}{3} d_{\text{model}}$ to maintain the same parameter count (different from the standard $4\times$)

5. Output Layer

Apply RMSNorm to the final transformer layer's output, then project to vocabulary size $|V|$ via a linear transformation (unembedding). Convert to a probability distribution via softmax to predict the next token.

Many LLMs use weight tying: $W_U = W_E^T$, sharing the embedding matrix $W_E$ with the output weight $W_U$. This reduces parameters by $|V| \times d$ and aligns the input/output semantic spaces.

Causal Mask in Decoders

In decoder-only models like GPT, a causal mask blocks attention to future tokens. Token at position $i$ can only attend to positions $1, 2, \ldots, i$.

Implementation-wise, the upper triangular portion of the score matrix is set to $-\infty$, which becomes zero after softmax. This ensures consistency with autoregressive generation (left-to-right, one token at a time).

Full Transformer Block Data Flow

Now let's trace how all these components work together by stepping through the data flow of a single Transformer block.

Self-Attention — The Core of Transformers

Self-attention dynamically computes relevance scores between all tokens, producing context-aware representations.

Formula

• $Q$ (Query): What each token is "looking for"
• $K$ (Key): What each token "contains" (how it wants to be found)
• $V$ (Value): The actual information each token passes along
• $\sqrt{d_k}$: Scaling factor to prevent dot products from growing too large, which would saturate softmax

Interactive Demo

Why Self-Attention Is Revolutionary

1. Long-range dependencies: Unlike RNNs where distant token information decays, self-attention directly accesses all tokens
2. Parallel computation: RNNs process sequentially; self-attention uses matrix operations that GPUs parallelize efficiently
3. Interpretability: Visualizing attention weights reveals what the model "focuses on"

Computational Complexity

Self-attention has $O(n^2)$ complexity with respect to sequence length $n$, which becomes problematic for long contexts. Active research addresses this:

Text Generation — Next Token Prediction

LLM inference is simple: given a context (input token sequence), output a probability distribution over the next token, sample from it, append it, and repeat — this is autoregressive generation.

Generation Strategies

Greedy decoding: Always pick the highest-probability token. Deterministic but tends toward monotonous output.

Temperature scaling: Divide logits by temperature $T$ before softmax:

• $T < 1$: Distribution sharpens → more deterministic (good for factual tasks)
• $T > 1$: Distribution flattens → more diverse/creative (good for creative writing)
• $T \to 0$: Converges to greedy decoding

Top-k sampling: Keep only the top $k$ tokens by probability, zero out the rest, and renormalize.

Top-p (Nucleus) sampling: Keep tokens until cumulative probability exceeds $p$ (e.g., 0.95). The candidate count varies dynamically, making it more flexible than top-k.

Interactive Demo

Training Pipeline — From Raw Model to Conversational AI

Stage 1: Pre-training

Goal: Acquire general language knowledge

Train on next-token prediction using a massive text corpus (web text, books, code) with trillions of tokens.

At this stage, the model can generate text continuations but cannot "answer questions" or "follow instructions."

Stage 2: Supervised Fine-Tuning (SFT)

Goal: Teach the model to follow instructions

Fine-tune on a dataset of high-quality (instruction, response) pairs (~100K examples). This teaches the model the format of "responding to user queries."

[Instruction] What is the capital of Japan?
[Response] The capital of Japan is Tokyo.

Stage 3: RLHF (Reinforcement Learning from Human Feedback)

Goal: Align with human preferences

SFT alone can still produce factually incorrect or harmful outputs. RLHF improves this:

1. Train a reward model: Human annotators rank multiple responses → train a reward model
2. PPO (Proximal Policy Optimization): Update the policy (LLM) to maximize the reward model's score, with a KL penalty to prevent excessive divergence from the original model

DPO (Direct Preference Optimization) simplifies RLHF by optimizing the policy directly from preference data without explicitly training a reward model. LLaMA 2 itself used PPO-based RLHF, but DPO has since been widely adopted in open models such as Zephyr and LLaMA 3.

Modern LLM Techniques

KV Cache

In autoregressive generation, recomputing all previous tokens' K and V at each step is wasteful. KV caching stores and reuses K, V from previous steps. This dramatically speeds up inference but increases memory consumption linearly with sequence length.

Quantization

Convert LLM parameters from 32/16-bit floating point to 4/8-bit integers to reduce memory and compute costs. Methods like GPTQ and AWQ can run 70B models on a single GPU with minimal accuracy loss.

MoE (Mixture of Experts)

Exemplified by Mixtral. Split FFN layers into multiple "experts," with a router selecting a few (e.g., 2 of 8) per token. Total parameters are large, but active parameters during inference are small — enabling efficient scaling.

Context Length Extension

Early Transformers handled ~512 tokens; models today process over 1 million tokens. Techniques include RoPE frequency scaling, YaRN, and Ring Attention.

Summary

We've explored LLM internals through six pillars:

1. History: Word2Vec → Transformer → BERT/GPT → scaling laws enabling massive models
2. Tokenization: BPE subword splitting converts arbitrary text to numerical sequences
3. Transformer: N layers of self-attention + FFN + residual connections
4. Self-attention: $\text{softmax}(QK^T/\sqrt{d_k})V$ dynamically computes relevance across all tokens
5. Text generation: Temperature, top-k, top-p sampling for stochastic autoregressive generation
6. Training pipeline: Pre-training → SFT → RLHF/DPO transforms a "next-token predictor" into a "conversational AI"

The seemingly simple principle of "predict the next token" — when scaled sufficiently — produces the remarkable capabilities that power modern AI.

References

• A. Vaswani et al. Attention Is All You Need. NeurIPS 2017.
• J. Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers. NAACL 2019.
• T. Brown et al. Language Models are Few-Shot Learners. NeurIPS 2020 (GPT-3).
• J. Kaplan et al. Scaling Laws for Neural Language Models. 2020.
• J. Hoffmann et al. Training Compute-Optimal Large Language Models. NeurIPS 2022 (Chinchilla).
• L. Ouyang et al. Training language models to follow instructions with human feedback. NeurIPS 2022 (InstructGPT / RLHF).
• R. Rafailov et al. Direct Preference Optimization. NeurIPS 2023.