# How a chain of failures produced the LLM (part 2) — from attention to the scaling revolution and alignment > From context to the finished LLM: the limits of RNNs, the birth of attention, the Transformer, GPT-3 scaling, RLHF alignment, and the era of efficiency. - Source: https://oharu121.com/blog/llm-birth-history-turing-shannon-transformer-gpt3-part2/ - Published: 2026-08-12T10:25:21+09:00 - Tags: LLM, Generative AI, Deep Learning --- ## Introduction This article continues from **[part 1](/blog/llm-birth-history-turing-shannon-perceptron-word2vec-part1/)**. Part 1 traced the path from the Turing machine through Shannon's information theory, the perceptron and backpropagation, GPU-accelerated training and overcoming vanishing gradients, and Word2Vec/Embedding's technique of "converting words into meaningful vectors". Part 2 starts from the question **how do you handle context** and follows the structural limits of RNNs, the birth of attention, the evolution into the Transformer, GPT-3's scaling revolution, and the present of RLHF alignment and efficiency. It is the second half of the story of how the LLM was completed "without being designed". ## 5. How do you handle context? — the birth of attention > **Note:** the LSTM (1997) and RNN research in this chapter began, chronologically, before AlexNet (2012) and Word2Vec (2013). But understanding "why ordered data is hard" presupposes the knowledge of chapter 3 (training neural networks and the depth wall) and chapter 4 (converting words into vectors), so I cover it in this order. ### Why machine translation was the main arena In the early 2010s, the application NLP researchers poured the most effort into was **machine translation**. The reason is simple. Translation is the task that most severely tests "does it really understand language?" You cannot translate by replacing words one by one. Word order changes, grammar changes, cultural nuance changes. You have to understand the meaning of the whole sentence and reconstruct it in another language. ``` English: The cat sat on the mat. Japanese: 猫がマットの上に座った。 Word-by-word replacement: The=その cat=猫 sat=座った on=の上に the=その mat=マット → "その猫座ったの上にそのマット" ← nonsense Correct translation needs reordering, added particles, dropped articles → presupposes understanding the whole sentence's meaning ``` So the limit of "how far a machine can handle language" was exposed earliest in translation. And the technique born to remove that bottleneck later became the foundation of the whole LLM. ### The bottleneck of translation models: the structural limit of RNNs (around 2014) Translation models of the day used **RNNs (Recurrent Neural Networks)**. The RNN was the best available approach for handling context. An RNN is not a new invention but a combination of the parts we have seen. Its internal neurons, weights, layers, and backpropagation are the same as chapter 3. What it takes as input is the Embedding vector of chapter 4. To understand the RNN's one new idea, you first need to know the limit of an ordinary neural network. **Why is language special: the difference from images** Neural networks succeeded so far on **data you can input all at once**, like images. When AlexNet classifies a photo of a cat, the million pixels enter the input layer **simultaneously**. It does not process "first the top-left pixel, then the one to its right..." in order. Because the whole thing is input in one shot, there is no need to "remember what came before". But language is fundamentally **data arranged in order**. The meaning of "the cat sat on the mat" depends on the word order. "The mat sat on the cat" has the same words but a completely different meaning. And to understand the second half of a sentence, you need to remember what was said in the first half. *Figure — OrderedVsAllAtOnce: Order is the thing images never had.* Put in a familiar analogy, this problem resembles **taking notes while listening to a lecture**. - **An ordinary network** = listens to the lecture but takes no notes. It forgets each sentence as soon as it hears the next. It can only react to the sentence it just heard. - **An RNN** (explained below) = listens while taking notes on a **single-page notepad**. When the notepad fills, it overwrites old notes to write new content. When the lecture ends, only that one page remains. With that analogy in mind, look at the technical mechanism. So what happens if you try to process a sentence with an ordinary neural network? *Figure — NoMemoryAcrossTime: Depth it had. Memory it did not.* You might think, "each layer takes the previous layer's output, so information carries over, doesn't it?" But that is the flow of **one input passing through multiple layers** (spatial flow). "cat" flowing input→hidden→output is one word progressing to deeper layers. The problem is the **temporal flow**. When "cat" finishes and "sat" begins, the network resets completely. No trace of "cat" remains in the hidden layer. So when processing "sat", there is no way to know "what sat (the cat)". #### The RNN's solution: the same layer carries memory over The RNN's idea is that **the same layer processes tokens from different time steps in order, carrying an internal state (the notepad) over to the next step**. *Figure — RnnAssembly: Two chapters of parts, and one idea that was not there before.* This "notepad" is technically called the **hidden state**. Remember the "hidden layer" from chapter 3, the internal workspace between the input and output layers, invisible to the user? The RNN's "hidden state" is the same "hidden". It is called hidden because it is **intermediate data inside the network** that appears directly in neither the input (the word being read) nor the output (the translation). Compare with an ordinary network: *Figure — NotepadCarry: One layer, run again, handed back what it wrote.* The point is that in an RNN **the same hidden layer is used repeatedly (recurrently)**. Not a relay of different layers, but one layer running repeatedly at each time step and carrying the notepad over, so it can remember "what was read before". And the notepad is **not held per neuron, but one for the whole layer**. If the hidden layer has 1,024 neurons, each neuron outputs one number, and the vector of those 1,024 numbers is the "notepad". At the next step, these 1,024 numbers are fed back to the whole layer as shared input. *Figure — NotepadIsShared: Sideways sharing is the whole reason it is one notepad, not 1,024.* **In React terms** If you know React, it may help to think of the RNN's hidden state as close to `useRef`. `ref.current` holds a value across renders and does not trigger a re-render when updated: an internal state "hidden" from React's system. The RNN's notepad is the same: across each time step (token processing), `ref.current` is overwritten, always holding only the latest state. Old values are overwritten and vanish. ```javascript const memo = useRef(new Float32Array(1024)); function processToken(embedding) { memo.current = computeNewState(embedding, memo.current); // the old memo.current is overwritten and vanishes } processToken(embed("cat")); // memo.current = [cat info] processToken(embed("sat")); // memo.current = [cat ... info] processToken(embed("down")); // memo.current = [cat ... sat down info] ``` At each step the RNN can use **only two things**: "the Embedding vector being read now" and "the notepad passed from the previous step". It cannot go back and re-read an earlier word. Information already read exists only inside the notepad. In the lecture analogy, it keeps writing on a one-page notepad and cannot rewind the lecture. **Why is the notepad "one for the whole layer" rather than "per neuron"?** Intuitively, each neuron having its own notepad seems more natural. But the reason the RNN's notepad is shared across the layer is that **each neuron's output becomes context for the other neurons**. When processing "the cat sat", suppose one neuron detects "the subject is the cat" and another detects "the verb is past tense". Processing the next token needs the composite understanding "the subject of the past-tense verb is the cat". No single neuron can hold this; it only forms by combining several neurons' outputs. By sharing the notepad as a vector of all neurons' outputs (1,024 numbers) and feeding it to all neurons at the next step, each neuron can reference **what the other neurons detected at the previous step**. A per-neuron independent notepad cannot do this "sideways information sharing". #### How a neuron uses the notepad At each step, each neuron receives, as one input vector, the concatenation of "the current token's Embedding" and "the previous step's notepad", and performs the same multiplication and addition as chapter 3. The neuron itself does not know "where the token ends and the notepad begins". But through training, **the weights for the token and the weights for the notepad learn different roles**. For example, a pattern like "if the current token is a verb, emphasise the information in the notepad corresponding to the subject" naturally emerges as weight values. *Figure — RnnNeuron: To the neuron there is no boundary. Training puts one there.* What pattern each neuron learns is **not decided in advance**. During training, each neuron finds by itself "some pattern useful for predicting the next token". That pattern may be abstract and uninterpretable to humans. This is one reason neural networks are called "black boxes". #### The RNN's real bottleneck: you cannot query a snapshot The notepad's size (1,024 numbers) is a parameter the designer sets before training. It could be 512 or 2,048, but once set it does not change, because the layer of neurons expects a fixed number of inputs (recall the structure of chapter 3). "Because the size is fixed, information overflows on long sentences." At first glance that looks like the problem. But the real problem is not the size, it is the **data structure**. The notepad resembles a **database snapshot (dump)**. A dump file reflects the cumulative result of all past transactions, but you cannot restore a specific INSERT statement. Whether you make it 10 times or 1,000 times larger, this property does not change. The RNN's notepad is the same. All past tokens contribute to the current state, but **the structural constraint of not being able to extract individual tokens' information does not change however large you make the notepad.** *Figure — NotepadAsSnapshot: A bigger dump is still a dump.* In a translation model, you finish reading the whole input sentence before starting to generate the output sentence. When the output RNN generates the translation, all it has is the **final snapshot**. For example, translating "she went to the market. There she met her friend's girlfriend". To translate the third "her", the output RNN wants to reference the corresponding place in the input sentence to confirm it is "the friend's partner". But **individual tokens have dissolved into the snapshot**. There is no way to pull one out precisely. What is really needed is not a snapshot but a **transaction log**: a data structure where each token's processing result is stored individually and can be queried. This shift becomes the core of attention, which appears later. #### Déjà vu from chapter 3: vanishing gradients, again The RNN's problem is not only information overflow. **There was a wall in learning itself.** In chapter 3 we saw how backpropagation works. Compare the output with the correct answer, propagate the error signal backwards, and adjust each layer's weights. That is the "blame" mechanism that tracks "which weight contributed how much to the error". The RNN uses the same mechanism. But whereas in chapter 3 the error signal flowed backwards **across layers**, in the RNN it flows backwards **across time steps**. *Figure — VanishingGradientTwoAxes: Same maths, new axis.* For example, if the prediction is wrong at the 50th token, the cause might be "the way the 1st token was processed was bad". But the error signal has to go back 49 times through the notepad chain, shrinking at each step. As a result, **the feedback for early tokens becomes near zero and the weights are not adjusted**, meaning the network cannot learn "how to process the start of a sentence well". The maths is exactly the same. In chapter 3 the signal vanished each time it passed through a deep **layer**. In the RNN it vanishes each time it goes back a long way in **time**. Only the axis changed, from space to time, and the same problem reappeared. #### LSTM: a ResNet for the time axis (invented 1997, the mainstay of translation from 2014) In chapter 3, **ResNet (residual connections)** solved layer-direction vanishing gradients. It adds the input to the output via a shortcut, making a "highway" through which the gradient flows directly. So what about time-direction vanishing gradients? In fact, 18 years before ResNet (2015), in 1997, a solution with the same idea had been proposed. It is the **LSTM (Long Short-Term Memory)** by Hochreiter and Schmidhuber. Long a niche, it flourished as the mainstay architecture for machine translation around 2014 when the deep-learning boom and GPUs coincided. The LSTM's core is having, in addition to the ordinary notepad, another memory called the **cell state**. This cell state is a **highway that passes through almost unchanged** between time steps. Where the ordinary RNN's notepad goes through a complex transform at every step and loses a quarter of the gradient each time, the cell state is joined only by additions, so the gradient arrives at roughly the strength it left with. The LSTM also has a mechanism called **gates**. The three of them, the "forget gate", the "input gate" and the "output gate", control for the cell state "what to forget", "what to write", and "what to read". This lets it hold important information for a long time while discarding unneeded information. *Figure — LstmCellState: ResNet's idea, eighteen years early, on the other axis.* The LSTM greatly improved the RNN's memory problem and was the mainstay of machine translation from around 2014 to 2017. But a fundamental constraint remained: **sequential processing**. Process token 1, then token 2; process token 2, then token 3. However much the memory improves, it cannot exploit GPU parallelism. ### A patch for a bug: attention Bahdanau and colleagues' solution was exactly the earlier shift "from snapshot to transaction log". #### Step 1: turn the snapshot into a transaction log When an RNN processes input, it updates the notepad (snapshot) at each step. Conventionally, only this final snapshot was passed to the output RNN. Bahdanau and colleagues' idea was to **save every step's snapshot as an independent copy**. The key is that what is saved is **not the difference (delta) but a complete snapshot at each time step**. Process 5 tokens and you save 5 sets of 1,024 numbers, 5,120 in total. *Figure — AttentionKeepsEveryStep: Stop throwing four of the five away.* **In React terms** In React terms, attention is like saving a spread copy with `history.push([...memo.current])` before overwriting `ref.current`. At output time you can access the state at any step with `history[i]`. Back to the lecture analogy, attention corresponds to **recording the lecture**. Instead of relying only on notes (the final notepad), you keep a recording of each moment of the lecture. But having a recording alone is not enough. You cannot re-listen to the whole recording of a two-hour lecture every time. You need **a mechanism to judge "which part to re-listen to right now"**. This is the core of attention. #### Step 2: learn which snapshot to attend to When the output RNN generates each word of the translation, the following process runs: 1. Take the output RNN's current state ("what am I trying to translate now") 2. Compute a relevance score against **each of the saved snapshots** 3. Normalise the scores with softmax into weights summing to 1.0 4. Take a weighted average of the snapshots by the weights to make a context vector 5. Use this context vector to generate the output word ``` the output RNN is about to generate "she": saved snapshots: copy₁ (after processing "she") copy₂ (after "went") copy₃ (after "market") copy₄ (after "friend") copy₅ (after "she") step 2 — compute relevance scores: score(current state, copy₁) = 4.2 ← high! the first "she" is relevant score(current state, copy₂) = 0.1 score(current state, copy₃) = 0.8 score(current state, copy₄) = 1.5 score(current state, copy₅) = 0.3 step 3 — normalise with softmax (sum = 1.0): weights = [0.70, 0.02, 0.08, 0.15, 0.05] step 4 — weighted average: context = 0.70×copy₁ + 0.02×copy₂ + 0.08×copy₃ + 0.15×copy₄ + 0.05×copy₅ = a vector strongly biased toward copy₁ (the first "she") step 5 — use this context to generate "she" ``` This weight that decides "how much to attend to which snapshot" is the origin of the name **attention**. When a human reads text, they do not pay equal attention to every word but concentrate attention on the parts relevant to the current context. The attention mechanism reproduces exactly this numerically. And the weights used in the score computation are also parameters **trained** by backpropagation. It automatically learns patterns like "when translating a verb, give a high score to the subject's snapshot" from data. *Figure — AttentionScoring: Attention is just this: deciding what to weight.* #### Attention also improves vanishing gradients Attention has another important effect. The **direct connection** from the output to each snapshot becomes a shortcut for the gradient. *Figure — AttentionGradientShortcut: Nobody was aiming at this one.* Because the gradient can reach any token in one step through attention's direct path, it no longer needs to go back up the RNN chain, and vanishing gradients are greatly relieved. This was not a grand vision. It was an engineering patch for the bug that translation accuracy dropped on long sentences. But as a result, it solved the snapshot constraint (unable to query a specific token) and even improved vanishing gradients. ### The patch became the architecture: the Transformer (2017) Vaswani and colleagues posed the question: "is an RNN (a sequential mechanism) needed at all? Couldn't we translate with attention alone?" Looking back at the improvements so far, even attention+RNN kept the RNN part's constraint: **snapshot generation is sequential**. In attention+RNN, referencing (re-reading) can be parallelised, but snapshot generation is still handled by the RNN. Because each snapshot depends on the previous one, they can only be made in order. ``` attention + RNN: referencing improved, but snapshot generation stays sequential snapshot₁ = f("cat", empty) ← compute this first snapshot₂ = f("sat", snapshot₁) ← cannot compute until snapshot₁ is done snapshot₃ = f("mat", snapshot₂) ← cannot compute until snapshot₂ is done each snapshot depends on the previous → a chain → not parallelisable ``` Vaswani and colleagues' insight was to **discard this chain (the notepad chain) itself**. #### How to understand context without an RNN In an RNN, token order was implicitly represented by the notepad chain. Precisely because you process "cat" before "sat", the notepad at "sat" contains "cat" information. Processing order = context order. The Transformer takes a completely different approach. **Each token has its own position number from the start.** *Figure — RnnVsTransformer: The Transformer is not cheaper. It is schedulable.* Because position information is contained in the data itself, processing all tokens **simultaneously** loses no order. And each token references every other token. This is the Transformer's **Self-Attention**. The earlier attention was "the output RNN references the input's snapshots". In self-attention, **the input tokens reference each other**. "cat" looks at "sat", "sat" looks at "cat", all at once. In an RNN there was a dependency chain "to compute notepad₃ you need notepad₂, and notepad₂ needs notepad₁". The Transformer has no such chain. Each token's Embedding does not depend on the processing result of other tokens, which is why they can be processed simultaneously. *Figure — SelfAttention: Order stopped being a consequence of processing and became data.* **The Transformer is still "deep"** Hearing "process all tokens simultaneously" might make the network sound "flat". But **the Transformer is still a deep multi-layer structure**. GPT-3 has 96 layers. What was parallelised is only the token direction. **The layer direction is still processed in order.** Each layer refines the representation. Layer 1 learns "sat = a verb", Layer 2 learns "the cat is the subject of sat", and deeper layers learn complex semantic relations. **Deeper means richer understanding**, the same principle as chapter 3. Backpropagation (blame) also flows backwards through the layers by the same mechanism as chapter 3. In addition, through self-attention's connections, **the blame signal reaches directly from any token to any token**, with no need to go back up a long chain like an RNN, so vanishing gradients are greatly relieved. And ResNet residual connections prevent layer-direction vanishing gradients. *Figure — DeepTransformer: Wide in tokens, deep in layers. Only one of those was parallelised.* #### Processing time: why the Transformer is fast Compare the processing time. An RNN needs **token count × layer count** sequential steps, so 100 tokens through 96 layers is 9,600 of them. A Transformer needs only as many as it has layers, because the tokens inside each layer are processed simultaneously: **96**. However, each Transformer step involves **more computation** than an RNN. In self-attention, all tokens reference all tokens, so it needs computation proportional to the square of the token count (n²). For 100 tokens, 10,000 pairs of score computation. | | Sequential steps | Computation per step | |----------------|------------------|----------------------| | RNN | token count × layer count | light (one token's worth) | | Transformer | layer count only | heavy (n² attention computation) | The total computation can be higher for the Transformer. But **the heavy computation can be parallelised on a GPU**. This is the same structure as AlexNet in chapter 3: 4,000 primary-school children solving additions at once. The n² score computations are independent of each other, so they can run simultaneously on a GPU's thousands of cores. Meanwhile, the RNN's sequential steps cannot be parallelised however powerful the GPU, because you cannot advance until the previous step finishes. As a result, the Transformer solved **all** the problems seen in this chapter at once: | | RNN | LSTM | Attention+RNN | Transformer | |--------------------|-----|------|---------------|-------------| | Snapshot problem | ✗ | ✗ | ✓ (logged) | ✓ | | Vanishing gradient (temporal) | ✗ | ✓ (highway) | ✓ (shortcut) | ✓ (direct) | | Parallel processing | ✗ | ✗ | ✗ (RNN part remains) | ✓ | Discarding the RNN and using attention alone made training dramatically faster. Faster means you can train a larger model on more data. **An architecture aimed at speeding up translation became the foundation of the whole LLM by making scale possible.** ## 6. Does bigger mean smarter? — scale and emergence ### The discovery of scaling laws (2020) "Making the model bigger improves performance" was known as a rule of thumb. OpenAI's Kaplan and colleagues showed this as a formula: **Performance improves according to a smooth, predictable law with respect to three quantities: parameter count, data volume, and compute.** Look at the three variables concretely: | Variable | Meaning | How to increase it | |------|------|-----------| | Parameter count (N) | Total number of the model's weights | Deepen layers, increase nodes per layer | | Data volume (D) | Number of tokens used for training | Collect more text (web, books, code, etc.) | | Compute (C) | Total operations the GPUs run during training (FLOPs) | Train on more GPUs for longer | *Figure — ScalingLaws: Three variables, but only one budget.* **What is compute?** Parameter count and data volume are intuitive, but **compute** needs a little explanation. Compute is the total number of floating-point operations (FLOP) the GPUs run during training. Every time the model processes one batch, a large amount of matrix computation runs in the forward and backward passes. The cumulative total of that computation is compute. Kaplan and colleagues showed the relationship of the three with an approximation: ``` C ≈ 6 × N × D ``` So compute is roughly proportional to **the product of parameter count and data volume**. Compute is less an independent third variable than **the total budget you can put in**. Once the budget is fixed, how large you make the model and how much data you feed it are determined. The way to increase compute is simple: **number of GPUs × training time**. Training GPT-3 spent about 3.14×10²³ FLOPs. That was the result of running thousands of GPUs for weeks. So increasing compute means, plainly, **putting in money and time**. > **Note: the Chinchilla law (2022)** > Kaplan's scaling law suggested "when the compute budget grows, allocate more to making the model bigger". But in 2022, DeepMind's Hoffmann and colleagues revised it to "allocate more to data volume" (the Chinchilla law). The finding is that even with the same compute budget, the optimal allocation ratio differs. What both agree on is that **compute is the fundamental limiting resource**. This was a "map". You can predict in advance "train a model of this size on this much data and you get roughly this performance". If the result is visible, you can make the call on a large investment. ### From Transformer to "GPT" — a shift in purpose The Transformer was born for translation. But GPT (Generative Pre-trained Transformer) set out to solve a different problem from translation. In 2018, the problem OpenAI's Radford and colleagues faced was this: **NLP has diverse tasks (question answering, document classification, sentiment analysis, etc.), but collecting a large amount of labelled data for each task is too costly.** Unlabelled text (writing on the internet) is abundant. Could you use it to learn "a general understanding of language" first, then adapt to individual tasks with a small amount of labelled data? This is GPT-1's core idea: 1. **Pre-training**: train on a large amount of text just to "predict the next word" 2. **Fine-tuning**: adjust the weights with a small amount of data for an individual task The architecture used only the Decoder part of the Transformer. The Encoder-Decoder structure for translation was unnecessary; the simple structure of "read text and predict the continuation" was enough. *Figure — GptDecoderOnly: Half the Transformer, because half was all the job needed.* ### GPT-1 → GPT-2 → GPT-3: the same blueprint, different scale Here is the key point. **GPT-1, GPT-2, and GPT-3 have almost the same architecture.** Keep the basic design and change only the scale to observe what happens. That is the essence of the three generations. | | GPT-1 (2018) | GPT-2 (2019) | GPT-3 (2020) | |---|---|---|---| | Parameter count | 117 million | 1.5 billion | 175 billion | | Training data | BookCorpus (~5GB) | WebText (~40GB) | CommonCrawl etc. (~570GB) | | Core finding | Pre-training + fine-tuning works | Solves tasks even without fine-tuning (zero-shot) | Handles unseen tasks from a few examples (few-shot) | | Idea in the paper title | "Improving Language Understanding by Generative Pre-Training" | "Language Models are Unsupervised Multitask Learners" | "Language Models are Few-Shot Learners" | **GPT-1** showed the effectiveness of "pre-training + fine-tuning". But it needed fine-tuning per task. **GPT-2** made the model 13 times bigger and increased the data. Then it could handle some tasks even without fine-tuning (zero-shot). As the paper title says, the finding is "language models are unsupervised multitask learners". But the performance was still limited. **GPT-3** made it 117 times bigger still. The architecture change is tiny (about the partial introduction of sparse attention). **With the same blueprint, scale alone produced a qualitative leap.** *Figure — GptGenerations: Two jumps, one blueprint.* **Few-shot learning: what happens after training** Here I answer a question readers easily have. > **"Is few-shot part of training? Or something that happens after training?"** The answer is **after training**. Few-shot is a technique at inference time (when the user uses the model). Organising GPT-3's training process: 1. **At training time**: it learns only to "predict the next word" from a large amount of text collected from the internet. It was not taught translation, nor taught coding. The goal is one thing: **raise the accuracy of next-word prediction** 2. **At inference time (few-shot)**: the user shows a few sample examples in the prompt. The model reads that pattern from the context and generates the continuation in the same pattern GPT-3 was not explicitly trained on translation. But because English-Japanese parallel text was in the training data, as the ability to "predict the next word" scaled, the power to grasp the pattern and continue correctly **emerged**. Code generation is the same. Because GitHub code was in the training data, showing a few examples of the pattern "function description → code" let it write functions it had never seen. *Figure — FewShotLearning: The examples never touch the weights.* ### Why did the "big bang" happen at GPT-3? Zero-shot partly worked even in GPT-2. So what changed at GPT-3? The answer is **crossing a threshold**. For many tasks, the relationship between model size and performance moved like this: - 125 million to 350 million parameters: near-random performance - 1.3 billion to 13 billion parameters: gradual improvement - **175 billion parameters: a sudden jump** This is **emergent abilities**. Like a phase transition in physics (the moment water becomes ice), a quantitative change beyond a certain threshold triggers a qualitative change. GPT-2 was just short of that threshold, and GPT-3 was beyond it. Why does a threshold exist? During training, to "predict the next word" the model encodes into itself every pattern contained in the text (grammar, logic, facts, task structure). A small model lacks the capacity and can only learn fragments of individual patterns. But when scale grows large enough, the patterns **combine** and surface as abilities that were not explicitly trained. This is the nature of emergence. GPT-3 did not design intelligence with a new architecture. **Scaling the same blueprint until it crossed a threshold made an intelligence that was not designed appear.** This is the starting point of the current LLM era. *Figure — EmergentAbilities: GPT-2 was short of the line. That was the whole difference.* ## 7. Challenges after getting smart — from fine-tuning to alignment, and beyond scale GPT-3 was astonishingly versatile but **hard to use**. It generated racist text, continued unrelated text instead of answering the question, and told lies with total confidence. The ability to "predict the next word" and the ability to "give a response useful to humans" were not the same. The history from here divides into three stages. ### How fine-tuning works Let us look a little more closely at the fine-tuning that appeared with GPT-1. A pre-trained model has learned general patterns of language but is not optimised for a specific task (sentiment analysis, question answering, etc.). Fine-tuning is the process of "tuning" this general-purpose model to a specific purpose. The mechanism is simple: 1. **Prepare task data**: e.g. hundreds to thousands of "review text → positive/negative" pairs 2. **Start from the pre-trained weights**: instead of learning from scratch, start from weights that already understand language 3. **Additionally train on the task data**: the same mechanism as ordinary training (forward pass → loss computation → backward pass), but with a smaller learning rate and fewer steps Why does it work with a small amount of data? Because pre-training already acquired grammar, vocabulary, and world knowledge. Fine-tuning is like "teaching medical terms to someone who already speaks English"; there is no need to re-teach the basics of language. *Figure — FineTuning: Teaching medical terms to someone who already speaks the language.* ### InstructGPT: "aligning" with human feedback (2022) GPT-3's problem was that it was "smart but not aligned with human intent". To solve this, OpenAI developed **RLHF (Reinforcement Learning from Human Feedback)**, which extended the concept of fine-tuning. RLHF is a three-stage pipeline: **Stage 1: supervised fine-tuning (SFT)** Human annotators (about 40) hand-wrote about 13,000 "ideal answers" to prompts. Ordinary fine-tuning on this teaches the basic behaviour "when asked, answer". **Stage 2: training a reward model** The model generates multiple answers to the same prompt, and humans rank them by "which is better". From this ranking data, a **reward model** that quantifies "what a good answer is" is trained. **Stage 3: optimisation by reinforcement learning** The original model generates answers, the reward model scores them, and those scores are used to update the original model's weights. The model is adjusted toward generating "answers humans prefer". The result was shocking. **A 1.3-billion-parameter InstructGPT generated answers humans preferred over the raw 175-billion-parameter GPT-3.** A model 130 times smaller in parameters overturned the result just by being "aligned". *Figure — RlhfPipeline: 130× smaller, and preferred anyway.* This is an important lesson: **making the model bigger is not the only path to better performance.** ### Where fine-tuning stands now: when it is needed and when not In the GPT-1 era, fine-tuning was mandatory for every task. Now that GPT-3 made few-shot usable and InstructGPT made it follow instructions even zero-shot, fine-tuning's role has changed. | | Cases where fine-tuning is unnecessary | Cases where fine-tuning is necessary | |---|---|---| | Typical example | Email drafting, summarisation, general Q&A | Medical-diagnosis support, legal-document review, understanding internal terminology | | Reason | A general model's prompt gives sufficient quality | Specialist terms, specific formats, and compliance requirements are strict | **Labelled data is still important**, but its use changed. In the GPT-1 era it was needed to "teach the task"; now it is needed mainly to "align the model's behaviour". RLHF's human ranking data is the typical example. Also, with the arrival of **parameter-efficient fine-tuning (PEFT)** such as LoRA (Low-Rank Adaptation) and QLoRA, you can now fine-tune by adding only a few parameters without updating all of them. The cost of fine-tuning has been cut by over 90% compared with the early 2020s. ### Beyond scale: how are LLMs evolving now? "Increase parameters, data, and compute and it gets smarter." The scaling law explained GPT-3's success. But since around 2024, this strategy has hit a wall. **Wall 1: data exhaustion** High-quality text data on the internet is finite. As of 2026, high-quality human-written text is said to be nearly used up. **Wall 2: the limit of cost** Pre-training a larger model needs an investment on the order of hundreds of millions of dollars, and diminishing returns are starting to show. At NeurIPS 2024, OpenAI co-founder Ilya Sutskever declared: > **"Pre-training as we know it will end. The 2010s were the age of scaling. Now we are back in the age of wonder and discovery."** *Figure — PretrainingWalls: The map ran out before the ambition did.* So what axes of evolution are there besides scale? Here are the main current approaches. #### Axis 1: deepening post-training RLHF, which began with InstructGPT, is now further refined. Additional training done after pre-training (acquiring basic ability from a large amount of text), called **post-training**, has become the main arena for drawing out a model's ability. - **RLHF / RLAIF**: alignment with human or AI feedback (Anthropic and OpenAI continually improve it) - **DPO (Direct Preference Optimization)**: an efficient method that optimises directly from human preference data without going through a reward model *Figure — PostTraining: Same outcome, one model fewer in the loop.* #### Axis 2: test-time compute Instead of scaling up training, the approach of **making it think more at inference time**. OpenAI's o1/o3 series and Anthropic's Claude (extended thinking mode) are representative. Before answering, the model generates a "chain of thought" and reasons step by step. Instead of increasing training-time compute, it raises performance by increasing inference-time compute. Where the scaling law was a law of "how much compute to put into training", this is a new axis of "how much compute to put into inference". *Figure — TestTimeCompute: The compute did not shrink. It moved.* #### Axis 3: synthetic data and distillation If human data is exhausted, **have AI make the data**. - **Distillation**: train a small model (the student) on answers generated by a large model (the teacher). DeepSeek-R1's distilled versions and Llama derivatives are made this way - **Self-play**: the model converses with itself, posing and solving problems to each other, improving its ability *Figure — SyntheticData: The corpus is finite. The models making it are not.* In a typical 2026 workflow, humans write 200 seed samples, a frontier model expands them to tens of thousands, a quality filter is applied, and a small model is trained. #### Summary: three eras | Era | Main strategy | Representative examples | |------|---------|--------| | The age of scale (~2023) | Increase parameters, data, compute | GPT-3, PaLM, LLaMA | | The age of alignment (2022~) | Align with human intent via post-training | InstructGPT, Claude, ChatGPT | | The age of efficiency (2024~) | Get smarter via test-time compute, synthetic data, distillation | o1/o3, DeepSeek-R1, Claude extended thinking | These are not mutually exclusive; current frontier models combine all three. But the centre of gravity has clearly shifted, from "bigger means smarter" to "used smartly, even small is strong". ## Summary The LLM is an accumulation of concepts that piled up as answers to separate problems, in an age when nobody was aiming at an LLM. *Figure — Timeline: Each answer solved its own problem. None of them were aiming at this.* | Person | Original purpose | Problem being solved | Unintended by-product | |---|---|---|---| | Turing (1936) | Prove the limits of mathematics | Do problems unsolvable by algorithm exist? | The birth of the concept of a computer | | Shannon (1948) | Reduce communication noise | How to send information accurately? | Information entropy → the LLM's loss function | | Rosenblatt (1958) | Implement a learning machine | Can machines learn? | The perceptron (the first neural net) | | Minsky & Papert (1969) | Mathematical analysis of NNs | What are the perceptron's abilities and limits? | Proof of the limit → the requirement for multi-layer networks | | Rumelhart et al. (1986) | Train multi-layer NNs | How to train deep networks? | Backpropagation | | Krizhevsky et al. (2012) | Improve image-recognition accuracy | Computation speed is insufficient | GPU use → breaking the training-speed wall | | He et al. (2015) | Even deeper networks | Gradients vanish when layers deepen | Residual connections (ResNet) → breaking the depth wall | | Mikolov et al. (2013) | Better word representation | How to turn text into numbers? | The geometry of meaning (Embedding) | | Hochreiter & Schmidhuber (1997) | Long-term memory for RNNs | How to solve temporal vanishing gradients? | LSTM → a ResNet for the time axis | | Bahdanau et al. (2014) | Long-sentence translation accuracy | How to prevent the information loss of a fixed-length vector? | The attention mechanism | | Vaswani et al. (2017) | Speed up translation | How to solve the slowness of RNN sequential processing? | The Transformer (a parallelisable architecture) | | Radford et al. (2018) | Solve the shortage of labelled data | How to solve diverse NLP tasks with little labelled data? | GPT-1: the pre-training + fine-tuning paradigm | | Radford et al. (2019) | Explore a general-purpose language model | Can tasks be solved without fine-tuning? | GPT-2: the germ of zero-shot ability | | Kaplan et al. (2020) | Optimise compute cost | What is the relationship of scale and performance? | Scaling laws → the basis for large-scale training | | Brown et al. (2020) | Investigate large-model ability | What happens when you scale up? | GPT-3 and the emergence of few-shot learning | | Ouyang et al. (2022) | Improve model safety and usefulness | How to align a smart model with human intent? | RLHF → small but "aligned" overturns the result | Each researcher was looking only at the problem in front of them. With no blueprint, the parts came together over 80-odd years. At this very moment, a paper someone is writing to get past a completely different wall may be a part of the AI of ten years from now.