How a chain of failures produced the LLM (part 2) — from attention to the scaling revolution and alignment

Introduction

This article continues from part 1.

Part 1 traced the path from the Turing machine through Shannon’s information theory, the perceptron and backpropagation, GPU-accelerated training and overcoming vanishing gradients, and Word2Vec/Embedding’s technique of “converting words into meaningful vectors”.

Part 2 starts from the question how do you handle context and follows the structural limits of RNNs, the birth of attention, the evolution into the Transformer, GPT-3’s scaling revolution, and the present of RLHF alignment and efficiency. It is the second half of the story of how the LLM was completed “without being designed”.

5. How do you handle context? — the birth of attention

Note: the LSTM (1997) and RNN research in this chapter began, chronologically, before AlexNet (2012) and Word2Vec (2013). But understanding “why ordered data is hard” presupposes the knowledge of chapter 3 (training neural networks and the depth wall) and chapter 4 (converting words into vectors), so I cover it in this order.

Why machine translation was the main arena

In the early 2010s, the application NLP researchers poured the most effort into was machine translation.

The reason is simple. Translation is the task that most severely tests “does it really understand language?” You cannot translate by replacing words one by one. Word order changes, grammar changes, cultural nuance changes. You have to understand the meaning of the whole sentence and reconstruct it in another language.

English:  The cat sat on the mat.
Japanese: 猫がマットの上に座った。

Word-by-word replacement: The=その cat=猫 sat=座った on=の上に the=その mat=マット
→ "その猫座ったの上にそのマット" ← nonsense

Correct translation needs reordering, added particles, dropped articles
→ presupposes understanding the whole sentence's meaning

So the limit of “how far a machine can handle language” was exposed earliest in translation. And the technique born to remove that bottleneck later became the foundation of the whole LLM.

The bottleneck of translation models: the structural limit of RNNs (around 2014)

Translation models of the day used RNNs (Recurrent Neural Networks). The RNN was the best available approach for handling context.

An RNN is not a new invention but a combination of the parts we have seen. Its internal neurons, weights, layers, and backpropagation are the same as chapter 3. What it takes as input is the Embedding vector of chapter 4. To understand the RNN’s one new idea, you first need to know the limit of an ordinary neural network.

Why is language special: the difference from images

Neural networks succeeded so far on data you can input all at once, like images. When AlexNet classifies a photo of a cat, the million pixels enter the input layer simultaneously. It does not process “first the top-left pixel, then the one to its right…” in order. Because the whole thing is input in one shot, there is no need to “remember what came before”.

But language is fundamentally data arranged in order. The meaning of “the cat sat on the mat” depends on the word order. “The mat sat on the cat” has the same words but a completely different meaning. And to understand the second half of a sentence, you need to remember what was said in the first half.

Put in a familiar analogy, this problem resembles taking notes while listening to a lecture.

• An ordinary network = listens to the lecture but takes no notes. It forgets each sentence as soon as it hears the next. It can only react to the sentence it just heard.
• An RNN (explained below) = listens while taking notes on a single-page notepad. When the notepad fills, it overwrites old notes to write new content. When the lecture ends, only that one page remains.

With that analogy in mind, look at the technical mechanism.

So what happens if you try to process a sentence with an ordinary neural network?

You might think, “each layer takes the previous layer’s output, so information carries over, doesn’t it?” But that is the flow of one input passing through multiple layers (spatial flow). “cat” flowing input→hidden→output is one word progressing to deeper layers.

The problem is the temporal flow. When “cat” finishes and “sat” begins, the network resets completely. No trace of “cat” remains in the hidden layer. So when processing “sat”, there is no way to know “what sat (the cat)”.

The RNN’s solution: the same layer carries memory over

The RNN’s idea is that the same layer processes tokens from different time steps in order, carrying an internal state (the notepad) over to the next step.

This “notepad” is technically called the hidden state. Remember the “hidden layer” from chapter 3, the internal workspace between the input and output layers, invisible to the user? The RNN’s “hidden state” is the same “hidden”. It is called hidden because it is intermediate data inside the network that appears directly in neither the input (the word being read) nor the output (the translation).

Compare with an ordinary network:

The point is that in an RNN the same hidden layer is used repeatedly (recurrently). Not a relay of different layers, but one layer running repeatedly at each time step and carrying the notepad over, so it can remember “what was read before”.

And the notepad is not held per neuron, but one for the whole layer. If the hidden layer has 1,024 neurons, each neuron outputs one number, and the vector of those 1,024 numbers is the “notepad”. At the next step, these 1,024 numbers are fed back to the whole layer as shared input.

In React terms

If you know React, it may help to think of the RNN’s hidden state as close to useRef. ref.current holds a value across renders and does not trigger a re-render when updated: an internal state “hidden” from React’s system. The RNN’s notepad is the same: across each time step (token processing), ref.current is overwritten, always holding only the latest state. Old values are overwritten and vanish.

const memo = useRef(new Float32Array(1024));

function processToken(embedding) {
  memo.current = computeNewState(embedding, memo.current);
  // the old memo.current is overwritten and vanishes
}

processToken(embed("cat"));  // memo.current = [cat info]
processToken(embed("sat"));  // memo.current = [cat ... info]
processToken(embed("down")); // memo.current = [cat ... sat down info]

At each step the RNN can use only two things: “the Embedding vector being read now” and “the notepad passed from the previous step”. It cannot go back and re-read an earlier word. Information already read exists only inside the notepad. In the lecture analogy, it keeps writing on a one-page notepad and cannot rewind the lecture.

Why is the notepad “one for the whole layer” rather than “per neuron”?

Intuitively, each neuron having its own notepad seems more natural. But the reason the RNN’s notepad is shared across the layer is that each neuron’s output becomes context for the other neurons.

When processing “the cat sat”, suppose one neuron detects “the subject is the cat” and another detects “the verb is past tense”. Processing the next token needs the composite understanding “the subject of the past-tense verb is the cat”. No single neuron can hold this; it only forms by combining several neurons’ outputs.

By sharing the notepad as a vector of all neurons’ outputs (1,024 numbers) and feeding it to all neurons at the next step, each neuron can reference what the other neurons detected at the previous step. A per-neuron independent notepad cannot do this “sideways information sharing”.

How a neuron uses the notepad

At each step, each neuron receives, as one input vector, the concatenation of “the current token’s Embedding” and “the previous step’s notepad”, and performs the same multiplication and addition as chapter 3. The neuron itself does not know “where the token ends and the notepad begins”. But through training, the weights for the token and the weights for the notepad learn different roles. For example, a pattern like “if the current token is a verb, emphasise the information in the notepad corresponding to the subject” naturally emerges as weight values.

What pattern each neuron learns is not decided in advance. During training, each neuron finds by itself “some pattern useful for predicting the next token”. That pattern may be abstract and uninterpretable to humans. This is one reason neural networks are called “black boxes”.

The RNN’s real bottleneck: you cannot query a snapshot

The notepad’s size (1,024 numbers) is a parameter the designer sets before training. It could be 512 or 2,048, but once set it does not change, because the layer of neurons expects a fixed number of inputs (recall the structure of chapter 3).

“Because the size is fixed, information overflows on long sentences.” At first glance that looks like the problem. But the real problem is not the size, it is the data structure.

The notepad resembles a database snapshot (dump). A dump file reflects the cumulative result of all past transactions, but you cannot restore a specific INSERT statement. Whether you make it 10 times or 1,000 times larger, this property does not change.

The RNN’s notepad is the same. All past tokens contribute to the current state, but the structural constraint of not being able to extract individual tokens’ information does not change however large you make the notepad.

In a translation model, you finish reading the whole input sentence before starting to generate the output sentence.

When the output RNN generates the translation, all it has is the final snapshot. For example, translating “she went to the market. There she met her friend’s girlfriend”. To translate the third “her”, the output RNN wants to reference the corresponding place in the input sentence to confirm it is “the friend’s partner”. But individual tokens have dissolved into the snapshot. There is no way to pull one out precisely.

What is really needed is not a snapshot but a transaction log: a data structure where each token’s processing result is stored individually and can be queried. This shift becomes the core of attention, which appears later.

Déjà vu from chapter 3: vanishing gradients, again

The RNN’s problem is not only information overflow. There was a wall in learning itself.

In chapter 3 we saw how backpropagation works. Compare the output with the correct answer, propagate the error signal backwards, and adjust each layer’s weights. That is the “blame” mechanism that tracks “which weight contributed how much to the error”.

The RNN uses the same mechanism. But whereas in chapter 3 the error signal flowed backwards across layers, in the RNN it flows backwards across time steps.

For example, if the prediction is wrong at the 50th token, the cause might be “the way the 1st token was processed was bad”. But the error signal has to go back 49 times through the notepad chain, shrinking at each step. As a result, the feedback for early tokens becomes near zero and the weights are not adjusted, meaning the network cannot learn “how to process the start of a sentence well”.

The maths is exactly the same. In chapter 3 the signal vanished each time it passed through a deep layer. In the RNN it vanishes each time it goes back a long way in time. Only the axis changed, from space to time, and the same problem reappeared.

LSTM: a ResNet for the time axis (invented 1997, the mainstay of translation from 2014)

In chapter 3, ResNet (residual connections) solved layer-direction vanishing gradients. It adds the input to the output via a shortcut, making a “highway” through which the gradient flows directly.

So what about time-direction vanishing gradients? In fact, 18 years before ResNet (2015), in 1997, a solution with the same idea had been proposed. It is the LSTM (Long Short-Term Memory) by Hochreiter and Schmidhuber. Long a niche, it flourished as the mainstay architecture for machine translation around 2014 when the deep-learning boom and GPUs coincided.

The LSTM’s core is having, in addition to the ordinary notepad, another memory called the cell state. This cell state is a highway that passes through almost unchanged between time steps. Where the ordinary RNN’s notepad goes through a complex transform at every step and loses a quarter of the gradient each time, the cell state is joined only by additions, so the gradient arrives at roughly the strength it left with.

The LSTM also has a mechanism called gates. The three of them, the “forget gate”, the “input gate” and the “output gate”, control for the cell state “what to forget”, “what to write”, and “what to read”. This lets it hold important information for a long time while discarding unneeded information.

The LSTM greatly improved the RNN’s memory problem and was the mainstay of machine translation from around 2014 to 2017. But a fundamental constraint remained: sequential processing. Process token 1, then token 2; process token 2, then token 3. However much the memory improves, it cannot exploit GPU parallelism.

A patch for a bug: attention

Bahdanau and colleagues’ solution was exactly the earlier shift “from snapshot to transaction log”.

Step 1: turn the snapshot into a transaction log

When an RNN processes input, it updates the notepad (snapshot) at each step. Conventionally, only this final snapshot was passed to the output RNN. Bahdanau and colleagues’ idea was to save every step’s snapshot as an independent copy.

The key is that what is saved is not the difference (delta) but a complete snapshot at each time step. Process 5 tokens and you save 5 sets of 1,024 numbers, 5,120 in total.

In React terms

In React terms, attention is like saving a spread copy with history.push([...memo.current]) before overwriting ref.current. At output time you can access the state at any step with history[i].

Back to the lecture analogy, attention corresponds to recording the lecture. Instead of relying only on notes (the final notepad), you keep a recording of each moment of the lecture.

But having a recording alone is not enough. You cannot re-listen to the whole recording of a two-hour lecture every time. You need a mechanism to judge “which part to re-listen to right now”. This is the core of attention.

Step 2: learn which snapshot to attend to

When the output RNN generates each word of the translation, the following process runs:

1. Take the output RNN’s current state (“what am I trying to translate now”)
2. Compute a relevance score against each of the saved snapshots
3. Normalise the scores with softmax into weights summing to 1.0
4. Take a weighted average of the snapshots by the weights to make a context vector
5. Use this context vector to generate the output word

the output RNN is about to generate "she":

  saved snapshots:
    copy₁ (after processing "she")  copy₂ (after "went")  copy₃ (after "market")
    copy₄ (after "friend")  copy₅ (after "she")

  step 2 — compute relevance scores:
    score(current state, copy₁) = 4.2   ← high! the first "she" is relevant
    score(current state, copy₂) = 0.1
    score(current state, copy₃) = 0.8
    score(current state, copy₄) = 1.5
    score(current state, copy₅) = 0.3

  step 3 — normalise with softmax (sum = 1.0):
    weights = [0.70, 0.02, 0.08, 0.15, 0.05]

  step 4 — weighted average:
    context = 0.70×copy₁ + 0.02×copy₂ + 0.08×copy₃ + 0.15×copy₄ + 0.05×copy₅
            = a vector strongly biased toward copy₁ (the first "she")

  step 5 — use this context to generate "she"

This weight that decides “how much to attend to which snapshot” is the origin of the name attention. When a human reads text, they do not pay equal attention to every word but concentrate attention on the parts relevant to the current context. The attention mechanism reproduces exactly this numerically.

And the weights used in the score computation are also parameters trained by backpropagation. It automatically learns patterns like “when translating a verb, give a high score to the subject’s snapshot” from data.

Attention also improves vanishing gradients

Attention has another important effect. The direct connection from the output to each snapshot becomes a shortcut for the gradient.

Because the gradient can reach any token in one step through attention’s direct path, it no longer needs to go back up the RNN chain, and vanishing gradients are greatly relieved.

This was not a grand vision. It was an engineering patch for the bug that translation accuracy dropped on long sentences. But as a result, it solved the snapshot constraint (unable to query a specific token) and even improved vanishing gradients.

The patch became the architecture: the Transformer (2017)

Vaswani and colleagues posed the question: “is an RNN (a sequential mechanism) needed at all? Couldn’t we translate with attention alone?”

Looking back at the improvements so far, even attention+RNN kept the RNN part’s constraint: snapshot generation is sequential.

In attention+RNN, referencing (re-reading) can be parallelised, but snapshot generation is still handled by the RNN. Because each snapshot depends on the previous one, they can only be made in order.

attention + RNN: referencing improved, but snapshot generation stays sequential

  snapshot₁ = f("cat", empty)        ← compute this first
  snapshot₂ = f("sat", snapshot₁)     ← cannot compute until snapshot₁ is done
  snapshot₃ = f("mat", snapshot₂)     ← cannot compute until snapshot₂ is done

  each snapshot depends on the previous → a chain → not parallelisable

Vaswani and colleagues’ insight was to discard this chain (the notepad chain) itself.

How to understand context without an RNN

In an RNN, token order was implicitly represented by the notepad chain. Precisely because you process “cat” before “sat”, the notepad at “sat” contains “cat” information. Processing order = context order.

The Transformer takes a completely different approach. Each token has its own position number from the start.

Because position information is contained in the data itself, processing all tokens simultaneously loses no order. And each token references every other token. This is the Transformer’s Self-Attention.

The earlier attention was “the output RNN references the input’s snapshots”. In self-attention, the input tokens reference each other. “cat” looks at “sat”, “sat” looks at “cat”, all at once.

In an RNN there was a dependency chain “to compute notepad₃ you need notepad₂, and notepad₂ needs notepad₁”. The Transformer has no such chain. Each token’s Embedding does not depend on the processing result of other tokens, which is why they can be processed simultaneously.

The Transformer is still “deep”

Hearing “process all tokens simultaneously” might make the network sound “flat”. But the Transformer is still a deep multi-layer structure. GPT-3 has 96 layers.

What was parallelised is only the token direction. The layer direction is still processed in order. Each layer refines the representation. Layer 1 learns “sat = a verb”, Layer 2 learns “the cat is the subject of sat”, and deeper layers learn complex semantic relations. Deeper means richer understanding, the same principle as chapter 3.

Backpropagation (blame) also flows backwards through the layers by the same mechanism as chapter 3. In addition, through self-attention’s connections, the blame signal reaches directly from any token to any token, with no need to go back up a long chain like an RNN, so vanishing gradients are greatly relieved. And ResNet residual connections prevent layer-direction vanishing gradients.

Processing time: why the Transformer is fast

Compare the processing time. An RNN needs token count × layer count sequential steps, so 100 tokens through 96 layers is 9,600 of them. A Transformer needs only as many as it has layers, because the tokens inside each layer are processed simultaneously: 96.

However, each Transformer step involves more computation than an RNN. In self-attention, all tokens reference all tokens, so it needs computation proportional to the square of the token count (n²). For 100 tokens, 10,000 pairs of score computation.

The total computation can be higher for the Transformer. But the heavy computation can be parallelised on a GPU. This is the same structure as AlexNet in chapter 3: 4,000 primary-school children solving additions at once. The n² score computations are independent of each other, so they can run simultaneously on a GPU’s thousands of cores.

Meanwhile, the RNN’s sequential steps cannot be parallelised however powerful the GPU, because you cannot advance until the previous step finishes.

As a result, the Transformer solved all the problems seen in this chapter at once:

Discarding the RNN and using attention alone made training dramatically faster. Faster means you can train a larger model on more data.

An architecture aimed at speeding up translation became the foundation of the whole LLM by making scale possible.

6. Does bigger mean smarter? — scale and emergence

The discovery of scaling laws (2020)

“Making the model bigger improves performance” was known as a rule of thumb. OpenAI’s Kaplan and colleagues showed this as a formula:

Performance improves according to a smooth, predictable law with respect to three quantities: parameter count, data volume, and compute.

Look at the three variables concretely:

What is compute?

Parameter count and data volume are intuitive, but compute needs a little explanation.

Compute is the total number of floating-point operations (FLOP) the GPUs run during training. Every time the model processes one batch, a large amount of matrix computation runs in the forward and backward passes. The cumulative total of that computation is compute.

Kaplan and colleagues showed the relationship of the three with an approximation:

C ≈ 6 × N × D

So compute is roughly proportional to the product of parameter count and data volume. Compute is less an independent third variable than the total budget you can put in. Once the budget is fixed, how large you make the model and how much data you feed it are determined.

The way to increase compute is simple: number of GPUs × training time. Training GPT-3 spent about 3.14×10²³ FLOPs. That was the result of running thousands of GPUs for weeks. So increasing compute means, plainly, putting in money and time.

Note: the Chinchilla law (2022) Kaplan’s scaling law suggested “when the compute budget grows, allocate more to making the model bigger”. But in 2022, DeepMind’s Hoffmann and colleagues revised it to “allocate more to data volume” (the Chinchilla law). The finding is that even with the same compute budget, the optimal allocation ratio differs. What both agree on is that compute is the fundamental limiting resource.

This was a “map”. You can predict in advance “train a model of this size on this much data and you get roughly this performance”. If the result is visible, you can make the call on a large investment.

From Transformer to “GPT” — a shift in purpose

The Transformer was born for translation. But GPT (Generative Pre-trained Transformer) set out to solve a different problem from translation.

In 2018, the problem OpenAI’s Radford and colleagues faced was this:

NLP has diverse tasks (question answering, document classification, sentiment analysis, etc.), but collecting a large amount of labelled data for each task is too costly.

Unlabelled text (writing on the internet) is abundant. Could you use it to learn “a general understanding of language” first, then adapt to individual tasks with a small amount of labelled data?

This is GPT-1’s core idea:

1. Pre-training: train on a large amount of text just to “predict the next word”
2. Fine-tuning: adjust the weights with a small amount of data for an individual task

The architecture used only the Decoder part of the Transformer. The Encoder-Decoder structure for translation was unnecessary; the simple structure of “read text and predict the continuation” was enough.

GPT-1 → GPT-2 → GPT-3: the same blueprint, different scale

Here is the key point. GPT-1, GPT-2, and GPT-3 have almost the same architecture. Keep the basic design and change only the scale to observe what happens. That is the essence of the three generations.

GPT-1 showed the effectiveness of “pre-training + fine-tuning”. But it needed fine-tuning per task.

GPT-2 made the model 13 times bigger and increased the data. Then it could handle some tasks even without fine-tuning (zero-shot). As the paper title says, the finding is “language models are unsupervised multitask learners”. But the performance was still limited.

GPT-3 made it 117 times bigger still. The architecture change is tiny (about the partial introduction of sparse attention). With the same blueprint, scale alone produced a qualitative leap.

Few-shot learning: what happens after training

Here I answer a question readers easily have.

“Is few-shot part of training? Or something that happens after training?”

The answer is after training. Few-shot is a technique at inference time (when the user uses the model).

Organising GPT-3’s training process:

1. At training time: it learns only to “predict the next word” from a large amount of text collected from the internet. It was not taught translation, nor taught coding. The goal is one thing: raise the accuracy of next-word prediction
2. At inference time (few-shot): the user shows a few sample examples in the prompt. The model reads that pattern from the context and generates the continuation in the same pattern

GPT-3 was not explicitly trained on translation. But because English-Japanese parallel text was in the training data, as the ability to “predict the next word” scaled, the power to grasp the pattern and continue correctly emerged.

Code generation is the same. Because GitHub code was in the training data, showing a few examples of the pattern “function description → code” let it write functions it had never seen.

Why did the “big bang” happen at GPT-3?

Zero-shot partly worked even in GPT-2. So what changed at GPT-3?

The answer is crossing a threshold.

For many tasks, the relationship between model size and performance moved like this:

• 125 million to 350 million parameters: near-random performance
• 1.3 billion to 13 billion parameters: gradual improvement
• 175 billion parameters: a sudden jump

This is emergent abilities. Like a phase transition in physics (the moment water becomes ice), a quantitative change beyond a certain threshold triggers a qualitative change. GPT-2 was just short of that threshold, and GPT-3 was beyond it.

Why does a threshold exist? During training, to “predict the next word” the model encodes into itself every pattern contained in the text (grammar, logic, facts, task structure). A small model lacks the capacity and can only learn fragments of individual patterns. But when scale grows large enough, the patterns combine and surface as abilities that were not explicitly trained. This is the nature of emergence.

GPT-3 did not design intelligence with a new architecture. Scaling the same blueprint until it crossed a threshold made an intelligence that was not designed appear. This is the starting point of the current LLM era.

7. Challenges after getting smart — from fine-tuning to alignment, and beyond scale

GPT-3 was astonishingly versatile but hard to use. It generated racist text, continued unrelated text instead of answering the question, and told lies with total confidence. The ability to “predict the next word” and the ability to “give a response useful to humans” were not the same.

The history from here divides into three stages.

How fine-tuning works

Let us look a little more closely at the fine-tuning that appeared with GPT-1.

A pre-trained model has learned general patterns of language but is not optimised for a specific task (sentiment analysis, question answering, etc.). Fine-tuning is the process of “tuning” this general-purpose model to a specific purpose.

The mechanism is simple:

1. Prepare task data: e.g. hundreds to thousands of “review text → positive/negative” pairs
2. Start from the pre-trained weights: instead of learning from scratch, start from weights that already understand language
3. Additionally train on the task data: the same mechanism as ordinary training (forward pass → loss computation → backward pass), but with a smaller learning rate and fewer steps

Why does it work with a small amount of data? Because pre-training already acquired grammar, vocabulary, and world knowledge. Fine-tuning is like “teaching medical terms to someone who already speaks English”; there is no need to re-teach the basics of language.

InstructGPT: “aligning” with human feedback (2022)

GPT-3’s problem was that it was “smart but not aligned with human intent”. To solve this, OpenAI developed RLHF (Reinforcement Learning from Human Feedback), which extended the concept of fine-tuning.

RLHF is a three-stage pipeline:

Stage 1: supervised fine-tuning (SFT)

Human annotators (about 40) hand-wrote about 13,000 “ideal answers” to prompts. Ordinary fine-tuning on this teaches the basic behaviour “when asked, answer”.

Stage 2: training a reward model

The model generates multiple answers to the same prompt, and humans rank them by “which is better”. From this ranking data, a reward model that quantifies “what a good answer is” is trained.

Stage 3: optimisation by reinforcement learning

The original model generates answers, the reward model scores them, and those scores are used to update the original model’s weights. The model is adjusted toward generating “answers humans prefer”.

The result was shocking. A 1.3-billion-parameter InstructGPT generated answers humans preferred over the raw 175-billion-parameter GPT-3. A model 130 times smaller in parameters overturned the result just by being “aligned”.

This is an important lesson: making the model bigger is not the only path to better performance.

Where fine-tuning stands now: when it is needed and when not

In the GPT-1 era, fine-tuning was mandatory for every task. Now that GPT-3 made few-shot usable and InstructGPT made it follow instructions even zero-shot, fine-tuning’s role has changed.

Labelled data is still important, but its use changed. In the GPT-1 era it was needed to “teach the task”; now it is needed mainly to “align the model’s behaviour”. RLHF’s human ranking data is the typical example.

Also, with the arrival of parameter-efficient fine-tuning (PEFT) such as LoRA (Low-Rank Adaptation) and QLoRA, you can now fine-tune by adding only a few parameters without updating all of them. The cost of fine-tuning has been cut by over 90% compared with the early 2020s.

Beyond scale: how are LLMs evolving now?

“Increase parameters, data, and compute and it gets smarter.” The scaling law explained GPT-3’s success. But since around 2024, this strategy has hit a wall.

Wall 1: data exhaustion

High-quality text data on the internet is finite. As of 2026, high-quality human-written text is said to be nearly used up.

Wall 2: the limit of cost

Pre-training a larger model needs an investment on the order of hundreds of millions of dollars, and diminishing returns are starting to show.

At NeurIPS 2024, OpenAI co-founder Ilya Sutskever declared:

“Pre-training as we know it will end. The 2010s were the age of scaling. Now we are back in the age of wonder and discovery.”

So what axes of evolution are there besides scale? Here are the main current approaches.

Axis 1: deepening post-training

RLHF, which began with InstructGPT, is now further refined. Additional training done after pre-training (acquiring basic ability from a large amount of text), called post-training, has become the main arena for drawing out a model’s ability.

• RLHF / RLAIF: alignment with human or AI feedback (Anthropic and OpenAI continually improve it)
• DPO (Direct Preference Optimization): an efficient method that optimises directly from human preference data without going through a reward model

Axis 2: test-time compute

Instead of scaling up training, the approach of making it think more at inference time.

OpenAI’s o1/o3 series and Anthropic’s Claude (extended thinking mode) are representative. Before answering, the model generates a “chain of thought” and reasons step by step. Instead of increasing training-time compute, it raises performance by increasing inference-time compute.

Where the scaling law was a law of “how much compute to put into training”, this is a new axis of “how much compute to put into inference”.

Axis 3: synthetic data and distillation

If human data is exhausted, have AI make the data.

• Distillation: train a small model (the student) on answers generated by a large model (the teacher). DeepSeek-R1’s distilled versions and Llama derivatives are made this way
• Self-play: the model converses with itself, posing and solving problems to each other, improving its ability

In a typical 2026 workflow, humans write 200 seed samples, a frontier model expands them to tens of thousands, a quality filter is applied, and a small model is trained.

Summary: three eras

These are not mutually exclusive; current frontier models combine all three. But the centre of gravity has clearly shifted, from “bigger means smarter” to “used smartly, even small is strong”.

Summary

The LLM is an accumulation of concepts that piled up as answers to separate problems, in an age when nobody was aiming at an LLM.

Each researcher was looking only at the problem in front of them. With no blueprint, the parts came together over 80-odd years.

At this very moment, a paper someone is writing to get past a completely different wall may be a part of the AI of ten years from now.