RAG vs fine-tuning for KB search — what fine-tuning actually learns, and when pure IR wins

Introduction

Plenty of people are building a knowledge base (KB) search system on RAG. I am one of them, and at some point a naive question came to me.

“If I fine-tuned a small local model on our internal documents, could it do the same job as RAG, with a smaller and cheaper model?”

Tailor the model to the documents it has to reference, and the search infrastructure and the vector database both disappear. Intuitively that seems to work. It does not, and the reason is that fine-tuning learns behaviour rather than facts, so it cannot supply what a KB search is actually asking for. What did hold up was the opposite lesson: for a narrow enough need, something simpler than full RAG often wins. Starting from that question, I worked through three others in order: can fine-tuning stand in for RAG, what is fine-tuning actually doing, and is RAG even the only option.

I hope it helps anyone holding the same question sort out their thinking.

Question 1: can fine-tuning replace RAG

Looking into the first question, it became clear the intuition does not hold up well. Fine-tuning and RAG are solving different problems in the first place.

Fine-tuning learns “behaviour”, not “facts”

Fine-tuning adjusts a model’s weights to change its style, output format, tone, and the patterns by which it carries out a task. What it is not good at is embedding specific facts so they can be pulled back out reliably afterwards.

Fine-tune on internal documents and the model learns to “talk like internal documents”, but against a specific question it will happily return a fluent, confident, wrong answer. RAG does the opposite: it keeps facts in a searchable store, in the original wording.

Why RAG suits KB search

Working through it, the reasons RAG suits a KB search system come down to three: grounding and citation, cheap updates, and hallucination held down by the retrieved context. In a KB, being able to trace a source is worth a lot.

The trap in the “cheaper and smaller” intuition

“A small model specialised to our documents would be cheap and small” is an appealing idea, but it has several holes.

• Fine-tuning does not shrink a model in any meaningful sense. Model size is set by the reasoning and language ability you need, not by how much domain text you showed it. A small model, tailored on your documents, is still a small model that is bad at synthesis and instruction-following.
• Building and maintaining fine-tuning has real costs: data preparation, compute, evaluation, and retraining on every document update. RAG avoids all of that.
• RAG’s inference cost, where retrieved chunks lengthen the prompt, is a genuine concern, but it usually still comes in cheaper than owning a training pipeline.

Where fine-tuning genuinely works

None of which makes fine-tuning pointless. Combined with RAG it is effective.

• Teaching a consistent output format, tone, and domain vocabulary (jargon)
• Improving how the model uses retrieved context (RAFT, or Retrieval-Augmented Fine-Tuning, and similar) and improving query understanding
• Baking repeated instructions into the weights so the prompt gets shorter

So the mature pattern is “facts by RAG, behaviour by (light, if needed) fine-tuning”. It is not a replacement for RAG.

Question 2: what is fine-tuning actually doing

I have written “fine-tuning changes behaviour” up to here, but honestly, part of it had not clicked for me.

“Fine-tuning is about nudging a product model like ChatGPT or Claude with your own data, right? If that is all the behaviour change you get, isn’t prompting enough?”

That question came from not knowing base models existed.

The models we use are already fine-tuned

What surprised me on investigation is that product models like ChatGPT and Claude are not the “finished article” of an LLM. They are what an LLM looks like after large-scale fine-tuning has already happened. The base model (the pre-trained model) that sits behind them is a completely different thing from what we touch day to day.

What a base model is

A base model is trained on the task of “predict the next word” over enormous amounts of internet text: web pages, books, code. That pre-training gives it the fundamentals of language, including grammar, knowledge, and reasoning patterns.

But a base model is only a text completion engine. Give it a question and it does not return an answer; it tries to write the continuation of the question. It does not follow instructions, does not answer questions, and does not refuse harmful requests. It has language ability but does not know how to hold a conversation.

Fine-tuning is what makes a model usable

Turning a base model into a useful assistant is the fine-tuning stage known as post-training.

1. Instruction tuning: teaching conversational patterns such as “answer when asked” and “follow instructions” from large numbers of (question, answer) pairs.
2. RLHF (reinforcement learning from human feedback): human raters judge which answers are good and bad, and the model’s output is adjusted against those judgements. This is what reinforces helpful, safe responses.

So the thing we type prompts at is itself the product of enormous fine-tuning. Thinking “prompting is enough” was standing on the benefits of fine-tuning.

Which one costs the money: pre-training vs fine-tuning

Building an LLM has two broad phases, and their costs differ by orders of magnitude.

Aspect	Pre-training (building the base model)	Fine-tuning (post-training)
What it does	Learns next-word prediction over huge text	Learns dialogue, instruction-following, safety
Rough compute cost	Tens of millions to hundreds of millions of dollars	Millions of dollars (including RLHF)
Infrastructure needed	Thousands to tens of thousands of GPUs for months	Tens to hundreds of GPUs for days to weeks
Data needed	Trillions of tokens of general text	Tens of thousands to hundreds of thousands of high-quality (instruction, answer) pairs, plus human ratings
Barrier to entry	Very high (capital, infrastructure, data, specialist staff)	Relatively low (an open base model can be used)

Pre-training is overwhelmingly the more expensive, and perhaps a few dozen companies worldwide can do it in-house. Fine-tuning, by contrast, can start from an open base model such as Llama or Mistral, which puts it within reach of newer companies.

Which one makes new models stronger

Given all that, another question follows. When a new model gets dramatically smarter, as in GPT-3 to GPT-4 or Claude 2 to Claude 3, is pre-training or fine-tuning contributing more?

The short answer: pre-training sets the ceiling on capability, and fine-tuning decides how close you get to that ceiling.

• Large jumps across generations (GPT-3 to GPT-4, Claude 2 to Claude 3) come almost entirely from advances in pre-training. More compute, better data, larger model size, architectural improvements: these raise the base model’s fundamentals, meaning depth of reasoning, breadth of knowledge, and code comprehension. Fine-tuning cannot bolt on a capability the base model does not have.
• Improvements within a generation (GPT-4 to GPT-4o, Claude 3 Opus to Claude 3.5 Sonnet) can owe a great deal to better fine-tuning and post-training. With the same or a similar base model, better RLHF data, more refined training methods, and clever distillation can move benchmark scores substantially.

So pre-training sets the upper bound on how smart a model can get, and fine-tuning determines how much of that intelligence you can draw out. The two are complementary, not substitutes. In one line, new models are strong because better fine-tuning sits on top of a better base model.

A realistic path for a new company to own a model

Once the cost structure is clear, the options for “we want our own model” come into view.

1. From pre-training: tens of millions of dollars and up. Requires frontier-lab funding and infrastructure. Realistically a few dozen companies.
2. Open base model plus in-house fine-tuning: tens of thousands to millions of dollars. Lets you build a model specialised to your domain. The route most AI startups take.
3. Product model API plus prompt engineering: hundreds to thousands of dollars per month. The easiest by far, but the model is not yours.

So “millions of dollars on fine-tuning” was never about changing a personality. It is the process that takes a base model which can only complete text and makes it viable as an assistant, and that process itself creates the core value of the product.

Why companies fine-tune further

Some do fine-tune on top of a product model. That is not a personality change; there are strong practical reasons.

• Reliability: a prompt is a request; fine-tuning is training. Across millions of production calls, a 5% failure rate can be fatal.
• New capabilities: tool-use patterns, structured extraction, domain reasoning. Things prompting alone cannot do reliably can be taught.
• Cost reduction: instead of putting 3,000 tokens of instructions in every prompt, bake them into the weights and get by with 50. At scale that is a large difference.
• Distillation: fine-tune a small, cheap model on the outputs of a huge, expensive one, and get most of the capability for a fraction of the cost.

The rule of thumb is prompt first, fine-tune when you hit a wall. Reliability that prompting cannot reach, behaviour too complex to write down, a scale at which long prompts get expensive: invest when you reach that point, and recoup it through lower inference cost.

Aspect	Prompt engineering	Fine-tuning
Number of examples you can give	A handful	Tens of thousands
Consistency and reliability	Request-based, so it varies	Training-based, so it is high
Granting new capabilities	Limited	Possible
Prompt length and inference cost	Tends to grow	Can be baked in and shortened
Transferring capability between models	Not possible	Possible via distillation

Question 3: is RAG even the only option

Having established “facts by RAG”, the next question was whether RAG is the only right answer. What helped here was breaking down what RAG actually is.

Splitting RAG into two stages

RAG is not monolithic. It is two stages glued together.

1. Retrieval: find the relevant text. This is classical ML and information retrieval (IR) exactly: BM25, embeddings, rerankers, vector search. No reasoning required.
2. Generation: the LLM reads the retrieved text and synthesises an answer.

So “ML or RAG” is a false dichotomy. ML (machine learning) is the retrieval half of RAG. The real design question was “which stages do I genuinely need, and how sophisticated does each have to be”.

Seen that way, alternatives by need come into focus.

Choosing by case

• If you only need to “find”, drop the LLM: if the goal is only finding the right document or passage, classical IR (BM25 plus a dense retriever plus a reranker) completes the system on its own. No generation, no hallucination, cheap, fast, auditable. Many “KB search” needs are in fact this, and some are over-engineered by bolting an LLM on afterwards.
• If the corpus is small, skip retrieval and use long context: with today’s large context windows, if the knowledge roughly fits, you can stuff the whole thing into the prompt (cheaply, with prompt caching). No vector database, no chunking, no retrieval misses. For a small, stable corpus it is simpler than RAG. It breaks down at scale on cost, latency, and “lost in the middle”, the drop in recall for material in the middle of a long input.
• If relationships matter, GraphRAG or a knowledge graph: for questions that require connecting facts across multiple documents (multi-hop), plain vector RAG struggles because it fetches each chunk in isolation. Building and querying a knowledge graph, or GraphRAG, fits better. Build and maintenance costs are higher.
• If queries are complex or multi-step, agentic retrieval: instead of one retrieve-then-generate pass, let the model iteratively search, read, refine the query, and search again. Strong on hard questions, but slower and more expensive per query.

Need	Recommended approach	Generation (LLM) required
Just find the document or passage	Pure IR (BM25 plus dense plus reranker)	No
Small, stable corpus	Long context (plus prompt caching)	Yes (retrieval not needed)
Multi-hop, relationships	GraphRAG or knowledge graph	Yes
Complex, multi-step queries	Agentic retrieval	Yes
Synthesised answers over a large, changing corpus	RAG	Yes

The honest summary

Honestly, for the specific combination of “return a grounded, current, synthesised answer over a large and changing corpus”, nothing replaces RAG. That is precisely RAG’s job.

But when the need is narrower, something simpler than RAG usually wins. That was the lesson. The mistake is not “choosing RAG”; it is reaching for fully-equipped RAG where a cheaper subset would do. So the question to ask in practice is not “what beats RAG” but “does the user actually need a synthesised answer, and how large and stable is the corpus”. Those two answers determine the architecture.

Squaring this with the earlier sections

The earlier point, that fine-tuning is bad at embedding facts because that is RAG’s job, was narrow and correct. Meanwhile, as Question 2 showed, fine-tuning’s “changing behaviour” covers a very wide range: from turning a base model into an assistant, through reliability and new skills, to cost reduction via distillation.

Going deeper: if cost reduction is the real goal

Back at the original motive, what I actually cared about was lowering cost. On that front there are moves that work better than fine-tuning.

• Swap the generation model behind RAG for something smaller and cheaper
• Improve retrieval quality (better chunking, reranking, hybrid search) so fewer tokens get sent
• Cache embeddings
• Self-host a small open model (in the 7 to 8B range, say) as RAG’s generator

That gets you the “small local model tailored to the use case” without giving up grounding.

One caveat: if your “KB search” is really lookup rather than generation, meaning finding the right passage, the biggest wins are almost entirely on the retrieval side, and the choice of LLM barely matters.

Summary

Starting from a naive “could fine-tuning replace RAG more cheaply”, this is where I landed:

• Fine-tuning learns “behaviour”, not “facts”. Supplying facts for KB search is RAG’s job, and the mature pattern is using both, not replacing one with the other.
• RAG is not monolithic. It is retrieval (classical ML and IR) plus generation (LLM). If the need is narrow, pure IR, long context, GraphRAG, or an agentic approach often wins on simplicity.
• The question to ask in practice is not “what beats RAG” but “is a synthesised answer required” and “how large and stable is the corpus”.
• The product models we use, ChatGPT and Claude, are themselves the result of large-scale fine-tuning applied to a base model. Pre-training (tens of millions to hundreds of millions of dollars) and fine-tuning (millions) differ by orders of magnitude in cost, and using open models lowers the barrier to entry. The rule of thumb is prompt first, fine-tune when you hit a wall.
• If cost reduction is the goal, a smaller generation model, better retrieval quality, cached embeddings, and self-hosting all work better.

The starting point for the decision is simple: does the user want an answer or a document, and how large and how frequently updated is the corpus. Settle those two first and the architecture choice gets much easier.