Fine-tuning is often proposed as the way to make a model know a specific subject. It is effective for some goals and a poor fit for that particular one.
Adjusting weights shapes behaviour
Fine-tuning continues training on additional examples, nudging the weights toward the patterns those examples display.
Because the examples are few relative to the original training data, the adjustment is a shift in tendency rather than the addition of a new capability.
What shifts most readily is style: response format, level of detail, terminology and how the model handles requests it should decline.
Facts do not stick reliably
Teaching a specific fact by including it in training examples requires many repetitions, and the result is a tendency rather than a lookup.
The model may produce a plausible variant of the fact instead of the fact itself, since it learned a pattern rather than stored a record.
Facts also change, and updating a fine-tuned model means retraining, whereas updating a document store means editing a file.
Retrieval handles knowledge better
Supplying relevant material alongside the question lets the model work from text it can read rather than from what it absorbed during training.
This makes answers traceable to a source, keeps information current and allows access control, since the retrieval step can respect permissions.
The two approaches combine well: fine-tuning for how the system should respond, retrieval for what it should respond about.
Narrow training narrows the model
Training heavily on a specialised set can degrade performance on tasks outside it, because the adjustment pulls general behaviour toward the narrow examples.
Safety behaviour is particularly susceptible, since it was established through careful training that a subsequent adjustment can partially undo.
Evaluating a fine-tuned model on general tasks, not only on the target task, is what catches this before it reaches users.
Data quality dominates data quantity
A small set of consistent, carefully constructed examples typically outperforms a large set assembled from whatever was available.
Inconsistency is the main failure: examples that handle similar cases differently teach the model that either response is acceptable.
Most of the effort in a successful fine-tune goes into constructing and reviewing that set, and the training run itself is the short part.
Teams that budget for data preparation rather than for compute tend to get usable results on the first attempt, while the reverse pattern usually produces several inconclusive runs.