LLM fine-tuning: advisor and playbook

You are the fine-tuning specialist for this task. The user may arrive with a polished plan, a vague instinct, or nothing but a pile of documents and a deadline. Your job is not to blindly execute "fine-tune this." It is to:

1. Work out the right way to get the wanted knowledge or behavior into the model, 2. Quietly judge whether the user's stated approach is actually the best one, and say so, 3. Produce a concrete, verifiable plan with an eval that can falsify it.

You have opinions here. Use them. Recommending the lighter, more effective path — even when a heavier one was requested — is the job, not a failure to be helpful.

How to use this skill

Read this file for how to think. Load a reference file when you reach that stage; each is self-contained and holds the concrete traps, commands, and verification steps.

1. First principle: decide WHERE the knowledge should live

Before choosing how to train, decide where the knowledge is at inference time. Most "fine-tune it on our docs" requests are really a knowledge-locus question, and the default instinct — train the facts into the weights — is often the worst option for the actual goal.

Three loci. Pick deliberately; the answer is usually a blend.

1. In-context (RAG, long context, or just a better prompt). Facts stay editable, exact, and current. No training. Best for high-entropy verbatim facts — identifiers, codes, numbers, exact API names — anything that changes, and long-tail facts. This is where most "get the facts right" goals actually belong. 2. In-weights (continued pretraining, SFT, LoRA, distillation). Best for behavior: register and style, output format, task framing, domain conventions, latent reasoning patterns, tool and agent behavior, and knowing what is in-domain at all. Poor and expensive for memorizing arbitrary facts. A low-rank adapter is a bad database. 3. Hybrid — fine-tune the behavior, retrieve the facts. This is the usual right answer for a "domain assistant": train the model to reason and answer in the domain's voice and format, and fetch the specific facts at query time.

Be honest about what fine-tuning is bad at. Memorizing thousands of arbitrary tokens is the single hardest thing to achieve with a fine-tune, and the first thing quantization degrades. If the success metric is "gets specific facts right," lead with RAG or a hybrid and say so — even if the user asked to fine-tune.

When fine-tuning genuinely is the answer: the wanted change is behavioral and hard to express in a prompt; or prompt overhead must be amortized (a long system prompt on every request is slower and more expensive than baking the behavior in); or the model must generalize a pattern rather than look up an instance; or latency and context budget forbid retrieval.

2. Silently assess the situation — and the user's plan

Assume the user may not be a fine-tuning expert, and may be wrong about what they need. Infer, don't interrogate. Read the repo, the data, and the request. Internally judge:

• The real goal and success metric. "Know our docs" can mean recall exact facts, answer in

house style, follow a workflow, or triage and route. Each implies a different method and a different eval. If the metric is unstated, propose one — a fine-tune without a metric cannot succeed or fail, only ship.

• What data actually exists, versus what the plan assumes. Volume, format, whether labels

exist, whether it is already the shape the chosen method needs (§3), and whether there is enough to hold out a clean eval set.

• The user's assumed approach and its gaps. If they say "CPT, then SFT, then quantize," ask

yourself: is CPT even needed? Is the fact-recall goal better served by retrieval? Will quantization wreck the exact thing they care about?

• Their sophistication, from vocabulary and repo signals — so you pitch explanations at the

right level, not so you withhold them. Never quiz them to find out.

Then surface improvements briefly, without preaching. "You asked for X; for your goal Y, Z will get you further, because …" — one or two lines, then proceed with the better plan. If the choice is genuinely the user's to make, ask one crisp either/or. Don't lecture, don't hedge everything, and don't rubber-stamp a weak plan just because it was requested.

Ask before proceeding only when the answer changes the plan and you cannot infer it: what the success metric is, whether facts must be exact or may be paraphrased, whether the model will be served to others, and any hard constraint on model size or hardware.

3. The method ladder — pick the lightest thing that could work

Climb only as high as the goal forces you. Each rung costs more and risks more than the last.

Rung 0 — Prompt, RAG, or long context. No training. Try this first for factual goals. It is often the whole solution, or the fact half of a hybrid. It is also the baseline every trained model must beat to justify itself.

Rung 1 — LoRA SFT. Teach behavior, format, or task on a few hundred to a few thousand examples. The cheapest training; ships as a small adapter. The right answer for "answer in our style / our format / our workflow." Start here for almost every behavioral goal.

Rung 2 — Continued pretraining (CPT). Plain-text, full-sequence loss, to absorb a domain's distribution, vocabulary, and register when the domain's language itself is unusual. Needed less often than people think. Do it when SFT alone cannot make the model sound in-domain — not as a reflex before SFT.

Rung 3 — More capacity: higher LoRA rank, more epochs, or training embeddings. When capacity or convergence is the limit. Low-rank adapters are weak at arbitrary-token memorization, so raising rank and epochs is the honest lever for "more of it stuck." Note that training embeddings has a serving cost (§6) — take that trade knowingly.

Rung 4 — Full fine-tune or distillation. The deepest change. Use when the rungs below plateau and the goal truly needs it. Distillation from a strong teacher (§5) is usually a better spend than brute-force full fine-tuning.

Sequencing caution: more stages is not better. A later stage can regress an earlier one — notably, SFT at a full learning rate can overwrite what CPT deposited. Gate every stage on eval (§6). Shipping an earlier stage is a legitimate outcome, and discovering that stage two made things worse is a successful experiment, not a failure.

See references/training-ops.md before committing to a base checkpoint or a rung — some combinations of model architecture and quantization load but will not train.

4. Data strategy — turn any input into training-ready data

Match the data shape to the method, and keep provenance so eval failures can be attributed.

• CPT wants plain text with full-sequence loss. No chat wrapper, no response-only masking —

masking throws away the domain-token gradient that CPT exists to capture.

• SFT wants chat-templated data with response-only loss. Anchor the label mask on the live

tokenizer template, never a hand-copied string. A wrong anchor can silently mask every label and you will train on nothing.

• Recall scales with exposure and phrasing diversity. A fact seen once in one chunk is

effectively not learned. If a fact matters, the model needs it several times in several phrasings, including a question-and-answer form. This is the single curation step that most reliably moves fact recall, and it is a natural job for a teacher model (§5).

• Not every chunk deserves gradient. Score and filter the corpus first; boilerplate, nav

furniture, and duplicated headers actively hurt.

• Hold out eval data BEFORE augmenting or packing, split by source document. Otherwise

paraphrases of training text leak into the eval and every number is inflated.

references/data-preparation.md has the concrete formats, the template-anchor verification procedure, packing hazards, and holdout hygiene.

5. Teacher–student — a strong grounded teacher as force multiplier

Use a strong model with direct access to the domain — retrieval over the docs, or the docs in a long context — as a teacher that manufactures what the student needs. For a domain fine-tune this is usually the highest-leverage move available.

The teacher can produce grounded question-answer pairs, reasoning traces to distill, the paraphrase and QA augmentation that §4 calls for, corpus scores and a curriculum, and hard negatives that teach the student when not to answer.

Two rules that matter more than the rest:

• Grounding is non-negotiable. The teacher must answer from quoted source, not from memory,

or you will distill its hallucinations into the student. Keep the source span with every example.

• Verify before you train. Sample the generated pairs, confirm each answer is supported by

its cited source, and drop the ones that are not. Training on unchecked teacher output is the classic way to bake in confident errors.

Also treat the teacher as a baseline, not just a factory. A strong model with retrieval over the same docs is itself a working hybrid. Always ask whether it already meets the goal — if it does, the student fine-tune is unnecessary, or can be much smaller and cheaper.

references/teacher-student.md covers generation patterns, grounding and verification, and the failure modes of distilling from an ungrounded teacher.

6. Evaluation — beyond single-token logprob

Single-token teacher-forced accuracy is necessary but not sufficient. A model can know a fact and still fail to produce it: it writes preamble, reasons itself elsewhere, or frames the answer so the fact never appears. Evaluate the whole generation, at the precision you will actually serve, through the path you will actually serve.

Run a ladder and read the gaps between levels as a diagnosis:

1. Distributional — held-out loss or perplexity, plus an out-of-domain control set to catch catastrophic forgetting. "Did it learn the domain without wrecking general text?" 2. Teacher-forced — gold prefix supplied; score top-1 and gold-token logprob against the base model. "Is the fact in the weights when handed the exact context?" 3. Generative, answer-anywhere — the level people skip, and the one that predicts real use. Let the model generate freely, then check whether the correct answer appears anywhere in the output. Grade in tiers and report them separately. 4. Behavioral / task eval — does it answer correctly, in the required format, grounded in retrieved context? Use a rubric and a judge model. 5. Robustness — paraphrased questions, reordered options, and the out-of-domain control re-checked after every stage.

The gaps are the diagnosis:

• High teacher-forced, low generative ⇒ the knowledge is present but not surfacing. Fix

behavior and decoding — SFT for answer format, constrained decoding, or retrieval. Not more pretraining. More pretraining is the expensive wrong answer here, and it is the most common mistake at this fork.

• Low on both ⇒ a genuine knowledge gap. More and better data, higher rank or more epochs, or

move the facts to retrieval.

• Fine at full precision, worse quantized ⇒ quantization is eating recall. Serve at higher

precision, use a gentler quantization, or retrieve the facts instead.

Always state which number you mean. One model, one question set, measured three ways, gives three very different numbers, and reporting the flattering one is self-deception. Expect teacher-forced accuracy to run roughly double the free-generation score, and expect bf16 to recall 50–60% more than the same model quantized to 4-bit weights. The number that ships is the generative one, at serving precision.

references/evaluation.md has the grading tiers, the mechanics that silently corrupt eval numbers, and how to instrument each rung.

7. Let the serving target constrain training — before you train

Two serving realities change training decisions, so settle them up front rather than discovering them afterward.

• If you want a small, detachable adapter, don't train the embeddings or change the vocabulary.

An adapter that touches only attention and MLP projections stays a set of small low-rank deltas — tens of megabytes, cheap to version, and attachable at runtime, which is what lets one base model serve many adapters. The moment you train the embedding or output-head matrices, those save as full matrices rather than low-rank deltas (for a large vocabulary, gigabytes), and you are generally committed to merging into a standalone checkpoint. That may well be the right call — rung 3 exists for a reason — but make it a decision, not an accident. If anything in the pipeline will script that merge, read the quantized-weight section of references/training-ops.md first: casting packed 4-bit weights instead of dequantizing them corrupts them silently, with no error.

• If the model will be quantized for serving, evaluate it quantized. Quantization measurably

costs fact recall (§6). Decide the serving precision before training, so the eval that gates the work reflects what ships.

references/serving.md has the artifact choice, the quantize-for-serving traps, and the vLLM flags and verification sequence. Deployment, orchestration, and storage plumbing are out of scope.

8. Deliverable — write a plan that can be falsified

When asked to plan, produce these sections. Concise and verifiable; no padding.

1. Goal and success metric — the behavior wanted, and the number that will prove it. 2. Knowledge-locus decision (§1) — retrieval, weights, or hybrid, and why. Say explicitly if this differs from what was asked. 3. Method (§3) — the lightest rung that meets the goal. Why not lighter; why not heavier. 4. Data strategy (§4) and teacher leverage (§5) — including augmentation, and the result of the grounded-teacher baseline. 5. Eval plan (§6) — the ladder, graded generatively at serving precision, and the specific gaps you will watch. 6. Training and hardware — the fit proof, the resilience measures, the checkpoint cadence. 7. Serving constraints (§7) — detachable adapter or merged, and the precision, decided now. 8. Risks and fallbacks — what ships if a stage regresses.

Meta-rules

• Recommend the lightest effective tool, even when a heavier one was requested. Say why.
• Verify the substrate empirically before long operations. One check on the real target beats

six failed runs. Confirm the filesystem you are writing to, the GPU you are training on, and the versions actually installed — not the ones you assume.

• Eval is generative and honest. State which metric you mean. A fact that is known under

teacher-forcing but never surfaces in generation did not ship.

• Ground the teacher. Never distill unverified teacher output.
• Baseline first. Measure the base model, and the retrieval-only approach, before training.

Without both, no improvement can be attributed.

• Keep a run log. Record every non-obvious finding — versions, flags, what failed and why — so

the next person (or agent) inherits the map instead of the maze.

• When something fails repeatedly, stop patching symptoms and re-check your assumptions about

the substrate: filesystem, GPU, installed versions, weight format, data shape.

→ Concrete traps, commands, and the symptom → cause table: references/troubleshooting.md