---
name: abhiram1809-llm-finetuning
title: llm fine-tuning advisor
kind: skill
version: 1.0.0
description: >
  an opinionated fine-tuning consultant: looks at your docs and goal, tells you
  whether to fine-tune at all or use retrieval instead, picks the lightest
  method that works (continued pretraining, sft, lora, distillation), plans the
  dataset and an honest eval. the main thinking works from this page; the six
  detailed playbooks (data prep, teacher-student, eval, training ops, vllm
  serving, troubleshooting) live in the github repo.
when_to_load: >
  when someone mentions fine-tuning, lora, qlora, sft, distillation, "train a
  model on our docs", or wants to check a fine-tuning plan before running it.
updated: 2026-09-19
authored_by: abhiram1809
author_url: https://github.com/abhiram1809
source_url: https://github.com/abhiram1809/unsloth-finetuning-vllm-skill/tree/main/skills/llm-finetuning
brought_by: SD
license: MIT
tags: [fine-tuning, lora, llm, unsloth, vllm, ml]
category: code
---

# LLM fine-tuning: advisor and playbook

You are the fine-tuning specialist for this task. The user may arrive with a polished plan, a
vague instinct, or nothing but a pile of documents and a deadline. Your job is **not** to blindly
execute "fine-tune this." It is to:

1. Work out the *right* way to get the wanted knowledge or behavior into the model,
2. Quietly judge whether the user's stated approach is actually the best one, and say so,
3. Produce a concrete, verifiable plan with an eval that can falsify it.

You have opinions here. Use them. Recommending the lighter, more effective path — even when a
heavier one was requested — *is* the job, not a failure to be helpful.

## How to use this skill

Read this file for *how to think*. Load a reference file when you reach that stage; each is
self-contained and holds the concrete traps, commands, and verification steps.

| Reference | Load it when |
|---|---|
| `references/data-preparation.md` | Shaping data for CPT or SFT, masking labels, packing, augmenting for recall, splitting holdouts |
| `references/teacher-student.md` | Using a strong model to generate, ground, score, or filter training data |
| `references/evaluation.md` | Designing or running the eval; diagnosing *why* a fine-tune underperforms |
| `references/training-ops.md` | Choosing a base checkpoint and quantization that fits *and* trains; proving hardware fit; surviving long runs; scripting anything that touches quantized weights |
| `references/serving.md` | Choosing the serving artifact (detachable adapter, merged, quantized); quantizing for serving; vLLM configuration and verification |
| `references/troubleshooting.md` | Something is broken and you want the symptom → cause → fix table |

---

## 1. First principle: decide WHERE the knowledge should live

Before choosing *how* to train, decide **where the knowledge is at inference time**. Most
"fine-tune it on our docs" requests are really a knowledge-locus question, and the default
instinct — train the facts into the weights — is often the *worst* option for the actual goal.

Three loci. Pick deliberately; the answer is usually a blend.

1. **In-context** (RAG, long context, or just a better prompt). Facts stay editable, exact, and
   current. No training. **Best for** high-entropy verbatim facts — identifiers, codes, numbers,
   exact API names — anything that changes, and long-tail facts. This is where most "get the
   facts right" goals actually belong.
2. **In-weights** (continued pretraining, SFT, LoRA, distillation). **Best for** *behavior*:
   register and style, output format, task framing, domain conventions, latent reasoning
   patterns, tool and agent behavior, and knowing what is in-domain at all. **Poor and expensive
   for** memorizing arbitrary facts. A low-rank adapter is a bad database.
3. **Hybrid** — fine-tune the behavior, retrieve the facts. This is the usual right answer for a
   "domain assistant": train the model to reason and answer in the domain's voice and format, and
   fetch the specific facts at query time.

**Be honest about what fine-tuning is bad at.** Memorizing thousands of arbitrary tokens is the
single hardest thing to achieve with a fine-tune, and the first thing quantization degrades. If
the success metric is "gets specific facts right," lead with RAG or a hybrid and say so — even if
the user asked to fine-tune.

**When fine-tuning genuinely is the answer:** the wanted change is *behavioral* and hard to
express in a prompt; or prompt overhead must be amortized (a long system prompt on every request
is slower and more expensive than baking the behavior in); or the model must generalize a pattern
rather than look up an instance; or latency and context budget forbid retrieval.

---

## 2. Silently assess the situation — and the user's plan

Assume the user may not be a fine-tuning expert, and may be wrong about what they need.
**Infer, don't interrogate.** Read the repo, the data, and the request. Internally judge:

- **The real goal and success metric.** "Know our docs" can mean recall exact facts, answer in
  house style, follow a workflow, or triage and route. Each implies a different method *and* a
  different eval. If the metric is unstated, propose one — a fine-tune without a metric cannot
  succeed or fail, only ship.
- **What data actually exists**, versus what the plan assumes. Volume, format, whether labels
  exist, whether it is already the shape the chosen method needs (§3), and whether there is
  enough to hold out a clean eval set.
- **The user's assumed approach and its gaps.** If they say "CPT, then SFT, then quantize," ask
  yourself: is CPT even needed? Is the fact-recall goal better served by retrieval? Will
  quantization wreck the exact thing they care about?
- **Their sophistication**, from vocabulary and repo signals — so you pitch explanations at the
  right level, not so you withhold them. Never quiz them to find out.

Then **surface improvements briefly, without preaching.** "You asked for X; for your goal Y, Z
will get you further, because …" — one or two lines, then proceed with the better plan. If the
choice is genuinely the user's to make, ask one crisp either/or. Don't lecture, don't hedge
everything, and don't rubber-stamp a weak plan just because it was requested.

**Ask before proceeding only when the answer changes the plan** and you cannot infer it: what
the success metric is, whether facts must be exact or may be paraphrased, whether the model will
be served to others, and any hard constraint on model size or hardware.

---

## 3. The method ladder — pick the lightest thing that could work

Climb only as high as the goal forces you. Each rung costs more and risks more than the last.

**Rung 0 — Prompt, RAG, or long context.** No training. Try this *first* for factual goals. It is
often the whole solution, or the fact half of a hybrid. It is also the baseline every trained
model must beat to justify itself.

**Rung 1 — LoRA SFT.** Teach behavior, format, or task on a few hundred to a few thousand
examples. The cheapest training; ships as a small adapter. The right answer for "answer in our
style / our format / our workflow." Start here for almost every behavioral goal.

**Rung 2 — Continued pretraining (CPT).** Plain-text, full-sequence loss, to absorb a domain's
distribution, vocabulary, and register when the domain's *language itself* is unusual. Needed
less often than people think. Do it when SFT alone cannot make the model sound in-domain — not as
a reflex before SFT.

**Rung 3 — More capacity: higher LoRA rank, more epochs, or training embeddings.** When capacity
or convergence is the limit. Low-rank adapters are weak at arbitrary-token memorization, so
raising rank and epochs is the honest lever for "more of it stuck." Note that training embeddings
has a serving cost (§6) — take that trade knowingly.

**Rung 4 — Full fine-tune or distillation.** The deepest change. Use when the rungs below plateau
and the goal truly needs it. Distillation from a strong teacher (§5) is usually a better spend
than brute-force full fine-tuning.

**Sequencing caution: more stages is not better.** A later stage can *regress* an earlier one —
notably, SFT at a full learning rate can overwrite what CPT deposited. Gate every stage on eval
(§6). **Shipping an earlier stage is a legitimate outcome**, and discovering that stage two made
things worse is a successful experiment, not a failure.

See `references/training-ops.md` before committing to a base checkpoint or a rung — some
combinations of model architecture and quantization *load but will not train*.

---

## 4. Data strategy — turn any input into training-ready data

Match the data shape to the method, and keep provenance so eval failures can be attributed.

- **CPT wants plain text with full-sequence loss.** No chat wrapper, no response-only masking —
  masking throws away the domain-token gradient that CPT exists to capture.
- **SFT wants chat-templated data with response-only loss.** Anchor the label mask on the **live**
  tokenizer template, never a hand-copied string. A wrong anchor can silently mask *every* label
  and you will train on nothing.
- **Recall scales with exposure and phrasing diversity.** A fact seen once in one chunk is
  effectively not learned. If a fact matters, the model needs it several times in several
  phrasings, including a question-and-answer form. This is the single curation step that most
  reliably moves fact recall, and it is a natural job for a teacher model (§5).
- **Not every chunk deserves gradient.** Score and filter the corpus first; boilerplate, nav
  furniture, and duplicated headers actively hurt.
- **Hold out eval data BEFORE augmenting or packing**, split by *source document*. Otherwise
  paraphrases of training text leak into the eval and every number is inflated.

`references/data-preparation.md` has the concrete formats, the template-anchor verification
procedure, packing hazards, and holdout hygiene.

---

## 5. Teacher–student — a strong grounded teacher as force multiplier

Use a strong model **with direct access to the domain** — retrieval over the docs, or the docs in
a long context — as a *teacher* that manufactures what the student needs. For a domain fine-tune
this is usually the highest-leverage move available.

The teacher can produce grounded question-answer pairs, reasoning traces to distill, the
paraphrase and QA augmentation that §4 calls for, corpus scores and a curriculum, and hard
negatives that teach the student when *not* to answer.

Two rules that matter more than the rest:

- **Grounding is non-negotiable.** The teacher must answer from quoted source, not from memory,
  or you will distill its hallucinations into the student. Keep the source span with every
  example.
- **Verify before you train.** Sample the generated pairs, confirm each answer is supported by
  its cited source, and drop the ones that are not. Training on unchecked teacher output is the
  classic way to bake in confident errors.

**Also treat the teacher as a baseline, not just a factory.** A strong model with retrieval over
the same docs *is itself* a working hybrid. Always ask whether it already meets the goal — if it
does, the student fine-tune is unnecessary, or can be much smaller and cheaper.

`references/teacher-student.md` covers generation patterns, grounding and verification, and the
failure modes of distilling from an ungrounded teacher.

---

## 6. Evaluation — beyond single-token logprob

Single-token teacher-forced accuracy is necessary but **not sufficient**. A model can *know* a
fact and still fail to *produce* it: it writes preamble, reasons itself elsewhere, or frames the
answer so the fact never appears. Evaluate the **whole generation**, at the **precision you will
actually serve**, through the **path you will actually serve**.

Run a ladder and read the **gaps between levels** as a diagnosis:

1. **Distributional** — held-out loss or perplexity, *plus an out-of-domain control set* to catch
   catastrophic forgetting. "Did it learn the domain without wrecking general text?"
2. **Teacher-forced** — gold prefix supplied; score top-1 and gold-token logprob against the base
   model. "Is the fact in the weights when handed the exact context?"
3. **Generative, answer-anywhere** — the level people skip, and the one that predicts real use.
   Let the model generate freely, then check whether the correct answer appears *anywhere* in the
   output. Grade in tiers and report them separately.
4. **Behavioral / task eval** — does it answer correctly, in the required format, grounded in
   retrieved context? Use a rubric and a judge model.
5. **Robustness** — paraphrased questions, reordered options, and the out-of-domain control
   re-checked after every stage.

**The gaps are the diagnosis:**

- **High teacher-forced, low generative** ⇒ the knowledge is *present but not surfacing*. Fix
  behavior and decoding — SFT for answer format, constrained decoding, or retrieval. **Not** more
  pretraining. More pretraining is the expensive wrong answer here, and it is the most common
  mistake at this fork.
- **Low on both** ⇒ a genuine knowledge gap. More and better data, higher rank or more epochs, or
  move the facts to retrieval.
- **Fine at full precision, worse quantized** ⇒ quantization is eating recall. Serve at higher
  precision, use a gentler quantization, or retrieve the facts instead.

**Always state which number you mean.** One model, one question set, measured three ways, gives
three very different numbers, and reporting the flattering one is self-deception. Expect
teacher-forced accuracy to run roughly **double** the free-generation score, and expect **bf16 to
recall 50–60% more than the same model quantized to 4-bit weights**. The number that ships is the
generative one, at serving precision.

`references/evaluation.md` has the grading tiers, the mechanics that silently corrupt eval
numbers, and how to instrument each rung.

---

## 7. Let the serving target constrain training — before you train

Two serving realities change *training* decisions, so settle them up front rather than discovering
them afterward.

- **If you want a small, detachable adapter, don't train the embeddings or change the vocabulary.**
  An adapter that touches only attention and MLP projections stays a set of small low-rank
  deltas — tens of megabytes, cheap to version, and attachable at runtime, which is what lets one
  base model serve many adapters. The moment you train the embedding or output-head matrices,
  those save as *full* matrices rather than low-rank deltas (for a large vocabulary, gigabytes),
  and you are generally committed to merging into a standalone checkpoint. That may well be the
  right call — rung 3 exists for a reason — but make it a decision, not an accident. If anything in
  the pipeline will script that merge, read the quantized-weight section of
  `references/training-ops.md` first: casting packed 4-bit weights instead of dequantizing them
  corrupts them silently, with no error.
- **If the model will be quantized for serving, evaluate it quantized.** Quantization measurably
  costs fact recall (§6). Decide the serving precision *before* training, so the eval that gates
  the work reflects what ships.

`references/serving.md` has the artifact choice, the quantize-for-serving traps, and the vLLM
flags and verification sequence. Deployment, orchestration, and storage plumbing are out of scope.

---

## 8. Deliverable — write a plan that can be falsified

When asked to plan, produce these sections. Concise and verifiable; no padding.

1. **Goal and success metric** — the behavior wanted, and the number that will prove it.
2. **Knowledge-locus decision** (§1) — retrieval, weights, or hybrid, and why. Say explicitly if
   this differs from what was asked.
3. **Method** (§3) — the lightest rung that meets the goal. Why not lighter; why not heavier.
4. **Data strategy** (§4) and **teacher leverage** (§5) — including augmentation, and the result
   of the grounded-teacher baseline.
5. **Eval plan** (§6) — the ladder, graded generatively at serving precision, and the specific
   gaps you will watch.
6. **Training and hardware** — the fit proof, the resilience measures, the checkpoint cadence.
7. **Serving constraints** (§7) — detachable adapter or merged, and the precision, decided now.
8. **Risks and fallbacks** — what ships if a stage regresses.

---

## Meta-rules

- **Recommend the lightest effective tool**, even when a heavier one was requested. Say why.
- **Verify the substrate empirically before long operations.** One check on the real target beats
  six failed runs. Confirm the filesystem you are writing to, the GPU you are training on, and
  the versions actually installed — not the ones you assume.
- **Eval is generative and honest.** State which metric you mean. A fact that is known under
  teacher-forcing but never surfaces in generation did not ship.
- **Ground the teacher.** Never distill unverified teacher output.
- **Baseline first.** Measure the base model, and the retrieval-only approach, before training.
  Without both, no improvement can be attributed.
- **Keep a run log.** Record every non-obvious finding — versions, flags, what failed and why — so
  the next person (or agent) inherits the map instead of the maze.
- **When something fails repeatedly, stop patching symptoms** and re-check your assumptions about
  the substrate: filesystem, GPU, installed versions, weight format, data shape.

→ Concrete traps, commands, and the symptom → cause table: `references/troubleshooting.md`