What Is SFT? A Practical Guide to Supervised Fine-Tuning Your Own Model

The Short Answer: What Is SFT

SFT (Supervised Fine-Tuning) means: take a batch of “input → ideal output” examples and adjust a model that can already talk, so it answers the way you want.

If pretraining is how a model “learns language,” SFT is how it “learns to follow instructions, play a role, and produce the output you expect.” It is the first step in training your own model—and the one with the best return on effort.

Your understanding of the flow is basically right, with one emphasis:

Pick a base model → prepare the dataset → SFT

The real barrier is not the training itself (modern tools run it from a few lines of config)—it is the dataset. You could say 90% of SFT success lives in the data.

Where SFT Fits in the Training Pipeline

Back to the three stages of LLM training:

Stage	What it does	Your use case
Pretraining	Learn language and world knowledge	Usually not done yourself
SFT	Learn instructions, style, behavior	Your first step
Alignment (DPO/RLHF)	Make outputs better match preferences	An enhancement after SFT

• Pretraining decides what the model “knows”;
• SFT decides how it “talks and acts”;
• Alignment decides how satisfied people are with its answers.

The order cannot be shuffled: SFT first, then alignment. Jumping straight to DPO on a poorly tuned base is building on a tilted foundation.

Step 1: Pick a Base Model

Default to the Instruct / Chat version, not the plain Base version.

• A plain Base model only “continues text”—it cannot hold a conversation;
• The Instruct version has already gone through general instruction tuning, so you are doing a second round of fine-tuning (domain/style SFT) on top of it.

Common choices:

Base	Notes
Qwen2.5-1.5B / 7B / 14B-Instruct	Strong Chinese, great ecosystem, commercially usable—top pick
Llama 3.1/3.2 8B / 3B-Instruct	Strong English, large community
GLM, Yi, DeepSeek series	Depends on language and data

How big? Start at 7B and get LoRA working first; only move to 14B or full fine-tuning if results fall short.

Step 2: Prepare the Dataset (the Core)

What the data looks like

SFT data is just “example conversations,” one by one. Three mainstream formats:

① Alpaca format (single-turn)

{
  "instruction": "Give a roleplay reply",
  "input": "Work was exhausting today",
  "output": "That sounds genuinely draining—want to tell me what happened?"
}

② Multi-turn format (ShareGPT / messages)—more common for assistant-style fine-tuning

{
  "messages": [
    { "role": "system", "content": "You are Xiaoya, a gentle, attentive virtual companion." },
    { "role": "user", "content": "Work was exhausting today" },
    {
      "role": "assistant",
      "content": "That sounds genuinely draining—want to tell me what happened?"
    },
    { "role": "user", "content": "My boss dumped new requirements on me again" },
    { "role": "assistant", "content": "Again… you must have wanted to roll your eyes so hard." }
  ]
}

③ ShareGPT format

{
  "conversations": [
    { "from": "human", "value": "Work was exhausting today" },
    { "from": "gpt", "value": "That sounds genuinely draining…" }
  ]
}

The three are just different wrappers; fundamentally they are all “context → desired reply.”

One key point you must understand: assistant-only loss

During training, only the assistant turns (what the model must generate) contribute to the loss; the user and system parts are masked out (labels set to -100).

Why? Because you are teaching “how the model should reply,” not “how the user talks.” Without masking, the model learns the user’s tone too—or even starts asking and answering itself.

Good news: mainstream frameworks (TRL, LLaMA-Factory) mask automatically when you use multi-turn formats. Just use the right data format.

Where the data comes from

• Manual annotation: highest quality, highest cost—good for a small set of polished examples
• Real production logs: large and authentic, but must be cleaned (dedup, PII removal, low-quality filtering, conversation reconstruction)
• Synthesis by a strong model (distillation): batch-generate with a strong model, then filter by rules/humans—great value
• Open datasets: ready to use, but generic—usually mixed with your own data

How much data

• A few hundred to a few tens of thousands of examples all show clear gains;
• Research has shown that roughly 1,000 high-quality examples can teach a model a specific task;
• Quality matters far more than quantity: tens of thousands of noisy examples often underperform a few thousand carefully chosen ones.

Cleaning checklist: dedupe, remove garbled/dead-short messages, unify role setups, balance categories, and exclude the eval set (data leakage makes you misjudge).

Step 3: Run It—Full Fine-Tuning or LoRA?

Method	Description	Best for
Full fine-tuning	Updates all parameters, higher ceiling	Lots of data, plenty of compute
LoRA / QLoRA	Trains a few extra parameters, low memory	Top choice for individuals/small teams

On tooling, you rarely write a training loop by hand:

• LLaMA-Factory: config-driven, easiest;
• Unsloth: faster and lighter on a single GPU;
• Hugging Face TRL: SFTTrainer, flexible;
• Axolotl: config-based, active community.

A LLaMA-Factory LoRA SFT config looks roughly like this:

model_name_or_path: Qwen/Qwen2.5-7B-Instruct
stage: sft
finetuning_type: lora
lora_rank: 8
lora_alpha: 16
dataset: my_sft_dataset
template: qwen # must match the base model's chat template
cutoff_len: 2048
per_device_train_batch_size: 2
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true

A few hyperparameters to watch:

• learning rate: LoRA commonly uses 1e-4; full fine-tuning needs much smaller (1e-5 range);
• epochs: 2–3 is usually enough; too many overfits;
• chat template: must match the base (Qwen uses ChatML), or all your training is wasted;
• cutoff_len / packing: truncation strategy must be sensible.

The Most Common Pitfalls

• Format or template mismatch → loss won’t drop, model learns garbage
• Catastrophic forgetting: full fine-tuning + small data + high LR wipes out existing ability; mix in a little general instruction data
• Overfitting: the model “memorizes answers” and fails on rephrased questions; cut epochs, add data
• Learning format but not content: data is too homogeneous
• Train/inference system prompt mismatch: training says “You are Xiaoya,” inference changes the setup, results don’t line up
• EOS / truncation issues: replies never stop, or the end token is missing

How to Tell SFT Worked

• Build a fixed evaluation set (never use training data);
• Use LLM-as-judge + human scoring to compare before/after SFT:
  • Does it match the target style/persona?
  • Is instruction following better?
  • Did general ability regress (catastrophic forgetting)?
• A loss curve alone is not enough—low loss does not mean good output.

The Boundaries of SFT: What It Can and Cannot Do

• ✅ SFT excels at: changing behavior, style, output format, roleplay
• ❌ SFT is bad at: injecting new knowledge (use continued pretraining or RAG)
• ❌ SFT is not responsible for: optimizing preferences, reducing disliked answers (use DPO and other alignment methods)

In one line: SFT teaches “how to answer,” not “what to know.”

The Minimal Working Pipeline

1. Choose a base: Qwen2.5-7B-Instruct
2. Prepare data: JSONL, multi-turn messages format, assistant-only loss
3. Pick a tool: LLaMA-Factory or TRL
4. Run a small QLoRA job first; check loss and outputs
5. Build an eval set; compare before/after SFT
6. Merge the LoRA, deploy (vLLM / Ollama)

The pipeline is that short—the hard part is the data quality and evaluation behind each step. Nail those two, and SFT itself is the easy part.