GyaanStack. Blog

Articles / fine-tuningllmcheat-sheet

The LLM fine-tuning cheat sheet: methods, LoRA numbers, data formats and failure modes

SFT, DPO, ORPO and GRPO on one page, LoRA and QLoRA settings with worked memory numbers for a 7B model, chat template traps, evals and a failure table.

By Raghunath · · 20 min read

Fine-tuning changes how a model behaves: its output format, its tone, the label set it picks from, how it reasons through one kind of task. It does not give the model fresh facts in any reliable way, and it is the most expensive of the three ways to change an LLM app. This sheet assumes you have already decided a fine-tune is the right move. If you haven't, read Fine-tuning, RAG or prompting? A decision guide first.

Everything below was checked against transformers 5.24.0, peft 0.22.0 and trl 1.18.2, the current PyPI releases on 22 November 2026. The code blocks ran on a MacBook; none of them downloads model weights.

The workflow in one line#

baseline prompt + eval set -> pick method (SFT / DPO / GRPO) -> format data with the model's chat template
  -> LoRA or QLoRA config -> short run -> eval vs base and vs best prompt -> regression check -> merge or serve adapter

The eval set comes first. Without a score for the base model and for your best prompt, you cannot tell whether the fine-tune helped.

Which training method#

Method What it learns from Data you need TRL class Use it when
SFT Good examples (imitation) Prompt and ideal answer, a few hundred to a few thousand SFTTrainer Format, style, a fixed label set, a narrow task
DPO Pairs of better and worse answers Prompt, chosen, rejected DPOTrainer SFT output is close but you prefer some answers over others
ORPO SFT and preference in one stage, no reference model Prompt, chosen, rejected trl.experimental.orpo.ORPOTrainer You want one run instead of SFT then DPO
KTO Single answers marked good or bad Prompt, completion, true/false label KTOTrainer You have thumbs up/down logs, not pairs
GRPO Rewards on sampled answers Prompts plus a reward function you write GRPOTrainer Output can be checked by code: math answers, JSON validity, unit tests

Notes that save time:

  • SFT first, almost always. DPO and GRPO start from a model that already produces the right kind of output. Running GRPO on a model that never emits valid JSON gives every sample a reward of zero, and with zero variance there is nothing to learn.
  • ORPO moved. In TRL 1.x it lives under trl.experimental.orpo, so old from trl import ORPOTrainer code fails to import.
  • DPO with LoRA needs no second model copy. When you pass a PEFT model and no ref_model, TRL computes the reference log-probs by running the same model with the adapter switched off.
  • GRPO has no KL penalty by default. GRPOConfig.beta defaults to 0.0 and loss_type to "dapo". The TRL docs cite recent papers that found the KL term unnecessary. Set beta above zero if the policy drifts into reward hacking.
  • The library defaults assume full fine-tuning. SFTConfig defaults to a learning rate of 2e-5; DPOConfig and GRPOConfig default to 1e-6 (DPO's beta is 0.1). With LoRA, raise them (see the hyperparameter table).

Full fine-tune, LoRA or QLoRA#

Full fine-tune LoRA QLoRA
What trains Every weight Small low-rank matrices beside frozen weights Same as LoRA, base stored in 4-bit NF4
7B memory floor (below) ~125 GB ~19 GB ~9 GB
Output A full checkpoint An adapter of 5 to 300 MB An adapter of 5 to 300 MB
Quality Highest ceiling Close to full on most SFT tasks when applied to all linear layers Close to LoRA; slower per step because weights are dequantized on the fly
Forgetting Most Less (Biderman et al., 2024) Same as LoRA

Start with LoRA if the bf16 model fits on your GPU with room to spare, and QLoRA if it does not. Full fine-tuning is for teams that have measured LoRA falling short. The full mechanics, including the memory derivation and training cost, are in LoRA and QLoRA fine-tuning: when it is worth it and what it costs.

Worked numbers for a 7B model#

The calculator below uses the public config.json of Qwen2.5-7B-Instruct: hidden size 3584, MLP size 18944, 28 layers, 28 query heads and 4 key/value heads, vocabulary 152,064. It counts LoRA parameters by hand, then builds the model on PyTorch's meta device (shapes only, no weights, no download) and lets PEFT count them, so the formula is checked against the library.

For one linear layer of shape in x out, LoRA adds r x (in + out) parameters. With grouped-query attention, K and V project to 4 x 128 = 512 values, not 3584, so their adapters are smaller.

pip install torch transformers peft trl datasets
import json, torch
from transformers import AutoConfig, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

cfg = json.load(open("qwen7b_config.json"))          # Qwen2.5-7B-Instruct config.json
h, ff, L = cfg["hidden_size"], cfg["intermediate_size"], cfg["num_hidden_layers"]
V, heads, kv = cfg["vocab_size"], cfg["num_attention_heads"], cfg["num_key_value_heads"]
kv_dim = kv * (h // heads)                            # grouped-query attention: K/V are narrow

shapes = {  # (in, out) for every linear in one decoder layer
    "q_proj": (h, h), "k_proj": (h, kv_dim), "v_proj": (h, kv_dim), "o_proj": (h, h),
    "gate_proj": (h, ff), "up_proj": (h, ff), "down_proj": (ff, h),
}

def lora_params(r, targets):
    return L * sum(r * (i + o) for m, (i, o) in shapes.items() if m in targets)

# 1. Check the hand formula against PEFT on a weightless (meta) copy of the model
with torch.device("meta"):
    model = AutoModelForCausalLM.from_config(AutoConfig.from_pretrained("qwen7b_config.json"))
total = sum(p.numel() for p in model.parameters())
peft_model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                                              task_type="CAUSAL_LM"))
trainable = sum(p.numel() for p in peft_model.parameters() if p.requires_grad)
print(f"base params        {total:,}")
print(f"PEFT trainable r16 {trainable:,}")
print(f"formula r16        {lora_params(16, shapes):,}\n")

# 2. Trainable params for common settings
print(f"{'setting':22s} {'trainable':>12s} {'% base':>7s} {'adapter bf16':>12s}")
for name, r, t in [("q,v r=8", 8, {"q_proj", "v_proj"}),
                   ("all-linear r=16", 16, shapes), ("all-linear r=64", 64, shapes)]:
    n = lora_params(r, t)
    print(f"{name:22s} {n:12,} {100*n/total:6.2f}% {2*n/1e6:9.0f} MB")

# 3. Memory floor for training (GB), seq 2048, batch 1, gradient checkpointing on.
#    Adam in mixed precision = 16 bytes per trainable param (2 weight + 2 grad + 12 optimizer).
GB, s = 1e9, 2048
lora = lora_params(16, shapes)
emb = 2 * V * h                                        # input embedding + untied lm_head
lin = total - emb                                      # (norms are tiny; counted with linears)
acts = 2 * s * h * L + 2 * s * V * 4                   # checkpointed layer inputs + fp32 logits & grad
rows = {
    "full fine-tune": total * 16,
    "LoRA, bf16 base": total * 2 + lora * 16,
    "QLoRA, NF4 base": lin * 4.127 / 8 + emb * 2 + lora * 16,
}
print(f"\n{'method':16s} {'weights+optim':>13s} {'activations':>11s} {'floor':>7s}")
for k, v in rows.items():
    print(f"{k:16s} {v/GB:10.1f} GB {acts/GB:8.1f} GB {(v+acts)/GB:5.1f} GB")
print(f"\nlogits alone at seq {s}: {2*s*V*4/GB:.2f} GB (vocab {V:,})")

Output:

base params        7,615,616,512
PEFT trainable r16 40,370,176
formula r16        40,370,176

setting                   trainable  % base adapter bf16
q,v r=8                   2,523,136   0.03%         5 MB
all-linear r=16          40,370,176   0.53%        81 MB
all-linear r=64         161,480,704   2.12%       323 MB

method           weights+optim activations   floor
full fine-tune        121.8 GB      2.9 GB 124.8 GB
LoRA, bf16 base        15.9 GB      2.9 GB  18.8 GB
QLoRA, NF4 base         6.2 GB      2.9 GB   9.1 GB

logits alone at seq 2048: 2.49 GB (vocab 152,064)

How to read it:

  • The formula matches PEFT exactly (40,370,176), so you can size an adapter from config.json before you rent a GPU.
  • 16 bytes per trainable parameter is the mixed-precision Adam accounting from the ZeRO paper. LoRA wins because only 40M parameters pay it; the 7.6B frozen ones cost 2 bytes each in bf16 and about half a byte in NF4.
  • 4.127 bits per quantized weight is 4-bit NF4 plus the double-quantized block scales described in the QLoRA paper. Embeddings and the output head are assumed to stay in bf16.
  • Qwen's 152k vocabulary makes the logits expensive. At 2,048 tokens the fp32 logits and their gradient take 2.5 GB, which is most of the activation line. Doubling the sequence length doubles it.
  • Rank barely moves the total. Going from r=16 to r=64 adds about 1.9 GB of optimizer state. Sequence length, batch size and gradient checkpointing move memory far more.
  • These are floors. The CUDA context, allocator fragmentation and dequantization buffers sit on top. Budget a 24 GB card for this QLoRA setup and a 40 to 48 GB card for bf16 LoRA.

LoRA and QLoRA hyperparameters#

Setting Start here Range Notes
r 16 8 to 64 Higher rank rarely fixes bad data. Raise it only if train loss stalls high
lora_alpha 2 x r r to 2 x r The update is scaled by alpha / r. PEFT's default is alpha 8 with r 8
use_rslora False Scales by alpha / sqrt(r) (rsLoRA). Worth turning on at r of 64 and above
target_modules "all-linear" Attention-only LoRA underperforms in LoRA Without Regret's experiments; include the MLP layers
lora_dropout 0.05 0 to 0.1 PEFT's default is 0.0
Learning rate (SFT) 2e-4 5e-5 to 3e-4 About 10x the full fine-tune rate (Thinking Machines, LoRA Without Regret)
Learning rate (DPO with LoRA) 5e-6 to 1e-5 The 10x rule applied to TRL's 1e-6 default. Watch the chosen log-probs as well as the reward margin
Epochs 1 to 3 Small datasets overfit by epoch 3. Stop on validation loss
Effective batch 16 to 32 Per-device batch x gradient accumulation x GPUs. LoRA Without Regret found LoRA pays a bigger penalty than full fine-tuning at large batches
max_length p95 of your token lengths TRL's default is 1024. Anything longer is truncated without an error
Warmup and schedule 3 to 5% warmup, cosine or linear decay
use_dora False DoRA splits magnitude and direction. Slower per step; try it after a plain LoRA baseline

Change one or two of these at a time, and compare runs on the eval set, not on training loss.

Data formats TRL accepts#

TRL's dataset formats page lists the shapes. These are the ones you will use, one JSONL row each:

SFT, conversational   {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
SFT, prompt-completion {"prompt": [{"role": "user", "content": "..."}], "completion": [{"role": "assistant", "content": "..."}]}
DPO / ORPO preference {"prompt": [...], "chosen": [{"role": "assistant", ...}], "rejected": [{"role": "assistant", ...}]}
KTO unpaired          {"prompt": [...], "completion": [...], "label": true}
GRPO prompt-only      {"prompt": [{"role": "user", "content": "..."}], "answer": "..."}   extra columns go to the reward function

Prefer the conversational form. TRL then renders each row with the model's own chat template, so training and inference see the same special tokens. Hand-written ### Instruction: strings train the model on a format it will never see at serving time.

What the template does to your data#

The script loads only the tokenizer of Qwen2.5-0.5B-Instruct (about 10 MB) and renders one training row.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")  # tokenizer files only, ~10 MB
messages = [
    {"role": "user", "content": "Classify: 'refund not received after 10 days'"},
    {"role": "assistant", "content": '{"intent": "refund_status"}'},
]
# What SFT trains on: the full conversation, rendered with the model's own template
print(tok.apply_chat_template(messages, tokenize=False))
# What you send at inference: the prompt plus the assistant header, nothing after it
print(repr(tok.apply_chat_template(messages[:1], tokenize=False, add_generation_prompt=True)[-40:]))

# assistant_only_loss=True in TRL needs a template with {% generation %} markers
out = tok.apply_chat_template(messages, return_dict=True, return_assistant_tokens_mask=True)
print("generation markers in template:", "{% generation %}" in tok.chat_template)
print("assistant tokens in mask:", sum(out["assistant_masks"]), "of", len(out["input_ids"]))

# TRL ships "training" versions of common templates and swaps one in when
# assistant_only_loss=True. Check what the loss will actually see:
from trl.chat_template_utils import get_training_chat_template
train_tpl = get_training_chat_template(tok, require_prefix_preservation=False)
out = tok.apply_chat_template(messages, chat_template=train_tpl, return_dict=True,
                              return_assistant_tokens_mask=True)
ids = [i for i, m in zip(out["input_ids"], out["assistant_masks"]) if m]
print("with TRL training template:", sum(out["assistant_masks"]), "tokens ->", repr(tok.decode(ids)))

Output:

[transformers] return_assistant_tokens_mask==True but chat template does not contain `{% generation %}` keyword.
<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
Classify: 'refund not received after 10 days'<|im_end|>
<|im_start|>assistant
{"intent": "refund_status"}<|im_end|>

"0 days'<|im_end|>\n<|im_start|>assistant\n"
generation markers in template: False
assistant tokens in mask: 0 of 51
with TRL training template: 10 tokens -> '\n{"intent": "refund_status"}<|im_end|>\n'

Three things in that output cause real bugs:

  1. A system prompt you never wrote. Qwen's template adds "You are Qwen, created by Alibaba Cloud..." when a row has no system message. If production sends a different system prompt, training and serving no longer match. Put the exact production system prompt in every training row.
  2. The stock template cannot mark assistant tokens. With the model's own template, the assistant mask is all zeros. TRL 1.18 swaps in a training template for known model families when you set assistant_only_loss=True, and raises an error if a row still has no assistant tokens. For a model it does not know, you have to add the markers yourself.
  3. The end-of-turn token must be inside the loss. The training template includes <|im_end|> in the 10 masked tokens, so the model learns when to stop. A model that never sees its stop token trained in the loss rambles at inference, while the training loss looks fine.

A minimal SFT run with LoRA#

This is a smoke test: a tiny randomly initialised Qwen2 (19.5M parameters) with the real tokenizer, so it checks that the data, template, LoRA config and trainer wire together, without downloading a model. For a real run, replace the from_config model with the model ID.

from datasets import Dataset
from peft import LoraConfig
from transformers import AutoTokenizer, AutoModelForCausalLM, Qwen2Config
from trl import SFTConfig, SFTTrainer

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
# Smoke test: a tiny randomly initialised Qwen2 so nothing big downloads.
# For a real run: model = model_id (TRL loads it), or AutoModelForCausalLM.from_pretrained(model_id)
model = AutoModelForCausalLM.from_config(Qwen2Config(
    vocab_size=len(tok), hidden_size=64, intermediate_size=128, num_hidden_layers=2,
    num_attention_heads=4, num_key_value_heads=2))

rows = [("refund not received after 10 days", "refund_status"),
        ("please cancel my order", "cancel_order"),
        ("where is my parcel", "track_order"),
        ("charged twice for one order", "billing_issue")]
train = Dataset.from_list([{"messages": [
    {"role": "user", "content": f"Classify: '{q}'"},
    {"role": "assistant", "content": f'{{"intent": "{a}"}}'}]} for q, a in rows])

args = SFTConfig(
    output_dir="out-sft", max_steps=4, per_device_train_batch_size=2,
    learning_rate=2e-4,            # LoRA wants roughly 10x the full fine-tune rate
    max_length=1024, assistant_only_loss=True, logging_steps=1, report_to="none")
peft_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                      target_modules="all-linear", task_type="CAUSAL_LM")
trainer = SFTTrainer(model=model, args=args, train_dataset=train,
                     processing_class=tok, peft_config=peft_cfg)
trainer.model.print_trainable_parameters()
trainer.train()

Output (progress bars removed):

trainable params: 32,768 || all params: 19,520,192 || trainable%: 0.1679
{'loss': '11.85', 'grad_norm': '1.404', 'learning_rate': '0.0002', 'entropy': '11.92', 'mean_token_accuracy': '0', 'num_tokens': '95', 'epoch': '0.5'}
{'loss': '11.84', 'grad_norm': '1.407', 'learning_rate': '0.00015', 'entropy': '11.92', 'mean_token_accuracy': '0', 'num_tokens': '193', 'epoch': '1'}
{'loss': '11.83', 'grad_norm': '1.431', 'learning_rate': '0.0001', 'entropy': '11.92', 'mean_token_accuracy': '0', 'num_tokens': '291', 'epoch': '1.5'}
{'loss': '11.82', 'grad_norm': '1.334', 'learning_rate': '5e-05', 'entropy': '11.92', 'mean_token_accuracy': '0', 'num_tokens': '386', 'epoch': '2'}

The starting loss is a useful sanity check. A random model over 151,665 tokens should score about ln(151,665) = 11.93, and it does. A pretrained instruct model starts far below that. If yours starts near ln(vocab), the weights did not load, or the labels are misaligned.

For a full run on a laptop, including dataset splits, before-and-after scores and the GGUF export, see Fine-tune a small language model on a MacBook.

A GRPO reward function#

GRPO samples several completions per prompt (num_generations, default 8), scores each with your reward functions, and pushes the model toward completions that beat their group's average. GRPOTrainer calls each reward function with the completions plus every extra dataset column as a keyword argument, and expects one float per completion.

import json

# GRPOTrainer calls each reward function with the batch of completions plus every
# extra dataset column as a keyword argument, and expects one float per completion.
def json_intent_reward(completions, answer, **kwargs):
    scores = []
    for completion, gold in zip(completions, answer):
        text = completion[0]["content"]          # conversational dataset: list of messages
        try:
            pred = json.loads(text)["intent"]
        except (json.JSONDecodeError, KeyError, TypeError):
            scores.append(0.0)                   # unparseable output earns nothing
            continue
        scores.append(1.0 if pred == gold else 0.2)  # valid JSON, wrong label: small credit
    return scores

group = [  # four sampled completions for one prompt (num_generations=4)
    [{"role": "assistant", "content": '{"intent": "refund_status"}'}],
    [{"role": "assistant", "content": '{"intent": "cancel_order"}'}],
    [{"role": "assistant", "content": 'Sure! The intent is refund_status.'}],
    [{"role": "assistant", "content": '{"label": "refund_status"}'}],
]
r = json_intent_reward(group, answer=["refund_status"] * 4)
mean = sum(r) / len(r)
std = (sum((x - mean) ** 2 for x in r) / (len(r) - 1)) ** 0.5   # TRL uses sample std
print("rewards   ", r)
print("advantages", [round((x - mean) / (std + 1e-4), 2) for x in r])

Output:

rewards    [1.0, 0.2, 0.0, 0.0]
advantages [1.47, -0.21, -0.63, -0.63]

The advantage line reproduces TRL's default group normalisation (scale_rewards="group"): subtract the group mean, divide by the group's sample standard deviation plus 1e-4. The correct answer gets pushed up, and the chatty answer and the wrong key get pushed down. Unit-test reward functions like this before training. A reward bug is the most common reason a GRPO run "learns" something odd, and the code is easy to check without a GPU. Pass it as GRPOTrainer(model=..., reward_funcs=[json_intent_reward], ...).

Evaluation#

Training loss tells you the model memorised the training set. It does not tell you the model got better at the job. The minimum eval:

                 held-out test set (never trained on, no near-duplicates of train rows)
                          |
        +-----------------+-----------------+
        |                 |                 |
   base + plain      base + best      fine-tuned
     prompt            prompt          (same prompt)
        |                 |                 |
        +------- task metric per arm -------+
                          |
            regression set: general skills the base had
  • Three arms. If the fine-tune only beats the plain prompt, a few-shot prompt may have been enough.
  • Pick the metric for the task. Classification: accuracy and per-class F1. JSON: parse rate, then field-level accuracy. Generation: an LLM judge checked against 30 to 50 of your own labels, or pairwise human review.
  • A regression set. 50 to 100 prompts outside your task (general questions, instruction following, refusals the base handled correctly). Fine-tunes lose things quietly.
  • Generate with production settings. Same template, same system prompt, same temperature, same stop tokens.
  • Split before deduplication checks, then check across splits. Near-duplicate rows in train and test inflate scores.

The LoRA guide's evaluation section covers sample sizes and serving-side checks; Evaluating an AI agent in Python shows an eval harness you can reuse.

When training goes wrong#

Symptom Likely cause Fix
Starting loss near ln(vocab) Weights not loaded, or labels shifted Check the model path and a decoded batch
Loss drops to near 0 quickly Duplicates, tiny dataset, or loss on prompt tokens that repeat in every row Deduplicate; assistant_only_loss=True; fewer epochs
Validation loss rises after epoch 1 Overfitting Stop early; more varied data; lower LR
Model never stops generating End-of-turn token not trained, or wrong EOS in the generation config Check the mask includes the stop token; set eos_token_id
Good in training eval, bad in production Template or system prompt differs at serving Render both with the same template; diff the strings
Long examples lose their endings Truncated at max_length Measure token lengths; set max_length to p95 or more
General skills got worse Too narrow data, too many epochs, LR too high Mix some general instruction data into train; fewer epochs; run the regression set
DPO margin grows but answers get worse Both chosen and rejected log-probs falling Lower LR or beta; start from a better SFT model
GRPO reward flat at zero Base model never produces a rewardable output SFT first; give partial credit in the reward
GRPO reward rises, outputs look strange Reward hacking Read samples every N steps; tighten the reward; try beta > 0
QLoRA out of memory at long context Activations and logits, not the weights Gradient checkpointing; shorter sequences; smaller batch with more accumulation
Merged model worse than adapter Merged into a quantized base Merge into the bf16 base, then quantize the merged model

Checklist#

  • [ ] Baseline scores for base + plain prompt and base + best prompt
  • [ ] Held-out test set and regression set, frozen before training starts
  • [ ] Data in conversational format, rendered once with the real template and read by eye
  • [ ] Production system prompt in every training row
  • [ ] Assistant mask checked: correct token count, stop token included
  • [ ] Token lengths measured; max_length set from them
  • [ ] LoRA on all linear layers, LR raised for LoRA, one change per run
  • [ ] Reward functions unit-tested (GRPO)
  • [ ] Three-arm eval plus regression set on every candidate adapter
  • [ ] Library versions, base model revision and data hash saved with the adapter

The first two boxes decide whether the rest is worth doing.

Sources#