GyaanStack. Blog

Articles / fine-tuningllmpython

Fine-tune a small language model on a MacBook: data, LoRA, eval and export

A 0.5B model fine-tuned end to end on an Apple Silicon laptop with MLX-LM. Real training time, memory, before/after scores and the GGUF export traps.

By Raghunath · · 17 min read

You can fine-tune a half-billion-parameter model on a laptop in under a minute. The training loop is the easy part. The parts that decide whether the result is any good are the dataset, the split, and a test that the model has never seen. This post walks through one full run on a MacBook with an Apple M5 and 24 GB of unified memory: build a dataset, score the base model, train a LoRA adapter with MLX-LM, score it again, fix what the score exposes, then fuse and export.

Every number below comes from this run on that machine. Your times will differ with chip and background load.

This post does not explain how LoRA works or how to size it for a 7B model on a GPU. LoRA and QLoRA fine-tuning: when it is worth it and what it costs covers that. If you are still deciding whether to fine-tune at all, read Fine-tuning, RAG or prompting? first. The task here is one where fine-tuning is the right call: a fixed output format and a fixed label set that a small model keeps getting wrong.

What we are building#

The task is turning a chat-style expense message into strict JSON:

"spent 1200 rupees on vitamin tablets today"
        |
        v
{"amount": 1200, "currency": "INR", "category": "health"}

Five categories (food, travel, bills, shopping, health) and two currencies. A parser downstream calls json.loads on the reply, so a reply wrapped in a Markdown fence counts as a failure.

The pipeline:

make_data.py ──> data/{train,valid,test}.jsonl
                        |
       evaluate.py (base model)  ──> baseline score
                        |
       mlx_lm.lora --train       ──> adapters/ (12 MB)
                        |
       evaluate.py (with adapter) ──> new score, error breakdown
                        |
       mlx_lm.fuse               ──> fused/ (safetensors, 953 MB)
                        |
       GGUF + Ollama             ──> see the export section, it has traps

Why MLX-LM on a Mac#

On Apple Silicon there are two realistic routes. One is PyTorch with the MPS backend plus Hugging Face PEFT. The other is MLX, Apple's array framework, with its MLX-LM package. MLX-LM ships a LoRA trainer as a command-line tool, reads chat-format JSONL directly, and loads Hugging Face checkpoints without conversion. Most Unsloth tutorials assume an NVIDIA GPU, and bitsandbytes 4-bit loading is built for CUDA, so the usual Colab QLoRA recipe does not carry over to a Mac as-is.

This post uses MLX-LM. The PEFT on MPS route was not run here, so it gets no numbers. The PyTorch MPS notes are the place to start if you want it.

Versions used, checked against PyPI on the day of the run: mlx-lm 0.32.2 and mlx 0.32.4, in a Python 3.12 virtual environment.

uv venv -p 3.12 .venv
uv pip install -p .venv/bin/python mlx-lm
.venv/bin/python -c "import mlx_lm, mlx.core as mx; print(mlx_lm.__version__, mx.__version__)"
0.32.2 0.32.4

The model is Qwen2.5-0.5B-Instruct. It was already in the local Hugging Face cache (953 MB on disk), so this run downloaded no weights.

The dataset: format, split and a held-out test#

MLX-LM's trainer wants a folder with train.jsonl, valid.jsonl and, for testing, test.jsonl (LORA.md). Each line can be a messages list in the usual chat format. With --mask-prompt, loss is computed only on the final assistant message, so the model is not trained to reproduce the system prompt.

The script below generates 400 synthetic examples. One decision matters more than the rest: the last item in each category (such as "vitamin tablets" or "the gas cylinder") never appears in training or validation. The test set uses only those held-out items. If the test drew from the same items as training, a model that memorised "BigBasket means food" would score perfectly and tell you nothing.

"""Build a small synthetic dataset: expense messages -> strict JSON."""
import json, random, sys
from pathlib import Path

random.seed(7)
SYSTEM = ("Extract the expense as JSON with keys amount (number), "
          "currency (INR or USD), category (food, travel, bills, shopping, health). "
          "Reply with JSON only.")

ITEMS = {
    "food": ["lunch at Meghana's", "groceries from BigBasket", "coffee with the team",
             "dinner at a dhaba", "a Swiggy order", "breakfast idli and vada"],
    "travel": ["an Uber to the airport", "a Rapido ride to office", "the metro card top-up",
               "a train ticket to Mysuru", "petrol for the bike", "an auto to Koramangala"],
    "bills": ["the electricity bill", "the broadband bill", "mobile recharge",
              "the water bill", "society maintenance", "the gas cylinder"],
    "shopping": ["a pair of running shoes", "a phone cover", "a new kurta",
                 "a USB-C cable", "a backpack", "books from Blossom"],
    "health": ["medicines from Apollo", "a dental check-up", "the gym fee",
               "a blood test", "a physio session", "vitamin tablets"],
}
TEMPLATES = [
    "Paid {cur_word}{amt} for {item}",
    "spent {amt} {cur_code} on {item} today",
    "{item} cost me {cur_word}{amt}",
    "Just paid {amt} {cur_code} for {item}, ugh",
    "{cur_word}{amt} gone on {item}",
    "Log this: {item}, {amt} {cur_code}",
]

# Round 2: more varied training items per category (test items unchanged).
EXTRA = {
    "food": ["pani puri", "a cake for the birthday", "fruit from the market", "biryani takeaway"],
    "travel": ["a cab to Whitefield", "an Ola outstation trip", "the toll at Electronic City", "a bus pass"],
    "bills": ["the credit card bill", "the DTH recharge", "rent for November", "the LPG refill"],
    "shopping": ["a novel from Sapna Book House", "earphones", "a notebook and pens", "a lamp from IKEA"],
    "health": ["eye drops", "a doctor consultation", "a yoga class", "cough syrup"],
}
ROUND2 = "--round2" in sys.argv

# The last item in each category never appears in train or valid.
# The test set uses only those, so it measures generalisation, not recall.
TRAIN_ITEMS = {c: v[:-1] + (EXTRA[c] if ROUND2 else []) for c, v in ITEMS.items()}
TEST_ITEMS = {c: v[-1:] for c, v in ITEMS.items()}

def one(pool):
    cat = random.choice(list(pool))
    item = random.choice(pool[cat])
    usd = random.random() < 0.25
    amt = random.choice([round(random.uniform(3, 120), 2), random.randint(5, 300)]) if usd \
        else random.choice([random.randint(40, 9000), random.randint(1, 90) * 50])
    cur = "USD" if usd else "INR"
    text = random.choice(TEMPLATES).format(
        amt=amt, item=item,
        cur_word="$" if usd else random.choice(["Rs ", "₹", "INR "]),
        cur_code=cur if random.random() < 0.6 else ("dollars" if usd else "rupees"))
    answer = {"amount": amt, "currency": cur, "category": cat}
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": text},
        {"role": "assistant", "content": json.dumps(answer)},
    ]}

def build(pool, n, seen):
    rows = []
    while len(rows) < n:
        r = one(pool)
        key = r["messages"][1]["content"]
        if key not in seen:          # no duplicate inputs across splits
            seen.add(key); rows.append(r)
    return rows

seen = set()
train_valid = build(TRAIN_ITEMS, 340, seen)
splits = {"train": train_valid[:300], "valid": train_valid[300:],
          "test": build(TEST_ITEMS, 60, seen)}
out = Path("data_round2" if ROUND2 else "data"); out.mkdir(exist_ok=True)
for name, part in splits.items():
    if ROUND2 and name == "test":
        continue                     # reuse data/test.jsonl so scores compare
    with open(out / f"{name}.jsonl", "w") as f:
        for r in part:
            f.write(json.dumps(r, ensure_ascii=False) + "\n")
    print(name, len(part))
print(json.dumps(splits["train"][0], ensure_ascii=False, indent=1))

Ignore the EXTRA block for now. It comes in after the first evaluation. Running python make_data.py gives 300 training rows, 40 validation rows and 60 test rows:

train 300
valid 40
test 60

One line of train.jsonl:

{"messages": [{"role": "system", "content": "Extract the expense as JSON with keys amount (number), currency (INR or USD), category (food, travel, bills, shopping, health). Reply with JSON only."}, {"role": "user", "content": "Paid $36 for a Swiggy order"}, {"role": "assistant", "content": "{\"amount\": 36, \"currency\": \"USD\", \"category\": \"food\"}"}]}

Synthetic data is fine for learning the workflow. For a real product, take a few hundred real messages, label them by hand, and split by time or by user so the test set looks like next month's traffic.

Score the base model first#

Without a baseline you cannot tell whether training helped. The evaluation script runs greedy decoding on the 60 test rows and reports three things. Strict JSON counts replies that json.loads accepts exactly as returned. JSON after fence strip counts replies that parse once a Markdown fence is removed, which separates format failures from content failures. The last is whether all three fields are correct, plus a breakdown of category mistakes.

"""Score a model on data/test.jsonl: valid JSON rate and exact field match."""
import json, re, sys, time
from collections import Counter
from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
adapter = sys.argv[1] if len(sys.argv) > 1 else None
model, tok = load(MODEL, adapter_path=adapter)
greedy = make_sampler(temp=0.0)

rows = [json.loads(l) for l in open("data/test.jsonl")]
def lenient(text):
    """Strip a ```json fence if present, so we score content, not wrapping."""
    m = re.search(r"\{.*\}", text, re.S)
    return m.group(0) if m else text

valid = strict = exact = 0
per_field = {"amount": 0, "currency": 0, "category": 0}
bad = []
wrong_cat = Counter()
t0 = time.time()
for r in rows:
    prompt = tok.apply_chat_template(r["messages"][:2], add_generation_prompt=True,
                                     tokenize=False)
    out = generate(model, tok, prompt=prompt, max_tokens=60, sampler=greedy).strip()
    gold = json.loads(r["messages"][2]["content"])
    try:
        json.loads(out); strict += 1       # parses exactly as returned
    except json.JSONDecodeError:
        pass
    try:
        pred = json.loads(lenient(out))
        valid += 1
    except json.JSONDecodeError:
        bad.append((r["messages"][1]["content"], out)); continue
    hits = {k: pred.get(k) == v for k, v in gold.items()} if isinstance(pred, dict) else {}
    if isinstance(pred, dict) and pred.get("category") != gold["category"]:
        wrong_cat[f'{gold["category"]}->{pred.get("category")}'] += 1
    for k, ok in hits.items():
        per_field[k] += ok
    if hits and all(hits.values()):
        exact += 1
    elif len(bad) < 3:
        bad.append((r["messages"][1]["content"], out))
n = len(rows)
print(f"adapter={adapter}  n={n}  {time.time()-t0:.1f}s")
print(f"strict JSON {strict}/{n}  JSON after fence strip {valid}/{n}  all 3 fields right {exact}/{n}")
print("per field:", {k: f"{v}/{n}" for k, v in per_field.items()})
print("category errors:", dict(wrong_cat.most_common()))
for text, out in bad[:3]:
    print("  IN :", text, "\n  OUT:", out.replace("\n", " ")[:120])

Base model, no adapter (python evaluate.py):

adapter=None  n=60  18.7s
strict JSON 0/60  JSON after fence strip 60/60  all 3 fields right 21/60
per field: {'amount': '60/60', 'currency': '49/60', 'category': '27/60'}
category errors: {'travel->shopping': 10, 'health->food': 9, 'bills->gas': 6, 'bills->food': 3, 'travel->food': 2, 'health->shopping': 2, 'health->bills': 1}
  IN : spent 8730 rupees on an auto to Koramangala today
  OUT: ```json {   "amount": 8730,   "currency": "INR",   "category": "shopping" } ```

The base model always pulls out the amount. Every reply comes wrapped in a code fence, though, even though the system prompt says "JSON only". It also labels "$289" as INR, and on six rows it invents a category ("gas") that is not in the allowed list. For a parser that calls json.loads, the score is 0 out of 60.

Train the adapter#

One command. The flags that matter: --mask-prompt (loss on the answer only), --num-layers 16 (LoRA on the top 16 of Qwen2.5-0.5B's 24 blocks, the default), batch size 4, 300 iterations (four passes over 300 rows), learning rate 1e-4. /usr/bin/time -l reports wall time and peak memory on macOS.

/usr/bin/time -l .venv/bin/mlx_lm.lora --model Qwen/Qwen2.5-0.5B-Instruct \
  --train --data data --mask-prompt --num-layers 16 --batch-size 4 \
  --iters 300 --learning-rate 1e-4 --steps-per-report 50 --steps-per-eval 100 \
  --adapter-path adapters --seed 0

Trimmed output from this run:

Trainable parameters: 0.594% (2.933M/494.033M)
  iter   train_loss     tok/s     tokens
     1    val 1.782    1.90s
    50    0.195 ▼        434      4.9k
   100    val 0.003    0.66s
   100    0.027 ▼        762      9.9k
   200    val 0.000    0.67s
   300    val 0.000    0.71s
   300    0.000 ▼        738     29.7k
       49.75 real         5.69 user        15.79 sys
          2469513328  peak memory footprint

The run took 50 seconds of wall time, model loading included, and peaked at 2.47 GB of memory, about a tenth of the machine's 24 GB. The adapter file is 12 MB.

Validation loss hit 0.000 by step 200. The validation set comes from the same items and templates as training, and the model has memorised them. Validation loss here can catch a broken run, but it cannot tell you about quality. Only the held-out test can.

Score it again#

adapter=adapters  n=60  16.1s
strict JSON 60/60  JSON after fence strip 60/60  all 3 fields right 36/60
per field: {'amount': '60/60', 'currency': '60/60', 'category': '36/60'}
category errors: {'shopping->books': 12, 'bills->travel': 11, 'bills->food': 1}
  IN : books from Blossom cost me $289
  OUT: {"amount": 289, "currency": "USD", "category": "books"}

Format went from 0 to 60 out of 60 strict JSON, and currency went from 49 to 60. Those are the things a few hundred examples teach well. Category on unseen items barely moved: 36 out of 60. The adapter still outputs "books", a label that never appears in the training data. The step-100 checkpoint did worse (29 out of 60), so stopping earlier would not have helped.

The error breakdown shows where the gap is. Each category had only five distinct items in training, so the model learned those five items rather than the idea of "shopping". This is the most common way small fine-tunes fail: the training data is too narrow, and the model ends up matching surface patterns.

Round 2: change the data#

Round 2 adds four more items per category (the EXTRA block), keeps the same 60-row test file, and retrains with the same command:

python make_data.py --round2
cp data/test.jsonl data_round2/
mlx_lm.lora ... --data data_round2 --adapter-path adapters_round2   # same flags as before
     1    val 1.758    0.69s
   300    val 0.002    0.66s
   300    0.004 ▲      772     29.7k
       42.48 real         5.39 user        15.48 sys
          2765703232  peak memory footprint

adapter=adapters_round2  n=60  16.1s
strict JSON 60/60  JSON after fence strip 60/60  all 3 fields right 44/60
per field: {'amount': '58/60', 'currency': '60/60', 'category': '44/60'}
category errors: {'bills->travel': 11, 'travel->bills': 5}
  IN : the gas cylinder cost me Rs 5633
  OUT: {"amount": 5633, "currency": "INR", "category": "travel"}

The before/after on the same held-out set, all from this run:

Strict JSON Currency Category All fields
Base Qwen2.5-0.5B-Instruct 0/60 49/60 27/60 21/60
LoRA, round 1 data 60/60 60/60 36/60 36/60
LoRA, round 2 data 60/60 60/60 44/60 44/60

The invented labels are gone. What remains is a real ambiguity: "the gas cylinder" goes to travel, most likely by association with petrol, and "an auto to Koramangala" sometimes goes to bills. Amount slipped from 60 to 58, which you would catch in the next round. Sixty test rows is small. A 44 versus 36 gap is eight rows, so treat it as a direction, not a precise effect size. The next steps would be more varied training items, a few examples on the travel and bills boundary, and a larger test set. The learning rate can stay as it is.

Fuse and run the merged model#

mlx_lm.fuse folds the adapter into the base weights and writes a normal Hugging Face folder:

mlx_lm.fuse --model Qwen/Qwen2.5-0.5B-Instruct \
  --adapter-path adapters_round2 --save-path fused
mlx_lm.generate --model fused --temp 0 \
  --system-prompt "Extract the expense as JSON with keys amount (number), currency (INR or USD), category (food, travel, bills, shopping, health). Reply with JSON only." \
  --prompt "spent 1200 rupees on vitamin tablets today"
{"amount": 1200, "currency": "INR", "category": "health"}
==========
Prompt: 60 tokens, 1682.943 tokens-per-sec
Generation: 23 tokens, 117.809 tokens-per-sec
Peak memory: 1.034 GB

The fused folder is 953 MB in bf16. fuse fetches the base model's README from the Hub for the model card, so it fails with HF_HUB_OFFLINE=1 set, even when the weights are cached.

Export to GGUF and Ollama: three traps from this run#

Every problem in this run showed up at the export step.

  1. mlx_lm.fuse --export-gguf rejects Qwen. The MLX-LM LoRA docs say GGUF export is limited to Mistral, Mixtral and Llama-style models in fp16. On the Qwen run it stopped with ValueError: Model type qwen2 not supported for GGUF conversion.

  2. Ollama 0.40.16 would not import the fused safetensors folder. The Ollama import guide shows FROM /path/to/safetensors/directory. This version answered Error: unsupported MLX architecture: model "Qwen2ForCausalLM". Its --quantize flag also accepted only int4, int8, nvfp4, mxfp4, mxfp8, not the familiar q8_0.

  3. A Llama GGUF exported by MLX did not load in llama.cpp. To test the Llama path, the same round-2 data was trained on Llama-3.2-1B-Instruct (300 iterations, 122 seconds, 4.99 GB peak, 5.6M trainable parameters). --export-gguf wrote a 2.47 GB f16 file, and Homebrew's llama-quantize (llama.cpp build b11429) turned it into an 803 MB Q4_K_M file in 7.4 seconds. Both Ollama and llama-completion then refused to load it: invalid gguf type for tokenizer.ggml.token_type. Ollama also did not pick up a chat template from that file. ollama show --template printed only {{ .Prompt }}, so you would need to supply a TEMPLATE in the Modelfile.

The route that matches current tooling is llama.cpp's own converter, convert_hf_to_gguf.py. Run it on the fused folder, then llama-quantize, then FROM ./model.gguf in an Ollama Modelfile. That step was not run on this machine, so there is no output to show. Check it against your own versions before relying on it. If you only need the model on a Mac, mlx_lm.server serves the fused folder over an OpenAI-compatible API and skips GGUF entirely.

Whatever format you ship, run evaluate.py against the exported model too. Quantizing to 4 bits can shift a small model's answers, and the 60-row test file takes about 16 seconds to run.

A checklist for your own run#

  • Write the evaluation before training, and score the base model with it.
  • Hold out whole items, users or time periods for the test set. Holding out random rows is not enough.
  • Score format and content separately. Most early gains come from format.
  • Treat validation loss near zero on synthetic data as a sign of memorisation.
  • When a field stays wrong, read the error breakdown and change the data before touching the learning rate or rank.
  • Export, then evaluate the exported file. Do not assume conversion preserved behaviour.

The adapter trains in under a minute and uses less than 3 GB of memory, so each data fix costs about a minute to test.

Sources#