GyaanStack. Blog

Articles / nlpfine-tuningllmpython

Text classification: a fine-tuned Hugging Face encoder vs a local LLM, measured

DistilBERT fine-tuned in minutes vs zero-shot and few-shot Qwen3 on the same 300 news stories. Real accuracy, macro F1, latency and cost per million items.

By Raghunath · · 20 min read

You have a few thousand labelled texts and a fixed set of categories. Should you fine-tune a small encoder like DistilBERT, or send each text to an LLM with the label list in the prompt? This post runs both on the same 300 held-out news stories on one laptop and reports what came out: accuracy, macro F1, milliseconds per item and a cost per million items.

The short version: DistilBERT fine-tuned on 1,800 rows (two minutes on the laptop GPU, nine on its CPU) and zero-shot Qwen3 4B tied on accuracy (87.3% and 86.0%, inside the noise of a 300-item test), and the encoder was about 50 times faster per item. The interesting differences are in setup cost, label changes and the ways each one fails.

The contenders, cheapest first:

  1. TF-IDF plus logistic regression, as the floor any model has to clear.
  2. A Hugging Face zero-shot-classification pipeline with an NLI model (no training).
  3. DistilBERT fine-tuned on 1,800 labelled stories.
  4. Qwen3 1.7B and Qwen3 4B Instruct (the 2507 release) through Ollama, both at Ollama's default Q4_K_M quantization, zero-shot and few-shot.

Everything ran on a MacBook with an Apple M5 and 24 GB of memory, with transformers 5.24.0, datasets 5.2.0, torch 2.15.0, scikit-learn 1.9.1 and Ollama 0.40.16. Other jobs were running on the machine at the same time (load average between 4 and 29), so treat the timings as rough and compare them with each other, not with a benchmark.

If you are still deciding between fine-tuning, retrieval and prompting in general, Fine-tuning, RAG or prompting? covers that choice. This post is the narrow case where the output is one label from a fixed set.

The setup#

The data is AG News: 120,000 training and 7,600 test stories, each a headline plus a sentence or two, in four classes (World, Sports, Business, Sci/Tech). It comes from Zhang et al., 2015. Short texts and four balanced classes make it a fair fight for an LLM, and it trains fast on a CPU.

To keep each run to minutes, I take 2,000 shuffled training rows and 300 shuffled test rows. All models are scored on the same 300 test rows, which none of them saw in training or in prompts.

AG News train (120k) --shuffle(seed=42)--> 2,000 rows
     |                                       |-- 1,800 train  -> TF-IDF, DistilBERT
     |                                       |--   200 val    -> pick best epoch
     |                                       '-- first 2 per class -> few-shot examples
AG News test (7.6k) --shuffle(seed=42)-->   300 rows  -> every model scored here
# data.py
from datasets import load_dataset

LABELS = ["World", "Sports", "Business", "Sci/Tech"]

def load_splits(n_train=2000, n_test=300, seed=42):
    ds = load_dataset("fancyzhx/ag_news")
    train = ds["train"].shuffle(seed=seed).select(range(n_train))
    test = ds["test"].shuffle(seed=seed).select(range(n_test))
    return train, test

The 300 test rows split 69 World, 73 Sports, 84 Business, 74 Sci/Tech. I report accuracy and macro F1 (the unweighted mean of per-class F1, see the scikit-learn docs). Macro F1 punishes a model that dumps everything into one class, which turned out to matter.

# metrics.py
from sklearn.metrics import accuracy_score, f1_score

def report(name, y_true, y_pred, seconds, n):
    acc = accuracy_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred, average="macro")
    print(f"{name:<28} acc={acc:.3f} macroF1={f1:.3f} "
          f"latency={1000*seconds/n:.2f} ms/item")

With 300 test items, one item is a third of a percentage point, and the 95% interval on an accuracy near 87% is roughly plus or minus 4 points. Gaps smaller than that are noise.

The floor: TF-IDF and logistic regression#

Start here every time. If a linear model on word counts gets 95%, you do not need a transformer.

# baseline.py
import time
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from data import load_splits
from metrics import report

train, test = load_splits()
clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2),
                    LogisticRegression(max_iter=1000))
t0 = time.perf_counter(); clf.fit(train["text"], train["label"])
print(f"train time {time.perf_counter()-t0:.1f}s")
t0 = time.perf_counter(); pred = clf.predict(test["text"])
report("tfidf + logreg", test["label"], pred, time.perf_counter() - t0, len(test))
train time 1.2s
tfidf + logreg               acc=0.800 macroF1=0.799 latency=0.20 ms/item

80% after about a second of training, and 0.2 ms per prediction. On 1,800 examples the bag of words misses stories whose topic words it never saw, which is where pretrained models earn their place.

No training: the zero-shot pipeline#

The zero-shot-classification pipeline turns each label into a sentence ("This news story is about sports.") and asks a natural language inference model whether the story entails it. You need label names and nothing else. I used MoritzLaurer/deberta-v3-base-zeroshot-v2.0, a 184M-parameter MIT-licensed model built for this pipeline.

# zeroshot_pipeline.py
import time
from transformers import pipeline
from data import load_splits
from metrics import report

_, test = load_splits()
clf = pipeline("zero-shot-classification",
               model="MoritzLaurer/deberta-v3-base-zeroshot-v2.0", device="cpu")
names = ["world politics", "sports", "business", "science and technology"]

t0 = time.perf_counter()
out = clf(list(test["text"]), candidate_labels=names,
          hypothesis_template="This news story is about {}.", batch_size=16)
secs = time.perf_counter() - t0
pred = [names.index(o["labels"][0]) for o in out]
report("deberta-v3 zero-shot (NLI)", test["label"], pred, secs, len(test))
deberta-v3 zero-shot (NLI)   acc=0.867 macroF1=0.864 latency=1276.95 ms/item

86.7% with zero labelled examples, which beats the TF-IDF model trained on 1,800. The cost is speed: the model runs one forward pass per label, so four labels means four passes per story, and on a busy CPU that came to 1.3 seconds per item. Twenty labels would be twenty passes.

Two things tripped me up here. First, test["text"] in datasets 5.x is a lazy Column object, not a list. Passed straight to the pipeline it raised TypeError: string indices must be integers deep in the post-processing, so wrap it in list(). Second, the label wording matters: "world politics" and "science and technology" describe the classes; the raw names "World" and "Sci/Tech" give the NLI model less to work with. Tune the wording on validation data, never on the test set.

Fine-tuning DistilBERT on 1,800 examples#

DistilBERT has 67M parameters, about 40% fewer than BERT base, and its authors report it keeps 97% of BERT's language understanding performance while running 60% faster (Sanh et al., 2019). AutoModelForSequenceClassification puts a fresh classification head on top; the sequence classification guide walks through the same pattern.

A few changes from older tutorials you will find online, checked against transformers 5.24.0 and the Trainer docs:

  • The argument is eval_strategy, not evaluation_strategy.
  • Pass the tokenizer as processing_class; tokenizer= is gone from Trainer.
  • Tokenize without padding and let DataCollatorWithPadding pad each batch to its own longest row. Padding all 2,000 rows to 512 tokens, as many tutorials do, makes training several times slower on short news text for no gain.
  • Set report_to="none" unless you want it to look for Weights & Biases or TensorBoard.
  • Old examples that set eval_steps=500 while leaving the evaluation strategy at its default never evaluate at all. Pick one strategy and keep save and eval on the same one if you want load_best_model_at_end.
# finetune.py
import time, torch
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
                          DataCollatorWithPadding, Trainer, TrainingArguments)
from sklearn.metrics import accuracy_score, f1_score
from data import load_splits, LABELS
from metrics import report

MODEL = "distilbert-base-uncased"
train, test = load_splits()
split = train.train_test_split(test_size=0.1, seed=42)   # 1,800 train / 200 val
tok = AutoTokenizer.from_pretrained(MODEL)

def encode(batch):
    return tok(batch["text"], truncation=True, max_length=128)

tr = split["train"].map(encode, batched=True)
va = split["test"].map(encode, batched=True)

model = AutoModelForSequenceClassification.from_pretrained(
    MODEL, num_labels=len(LABELS),
    id2label=dict(enumerate(LABELS)), label2id={l: i for i, l in enumerate(LABELS)})

def compute_metrics(p):
    preds = p.predictions.argmax(-1)
    return {"accuracy": accuracy_score(p.label_ids, preds),
            "macro_f1": f1_score(p.label_ids, preds, average="macro")}

args = TrainingArguments(
    output_dir="out", num_train_epochs=2, learning_rate=5e-5,
    per_device_train_batch_size=16, per_device_eval_batch_size=64,
    weight_decay=0.01, warmup_steps=20, eval_strategy="epoch",
    save_strategy="epoch", load_best_model_at_end=True,
    metric_for_best_model="macro_f1", save_total_limit=1,
    logging_steps=25, report_to="none", use_cpu=True, seed=42)

trainer = Trainer(model=model, args=args, train_dataset=tr, eval_dataset=va,
                  processing_class=tok, data_collator=DataCollatorWithPadding(tok),
                  compute_metrics=compute_metrics)
t0 = time.perf_counter(); trainer.train()
print(f"train time {time.perf_counter()-t0:.0f}s")
trainer.save_model("distilbert-agnews")

# held-out test, one item at a time (what a request handler sees)
model.eval(); preds = []
t0 = time.perf_counter()
with torch.inference_mode():
    for text in test["text"]:
        x = tok(text, truncation=True, max_length=128, return_tensors="pt")
        preds.append(int(model(**x).logits.argmax(-1)))
report("distilbert fine-tuned", test["label"], preds, time.perf_counter()-t0, len(test))

Loading prints a table of MISSING weights (classifier.weight, pre_classifier.bias and so on). That is expected: the classification head is new and gets trained here. The training log, trimmed:

{'eval_loss': '0.2592', 'eval_accuracy': '0.92', 'eval_macro_f1': '0.9184', 'epoch': '1'}
{'eval_loss': '0.208', 'eval_accuracy': '0.94', 'eval_macro_f1': '0.9382', 'epoch': '2'}
train time 530s
distilbert fine-tuned        acc=0.873 macroF1=0.874 latency=22.64 ms/item

On CPU, two epochs took 530 seconds on a loaded machine. The same script with use_cpu=False, which picks up the Apple GPU through PyTorch's MPS backend, trained in 124 seconds and reached 93.5% on validation. On a free Colab or Kaggle GPU it would be faster again.

Notice the gap: 94% on the 200 validation rows, 87.3% on the 300 test rows. Both sets are small, so part of it is noise, but it is also why you hold out a test set you never tune against. If I had reported the validation number, this model would look 7 points better than it is.

Inference is 23 ms per story on CPU, one at a time with no batching. That is about 55 times faster than the NLI pipeline, because it is one forward pass of a smaller model instead of four passes of a bigger one.

LLM classification with Qwen3 through Ollama#

The LLM side uses Ollama's chat API with three settings that matter for classification:

  • format set to a JSON schema whose label field is an enum of the four classes. Ollama's structured outputs constrain decoding to that schema, so the reply always parses.
  • think: False. Qwen3 models reason before answering by default (Ollama thinking docs); for a four-way label that adds hundreds of tokens of latency for little gain.
  • temperature: 0, so reruns give the same labels.

The classifier sits behind a small class. The layout argument switches between two prompt layouts, explained in the next section; v2 is the one to use. Ollama runs locally, so there is no API key; to use a hosted model you replace the chat() method and keep everything else.

# llm.py   run as: python llm.py qwen3:4b-instruct zero v2
import json, sys, time, requests
from data import load_splits, LABELS
from metrics import report

SCHEMA = {"type": "object",
          "properties": {"label": {"type": "string", "enum": LABELS}},
          "required": ["label"]}
SYSTEM = ("Classify the news story into exactly one category: "
          "World, Sports, Business, Sci/Tech. Reply as JSON.")
# v2: story first, question last. Same labels, same schema.
QUESTION = ("Which category fits this story best: World, Sports, Business "
            "or Sci/Tech? Reply as JSON.")

def turn(text, layout):
    if layout == "v1":
        return text
    return f"News story: {text}\n\n{QUESTION}"

class OllamaClassifier:
    """Local model through Ollama. A hosted API would replace .chat() only."""
    def __init__(self, model, shots=(), layout="v2"):
        self.model, self.shots, self.layout = model, shots, layout
        self.tokens_in = self.tokens_out = 0

    def chat(self, messages):
        r = requests.post("http://localhost:11434/api/chat", json={
            "model": self.model, "messages": messages, "stream": False,
            "think": False, "format": SCHEMA,
            "options": {"temperature": 0, "num_ctx": 4096}}, timeout=120)
        r.raise_for_status(); d = r.json()
        self.tokens_in += d.get("prompt_eval_count", 0)
        self.tokens_out += d.get("eval_count", 0)
        return d["message"]["content"]

    def predict(self, text):
        msgs = [{"role": "system", "content": SYSTEM}] if self.layout == "v1" else []
        for t, y in self.shots:
            msgs += [{"role": "user", "content": turn(t, self.layout)},
                     {"role": "assistant", "content": json.dumps({"label": LABELS[y]})}]
        msgs.append({"role": "user", "content": turn(text, self.layout)})
        try:
            return LABELS.index(json.loads(self.chat(msgs))["label"])
        except (ValueError, KeyError, json.JSONDecodeError):
            return -1          # count as wrong, never silently guess

def pick_shots(train, per_class=2):
    shots, seen = [], {i: 0 for i in range(len(LABELS))}
    for row in train:
        if seen[row["label"]] < per_class:
            shots.append((row["text"], row["label"])); seen[row["label"]] += 1
    return shots

if __name__ == "__main__":
    model, mode, layout = sys.argv[1], sys.argv[2], sys.argv[3]
    train, test = load_splits()
    clf = OllamaClassifier(model, pick_shots(train) if mode == "few" else (), layout)
    clf.predict(test[0]["text"])                  # warm-up: load weights
    clf.tokens_in = clf.tokens_out = 0
    t0 = time.perf_counter()
    preds = [clf.predict(t) for t in test["text"]]
    secs = time.perf_counter() - t0
    report(f"{model} {mode} {layout}", test["label"], preds, secs, len(test))
    from collections import Counter
    print("  predicted label counts:", dict(Counter(preds)))
    print(f"  invalid={preds.count(-1)} tokens_in/item={clf.tokens_in/len(test):.0f} "
          f"tokens_out/item={clf.tokens_out/len(test):.1f}")

Few-shot uses two training stories per class (eight in total), taken from the training split, never from the test rows. Each run starts with one warm-up call so model loading is not counted in latency.

The prompt layout bug that cost 60 points#

My first version put the instruction in a system message ("Classify the news story into exactly one category: World, Sports, Business, Sci/Tech. Reply as JSON.") and sent the bare story as the user message. Qwen3 1.7B scored this:

qwen3:1.7b zero-shot         acc=0.260 macroF1=0.156 latency=1212.19 ms/item
  invalid=0 tokens_in/item=99 tokens_out/item=7.0

Every reply was valid JSON, so nothing looked broken. But 26% on four balanced classes is chance level. A spot check showed why: "Stocks fell sharply on Wall Street as oil prices hit a record high" came back World, and so did a story about a new iPod. Dropping the JSON schema did not change the answer. Moving the story first and the question last, in the same user turn, fixed it: those two headlines came back Business and Sci/Tech, and a tennis headline came back Sports.

Two lessons. Valid output tells you nothing about correct output, so always score against labels. And small models are sensitive to where the instruction sits; test the layout on validation data before blaming the model.

With the story-first layout, the full runs on the 300 test rows:

qwen3:1.7b zero v2           acc=0.820 macroF1=0.808 latency=1325.44 ms/item
  predicted label counts: {1: 76, 2: 117, 3: 38, 0: 69}
  invalid=0 tokens_in/item=94 tokens_out/item=10.1
qwen3:1.7b few v2            acc=0.790 macroF1=0.789 latency=298.49 ms/item
  predicted label counts: {2: 106, 1: 70, 3: 57, 0: 67}
  invalid=0 tokens_in/item=755 tokens_out/item=7.4
qwen3:4b-instruct zero v1    acc=0.823 macroF1=0.820 latency=1065.48 ms/item
  predicted label counts: {1: 72, 2: 96, 3: 56, 0: 76}
  invalid=0 tokens_in/item=91 tokens_out/item=7.4
qwen3:4b-instruct zero v2    acc=0.860 macroF1=0.858 latency=663.79 ms/item
  predicted label counts: {1: 75, 2: 102, 3: 72, 0: 51}
  invalid=0 tokens_in/item=86 tokens_out/item=10.5
qwen3:4b-instruct few v2     acc=0.850 macroF1=0.849 latency=566.39 ms/item
  predicted label counts: {1: 76, 2: 101, 3: 69, 0: 54}
  invalid=0 tokens_in/item=747 tokens_out/item=7.5
qwen3:1.7b zero v1           acc=0.260 macroF1=0.156 latency=273.01 ms/item
  predicted label counts: {0: 288, 1: 10, 3: 1, 2: 1}
  invalid=0 tokens_in/item=99 tokens_out/item=7.0

The last line is the old layout rerun with label counts printed: 288 of 300 stories came back World (label 0). The 4B Instruct model did not collapse under the old layout but still lost about 4 points to the new one.

The 1.7B model's main error is over-calling Business (117 predictions against 84 true) and under-calling Sci/Tech (38 against 74): stories about tech companies' earnings or deals get filed as Business. Few-shot examples did not help either model here. Eight examples cost 650 extra input tokens per call and moved accuracy by about one point down, which is inside the noise on 300 items.

The mean latencies above swing with machine load (the 1.7B old-layout run took 1,212 ms per item the first time and 273 ms the second). For a fairer comparison I ran every model back to back on the same 50 stories and took the median:

distilbert fine-tuned (CPU)       9.4 ms
qwen3:1.7b zero-shot            292.9 ms
qwen3:4b-instruct zero-shot     468.1 ms
qwen3:4b-instruct few-shot      525.9 ms

Ollama runs the Qwen models on the Apple GPU; DistilBERT ran on the CPU. Even so, the encoder is about 50 times faster than the 4B model per item.

All results side by side#

Approach Labelled rows used Accuracy Macro F1 Latency per item
TF-IDF + logistic regression 1,800 80.0% 0.799 0.2 ms
DeBERTa-v3 NLI zero-shot pipeline 0 86.7% 0.864 about 1,280 ms (CPU, 4 passes)
DistilBERT fine-tuned 1,800 87.3% 0.874 9.4 ms median
Qwen3 1.7B zero-shot 0 82.0% 0.808 293 ms median
Qwen3 1.7B few-shot 8 79.0% 0.789 not timed separately
Qwen3 4B Instruct zero-shot 0 86.0% 0.858 468 ms median
Qwen3 4B Instruct few-shot 8 85.0% 0.849 526 ms median

The top three (DistilBERT, the NLI pipeline and Qwen3 4B zero-shot) sit within 1.3 points of each other, well inside the 4-point uncertainty. On this test set they are tied on quality. They are not tied on speed, setup or cost.

Cost per million items#

Latency on a laptop is one number; what you pay at scale is another. Here is the method, with the inputs from this run.

For a hosted LLM, cost per item is input tokens times the input price plus output tokens times the output price. I did not run a hosted model, so only the token counts below come from this experiment. Claude Haiku 5.5, which Anthropic lists for classification and routing, costs $0.10 per million input tokens and $0.50 per million output tokens for prompts under 100,000 tokens, and half that through the Batch API.

zero-shot: 86 in, 10.5 out per item
  86 x $0.10/1M + 10.5 x $0.50/1M = $0.0000139  ->  about $14 per 1M items
  Batch API (half price)                         ->  about $7 per 1M items

few-shot:  747 in, 7.5 out per item
  747 x $0.10/1M + 7.5 x $0.50/1M = $0.0000785  ->  about $78 per 1M items
  ~660-token shared prefix read from cache at $0.01/1M
  660 x $0.01/1M + 87 x $0.10/1M + 7.5 x $0.50/1M
                                  = $0.0000191  ->  about $19 per 1M items

Three caveats. Token counts here come from Qwen's tokenizer, and Claude's tokenizer will count differently, so count with the provider's own token counter before budgeting. Caching only applies above a minimum prompt length, which is 512 tokens for Haiku 5.5 and 4,096 for Haiku 4.5 (prompt caching docs); the few-shot prefix here clears the first and not the second. And cache writes cost more than normal input, which the estimate above ignores.

For the fine-tuned encoder, the cost is machine time. At the 9.4 ms median, one process handles about 100 items a second, or a million items in under 3 hours, unbatched (the 22.6 ms mean from the loaded full run gives about 6 hours). Batching 32 or 64 items per forward pass raises throughput further. The saved model is 256 MB, so it runs on a small CPU-only machine, and the bill for a million items is a few hours of that machine.

For the local LLM the same arithmetic applies with worse numbers: at 468 ms per item, Qwen3 4B needs about 5.4 days of one laptop for a million items.

The one-off costs go the other way. The encoder needs labelled data (here 1,800 rows) and a training run of 2 to 9 minutes. The LLM needs a label list and a prompt.

Failure modes to test for#

  • Label collapse. A model that returns one class for everything can still produce valid output. Print the predicted label counts next to accuracy, and watch macro F1, which drops much faster than accuracy when one class swallows the rest.
  • Validation leakage. Choosing the prompt wording, the label names or the best epoch on the test set inflates the score. Keep the 300 test rows untouched until the end.
  • Long inputs. I truncate at 128 tokens for DistilBERT, which is fine for headlines. For emails or support tickets, check how many rows exceed the limit before choosing max_length; DistilBERT stops at 512 tokens, while ModernBERT accepts 8,192.
  • New labels. The fine-tuned head knows four classes. Adding a fifth means relabelling and retraining. The LLM and the NLI pipeline take a new label as a prompt change, which is their biggest practical advantage.
  • Drift. AG News stories were collected from 2004 onward, so they do not mention the products of 2026. Re-score a fresh labelled sample every few months whichever approach you choose.

Which one to use#

Fine-tune a small encoder (DistilBERT, or ModernBERT for long inputs) when:

  • The label set is fixed and you have, or can label, a few hundred to a few thousand examples per task. Here 1,800 rows were enough to tie a 4B LLM.
  • Volume is high or latency matters. 9 ms on a CPU against 300 to 500 ms on a GPU is the difference between classifying inside a request and queueing it.
  • Data cannot leave your machines and you do not want to run a GPU server for a 4B model.
  • You need a score per class. The softmax over the logits gives one you can threshold to route unsure items to a human (check its calibration on validation data first); an LLM's JSON label comes with no score unless you also request log probabilities.

Use an LLM (or the NLI zero-shot pipeline) when:

  • You have no labelled data yet. Zero-shot Qwen3 4B and the NLI pipeline both reached about 86% with nothing but label names.
  • Labels change often. A new category is a prompt edit, not a relabel-and-retrain cycle.
  • The decision needs reading comprehension beyond topic words, such as "is this ticket a refund request that mentions a legal threat", or an explanation alongside the label.
  • Volume is low. At a few thousand items a month, the hosted bill is pennies and nobody has to maintain a model.

A common path is to start with an LLM, review and correct its labels, and use those labels to train the encoder once volume justifies it. Keep the LLM for the cases the encoder scores with low confidence. Whichever you pick, keep a labelled test set the size of this one and rerun it whenever the prompt, model or data changes.

If you go the fine-tuning route with a generative model instead of an encoder, Fine-tune a small language model on a MacBook walks through a LoRA run end to end, and the LLM fine-tuning cheat sheet has the hyperparameters and failure modes in one place.

Sources#