Articles / synthetic-datanlppython
Data augmentation with LLMs: generate, filter, and test whether it helped
Paraphrases, label-conditioned generation and hard negatives from a local LLM, four quality filters, and a real experiment on whether the synthetic text helped.
You have 40 labelled examples and a classifier to train. Labelling 400 more means somebody reading and tagging 400 rows by hand. An LLM will write 400 plausible examples while you get lunch. The open question is whether those 400 examples teach your classifier anything the 40 real ones did not, and the only way to answer it is to train with and without them and score both on real data the generator never saw.
This post builds that whole loop for a text classification task: four ways to generate synthetic examples, four filters meant to throw away the bad ones, the leakage check people skip, the licence question, and a small experiment with real numbers from a local qwen3:4b-instruct model through Ollama. Results in one line: LLM-written examples lifted accuracy from 54.3% to 62.8%, paraphrases added nothing, the quality filters made the result slightly worse, and 200 extra real labels beat everything at 73.3%.
Where augmentation fits#
Augmentation means adding training rows you made instead of collected. For text there are two broad families:
- Transform an existing row so its label stays the same: paraphrase it, swap synonyms, back-translate it through another language.
- Write a new row from a label: give the model the label, a few real examples and some steering, and ask for fresh examples.
LLMs are good at both, and both have the same weakness. The model writes what it thinks a "Sports headline" looks like, which is narrower, cleaner and more stereotyped than what a real newswire produces. That gap is the reason augmentation sometimes helps a lot and sometimes does nothing. A 2023 study of LLM-generated training data for classification found that the more subjective the task (sarcasm, humour, stance), the worse models trained on synthetic data did (Li et al., 2023). Topic classification, which is what the experiment below uses, sits at the easy, objective end.
This post covers small classifiers and fine-tuning data. If you are choosing between a Hugging Face classifier and prompting an LLM directly for the classification itself, that is a separate decision; this post assumes you have already decided to train something.
The pipeline#
real labelled seeds (small)
|
+--> paraphrase each seed ----------+
| |
+--> label-conditioned generation --+--> raw synthetic rows
| (few-shot + angle/persona) |
+--> hard negatives (optional) -----+
|
v
filters, in this order:
1. length / format
2. label check by a second blind pass
3. leakage check against the eval set
4. near-duplicate removal (embeddings)
|
v
seeds + kept rows --> train --> score on REAL held-out test
seeds only --> train --> score on the same test
The last two lines are the part that matters. If you cannot show the augmented model beating the seeds-only model on real data, you have not augmented anything; you have made your training set bigger.
Four ways to generate#
1. Paraphrase#
Paraphrasing keeps the facts and the label and changes the wording. It is the safest technique, since the label is very unlikely to flip, and the least informative one, because it adds no new content. A TF-IDF model learns new word variants from it; a model that already understands synonyms learns very little.
Batching several seeds into one call cuts the number of requests by a factor of five:
def paraphrase(gen, texts, n=2):
"""One call rewrites a batch of snippets n ways each."""
numbered = "\n".join(f"{i+1}. {t}" for i, t in enumerate(texts))
p = (f"Rewrite each news snippet below {n} different ways. Keep the facts and the topic, change the "
f"wording and sentence structure. Return JSON with key 'items': a flat list of "
f"{n * len(texts)} strings, the rewrites of snippet 1 first.\n\n{numbered}")
out = gen.chat(p, LIST_SCHEMA, 0.8)["items"]
return [out[i * n:(i + 1) * n] for i in range(len(texts))]
2. Label-conditioned generation#
Give the model a label, three real examples of it and a request for new ones. Three examples, drawn at random from the seeds on every call, are enough to set the style without inviting the model to copy one of them.
3. Seed and persona diversity#
Ask the same "write 8 Sports headlines" prompt 20 times and you get 160 headlines about the same three football clubs. The fix is to vary something on every call: a sub-topic ("tennis", "baseball"), a persona ("a local reporter in Pune", "a wire service editor"), a reading level, a length. The experiment uses a sub-topic list per class:
ANGLES = {
"World": ["elections", "armed conflict", "diplomacy", "natural disasters", "courts and law"],
"Sports": ["football (soccer)", "American football", "baseball", "tennis", "Olympics and athletics"],
"Business": ["stock markets", "mergers", "oil prices", "company earnings", "central banks"],
"Sci/Tech": ["software", "space", "internet companies", "computer hardware", "biology research"],
}
def generate(gen, label, examples, angle, n=8):
shots = "\n".join(f"- {e}" for e in examples)
p = (f"Write {n} new, realistic news-wire snippets (title plus one sentence, 20 to 45 words) for the "
f"category '{label}', focused on {angle}. Invent different people, places and numbers for each. "
f"Match the style of these real examples but do not copy them:\n{shots}\n"
f"Return JSON with key 'items'.")
return gen.chat(p, LIST_SCHEMA, 1.0)["items"][:n]
Your angle list is a hidden prior. If the real Sci/Tech class is 40% about one software company and your list never mentions software, the synthetic rows shift the class away from what the test set contains.
4. Hard negatives#
A hard negative is an example that looks like class A on the surface but belongs to class B: a business story about a football club's share price, a tech story about a space agency's budget vote. These teach the decision boundary, which is usually where a small classifier fails. The prompt names both classes and the confusion:
Write 8 news snippets that belong to Business but mention sports teams, players
or leagues. The main topic must be money: earnings, deals, sponsorships, share prices.
Hard negatives have the highest rate of wrong labels, because the model is being asked to write right at the boundary. Run them through the label check below every time. The experiment here does not include them, so there is no number for them in this post.
The model behind an interface#
All generation goes through one tiny interface. The experiment uses a local model through Ollama's chat API, with a JSON schema passed as format so the model has to return valid JSON (Ollama structured outputs). To use a hosted API instead, write another class with the same chat method; no other code changes.
class Generator:
"""Anything with .chat(prompt, schema, temperature) -> dict works here.
Swap OllamaGenerator for a hosted API client; nothing else changes."""
def chat(self, prompt, schema, temperature):
raise NotImplementedError
class OllamaGenerator(Generator):
def __init__(self, model=GEN_MODEL):
self.model = model
def chat(self, prompt, schema, temperature, retries=2):
for _ in range(retries + 1):
r = requests.post(f"{OLLAMA}/api/chat", json={
"model": self.model, "stream": False, "format": schema,
"options": {"temperature": temperature, "num_predict": 3000},
"messages": [{"role": "user", "content": prompt}]}, timeout=600)
r.raise_for_status()
try: # a schema does not stop truncation at num_predict
return json.loads(r.json()["message"]["content"])
except json.JSONDecodeError:
continue
raise RuntimeError("no valid JSON after retries")
The retry is there for a reason. The first run of this experiment crashed on a paraphrase batch: the model hit the num_predict token cap halfway through a string, and the JSON came back unterminated. A schema constrains the tokens the model may emit; it does not make the output finish. Set the cap well above the longest answer you expect and retry on a decode error.
Structured output removes one whole class of failure, the "parse the model's free text with a regex and hope" step. Generating tabular rows by asking a base model to continue a CSV is the worst version of that: a completion model has no idea where the table ends, invents new categories, and drifts into prose. For tables, define a JSON schema with an enum for each categorical column and numeric ranges where you know them, then check each row against the schema before it goes anywhere near training.
Four filters#
Generation is cheap, so be strict about what you keep. The filters run in order from cheapest to most expensive.
1. Length and format. Drop rows under 8 or over 80 words and empty strings. In a real pipeline also drop rows that still contain instruction text ("Here are 8 snippets"), leftover list numbering and rows in the wrong language; the experiment code below checks length only. This costs nothing.
2. Label check by a second blind pass. Send each synthetic row back to a model with the label list and no hint of the intended label, at temperature 0. Keep the row only if the answer matches. The same model checking itself is weaker than a different model, but it still catches rows that drifted off topic. Batch it, ten rows per call:
def verify(gen, texts):
"""Second pass at temperature 0: label a batch blind, without the intended label."""
numbered = "\n".join(f"{i+1}. {t}" for i, t in enumerate(texts))
p = (f"Classify each news snippet into exactly one category: {', '.join(LABELS)}.\n\n{numbered}\n\n"
f"Return JSON with key 'labels': a list of {len(texts)} categories in order.")
out = gen.chat(p, BATCH_LABEL_SCHEMA, 0.0)["labels"]
if len(out) != len(texts): # model miscounted: fall back to one call per item
out = [gen.chat(f"Classify into one of {', '.join(LABELS)}: {t}\nReturn JSON with key 'label'.",
LABEL_SCHEMA, 0.0)["label"] for t in texts]
return out
The check has a cost. It removes the rows the model finds ambiguous, which are often the most useful ones near the boundary. Look at a sample of what it drops before you trust it.
3. Leakage against the eval set. Embed every synthetic row and every test row, and drop any synthetic row whose cosine similarity to a test row is at or above 0.95. Leakage happens more often than people expect: if you generate from seeds that came from the same pool as your test set, or if the generator memorised a public benchmark like AG News during pre-training, it can reproduce test items nearly word for word. A leaked row inflates your test score and the inflation disappears in production. Run this check against every set you report numbers on.
4. Near-duplicates. Drop any row with cosine similarity of 0.92 or more to a seed or to a row already kept. This keeps the model from padding the set with ten copies of the same headline.
def quality_filter(rows, seed_texts, dup_thresh=0.92, leak_thresh=0.95):
"""rows: (text, label, method, verified_label). Returns kept rows and drop counts."""
drops = {"length": 0, "label_mismatch": 0, "near_dup": 0, "test_leak": 0}
stage = []
for t, y, m, v in rows:
n = len(t.split())
if n < 8 or n > 80:
drops["length"] += 1
elif v != y:
drops["label_mismatch"] += 1
else:
stage.append((t, y, m))
if not stage:
return [], drops
vecs = embed([t for t, _, _ in stage])
seed_vecs = embed(seed_texts)
kept, kept_vecs = [], []
for (t, y, m), v in zip(stage, vecs):
if float((test_vecs @ v).max()) >= leak_thresh:
drops["test_leak"] += 1
continue
pool = [seed_vecs] + ([np.array(kept_vecs)] if kept_vecs else [])
if max(float((p @ v).max()) for p in pool) >= dup_thresh:
drops["near_dup"] += 1
continue
kept.append((t, y, m)); kept_vecs.append(v)
return kept, drops
The embeddings come from nomic-embed-text through Ollama's /api/embed endpoint (Ollama embeddings), normalised so a dot product is the cosine similarity. The thresholds 0.92 and 0.95 are starting points, not tuned values; print the pairs just above and below each threshold for your data and move them.
The experiment#
The question: does LLM-generated text help a small classifier trained on very few real labels?
Data. AG News, four classes (World, Sports, Business, Sci/Tech). The seed set is 10 real training rows per class, 40 in total. The test set is 2,000 rows sampled from the official AG News test split, fixed across all runs and never shown to the generator.
Generator. qwen3:4b-instruct (Q4_K_M, 4.0B parameters) through Ollama 0.40.16 on an Apple M5 laptop with 24 GB of memory. Per seed set: 2 paraphrases of every seed (about 80 rows) and 8 new rows for each of 3 randomly chosen angles per class (96 rows), then a blind label check on every row. The three seed sets took 37 minutes of generation in total.
Classifier. TF-IDF over word unigrams and bigrams plus logistic regression from scikit-learn 1.9.1, trained in under a second on CPU. A weak model on purpose: it cannot generalise from synonyms the way a pre-trained encoder can, so any gain has to come from the data.
Runs. Three different random seed sets (0, 1, 2), each with its own synthetic rows. The table shows the mean accuracy over the three and the spread (population standard deviation, in percentage points). Macro-F1 tracked accuracy within one point in every row, so it is left out.
def score(texts, labels):
vec = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True, min_df=1)
X = vec.fit_transform(texts)
clf = LogisticRegression(max_iter=2000, C=10).fit(X, labels)
pred = clf.predict(vec.transform(test.text))
return accuracy_score(test.label, pred), f1_score(test.label, pred, average="macro")
Per-seed filter counts and the checker's agreement with the real seed labels, as printed:
seed 0: raw=166 kept=98 drops={'length': 0, 'label_mismatch': 37, 'near_dup': 31, 'test_leak': 0} checker-vs-real-seed-labels=0.60
seed 1: raw=160 kept=98 drops={'length': 0, 'label_mismatch': 17, 'near_dup': 45, 'test_leak': 0} checker-vs-real-seed-labels=0.72
seed 2: raw=171 kept=103 drops={'length': 0, 'label_mismatch': 25, 'near_dup': 43, 'test_leak': 0} checker-vs-real-seed-labels=0.72
| Training data | Rows | Accuracy on 2,000 real test rows |
|---|---|---|
| Real 40 only | 40 | 54.3% (spread 3.0) |
| Real 40 + paraphrases | ~112 | 54.6% (2.0) |
| Real 40 + generated rows | ~134 | 62.8% (1.7) |
| Real 40 + all synthetic, unfiltered | ~206 | 63.6% (0.7) |
| Real 40 + synthetic, label check only | ~179 | 62.1% (1.5) |
| Real 40 + synthetic, dedup and leak check only | ~158 | 62.7% (0.5) |
| Real 40 + synthetic, all four filters | ~140 | 60.8% (0.7) |
| Real 240 (200 more real labels, no synthetic) | 240 | 73.3% (0.9) |
What the numbers say:
Label-conditioned generation helped; paraphrasing did not. Generated rows added 8.5 points over the seeds alone, in all three seed sets. Paraphrases added 0.3 points, inside the noise. A paraphrase of a seed carries the seed's topic words, and the topic words are what TF-IDF keys on, so the classifier learns almost nothing new from it. The generated rows bring new entities, new sub-topics and new vocabulary for each class.
The filters cost accuracy here. Each filter on its own lost one to one and a half points against the unfiltered set, and all four together lost almost three. Two reasons show up in the logs. First, the label checker is the same 4B model, and on the 40 real seed rows it agreed with the human AG News labels only 60% to 72% of the time. AG News has many borderline rows (a Canadian government budget story is labelled World; the checker says Business), and the paraphrases of those rows got thrown out for disagreeing with a label that is right by the dataset's own rules. A checker that weak removes good rows along with bad ones. Second, the near-duplicate filter compares every row against the seeds, and a paraphrase is a near-duplicate of its seed by design, so every one of the 31 to 45 near-duplicate drops per seed set was a paraphrase.
No leakage at 0.95. None of the roughly 500 synthetic rows came within 0.95 cosine similarity of a test row. That is a measurement for this model and this dataset; a larger model that has memorised AG News could behave differently, so keep the check.
Real labels won by a wide margin. 200 more real rows (50 per class) reached 73.3%, almost ten points above the best synthetic run. With a 4B local model, synthetic data was a cheap way to recover part of the gap, not a replacement for labelling.
Two cautions before you generalise. This is topic classification, the objective end of the scale where synthetic data tends to work best. And the filter result is specific to a weak checker: with a stronger, different model as the checker, or with human review of a sample, the label check could pay off. The way to find out on your data is the same table: run it with the filter on and off.
One format failure got past the length filter. In two of the three seed sets, 10 paraphrases came back with the batch numbering still attached ("1. Topics A, B and C in Run-up to Debate..."). A regex for a leading ^\d+\.\s catches them; add it to the format check if you batch prompts the same way.
Failure modes#
| Symptom | Likely cause | What to check |
|---|---|---|
| Test score jumps far above the seeds-only baseline | Synthetic rows leaked test items | Leakage filter against the test set; search synthetic rows for exact test strings |
| No gain at all | Synthetic rows say what the seeds already say | Diversity: count distinct entities and bigrams per class |
| One class gets worse | Generator's idea of that class differs from the data | Read 20 synthetic rows of that class next to 20 real ones |
| Label check drops most rows | Prompt or class definitions are vague | Tighten the label descriptions; give the checker the same definitions |
| Gains vanish on a newer test set | Synthetic data matched the old distribution | Re-score on recent real data before each retrain |
| Model sounds like the generator | Synthetic share too high | Cap synthetic rows at a fixed ratio to real ones and test the ratio |
Licences and terms#
Check the terms of the model that writes your data before you train on it, especially if what you train is itself a language model.
qwen3:4b-instructon Ollama is the Qwen3-4B-Instruct-2507 model, released under Apache 2.0 according to its model card. Apache 2.0 puts no restriction on what you do with outputs.- OpenAI's terms of use list "Use Output to develop models that compete with OpenAI" among the things you may not do.
- Anthropic's commercial terms say customers may not use the services "to build a competing product or service, including to train competing AI models".
A small topic classifier is not a competing model by any normal reading, but if you are generating instruction data to fine-tune an LLM, read the clause with your legal team. The seed data matters too. Whatever you paste into the prompt carries its own terms, and they flow into the synthetic rows. The AG News dataset card lists its licence as unknown and describes the underlying corpus as provided for research and non-commercial use, so the experiment below is fine for a blog post and not something to ship in a product.
Checklist#
- Freeze a real, held-out test set before generating anything, and never show it to the generator.
- Train the seeds-only baseline first and write its score down.
- Generate with varied angles or personas, and use structured output so parsing never fails.
- Filter: length, blind label check, leakage against the test set, near-duplicates.
- Train seeds plus synthetic rows, score on the same test set, run several random seeds.
- Compare against the cost of labelling more real rows, not only against the baseline.
- Keep the synthetic rows, the prompts and the model version in version control next to the training data.
If the next step is fine-tuning an LLM on synthetic instruction data, the LLM fine-tuning cheat sheet covers data formats and LoRA settings. If you are generating synthetic questions to evaluate a RAG system, the same leakage and real-data rules apply, and Evaluate a RAG app in Python shows how to score the retrieval and generation halves separately.
Sources#
- Li, Zhu, Lu, Yin. Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations, 2023.
- Dai et al. AugGPT: Leveraging ChatGPT for Text Data Augmentation, 2023.
- Wang et al. Self-Instruct: Aligning Language Models with Self-Generated Instructions, 2022.
- Zhang, Zhao, LeCun. Character-level Convolutional Networks for Text Classification, 2015 (introduces the AG News benchmark split).
- AG News dataset card, Hugging Face.
- Qwen3-4B-Instruct-2507 model card and Ollama qwen3 library page.
- Ollama structured outputs and Ollama embeddings.
- scikit-learn TfidfVectorizer.
- OpenAI terms of use and Anthropic commercial terms.