LLM terms explained simply: a working glossary for engineers
Tokens, context windows, embeddings, temperature, KV cache, quantization, RAG, LoRA, tool calling, MoE and more, each with a number and a runnable demo.
Most LLM jargon names a simple idea. A token is a chunk of text. A context window is a size limit. Temperature is one division before a softmax. The words feel hard because they show up all at once, in docs and pricing pages and interview questions, usually without a number attached.
This glossary gives each term a one-line definition, a concrete number or example, and the place it bites in a real system. Where the blog has a longer post on the term, the entry links to it. The terms are grouped by where they sit in the stack: text going in, the model itself, choosing the next token, memory and speed, changing what the model knows, and building apps on top. A short Python script near the end computes several of the numbers so you can change them and rerun.
your text ──► tokenizer ──► tokens ──► embeddings ──► transformer layers ──► logits
(attention, KV cache) │
▼
app layer: RAG, tools, agents, caching ◄── text ◄── detokenize ◄── sampling (temperature, top-p)
Text going in#
Token#
Definition: the unit of text a model reads and writes, usually a word, part of a word or a punctuation mark.
"unbelievable" might be one token or three (un, believ, able), depending on the tokenizer. Every limit and every price is counted in tokens, not words or characters. English prose averages roughly three quarters of a word per token, while code, JSON and non-Latin scripts such as Hindi cost more tokens for the same meaning. The tokens post builds a BPE tokenizer by hand and shows real counts for each of those cases.
Tokenizer#
Definition: the fixed program that maps text to token IDs and back.
Each model family ships its own. The same sentence can be 12 tokens for one model and 15 for another, so a token count from OpenAI's tiktoken is an estimate for a Claude or Llama model, not an exact figure. For billing, use the provider's own token counting endpoint.
Context window#
Definition: the maximum number of tokens the model can see in one request, counting your input and the reply it writes.
Current Claude models list a 1M-token context window and up to 128K output tokens on Anthropic's models overview; older ones such as Claude Opus 4.1 had 200K. What happens at the limit depends on the API. On Claude, an input that alone is too long gets a 400 "prompt is too long" error, and on newer models a reply that runs into the limit stops with stop_reason: "model_context_window_exceeded" (context windows docs). Nothing gets forgotten quietly inside the model. If old turns disappear in a chat app, the app trimmed them. The chatbot memory post shows how apps resend, trim and summarize history.
A bigger window is not free accuracy. The same Anthropic page warns that accuracy and recall can drop as token count grows, so a 300-page dump in the prompt often loses to retrieving the right five pages.
Embedding#
Definition: a list of numbers (a vector) that represents a piece of text so that similar meanings land close together.
The small all-MiniLM-L6-v2 model maps a sentence to 384 numbers. Inside an LLM, every token also gets an embedding as the first step, but when engineers say "embeddings" they usually mean sentence vectors used for search, clustering and RAG.
Cosine similarity#
Definition: a score from -1 to 1 for how closely two vectors point in the same direction.
In the demo below, toy vectors for "dog" and "puppy" score 0.995 and "dog" and "car" score 0.123. Real embedding models rarely give such a clean split, and the useful threshold differs by model, so pick one by testing on your own labelled pairs rather than copying a number.
Inside the model#
Transformer#
Definition: the neural network design behind almost every current LLM: a stack of identical layers, each mixing information across tokens (attention) and then transforming each token on its own (a feed-forward block).
It comes from the 2017 paper Attention Is All You Need. Llama 3 8B, for example, stacks 32 such layers (Llama 3 paper, Table 3).
Attention#
Definition: the step where each token looks at earlier tokens and decides how much of each to pull into its own representation.
In "The trophy did not fit in the suitcase because it was too big", attention is how the model links "it" to "trophy". Each token produces a query, a key and a value vector; the query is scored against every earlier key, and the scores weight the values. That all-pairs comparison is why cost grows with context length.
Parameters#
Definition: the learned numbers (weights) in the network. "8B" means about eight billion of them.
Parameter count sets the memory needed just to load the model. At 2 bytes each in fp16 or bf16, 8B parameters take 16 GB before any request is served.
Logits#
Definition: the raw score the model gives every token in its vocabulary for "what comes next", before they become probabilities.
After "The capital of France is", a model might score Paris at 6.0, Lyon at 4.0, London at 3.0 and pizza at 0.5. A softmax turns those scores into probabilities that add up to 1. With those four toy logits, Paris gets 0.841.
Mixture of experts (MoE)#
Definition: a model whose feed-forward block is split into several "expert" networks, with a small router sending each token to only a few of them.
You get a large total parameter count while paying compute for a fraction of it. Mixtral 8x7B has 8 experts per layer and uses 2 per token, so each token "has access to 47B parameters" but uses 13B during inference (Mixtral paper). The catch is memory: all 47B still have to sit in GPU memory.
Choosing the next token#
Temperature#
Definition: a number that divides the logits before the softmax. Below 1 sharpens the distribution toward the top token; above 1 flattens it.
With the toy logits above, Paris gets probability 1.000 at T=0.2, 0.841 at T=1.0 and 0.604 at T=2.0, while pizza climbs from 0.000 to 0.039. A 4 percent chance of nonsense per token adds up fast over a 500-token answer. Low temperature suits extraction, code and JSON; higher values suit brainstorming. Temperature 0 makes output much more repeatable but not guaranteed identical, because batching and floating-point order on the server can still change results.
Some newer models restrict or ignore sampling parameters when thinking is on, so check the model's docs before tuning them.
Top-p (nucleus sampling)#
Definition: sample only from the smallest set of top tokens whose probabilities add up to p.
At T=1.0 in the demo, top_p=0.5 leaves only Paris, 0.9 keeps Paris and Lyon, and 0.99 adds London. pizza never makes the cut. The method comes from Holtzman et al., who showed it avoids the dull repetition of always taking the top token and the junk of sampling from the full tail. A common rule is to tune temperature or top-p, not both at once.
Greedy decoding#
Definition: always pick the single highest-probability token. It is what very low temperature approaches.
Memory and speed at inference#
Prefill and decode#
Definition: the two phases of a request. Prefill reads the whole prompt in one parallel pass; decode then writes the reply one token at a time.
Prefill decides time to first token (TTFT). Decode decides tokens per second. A long prompt with a short answer is prefill-bound; a short prompt that asks for a long essay is decode-bound.
KV cache#
Definition: the stored key and value vectors for every token already processed, so the model does not recompute them for each new token.
Its size per token is 2 x layers x KV heads x head dimension x bytes per value. For the Llama 3 8B shape (32 layers, 8 KV heads, head dimension 128) in fp16, that is 128 KiB per token, 1 GiB at 8K tokens and 16 GiB at 128K, for a single sequence. The KV cache, not the weights, is often what limits how many users one GPU can serve at long context.
Grouped-query attention (GQA)#
Definition: letting several query heads share one key/value head, which shrinks the KV cache.
Llama 3 8B has 32 query heads but only 8 KV heads. The demo shows the same model shape with 32 KV heads would need 512 KiB per token, four times as much. GQA is described in Ainslie et al..
Quantization#
Definition: storing weights (and sometimes activations or the KV cache) in fewer bits, such as 8 or 4 instead of 16.
For 8B parameters, the weights take 16 GB at 16 bits, 8 GB at 8 bits and 4 GB at 4 bits, plus some overhead for scales. That is the difference between needing a data-centre GPU and fitting on a laptop. Quality usually holds well at 8 bits and drops more at 4 bits, depending on the method and the task, so run your own eval on the quantized model before shipping it.
Speculative decoding#
Definition: a small draft model guesses several next tokens, and the big model checks them all in one pass, keeping the ones it agrees with.
The output distribution stays the same as the big model's; you get speed when the draft guesses well. Leviathan et al. reported 2x to 3x speedups on T5-XXL.
Prompt caching#
Definition: the provider stores the processed prefix of a prompt so a later request that starts with the same prefix is cheaper and faster.
On Claude, writing to the 5-minute cache costs 1.25x the base input price and a cache hit costs 0.1x on most models (0.05x on the newest ones), per the prompt caching docs. It only works when the prefix is byte-identical, so put stable content (system prompt, tool definitions, documents) first and the changing user turn last. The Claude token-saving post works through the cost math.
Changing what the model knows#
Pre-training#
Definition: the first, most expensive training stage, where the model learns to predict the next token over a huge text corpus.
Fine-tuning#
Definition: further training on a smaller, targeted dataset to change behaviour, format or domain style.
Fine-tuning is good at teaching a format or tone and poor at adding facts that change weekly. The fine-tuning vs RAG vs prompting post has the decision table.
LoRA and QLoRA#
Definition: LoRA freezes the original weights and trains small low-rank matrices added beside them; QLoRA does the same on a 4-bit quantized base model.
The LoRA paper reports cutting trainable parameters for GPT-3 175B by 10,000x. QLoRA fine-tuned a 65B model on a single 48 GB GPU. The LoRA and QLoRA guide works out parameters, memory and cost for a 7B model.
Distillation#
Definition: training a small "student" model to copy the outputs of a large "teacher" model.
The idea of training on the teacher's full probability distribution, not only the right answer, comes from Hinton et al.. Many small open models today are trained partly on outputs from larger ones.
Knowledge cutoff#
Definition: the date after which the training data stops. The model knows nothing later unless you put it in the prompt.
Building on top#
Hallucination#
Definition: output that reads as fluent and confident but is false or unsupported, such as an invented API parameter or a citation to a paper that does not exist.
It follows from how the model works: it produces likely text, and likely is not the same as true. You cannot prompt it away fully. You measure it and reduce it. The hallucination detection post shows claim-level checks and how to report a rate.
Grounding#
Definition: tying the answer to sources you supplied, and ideally making the model cite them, so each claim can be checked.
A grounded answer that says "per section 4.2 of the uploaded policy" can be verified in seconds. An ungrounded one has to be trusted.
RAG (retrieval-augmented generation)#
Definition: retrieve relevant passages from your own data at query time, put them in the prompt, then generate the answer from them.
The term comes from Lewis et al.. Most RAG failures happen at retrieval, not generation: the right chunk never reached the prompt. The RAG cheat sheet maps symptoms to fixes, and RAG architectures compared covers the variants.
System prompt#
Definition: instructions placed before the conversation that set role, rules and output format for every turn.
Tool calling (function calling)#
Definition: the model replies with a structured request to run a function you described, such as get_weather(city="Bengaluru"), your code runs it, and you send the result back.
The model never executes anything itself. Your code decides whether to run the call. Anthropic's tool use overview has the request shape, and the function calling tutorial walks through a tested loop.
MCP (Model Context Protocol)#
Definition: an open protocol for exposing tools and data to LLM apps through a standard server, so one tool server works with many clients (modelcontextprotocol.io). The MCP from zero post builds a server and client.
Agent#
Definition: an LLM in a loop that picks a tool, sees the result and decides the next step until the task is done or a limit is hit.
Anthropic's Building effective agents separates workflows (code decides the steps) from agents (the model decides). Start with a workflow when the steps are known; an agent costs more tokens and fails in harder-to-predict ways. See AI agent architectures explained for the common patterns.
Reasoning or thinking tokens#
Definition: tokens the model writes to work through a problem before its final answer.
They take up room in the context window and are billed as output tokens. Claude's extended thinking docs point to usage.output_tokens_details.thinking_tokens for how many billed output tokens were reasoning. A hard question that triggers 5,000 thinking tokens before a 200-token answer is billed for 5,200 output tokens, so track this field per request.
Run the numbers yourself#
The script below computes the temperature, top-p, cosine, KV cache and quantization figures quoted above. It uses only the standard library and no model or API key; the logits and vectors are made up, and the model shapes come from the Llama 3 paper. To use real logits, most local inference libraries return them per step, and the softmax function below applies to them unchanged.
"""Toy numbers for four LLM terms: softmax + temperature, top-p, cosine similarity, KV cache size.
Pure standard library, no model and no API key."""
import math
# 1. Logits -> probabilities, at three temperatures
logits = {"Paris": 6.0, "Lyon": 4.0, "London": 3.0, "pizza": 0.5}
def softmax(scores, temperature=1.0):
scaled = {k: v / temperature for k, v in scores.items()}
m = max(scaled.values()) # subtract max for numerical stability
exps = {k: math.exp(v - m) for k, v in scaled.items()}
total = sum(exps.values())
return {k: v / total for k, v in exps.items()}
print("Next token after 'The capital of France is'")
for t in (0.2, 1.0, 2.0):
probs = softmax(logits, t)
row = " ".join(f"{k}={p:.3f}" for k, p in probs.items())
print(f" T={t:<4} {row}")
# 2. Top-p (nucleus): keep the smallest set of tokens whose probability adds up to p
def top_p(probs, p):
kept, running = [], 0.0
for tok, pr in sorted(probs.items(), key=lambda x: -x[1]):
kept.append(tok)
running += pr
if running >= p:
break
return kept
probs = softmax(logits, 1.0)
for p in (0.5, 0.9, 0.99):
print(f" top_p={p}: sample only from {top_p(probs, p)}")
# 3. Cosine similarity on made-up 4-d "embeddings"
vecs = {
"dog": [0.9, 0.8, 0.1, 0.0],
"puppy": [0.85, 0.9, 0.15, 0.05],
"car": [0.1, 0.0, 0.9, 0.8],
}
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
print("\nCosine similarity")
print(f" dog vs puppy: {cosine(vecs['dog'], vecs['puppy']):.3f}")
print(f" dog vs car: {cosine(vecs['dog'], vecs['car']):.3f}")
# 4. KV cache size per token = 2 (K and V) * layers * kv_heads * head_dim * bytes
def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_val=2):
return 2 * layers * kv_heads * head_dim * bytes_per_val
gqa = kv_bytes_per_token(layers=32, kv_heads=8, head_dim=128) # Llama 3 8B shape, GQA
mha = kv_bytes_per_token(layers=32, kv_heads=32, head_dim=128) # same shape, one KV head per query head
print("\nKV cache in fp16, one sequence")
for name, b in (("Llama 3 8B (8 KV heads)", gqa), ("Same shape, 32 KV heads", mha)):
print(f" {name}: {b/1024:.0f} KiB per token, "
f"{b*8_192/1024**3:.1f} GiB at 8k tokens, {b*131_072/1024**3:.0f} GiB at 128k")
# 5. Weight memory for an 8B-parameter model at different precisions
params = 8.0e9
print("\nWeights only, 8B parameters")
for label, bits in (("fp16/bf16", 16), ("int8", 8), ("4-bit", 4)):
print(f" {label:<9} {params*bits/8/1e9:.1f} GB")
Output from python3 demo.py:
Next token after 'The capital of France is'
T=0.2 Paris=1.000 Lyon=0.000 London=0.000 pizza=0.000
T=1.0 Paris=0.841 Lyon=0.114 London=0.042 pizza=0.003
T=2.0 Paris=0.604 Lyon=0.222 London=0.135 pizza=0.039
top_p=0.5: sample only from ['Paris']
top_p=0.9: sample only from ['Paris', 'Lyon']
top_p=0.99: sample only from ['Paris', 'Lyon', 'London']
Cosine similarity
dog vs puppy: 0.995
dog vs car: 0.123
KV cache in fp16, one sequence
Llama 3 8B (8 KV heads): 128 KiB per token, 1.0 GiB at 8k tokens, 16 GiB at 128k
Same shape, 32 KV heads: 512 KiB per token, 4.0 GiB at 8k tokens, 64 GiB at 128k
Weights only, 8B parameters
fp16/bf16 16.0 GB
int8 8.0 GB
4-bit 4.0 GB
Two things to try. Change pizza to 5.5 and watch top-p 0.9 start admitting it. Then set kv_heads=8 and bytes_per_val=1 to see what an 8-bit KV cache buys on top of GQA.
Where these terms cause bugs#
Most of these words map to a specific failure you will meet in production:
| Term | Typical bug | First check |
|---|---|---|
| Token | Cost estimate off by 30 percent for Hindi or JSON input | Count with the provider's tokenizer, not word count |
| Context window | 400 error or a cut-off reply on long documents | Input tokens plus max_tokens against the model limit |
| Temperature | JSON that parses on Monday and breaks on Tuesday | Lower temperature, and validate output against a schema |
| KV cache | GPU out of memory at long context with few users | KV bytes per token times total tokens in flight |
| Quantization | 4-bit model fails on maths that fp16 passes | Run the same eval set on both |
| Prompt caching | Cache hit rate near zero | A timestamp or user ID placed before the stable prefix |
| RAG | Confident wrong answer with real-looking citations | Was the right chunk in the retrieved set at all? |
| Tool calling | Model calls a tool with an invented argument | Strict schema and validation before running |
| Thinking tokens | Bill jumps after enabling thinking | Log thinking token counts per request |
To check you have the terms down, pick any row and explain the bug and the check without looking at the glossary. Then answer three numeric questions: how much memory 70B parameters need at 4 bits, what a KV cache costs per token for your own model's config, and what probability your top token gets at T=0.5. The script answers the last two once you put in your own numbers.
Sources#
- Vaswani et al., Attention Is All You Need (2017): the transformer and attention.
- Llama Team, Meta, The Llama 3 Herd of Models (2024): layer count, heads and KV heads for Llama 3 8B.
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023).
- Jiang et al., Mixtral of Experts (2024): 47B total, 13B active parameters.
- Holtzman et al., The Curious Case of Neural Text Degeneration (2019): nucleus (top-p) sampling.
- Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2022).
- Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021).
- Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023).
- Hinton et al., Distilling the Knowledge in a Neural Network (2015).
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020).
- Anthropic, Models overview, Context windows, Prompt caching, Extended thinking and Tool use overview.
- Anthropic, Building effective agents.
- Model Context Protocol and sentence-transformers/all-MiniLM-L6-v2 model card.
- OpenAI, tiktoken.