GyaanStack. Blog

Articles / llmtokenizationgenai-basics

What are tokens in LLMs? BPE by hand, real counts and the cost math

How LLM tokenizers split text, a BPE tokenizer built by hand in Python, real tiktoken counts for code, JSON, Hindi and emoji, and what tokens do to cost.

By Raghunath · · 16 min read

A language model never sees your words. Before your prompt reaches the network, a tokenizer cuts it into pieces called tokens and swaps each piece for an integer ID. The model reads a list of integers, predicts the next integer, and the tokenizer turns that back into text. Everything you pay for, every context limit you hit, and a few odd failures (miscounted letters, shaky arithmetic) trace back to this step.

This post builds a byte pair encoding (BPE) tokenizer by hand on a tiny corpus, then measures the real tokenizers used by OpenAI models on English, code, JSON, numbers, Hindi and emoji. After that comes the cost and context math with a current price, and the quirks tokens cause. All code here ran with Python 3 and tiktoken 0.14.0; the outputs are pasted as printed.

From text to integers#

Here is the whole pipeline for one short prompt, using the o200k_base tokenizer that GPT-4o, GPT-4.1 and the GPT-5 family use:

"Hello world"
     |
     v   tokenizer splits into known pieces
["Hello", " world"]
     |
     v   each piece maps to a row in the vocabulary
[13225, 2375]
     |
     v   each ID picks one row of the embedding matrix
two vectors  ->  transformer  ->  scores for all ~200k IDs  ->  next ID

Two details in that picture explain a lot later. The space belongs to the token after it: " world" (with the space) is ID 2375, while "world" with no space is a different token, ID 24169. And the vocabulary is fixed. o200k_base has about 200,000 entries, the older cl100k_base (GPT-4, GPT-3.5 Turbo) about 100,000. Any text, in any language, has to be spelled out with those pieces.

Why not use whole words? A word vocabulary cannot handle a word it has never seen, such as a new product name, a typo or a variable called user_id_v2. Why not single characters? Then every sentence becomes very long, and attention cost grows with sequence length. Subword tokens sit between the two: common words get one token, rare words get built from a few parts, and nothing is ever unknown.

Build a BPE tokenizer by hand#

BPE was adapted for neural translation by Sennrich, Haddow and Birch in 2015, and GPT-2 moved it to raw bytes. The training loop is short:

  1. Split every word into characters, with an end-of-word marker.
  2. Count every adjacent pair of symbols across the corpus.
  3. Merge the most frequent pair into one new symbol and record the merge.
  4. Repeat until the vocabulary is the size you want.

Here it is on a 17-word corpus:

from collections import Counter

corpus = "low low low low low lower lower newest newest newest newest newest newest widest widest widest"

# Start: every word is a tuple of characters, with an end-of-word marker.
words = Counter(tuple(w) + ("</w>",) for w in corpus.split())

def pair_counts(words):
    pairs = Counter()
    for word, freq in words.items():
        for a, b in zip(word, word[1:]):
            pairs[(a, b)] += freq
    return pairs

def merge(words, pair):
    out = Counter()
    for word, freq in words.items():
        new, i = [], 0
        while i < len(word):
            if i < len(word) - 1 and (word[i], word[i + 1]) == pair:
                new.append(word[i] + word[i + 1])
                i += 2
            else:
                new.append(word[i])
                i += 1
        out[tuple(new)] += freq
    return out

merges = []
for step in range(8):
    pairs = pair_counts(words)
    best, count = pairs.most_common(1)[0]
    words = merge(words, best)
    merges.append(best)
    print(f"merge {step + 1}: {best[0]!r} + {best[1]!r}  (seen {count}x)")

print("\nfinal segmentation:")
for word, freq in words.items():
    print(f"  {' '.join(word):<22} x{freq}")

def encode(word, merges):
    parts = list(word) + ["</w>"]
    for pair in merges:          # apply merges in the order they were learned
        i, new = 0, []
        while i < len(parts):
            if i < len(parts) - 1 and (parts[i], parts[i + 1]) == pair:
                new.append(parts[i] + parts[i + 1]); i += 2
            else:
                new.append(parts[i]); i += 1
        parts = new
    return parts

print("\nunseen words:")
for w in ["lowest", "newer", "slow"]:
    print(f"  {w:<8} -> {encode(w, merges)}")

Output:

merge 1: 'e' + 's'  (seen 9x)
merge 2: 'es' + 't'  (seen 9x)
merge 3: 'est' + '</w>'  (seen 9x)
merge 4: 'l' + 'o'  (seen 7x)
merge 5: 'lo' + 'w'  (seen 7x)
merge 6: 'n' + 'e'  (seen 6x)
merge 7: 'ne' + 'w'  (seen 6x)
merge 8: 'new' + 'est</w>'  (seen 6x)

final segmentation:
  low </w>               x5
  low e r </w>           x2
  newest</w>             x6
  w i d est</w>          x3

unseen words:
  lowest   -> ['low', 'est</w>']
  newer    -> ['new', 'e', 'r', '</w>']
  slow     -> ['s', 'low', '</w>']

Read the merges as a record of what the corpus repeats. The suffix est at the end of a word showed up 9 times (newest 6, widest 3), so it got merged first. newest appeared often enough to become a single token, while widest still needs four pieces because w, i and d never formed a frequent pair. lowest was never in the training text, yet it encodes cleanly as low + est</w>. That is the property that matters: no word is unknown, frequent strings are cheap, and rare strings cost more tokens.

Production tokenizers make three changes to this toy:

  • Bytes instead of characters. GPT-2 and its successors start from the 256 possible UTF-8 bytes, so every string, including emoji and any script, can be encoded. You will see the cost of this in the emoji row below.
  • A pre-split step. A regular expression first cuts text into chunks (words with their leading space, runs of digits, punctuation), and merges happen only inside a chunk. In o200k_base digit runs are capped at three, which is why numbers split the way they do later in this post.
  • Scale. Training runs over a huge corpus until the vocabulary reaches 100k or 200k entries. Encoding uses a learned rank per merge, implemented in Rust inside tiktoken for speed.

If you want to write a full byte-level BPE from scratch, the Build a GPT yourself video path linked at the end walks through one.

Real counts with tiktoken#

tiktoken is OpenAI's open-source tokenizer library. This script counts tokens for seven kinds of text on both the older and newer encoding:

import json
import tiktoken

old = tiktoken.get_encoding("cl100k_base")   # GPT-4, GPT-3.5-turbo
new = tiktoken.get_encoding("o200k_base")    # GPT-4o, GPT-4.1, o-series

samples = {
    "English": "The quick brown fox jumps over the lazy dog near the river bank.",
    "Python": "def add(a, b):\n    return a + b\n",
    "JSON": json.dumps({"user_id": 48213, "name": "Asha", "active": True}),
    "Numbers": "Order 9876543210 shipped on 2026-11-22 for Rs 14,999.50",
    "Hindi": "मुझे कल सुबह बैंगलोर जाना है।",
    "Hinglish": "Mujhe kal subah Bangalore jaana hai.",
    "Emoji": "🙂🚀🇮🇳👍🏽",
}

print(f"{'sample':<9} {'chars':>5} {'cl100k':>6} {'o200k':>6} {'chars/token':>11}")
for name, text in samples.items():
    a, b = len(old.encode(text)), len(new.encode(text))
    print(f"{name:<9} {len(text):>5} {a:>6} {b:>6} {len(text)/b:>11.1f}")

Output:

sample    chars cl100k  o200k chars/token
English      64     14     14         4.6
Python       32     12     12         2.7
JSON         50     20     20         2.5
Numbers      55     23     23         2.4
Hindi        29     32     10         2.9
Hinglish     36     11     10         3.6
Emoji         6     17     10         0.6

("chars" here means Python string length, which counts Unicode code points. A flag emoji is two code points.)

What the table shows:

  • English prose is the cheap case. About 4.6 characters per token, which is where the common "1 token is about 4 characters" rule comes from.
  • Code, JSON and numbers cost close to twice as much per character. Quotes, braces, colons, _ characters and digit groups often become separate tokens. A JSON tool result is denser in tokens than the English sentence that describes it.
  • The tokenizer version changes the bill. The Hindi sentence took 32 tokens on cl100k_base and 10 on o200k_base, a 3.2x drop for the same text, because the larger vocabulary learned Devanagari pieces. English did not move. Anthropic's docs report a change in the other direction: Claude Opus 4.7 and later use a newer tokenizer that produces about 30 percent more tokens for the same input, so counts measured on an older model are not valid for a newer one.
  • Emoji are the worst case. Six code points became 10 tokens. The flag 🇮🇳 is 8 UTF-8 bytes and costs 4 tokens on its own.

Hindi and Hinglish came out close here (10 tokens each), but one sentence is not a benchmark, and other scripts and other tokenizers behave differently. If your product serves Indian languages, measure your own traffic. The multilingual side of retrieval deserves its own post, so this one stops there.

Two warnings about these numbers. First, they are for OpenAI tokenizers only. Claude, Gemini and Llama each have their own tokenizer, and the same string gives a different count on each. For Claude, use the free token counting endpoint (client.messages.count_tokens), which counts under the model you name. Second, a chat request carries more than your text: message roles, tool definitions and system formatting add tokens, so a raw encode() count is a lower bound on what the API bills.

Why token count drives cost, limits and latency#

Cost#

APIs bill per token, with separate rates for input and output, and output is the expensive side. Here is a worked example with gpt-5.4-mini, whose standard-tier price on 22 November 2026 is $0.75 per million input tokens, $0.075 per million cached input tokens and $4.50 per million output tokens. Picture a support bot where each request carries a 1,200-token system prompt, 3,000 tokens of retrieved documents, 2,000 tokens of chat history and a 60-token question, and gets a 350-token answer:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

# gpt-5.4-mini, standard tier, USD per 1M tokens (OpenAI pricing page, 22 Nov 2026)
PRICE_IN, PRICE_CACHED, PRICE_OUT = 0.75, 0.075, 4.50
CONTEXT, MAX_OUT = 400_000, 128_000

def request_cost(input_tokens, output_tokens, cached_tokens=0):
    fresh = input_tokens - cached_tokens
    return (fresh * PRICE_IN + cached_tokens * PRICE_CACHED + output_tokens * PRICE_OUT) / 1e6

# One support-bot request
system, retrieved, history, question, answer = 1_200, 3_000, 2_000, 60, 350
inp = system + retrieved + history + question
per_req = request_cost(inp, answer)
print(f"input {inp:,} tokens, output {answer} tokens")
print(f"cost per request:     ${per_req:.6f}")
print(f"100,000 requests/day: ${per_req * 100_000:,.2f}/day, ${per_req * 100_000 * 30:,.0f}/month")
print(f"output share of cost: {answer * PRICE_OUT / (inp * PRICE_IN + answer * PRICE_OUT):.0%}")

# Same request with the 1,200-token system prompt served from the prompt cache
cached = request_cost(inp, answer, cached_tokens=system)
print(f"with system prompt cached: ${cached * 100_000:,.2f}/day")

def fits(prompt_text, max_output):
    n = len(enc.encode(prompt_text))
    return n + max_output <= CONTEXT and max_output <= MAX_OUT, n

for words in (300_000, 398_000):
    ok, n = fits("word " * words, max_output=4_000)
    print(f"{words:,}-word prompt: {n:,} tokens + 4,000 output, fits: {ok}")

print()
# How good is the 4-characters-per-token rule?
for label, text in [("English", "The quick brown fox jumps over the lazy dog near the river bank. " * 20),
                    ("JSON", '{"user_id": 48213, "name": "Asha", "active": true}' * 20),
                    ("Hindi", "मुझे कल सुबह बैंगलोर जाना है। " * 20)]:
    real = len(enc.encode(text)); guess = len(text) // 4
    print(f"{label:<8} rule-of-thumb {guess:>4}  real {real:>4}  error {100 * (guess - real) / real:+.0f}%")

Output:

input 6,260 tokens, output 350 tokens
cost per request:     $0.006270
100,000 requests/day: $627.00/day, $18,810/month
output share of cost: 25%
with system prompt cached: $546.00/day
300,000-word prompt: 300,001 tokens + 4,000 output, fits: True
398,000-word prompt: 398,001 tokens + 4,000 output, fits: False

English  rule-of-thumb  325  real  281  error +16%
JSON     rule-of-thumb  250  real  400  error -38%
Hindi    rule-of-thumb  150  real  182  error -18%

A fraction of a cent per request becomes about $18,800 a month at 100,000 requests a day. The 350 output tokens are 5% of the tokens but 25% of the bill, because each one costs six times an input token. Chat history is the line that grows: every turn resends the whole conversation, so turn 20 pays again for turns 1 to 19.

Prompt caching cut the bill by $81 a day here, but only because the system prompt is an exact, unchanging prefix. OpenAI caches automatically, requires the entire rendered prefix to match, and has a minimum cacheable length (1,024 tokens on GPT-5.6 and later). Put a timestamp at the top of the system prompt and the saving disappears. For a longer list of ways to trim input without hurting answers, see How to prompt Claude with fewer tokens and the same quality; the ideas apply to any provider.

The last block tests the 4-characters rule. It overestimated English by 16% and underestimated JSON by 38%. A budget built on the rule for a JSON-heavy agent would come in well under the real bill. Count with the real tokenizer.

Context limits#

The context window is the maximum number of tokens one request can hold, and it covers input and output together. gpt-5.4-mini lists a 400,000-token context window and 128,000 max output tokens. The fits() check above shows the arithmetic: a 398,000-token prompt plus a 4,000-token answer does not fit, even though the prompt alone does. Some models also cap input below the full window (the GPT-5 mini page lists a 272,000-token maximum input inside its 400,000 window), so read the model page instead of assuming.

When a chat app seems to forget the start of a long conversation, the app has usually trimmed or summarized older turns to stay under this limit. The model has no memory outside the tokens in the current request.

Latency#

A model generates output one token at a time, so a long answer takes longer to finish in proportion to its length. Input affects the wait before the first token: OpenAI's latency guide notes that a long prompt can delay the first token, especially when much of it is new, and recommends filtering search results and dropping markup you do not need. Asking for "the three fields we display" instead of "a detailed explanation" shortens the answer, which cuts latency and output cost in the same step.

Surprising effects of tokenization#

These come from a separate quirks script run on o200k_base:

'12345 + 67890'         6 tokens  ['123', '45', ' +', ' ', '678', '90']
'1234567'               3 tokens  ['123', '456', '7']
'1,234,567'             5 tokens  ['1', ',', '234', ',', '567']
'strawberry'            3 tokens  ['st', 'raw', 'berry']
' strawberry'           1 tokens  [' strawberry']
'Strawberry'            3 tokens  ['Str', 'aw', 'berry']
'🇮🇳'                    4 tokens  ['�', '�', '�', '�']
'👍🏽'                    3 tokens  ['👍', '�', '�']
'Hello world'           2 tokens  ['Hello', ' world']
'Hello world '          3 tokens  ['Hello', ' world', ' ']

Arithmetic. Digits are grouped in threes from the left, so 1234567 becomes 123 | 456 | 7. The groups do not line up with place value: in 12345 the token 123 holds the ten-thousands, thousands and hundreds digits, while in 1234567 it holds the millions, hundred-thousands and ten-thousands. To add two numbers digit by digit, the model has to work out place value from token boundaries that shift with the length of the number. Writing the number with commas, 1,234,567, groups digits by place value, which is one reason formatted numbers sometimes give better results. For exact math in production, give the model a calculator tool instead of trusting it.

Spelling and letter counting. Ask how many r's are in "strawberry" and the model sees either one token, ' strawberry', or st | raw | berry. The letters are not visible as separate units; the model has to have learned the spelling of each token from training data. The same word also tokenizes three different ways depending on a leading space and a capital letter, so spelling knowledge learned for one form does not automatically carry to the others. Character-level tasks (counting letters, reversing strings, checking rhymes) are weak spots for this reason.

Trailing spaces. "Hello world " adds a lone space token (ID 220). Because spaces normally attach to the front of the next word, text in training almost never shows a bare space token followed by a word token. A prompt or completion prefix that ends in a space puts the model in an unusual position, and output quality can drop. Strip trailing whitespace from prompts and from any text you pre-fill for the model to continue.

Emoji and rare symbols. The four � tokens for 🇮🇳 are raw byte fragments that only form a valid character when decoded together. If you cut a token stream at an arbitrary point (when truncating output, or streaming and printing token by token) you can split a character and print garbage. Decode on character boundaries.

A quick test to add to your project#

Token counts belong in tests, because they change silently when someone edits a prompt or switches models. Two cheap checks:

  1. Budget test. Render your largest realistic prompt (longest history you allow, maximum retrieved chunks, all tool definitions), count it with the tokenizer for the model you deploy, and assert it plus max_output is under the context limit with a margin. Run it in CI, so a long new tool description fails a build instead of failing a user request.
  2. Cost regression. Keep a fixed set of 20 to 50 sample requests. Record their token counts in a file, and fail the build when the total rises more than, say, 10%. Rerun the counts whenever you switch models, since a new tokenizer can move every number.

Neither test needs an API key for OpenAI models, since tiktoken runs locally. For Claude, the counting endpoint needs a key but is free to call. To watch real spend per feature once you are live, log the usage field from each response; Build an LLM cost and latency tracker in Python shows one way.

Where to go next#

Tokenization is the first layer of the stack. To see what happens to those integer IDs inside the transformer (embeddings, attention, next-token sampling), the free videos in Learn how LLMs work from free videos: a weekend plan cover it in about eight hours, including a full lecture on building a GPT tokenizer from scratch. Then run the BPE script above with 20 merges instead of 8 on a paragraph of your own text, and check which words end up as single tokens.

Sources#