GyaanStack. Blog

Articles / llmopspythonprojects

Build an LLM cost and latency tracker in Python

A decorator that logs tokens, latency and cost for every LLM call to SQLite, with a p50/p95 report and a daily budget that stops runaway spend.

By Raghunath · · 9 min read

LLM bills are hard to explain when nobody recorded the calls. The provider dashboard shows one total per day, but it can't tell you which feature spent the money, which prompt got longer last week, or why the p95 latency doubled.

This post builds a small tracker that records every call. It's one decorator, about 70 lines, and SQLite. For every call it stores:

  • the feature that made it (summarize, classify, …),
  • the model, input tokens and output tokens,
  • latency in milliseconds,
  • cost in dollars from a price table,
  • whether the call failed.

On top of that it gives you a report with p50 and p95 latency per feature, and a daily budget that refuses calls once a feature has spent its limit.

It needs Python 3.10 or newer. I tested it on Python 3.15 with the ollama Python package 0.6.3 and gemma3:12b running locally. Local models cost nothing, so the demo also uses a fake "hosted" model with made-up prices to show the cost and budget side.

The price table and the database#

Save this block and the decorator below as one file, tracker.py. The demo imports from it.

import functools
import sqlite3
import statistics
import time
from dataclasses import dataclass

# USD per 1M tokens. Keep this in one place and date it: prices change.
PRICES = {  # model: (input, output)
    "gemma3:12b": (0.0, 0.0),          # local
    "fake-pro": (3.00, 15.00),          # made-up hosted model for the demo
}

DB = sqlite3.connect("llm_calls.db")
DB.execute("""CREATE TABLE IF NOT EXISTS calls (
    ts REAL, feature TEXT, model TEXT, in_tok INTEGER, out_tok INTEGER,
    latency_ms REAL, cost_usd REAL, ok INTEGER)""")

Input and output tokens are priced separately because most providers charge several times more for output. A feature that writes long answers costs more than one that reads long documents, even at the same total token count. If you store only total tokens, you lose that.

The cost is stored at the time of the call, not calculated when you run the report. When the provider changes its prices, old rows keep the cost you actually paid.

The decorator#

@dataclass
class Usage:
    model: str
    in_tok: int
    out_tok: int

class BudgetExceeded(RuntimeError):
    pass

def cost(u: Usage) -> float:
    pin, pout = PRICES.get(u.model, (0.0, 0.0))
    return (u.in_tok * pin + u.out_tok * pout) / 1_000_000

def spent_today(feature: str) -> float:
    row = DB.execute("SELECT COALESCE(SUM(cost_usd), 0) FROM calls WHERE feature = ? AND ts > ?",
                     (feature, time.time() - 86400)).fetchone()
    return row[0]

def tracked(feature: str, daily_budget_usd: float | None = None):
    """Decorate a function that returns (text, Usage). Logs every call, even failures."""
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            if daily_budget_usd is not None and spent_today(feature) >= daily_budget_usd:
                raise BudgetExceeded(f"{feature}: daily budget ${daily_budget_usd} used up")
            t0 = time.perf_counter()
            try:
                text, usage = fn(*args, **kwargs)
            except Exception:
                DB.execute("INSERT INTO calls VALUES (?,?,?,?,?,?,?,0)",
                           (time.time(), feature, "?", 0, 0, (time.perf_counter() - t0) * 1000, 0.0))
                DB.commit()
                raise
            ms = (time.perf_counter() - t0) * 1000
            DB.execute("INSERT INTO calls VALUES (?,?,?,?,?,?,?,1)",
                       (time.time(), feature, usage.model, usage.in_tok, usage.out_tok, ms, cost(usage)))
            DB.commit()
            return text
        return wrapper
    return deco

A few design choices:

The wrapped function returns (text, Usage). The decorator doesn't try to guess token counts. Every provider reports real counts in its response: Ollama has prompt_eval_count and eval_count, and hosted APIs have a usage field. The function that makes the call knows where its provider puts them. The decorator removes the Usage and returns only the text, so callers don't see the tracking.

Failures are logged too. A timeout that takes 30 seconds and then raises still costs your user 30 seconds. If you log only successful calls, your latency numbers look better than what users feel, and you never see the error rate. The failed row has ok = 0, and then the exception is raised again so the caller still handles it.

The budget check runs before the call. "Today" here means the last 24 hours (time.time() - 86400), a rolling window, not the calendar day. It's a "stop starting new work" check, not a hard cap, so the last call can push spend slightly over the limit. You'll see that in the demo output. For a hard cap, estimate the cost of the next call (input tokens are known before you send) and refuse it if it would cross the line.

The report#

def pct(values: list[float], p: float) -> float:
    if len(values) == 1:
        return values[0]
    return statistics.quantiles(values, n=100, method="inclusive")[int(p) - 1]

def report():
    print(f"{'feature':<12}{'model':<12}{'calls':>6}{'fail':>6}{'p50 ms':>9}{'p95 ms':>9}{'tokens':>9}{'cost $':>10}")
    for feature, model in DB.execute("SELECT DISTINCT feature, model FROM calls WHERE ok = 1"):
        rows = DB.execute("SELECT latency_ms, in_tok + out_tok, cost_usd FROM calls "
                          "WHERE feature = ? AND model = ? AND ok = 1", (feature, model)).fetchall()
        fails = DB.execute("SELECT COUNT(*) FROM calls WHERE feature = ? AND ok = 0", (feature,)).fetchone()[0]
        lat = [r[0] for r in rows]
        print(f"{feature:<12}{model:<12}{len(rows):>6}{fails:>6}{pct(lat, 50):>9.0f}{pct(lat, 95):>9.0f}"
              f"{sum(r[1] for r in rows):>9}{sum(r[2] for r in rows):>10.4f}")

I report p50 and p95 rather than the average. LLM latency has a long tail: most calls are quick, and a few take many times longer because of long outputs, queueing at the provider, or retries. An average mixes both and describes neither. p50 is a typical call. p95 is what your unluckiest regular users see, and it's usually the number that needs fixing.

The demo#

Two features: summarize calls the real local model, and classify calls a fake hosted model with a $0.05 daily budget. The fake model sleeps for a random 50 to 300 ms and fails 10% of the time, which is enough to exercise every part of the tracker.

import random
import time

import ollama

from tracker import BudgetExceeded, Usage, report, tracked

MODEL = "gemma3:12b"

@tracked("summarize")
def summarize_local(text: str):
    r = ollama.chat(model=MODEL, messages=[{"role": "user", "content": f"Summarize in one sentence: {text}"}])
    return r.message.content, Usage(MODEL, r.prompt_eval_count, r.eval_count)

@tracked("classify", daily_budget_usd=0.05)
def classify_fake(text: str):
    # Stand-in for a hosted API: random latency, token counts from word counts.
    time.sleep(random.uniform(0.05, 0.3))
    if random.random() < 0.1:
        raise TimeoutError("upstream timeout")
    return "billing", Usage("fake-pro", in_tok=len(text.split()) * 4 + 900, out_tok=random.randint(200, 600))

docs = [
    "The invoice was charged twice and the customer wants one refund.",
    "Our deploy failed because the container ran out of memory during the build step.",
    "The new onboarding flow cut the time to first project from ten minutes to four.",
]
for d in docs:
    print("summary:", summarize_local(d))

random.seed(7)
calls = fails = 0
try:
    for i in range(100):
        try:
            classify_fake(docs[i % 3])
            calls += 1
        except TimeoutError:
            fails += 1
except BudgetExceeded as e:
    print(f"stopped after {calls} ok + {fails} failed calls: {e}")

report()

Output from a fresh database:

summary: A customer is requesting a refund because their invoice was mistakenly charged twice.
summary: The deployment failed due to insufficient memory allocated to the container during the build process.

summary: The revamped onboarding process dramatically reduced the time it takes for new users to complete their first project, cutting it from ten minutes to just four.
stopped after 6 ok + 5 failed calls: classify: daily budget $0.05 used up
feature     model        calls  fail   p50 ms   p95 ms   tokens    cost $
summarize   gemma3:12b       3     0     1671     2187      153    0.0000
classify    fake-pro         6     5       98      132     8023    0.0519

What the numbers show:

  • The budget did its job. The loop asked for 100 classifications and the tracker stopped it after 11 attempts. Spend ended at $0.0519, slightly over $0.05, because the check runs before each call (as described above).
  • Failures are visible. 5 of 11 classify calls failed. That's high for a 10% failure rate, and it's chance with this seed and so few calls. The table shows it, which is the point: in a real system, a "fail" column that jumps tells you about an outage before your users do.
  • Output tokens drive cost. Each fake call had about 900 to 950 input tokens and 200 to 600 output tokens. At $3 and $15 per million, the output tokens make up most of the bill even though there are fewer of them.
  • The local model is free but slow. gemma3:12b cost $0 and took 1.7 seconds at p50 for a one-sentence summary on my laptop. That's the trade-off the table helps you see: you can compare cost and latency per feature, side by side, when you decide which model each feature should use.

The second summary also has a stray blank line after it, because the model returned a trailing newline. Small output differences like this are another reason to log raw responses somewhere when you debug.

Using it in a real app#

  • One decorator per feature, not per model. The question you'll be asked is "what does the search feature cost?", not "what does model X cost?". The table stores both, so you can group either way.
  • Date your price table and keep it in code review. When a provider changes prices, update the table in one place. Old rows keep their old cost.
  • Swap SQLite for your metrics stack when you outgrow it. The calls table maps directly onto Prometheus histograms (latency), counters (tokens, cost, failures), or OpenTelemetry spans with attributes. Keep the decorator interface the same and change only where it writes.
  • Alert on p95 and the failure rate, not only on spend. A model that starts timing out gets cheaper (failed calls cost little) and worse at the same time. A budget alarm alone won't catch that.
  • Add prompt_version as a column. When a prompt change makes outputs 40% longer, you'll see it as a step change in output tokens for that version, and you'll know which commit to look at.

The whole tracker is one file you can drop into any project that calls an LLM. Start recording before the first surprise bill, not after it.