Articles / llmopspythonprojects
A production LLM API with FastAPI: streaming, retries, rate limits and load tests
A FastAPI service in front of an LLM with SSE streaming, timeouts, retries with jitter, per-user rate limits, structured errors, caching and load-test numbers.
Wrapping a model call in @app.post("/generate") takes ten lines. Those ten lines fall over the first time the provider returns a 529, a user sends a 40,000-character prompt, one customer's script fires 500 requests a minute, or a sync SDK call sits inside an async def and freezes every other request on the process.
This post builds the layer that sits between your users and the model. It covers server-sent event (SSE) streaming, a hard timeout, retries with backoff and jitter, a per-user rate limit, request validation, one JSON error shape for every failure, a cache for deterministic requests, a token-usage log line per request, and a cap on concurrent upstream calls. Seven tests check the behaviour, and a load test against a local server gives real latency numbers.
The model is a stub. It sleeps 150 ms before the first token and 10 ms per token after that, which is close enough to a hosted model's shape to make the concurrency results meaningful, and it needs no API key. The comments show where a real SDK call goes.
Versions used: FastAPI 0.165.0, Starlette 1.10.0, Pydantic 2.14.4, Uvicorn 0.54.0, httpx 0.28.1, Python 3.15, on an Apple M5 laptop.
The request path#
client
| POST /v1/generate (X-API-Key, JSON body)
v
middleware: request id, access log
|
v
validation (Pydantic) ---- bad body ----> 422 {error: invalid_request}
|
v
auth + per-user token bucket ---- empty -> 429 + Retry-After
|
v
cache lookup (temperature == 0 only) -- hit --> 200, cached: true
|
v
concurrency slot (semaphore, 2 s wait) -- full --> 503 + Retry-After
|
v
asyncio.timeout(10 s)
| retry loop: 3 attempts, backoff 0.2 s, 0.4 s with full jitter
v
LLM client (stub here, SDK in production)
|
v
usage log line -> response
The order is deliberate. Cheap checks run first, so a malformed or over-limit request never takes a concurrency slot. The cache sits before the slot because a hit costs no model time. The timeout wraps the whole retry loop, so three slow attempts cannot add up to 30 seconds.
The model interface#
Everything above the model talks to a two-method interface. That keeps the stub and the real client interchangeable and makes the tests fast.
class TransientLLMError(Exception):
"""Provider said 429, 5xx or overloaded. Safe to retry."""
class PermanentLLMError(Exception):
"""Provider rejected the request itself (400, 401). Do not retry."""
@dataclass
class Completion:
text: str
input_tokens: int
output_tokens: int
class LLMClient(Protocol):
async def complete(self, prompt: str, max_tokens: int, temperature: float) -> Completion: ...
def stream(self, prompt: str, max_tokens: int, temperature: float) -> AsyncIterator[str]: ...
The stub implements both methods with asyncio.sleep. In production, complete() would call something like await anthropic.AsyncAnthropic().messages.create(...) and stream() would iterate the SDK's stream helper. The one piece of real work is the error mapping. Anthropic's API returns 429 rate_limit_error and 529 overloaded_error for load problems; those become TransientLLMError. A 400 invalid_request_error becomes PermanentLLMError, because sending the same bad request again will fail the same way. Other providers document the same split; OpenAI's rate limit guide also recommends backoff for 429s.
The interface has to be async. If your SDK only has a sync client, either call it from a plain def endpoint (FastAPI runs those in a threadpool) or wrap the call in anyio.to_thread.run_sync. The load test further down shows what happens when you do neither.
Validation and one error shape#
Pydantic rejects bad input before any of your code runs. The limits are the part people skip. Without max_length on the prompt, a single request can carry a whole book and you pay for every token of it.
class GenerateRequest(BaseModel):
prompt: str = Field(min_length=1, max_length=8_000)
max_tokens: int = Field(default=256, ge=1, le=1024)
temperature: float = Field(default=0.7, ge=0.0, le=1.0)
FastAPI's default 422 body is a list of Pydantic errors, and HTTPException produces {"detail": "..."}. Clients then need two parsers. I prefer one shape for every failure, with a stable machine-readable code, a human message and the request id that also appears in the logs. RFC 9457 (problem details) is a standard alternative if your clients already expect it.
class APIError(Exception):
def __init__(self, status, code, message, retry_after=None):
self.status, self.code, self.message, self.retry_after = status, code, message, retry_after
def error_body(request, code, message, details=None):
body = {"error": {"code": code, "message": message,
"request_id": getattr(request.state, "request_id", None)}}
if details:
body["error"]["details"] = details
return body
@app.exception_handler(APIError)
async def api_error_handler(request, exc):
headers = {"Retry-After": str(max(1, round(exc.retry_after)))} if exc.retry_after else None
return JSONResponse(error_body(request, exc.code, exc.message), exc.status, headers=headers)
@app.exception_handler(RequestValidationError)
async def validation_handler(request, exc):
details = [{"field": ".".join(map(str, e["loc"][1:])), "issue": e["msg"]} for e in exc.errors()]
return JSONResponse(error_body(request, "invalid_request", "Request body failed validation.", details), 422)
@app.exception_handler(Exception)
async def unhandled(request, exc):
log.exception("unhandled")
return JSONResponse(error_body(request, "internal_error", "Something went wrong on our side."), 500)
The catch-all handler never puts str(exc) in the response. A raw exception string can leak a file path, an upstream URL or part of a prompt from another code path. It goes to the log, and the client gets the request id to quote in a bug report. FastAPI's error handling docs cover overriding the validation handler.
Sending an empty prompt and a temperature of 5 to the running server returns:
{"error":{"code":"invalid_request","message":"Request body failed validation.","request_id":"9034b11b51e4","details":[{"field":"prompt","issue":"String should have at least 1 character"},{"field":"temperature","issue":"Input should be less than or equal to 1"}]}}
Per-user rate limiting with a token bucket#
Each API key maps to a user and a requests-per-minute allowance. A token bucket per user refills continuously, so a user with 10 per minute gets one new request every 6 seconds rather than a burst of 10 at the top of each minute.
class TokenBucket:
def __init__(self, per_minute: int):
self.capacity = per_minute
self.tokens = float(per_minute)
self.rate = per_minute / 60.0
self.updated = time.monotonic()
def take(self) -> float:
"""Return 0 if allowed, else seconds until the next token."""
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
self.updated = now
if self.tokens >= 1:
self.tokens -= 1
return 0.0
return (1 - self.tokens) / self.rate
def current_user(x_api_key: Annotated[str | None, Header()] = None) -> str:
if x_api_key not in API_KEYS:
raise APIError(401, "invalid_api_key", "Send a valid X-API-Key header.")
user, per_min = API_KEYS[x_api_key]
wait = buckets.setdefault(user, TokenBucket(per_min)).take()
if wait:
raise APIError(429, "rate_limited", f"Limit is {per_min} requests/min.", retry_after=wait)
return user
current_user is a dependency, so each endpoint declares user: str = Depends(current_user) and gets both checks. Eleven quick requests from one key against the live server:
200 200 200 200 200 200 200 200 200 200 429
HTTP/1.1 429 Too Many Requests
retry-after: 4
{"error":{"code":"rate_limited","message":"Limit is 10 requests/min.","request_id":"bf2e48d1f9d3"}}
The Retry-After header tells a well-behaved client exactly how long to wait.
Two limits of this version. The buckets live in process memory, so four Uvicorn workers give each user four times the allowance; move the counters to Redis once you run more than one process. And requests are a weak proxy for cost. A request with max_tokens=1024 costs about 1,000 times one with max_tokens=1, so a real service also budgets tokens per user per day. My LLM cost and latency tracker builds that budget check on top of a per-call log.
Timeouts and retries#
The retry loop retries only transient errors, sleeps with exponential backoff and full jitter, and gives up after three attempts:
async def with_retries(fn, *, attempts=MAX_ATTEMPTS, base=0.2, cap=2.0):
for attempt in range(1, attempts + 1):
try:
return await fn()
except TransientLLMError:
if attempt == attempts:
raise
delay = random.uniform(0, min(cap, base * 2 ** (attempt - 1)))
log.info(json.dumps({"event": "retry", "attempt": attempt, "sleep_s": round(delay, 3)}))
await asyncio.sleep(delay)
Full jitter means the sleep is a random value between zero and the backoff ceiling. When a provider has a bad minute, every one of your in-flight requests fails at once. Without jitter they all retry at 200 ms, then all again at 400 ms, and hit the provider in waves. AWS's exponential backoff and jitter write-up simulates competing clients and shows that adding jitter cuts both the total number of calls and the time to finish compared with plain exponential backoff.
Check what your SDK already does before adding this loop. Anthropic's docs state that the official SDK retries connection errors, rate limits and 5xx errors twice by default, with backoff, and honours retry-after. Stack three attempts in your code on top of three in the SDK and one user request can become nine upstream calls. Pick one layer. I keep the loop in the service, where it can log each retry, and set the SDK's max_retries=0.
The timeout wraps the loop, using asyncio.timeout:
async with upstream_slot(request):
try:
async with asyncio.timeout(MODEL_TIMEOUT_S):
c = await with_retries(lambda: llm.complete(body.prompt, body.max_tokens, body.temperature))
except TimeoutError:
raise APIError(504, "upstream_timeout", f"Model did not answer in {MODEL_TIMEOUT_S:.0f}s.")
except TransientLLMError:
raise APIError(502, "upstream_unavailable", "Model provider failed after retries.", retry_after=5)
except PermanentLLMError as e:
raise APIError(400, "upstream_rejected", str(e))
Each upstream failure maps to a different status. 502 means the provider failed, 504 means it was too slow and 400 means the request was wrong. On-call engineers can tell those apart from a dashboard without opening a single log line.
Ten seconds is right for short completions. A request for 1,000 output tokens at 50 tokens a second needs 20 seconds before any overhead, so either size the timeout from max_tokens or push long generations to the streaming endpoint.
Capping concurrent upstream calls#
A semaphore limits how many model calls one process makes at a time. A request that cannot get a slot in 2 seconds gets a 503 instead of waiting in a queue that only grows:
@asynccontextmanager
async def upstream_slot(request):
slots = request.app.state.slots # asyncio.Semaphore(MAX_INFLIGHT)
try:
async with asyncio.timeout(QUEUE_WAIT_S):
await slots.acquire()
except TimeoutError:
raise APIError(503, "overloaded", "Too many requests in flight. Retry shortly.", retry_after=1)
try:
yield
finally:
slots.release()
Set MAX_INFLIGHT from your provider quota, not from your server. If your account allows 50 concurrent requests, a cap of 100 per process means the provider does your load shedding for you, with 429s you then retry.
Caching deterministic requests#
Caching only makes sense when the same input should give the same output. With temperature above zero, two identical requests are supposed to differ, so this service caches only temperature == 0 requests, keyed on a SHA-256 of the whole validated body:
key = cache_key(body) if body.temperature == 0 else None
if key and (hit := request.app.state.cache.get(key)):
log_usage(request, user, hit.usage, True, started)
return hit.model_copy(update={"cached": True, "request_id": request.state.request_id})
The cache is a small in-memory LRU with a 5-minute TTL. The key includes every field of the body, so changing max_tokens is a miss. In production, add the model name and the system prompt version to the key, or a prompt change will keep serving old answers until the TTL runs out. Provider-side prompt caching is a different feature; it discounts a repeated prompt prefix and still runs the model.
Logging token usage#
Every successful request writes one JSON line with the user, request id, tokens and wall time. Real lines from the local run:
{"event": "usage", "user": "alice", "request_id": "c962a2b49811", "input_tokens": 1, "output_tokens": 1, "cached": false, "ms": 162.6}
{"event": "usage", "user": "alice", "request_id": "b47563448718", "input_tokens": 1, "output_tokens": 1, "cached": false, "ms": 164.3}
The stub estimates input tokens as characters divided by four. A real client should read the usage field the provider returns, which is exact and is what you are billed for. The request_id comes from a middleware that also sets the x-request-id response header, so one id links the client's error message, the access log and the usage line. Prompts and outputs stay out of this log on purpose; what to log, how to redact it and how to turn these lines into traces is covered in my guardrails and observability post.
Streaming with server-sent events#
Streaming changes what the user waits for. A 40-token answer from the stub takes about 550 ms in full, but the first token arrives at about 150 ms. With a real model and a 500-token answer the gap is several seconds.
FastAPI added native SSE support in version 0.135.0. Set response_class=EventSourceResponse and yield ServerSentEvent objects from an async generator. FastAPI sends a keep-alive comment every 15 seconds of silence, sets Cache-Control: no-cache, and sets X-Accel-Buffering: no so Nginx does not buffer the stream. Older tutorials use sse-starlette or a hand-built StreamingResponse; on a current FastAPI neither is needed.
@app.post("/v1/generate/stream", response_class=EventSourceResponse)
async def generate_stream(body: GenerateRequest, request: Request,
user: str = Depends(current_user)) -> AsyncIterator[ServerSentEvent]:
started = time.perf_counter()
llm: LLMClient = request.app.state.llm
async with upstream_slot(request):
out_tokens = 0
try:
async with asyncio.timeout(MODEL_TIMEOUT_S):
async for chunk in llm.stream(body.prompt, body.max_tokens, body.temperature):
out_tokens += 1
yield ServerSentEvent(data={"delta": chunk}, event="delta")
except (TimeoutError, TransientLLMError) as e:
# headers are already sent, so the status code can't change: report in-band
code = "upstream_timeout" if isinstance(e, TimeoutError) else "upstream_unavailable"
yield ServerSentEvent(data=error_body(request, code, "Stream ended early.")["error"], event="error")
return
usage = Usage(input_tokens=len(body.prompt) // 4 + 1, output_tokens=out_tokens)
log_usage(request, user, usage, False, started)
yield ServerSentEvent(data={"usage": usage.model_dump(), "request_id": request.state.request_id},
event="done")
What curl -N receives for a 3-token request:
event: delta
data: {"delta": "tok0 "}
event: delta
data: {"delta": "tok1 "}
event: delta
data: {"delta": "tok2 "}
event: done
data: {"usage": {"input_tokens": 1, "output_tokens": 3}, "request_id": "19ff5ec87523"}
Streaming has failure modes the plain endpoint does not:
- Errors after the first byte. The 200 status and headers are already on the wire, so a mid-stream failure cannot become a 502. The endpoint sends an
errorevent and closes. Clients must treat a stream that ends without adoneevent as failed. - Retries. You can retry safely only before the first token goes out. After that the user has seen half an answer, and a retry would produce a different one. This endpoint does not retry at all; a production version would retry only the call that opens the stream.
- POST versus EventSource. The browser's
EventSourceAPI only sends GET requests. A POST endpoint with a JSON body needsfetch()with a stream reader on the client, which is what most chat UIs do anyway. - Rate limits on streams. The rate limit and concurrency slot apply when the stream opens. A stream holds its slot for its whole length, so long streams use up
MAX_INFLIGHTfaster than short calls.
Tests#
Seven pytest tests run against FastAPI's TestClient in under 3 seconds. Because the stub is swappable, failure paths are one line each:
def test_retry_then_give_up():
with client() as c:
c.app.state.llm = StubLLM(ttft=0.001, fail_rate=1.0)
r = c.post("/v1/generate", json={"prompt": "x"}, headers=LOAD)
assert r.status_code == 502 and r.headers["retry-after"] == "5"
assert c.app.state.llm.calls == 3
def test_timeout_maps_to_504(monkeypatch):
monkeypatch.setattr(main, "MODEL_TIMEOUT_S", 0.05)
with client() as c:
c.app.state.llm = StubLLM(ttft=1.0)
r = c.post("/v1/generate", json={"prompt": "x"}, headers=LOAD)
assert r.status_code == 504 and r.json()["error"]["code"] == "upstream_timeout"
def test_cache_only_when_temperature_zero():
with client() as c:
body = {"prompt": "same", "max_tokens": 5, "temperature": 0}
a, b = c.post("/v1/generate", json=body, headers=LOAD), c.post("/v1/generate", json=body, headers=LOAD)
assert (a.json()["cached"], b.json()["cached"]) == (False, True)
assert c.app.state.llm.calls == 1
The other four check the validation error shape, the missing-key 401, that one user's 429 does not block another user, and that the stream returns three delta events and a done.
....... [100%]
7 passed, 1 warning in 2.85s
The with client() as c form matters. TestClient runs the lifespan handler only inside a with block, and the lifespan is where the client, cache and semaphore get created.
Load test numbers#
The load generator is an httpx AsyncClient that keeps a fixed number of requests in flight and records each latency. Each request asks for 40 tokens, so the ideal latency is about 550 ms. Server and client ran on the same 10-core M5 laptop, with one Uvicorn worker and no caching (every prompt is unique, temperature 0.7).
c=1 n=20 rps= 1.8 p50= 557ms p95= 566ms codes={200: 20}
c=10 n=50 rps= 17.7 p50= 562ms p95= 568ms codes={200: 50}
c=50 n=250 rps= 81.3 p50= 556ms p95= 752ms codes={200: 250}
c=100 n=500 rps= 72.7 p50= 827ms p95= 3297ms codes={200: 500}
c=200 n=1000 rps= 53.2 p50= 2380ms p95= 6250ms codes={200: 1000}
c=400 n=2000 rps= 64.8 p50= 5245ms p95= 8931ms codes={200: 2000}
Up to 50 concurrent requests, throughput grows in line with concurrency and the p50 stays at the stub's 556 ms, so the service adds only a few milliseconds. Past that, throughput stops at about 70 to 80 requests per second and latency climbs. At 100 concurrent requests the ideal would be about 180 rps. The stub sleeps without using CPU and the semaphore allows 100 calls, so the most likely limit is CPU on one Python process (JSON parsing, Pydantic, logging, the event loop), with the load generator competing for the same laptop. I did not profile it. More Uvicorn workers would raise that ceiling. At 200 and 400 the p95 runs to 6 to 9 seconds with no errors. The 503 fires only when a request waits more than 2 seconds for a semaphore slot, and the queueing here happened elsewhere (in the socket backlog and the event loop), so the check never tripped. A queue-depth or p95 alert would catch this earlier than an error-rate alert.
The second run swaps the stub's asyncio.sleep for time.sleep, which is what a sync SDK call inside async def does:
c=1 n=20 rps= 1.8 p50= 567ms p95= 582ms codes={200: 20}
c=10 n=50 rps= 1.8 p50= 5629ms p95= 6195ms codes={200: 50}
At 10 concurrent requests throughput stays at 1.8 rps, the same as one user, because each call blocks the event loop and every request waits its turn. At 50 concurrent requests the load generator's 30-second client timeout fired and the run died with httpx.ReadTimeout. Nothing in the server logs pointed at the cause. Use an async client in async def, or a plain def endpoint for sync code. FastAPI runs def endpoints in AnyIO's threadpool, which allows 40 threads by default, so that route caps out at 40 concurrent calls per process unless you raise the limit. FastAPI's async page explains the rule.
What this does not cover#
Authentication here is a dictionary of API keys; a real service stores hashed keys and rotates them. Input and output checks for prompt injection or PII belong in a guardrail layer before and after the model call. Packaging and deployment are separate topics. Run the tests and the load test against your own provider's latency before you pick MAX_INFLIGHT, the timeout and the retry count, because the right values depend on that provider's numbers.
Sources#
- FastAPI: Server-Sent Events
- FastAPI: Lifespan events
- FastAPI: Concurrency and async / await
- FastAPI: Handling errors
- FastAPI: Dependencies
- Anthropic API: Errors
- Anthropic API: Rate limits
- OpenAI: Rate limits
- AWS Architecture Blog: Exponential backoff and jitter
- Python docs: asyncio.timeout
- AnyIO: Working with threads
- HTML Standard: Server-sent events
- RFC 9110: Retry-After
- RFC 9457: Problem Details for HTTP APIs