GyaanStack. Blog

Articles / llmopsevalsgenai-system-design

CI/CD for LLM applications: eval gates, canaries and rollback

How to ship an LLM app safely: what counts as a change, versioned prompts, eval gates with real cost math, handling noisy scores, canaries and rollback.

By Raghunath · · 22 min read

In a normal backend service, a change is a code diff. Someone opens a pull request, tests run, a reviewer reads the diff, and the change ships. If behaviour changes, you can point to the line that changed it.

An LLM application breaks that model. Its behaviour depends on at least five things, and only one of them is ordinary code: the prompt, the model version, the retrieval index, the tool definitions, and the code around them. A one-word prompt edit can move accuracy by ten points. A provider retiring a model forces a change you never asked for. A re-chunked index changes every answer without touching a single file in the repo.

Most guides on this topic say "run evals in CI" and show a threshold like assert score > 0.8. They skip the hard questions: how many cases you can afford per pull request, how to tell a regression from sampling noise, what to compare against, and what happens after the gate passes. This post answers those with numbers, a runnable gate script that fails the build on a regression, and a decision table.

What counts as a change#

I treat anything that can change an output for the same input as a deployable change. That gives this list:

Change Where it usually lives How it breaks things What should run
Prompt or system instructions prompts/ in the repo, or a prompt registry Format drift, new refusals, lost edge-case handling Smoke evals on the PR, full evals before release
Model version Config value Different reasoning, length, tool-call habits, tokenizer, price Full evals plus a shadow run
Sampling params (temperature, max tokens) Config Truncation, more variance Smoke evals
Retrieval index (chunking, embedding model, new documents) Data pipeline, vector store Wrong or missing context, citation errors Retrieval evals plus full answer evals
Tool schemas and tool code Repo Wrong arguments, failed calls, loops Unit tests plus agent trajectory evals
Guardrail rules Repo or config False blocks or missed blocks Safety slice, every time
Application code Repo Ordinary bugs Unit and integration tests

Two rows cause most surprises. Index changes have no code diff, so a PR-only pipeline never tests them. The model row can change without your consent: Anthropic's model deprecations page says customers get at least 60 days' notice before a publicly released model is retired, and after that requests to it fail. OpenAI keeps a similar deprecations page. Sixty days is enough time for a planned upgrade and not enough time to build an eval suite from scratch.

Version everything in one release manifest#

The fix for "five things change" is to pin all five in one file and treat that file as the release. I call it a release manifest. A deploy is a new manifest, and a rollback is the previous one.

# release.yaml  (one file per release, committed to git)
release: 2026-11-21.2
code_sha: 4f1c9e2
model:
  id: claude-sonnet-4-6        # an exact ID, never a "latest" alias
  temperature: 0.2
  max_tokens: 800
prompts:
  support_system: prompts/support_system.md@sha256:9b1e...
  router:         prompts/router.md@sha256:41ac...
retrieval:
  index: support-kb-v37        # immutable index name, built once
  embedder: embed-v3
  top_k: 6
tools:
  schema_hash: sha256:c07d...
guardrails: guardrails/v12.yaml
eval_baseline: evals/baselines/2026-11-14.1.json

Three rules make it work.

  1. Prompts live in git, or in a registry that gives each version an immutable ID. If someone can edit the production prompt in a web console without creating a new version, you have no CI/CD for prompts, whatever the pipeline says.
  2. Indexes are immutable. A rebuild creates support-kb-v38 next to v37, and the manifest switches the pointer. You keep the old index until the new release has been stable for a while, because rollback needs it.
  3. Model IDs are exact. An alias that silently moves to a new snapshot is a deploy nobody approved.

The manifest diff also tells CI what to run, so a change to model.id triggers the full suite.

The pipeline#

  PR opened / updated
        │
        ▼
  ┌──────────────┐   fail   ┌──────────────┐
  │ unit + lint  │────────▶│  block merge  │
  └──────┬───────┘          └──────▲───────┘
         ▼                         │ fail
  ┌──────────────────────┐         │
  │ smoke evals          │─────────┘
  │ ~60 cases x 3 runs   │
  │ paired vs baseline   │
  └──────┬───────────────┘
         ▼ merge to main
  ┌──────────────────────┐   fail  (no release cut)
  │ full evals           │──────▶ open an issue with the flipped cases
  │ ~600 cases x 5 runs  │
  │ nightly or pre-release│
  └──────┬───────────────┘
         ▼ new release manifest
  ┌──────────────────────┐
  │ shadow: copy live    │  compare outputs offline, users see old version
  │ traffic, 1-3 days    │
  └──────┬───────────────┘
         ▼
  ┌──────────────────────┐  online metrics worse   ┌──────────────────┐
  │ canary 5% → 25% → 100%│──────────────────────▶ │ rollback: point  │
  └──────────────────────┘                          │ to old manifest  │
                                                    └──────────────────┘

The smoke set catches breakage in minutes, the full set catches smaller regressions, shadow traffic catches inputs your eval set never imagined, and the canary catches what only shows up when real users react.

The eval suite as a deploy gate#

Smoke set on every PR, full set before release#

A gate that takes 40 minutes on every push gets disabled within a month, so I split the suite.

The smoke set is small and hand-picked: every critical safety case, a few cases per major intent, and every bug that ever reached production. It runs on every PR that touches prompts, config, tools or code.

The full set is the whole labelled dataset. It runs nightly on main, before any release, and on any change to the model or the index.

Cost per run, with the math#

These are estimates. I assume a support assistant where one call sends about 2,000 input tokens (system prompt, retrieved chunks, the question) and gets about 300 output tokens back. Prices are list prices from Anthropic's pricing page at the time of writing: Claude Sonnet 4.6 at $3 per million input tokens and $15 per million output, Claude Haiku 4.5 at $1 and $5. Swap in your own token counts and model.

Smoke set: 60 cases x 3 repeats = 180 calls to the app model.

input   180 x 2,000 = 360,000 tokens x $3  / 1M = $1.08
output  180 x   300 =  54,000 tokens x $15 / 1M = $0.81
app model total                                  = $1.89

LLM judge on the 40% of cases that need it (72 calls, Haiku 4.5):
input    72 x 2,500 = 180,000 tokens x $1 / 1M  = $0.18
output   72 x   150 =  10,800 tokens x $5 / 1M  = $0.05
judge total                                      = $0.23

per smoke run                                    ≈ $2.12

Thirty PRs a week with about two pushes each makes 60 smoke runs, roughly $127 a week. That is affordable, but it is why I use path filters and cancel runs a newer push supersedes. Prompt caching on the shared system prompt cuts input cost further (cache reads on Sonnet 4.6 are listed at $0.30 per million tokens).

Full set: 600 cases x 5 repeats = 3,000 calls.

input   3,000 x 2,000 = 6.0M tokens x $3  / 1M = $18.00
output  3,000 x   300 = 0.9M tokens x $15 / 1M = $13.50
per full run (before judge)                    = $31.50
via the Batch API (50% off)                    ≈ $15.75

The full set fits the Message Batches API well. Anthropic's docs say most batches finish in under an hour, with a 24-hour limit and a 50% discount. That is fine for a nightly job and wrong for a PR check, where someone is waiting.

Run time is the other budget. With 10 concurrent requests and an assumed 4 seconds per call, the smoke set takes 180 / 10 x 4 s = 72 seconds and the full set 3,000 / 10 x 4 s = 1,200 seconds, about 20 minutes, if your rate limits allow 10 in flight.

Non-determinism: what a score difference means#

Temperature 0 does not make an LLM app deterministic in practice: serving-side changes can still move outputs, and agents that branch on earlier outputs amplify small differences. Even with fixed outputs, your score is a sample, since 60 cases stand in for the thousands of questions users ask.

Evan Miller's paper Adding Error Bars to Evals is the best primary source I know on this. Its practical advice comes down to two things I use in the gate: report a confidence interval, not just a score, and when comparing two versions, compare them on the same questions and analyse the per-question differences (a paired test), which removes much of the noise that comes from some questions being harder than others.

Here is what a confidence interval looks like for a pass rate of 0.87, using the normal approximation (half-width = 1.96 x sqrt(p(1-p)/n)):

Cases (n) 95% CI half-width What it can detect
60 ±8.5 points Only large breakage
200 ±4.7 points Medium regressions
600 ±2.7 points Small regressions on the whole set
1,000 ±2.1 points Small regressions, some slice-level

For an unpaired comparison of two versions, the standard error of the difference is larger by a factor of about 1.4 (the square root of 2). Detecting a 3-point drop with unpaired runs at this pass rate needs about 2 x 0.113 x (1.96 / 0.03)² ≈ 965 cases. Pairing helps a lot when most cases pass or fail consistently across versions, which is the usual situation, so a paired comparison on 600 cases is a reasonable full set.

The conclusion for the smoke set: 60 cases can't tell you that quality dropped by 2 points, so do not ask them to. Ask them to catch breakage, plus any failure on a critical case.

Repeating each case (3 runs on the smoke set, 5 on the full set) handles a different noise source: run-to-run variation on the same case. It does nothing about having too few cases, so you need both.

Gate rules that hold up#

A single mean > 0.8 threshold has two problems. It passes a release that drops from 0.95 to 0.81, and it fails a release that sits at 0.79 because the baseline was 0.79 all along. I use three rules instead:

  1. Overall, paired. Fail if the 95% CI of the mean per-case difference (candidate minus baseline) sits entirely below zero.
  2. Per slice. Fail if any slice (refunds, billing, safety) drops by more than a fixed limit. This catches a regression that the overall mean hides because another slice improved.
  3. Critical cases. Fail if any case marked critical fails on any run. No statistics here: a model that leaks another customer's data once in three runs is not shipped.

A runnable eval gate#

This script is standard library only. The model is a stub: a seeded random function that returns the right answer with some probability, so you can run it with no API key and see the gate pass and fail. Seeding by config and run index means each repeat gives a different draw, the way sampling does, but CI runs are reproducible. In a real repo you replace stub_model with your app's entry point and grade with your checks or judge.

"""Eval gate for CI: run a stubbed LLM app on a fixed case set, compare to the
last release's baseline, exit 1 on regression. Standard library only."""
import argparse, json, math, random, statistics, sys

# --- 1. The case set. In a real repo this is evals/cases.jsonl, versioned in git.
SLICES = {"refund": 24, "billing": 24, "safety": 12}
CASES = []
for slice_name, count in SLICES.items():
    for i in range(count):
        CASES.append({"id": f"{slice_name}-{i:02d}", "slice": slice_name,
                      "expected": f"ANSWER-{slice_name}-{i}",
                      "critical": slice_name == "safety",
                      "difficulty": (i % 4) * 0.08})   # 0.00 .. 0.24

# --- 2. The "release config": everything that can change behaviour.
CONFIGS = {
    "v1":      {"prompt": "support-v1", "model": "stub-2026-01", "slice_penalty": {}},
    "v2-good": {"prompt": "support-v2", "model": "stub-2026-01", "slice_penalty": {"refund": -0.05}},
    "v2-bad":  {"prompt": "support-v2", "model": "stub-2026-01", "slice_penalty": {"billing": 0.30}},
}

def stub_model(case, config, run):
    """Stands in for the real LLM call. Seeded so CI is reproducible, but each
    run index gives a different draw, like sampling at temperature > 0."""
    rng = random.Random(f"{config['prompt']}|{config['model']}|{case['id']}|{run}")
    p_wrong = 0.04 + case["difficulty"] + config["slice_penalty"].get(case["slice"], 0.0)
    if case["critical"]:
        p_wrong = 0.0 if config["prompt"] == "support-v1" else config["slice_penalty"].get("safety", 0.0)
    return "I am not sure." if rng.random() < max(p_wrong, 0.0) else case["expected"]

def grade(case, output):
    return 1.0 if case["expected"] in output else 0.0

def run_suite(config, repeats):
    """Per-case score = mean over `repeats` runs, so one unlucky sample counts less."""
    return {c["id"]: statistics.mean(grade(c, stub_model(c, config, r)) for r in range(repeats))
            for c in CASES}

def paired_ci(base, cand, z=1.96):
    """Mean of per-case differences with a normal-approximation 95% CI.
    Pairing on the same cases removes most of the case-difficulty noise."""
    diffs = [cand[k] - base[k] for k in base]
    mean = statistics.mean(diffs)
    se = statistics.stdev(diffs) / math.sqrt(len(diffs)) if len(diffs) > 1 else 0.0
    return mean, mean - z * se, mean + z * se

def gate(base, cand, slice_drop_limit=0.10):
    failures = []
    mean, lo, hi = paired_ci(base, cand)
    print(f"overall: baseline {statistics.mean(base.values()):.3f}  candidate "
          f"{statistics.mean(cand.values()):.3f}  diff {mean:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")
    if hi < 0:
        failures.append("overall score is lower and the whole CI is below zero")
    for s in SLICES:
        ids = [c["id"] for c in CASES if c["slice"] == s]
        d = statistics.mean(cand[i] for i in ids) - statistics.mean(base[i] for i in ids)
        print(f"  slice {s:<8} diff {d:+.3f}")
        if d < -slice_drop_limit:
            failures.append(f"slice '{s}' dropped {d:+.3f} (limit -{slice_drop_limit})")
    for c in CASES:
        if c["critical"] and cand[c["id"]] < 1.0:
            failures.append(f"critical case {c['id']} failed at least one run")
    return failures

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--config", required=True, choices=CONFIGS)
    ap.add_argument("--repeats", type=int, default=3)
    ap.add_argument("--baseline", default="baseline.json")
    ap.add_argument("--write-baseline", action="store_true")
    args = ap.parse_args()

    cfg = CONFIGS[args.config]
    print(f"config {args.config}: prompt={cfg['prompt']} model={cfg['model']} "
          f"cases={len(CASES)} repeats={args.repeats} calls={len(CASES) * args.repeats}")
    cand = run_suite(cfg, args.repeats)
    if args.write_baseline:
        with open(args.baseline, "w") as f:
            json.dump({"config": args.config, "scores": cand}, f, indent=1)
        print(f"baseline written: mean {statistics.mean(cand.values()):.3f}")
        return 0
    with open(args.baseline) as f:
        base = json.load(f)["scores"]
    failures = gate(base, cand)
    for msg in failures:
        print("FAIL:", msg)
    print("GATE:", "FAILED" if failures else "PASSED")
    return 1 if failures else 0

if __name__ == "__main__":
    sys.exit(main())

Running it on Python 3, first to record the last release as the baseline, then for two candidates:

$ python3 eval_gate.py --config v1 --write-baseline
config v1: prompt=support-v1 model=stub-2026-01 cases=60 repeats=3 calls=180
baseline written: mean 0.872

$ python3 eval_gate.py --config v2-good; echo "exit $?"
config v2-good: prompt=support-v2 model=stub-2026-01 cases=60 repeats=3 calls=180
overall: baseline 0.872  candidate 0.861  diff -0.011  95% CI [-0.086, +0.063]
  slice refund   diff +0.042
  slice billing  diff -0.069
  slice safety   diff +0.000
GATE: PASSED
exit 0

$ python3 eval_gate.py --config v2-bad; echo "exit $?"
config v2-bad: prompt=support-v2 model=stub-2026-01 cases=60 repeats=3 calls=180
overall: baseline 0.872  candidate 0.728  diff -0.144  95% CI [-0.231, -0.057]
  slice refund   diff -0.028
  slice billing  diff -0.333
  slice safety   diff +0.000
FAIL: overall score is lower and the whole CI is below zero
FAIL: slice 'billing' dropped -0.333 (limit -0.1)
GATE: FAILED
exit 1

The v2-good run is the one I learn from. I gave it no billing penalty at all, so its true billing quality equals the baseline's, yet billing came out 6.9 points lower. That is pure sampling noise on 24 cases with 3 repeats each. The slice limit of 10 points sits only a little above the noise, which means a slice rule on a smoke set this small will sometimes fail a good change. You have two honest options: raise the limit on small slices, or grow the slices. I would rather grow them for the slices that matter and keep loose limits on the rest.

The overall CI on the good run is about ±7.5 points, close to the table above. The smoke set saw a 14-point drop clearly and would have missed a 3-point one.

Comparing against the last release#

The baseline is the scored output of the release currently in production, stored as an artifact next to its manifest (eval_baseline in the manifest above). Three details matter.

First, store a score per case ID, so the paired test works and the report can list cases that flipped from pass to fail. Five flipped cases tell a reviewer more than "0.872 to 0.861".

Second, refresh the baseline. A baseline scored three weeks ago against a provider endpoint that has since changed is not a fair comparison. When the nightly full run on main is clean, re-score the production manifest on the same night and store that as the baseline, so both sides of the comparison saw the same provider behaviour.

Third, when someone adds cases, the old baseline has no scores for them. Compare only on case IDs present in both and report new cases separately until the next baseline includes them.

For scorers, I use code checks (JSON schema, citations present, right tool arguments) first and an LLM judge only where code can't decide. Pin the judge's model and prompt in the manifest too. My agent evaluation walkthrough covers scorers in more depth.

A sample GitHub Actions workflow#

This is untested config. I wrote it against the GitHub Actions workflow syntax docs but have not run it in a repository. It assumes the gate script lives at evals/eval_gate.py, has been extended with a --suite flag, and that the API key is stored as a repository secret.

# .github/workflows/llm-evals.yml  (UNTESTED: adapt before use)
name: llm-evals
on:
  pull_request:
    paths: ["prompts/**", "src/**", "tools/**", "release.yaml", "evals/**"]
  schedule:
    - cron: "30 20 * * *"      # 02:00 IST nightly full run
  workflow_dispatch:

concurrency:
  group: evals-${{ github.ref }}
  cancel-in-progress: true      # a new push cancels the stale run

jobs:
  smoke:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python evals/eval_gate.py --suite smoke --repeats 3 --baseline evals/baselines/current.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

  full:
    if: github.event_name != 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python evals/eval_gate.py --suite full --repeats 5 --baseline evals/baselines/current.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The gate script's exit code is what blocks the merge, once you mark the smoke job as a required check in branch protection. Index rebuilds do not open PRs, so the nightly run (or a dispatch from the index pipeline) is what tests them.

After the gate: shadow, canary and online metrics#

Offline evals test the questions you wrote down. Production asks the others.

A shadow run sends a copy of live requests to the new manifest and logs its outputs without showing them to users. It costs a second model call per mirrored request, so mirror a sample. Use it to diff length, refusal rate, tool-call patterns, cost and latency. It can't measure user reaction, and a shadow agent must run against read-only or stubbed tools, or it will send real emails twice.

A canary shows the new version to a small share of real users. The Google SRE workbook chapter on canarying describes the general method: compare the canary population against a control population on the same metrics over the same period, and roll forward in steps. For LLM apps, the metrics I watch are:

Metric Why Alarm example
Thumbs-down or "not helpful" rate Direct user signal, sparse Up 1 point vs control
Escalation to human / retry rate Users who did not get an answer Up 2 points
Tool error rate Schema or argument breakage Any rise above control noise
Refusal rate Over-cautious prompt or model Up 2x
Guardrail trigger rate Safety drift either direction Big change either way
p95 latency and tokens per request Cost and user experience Up 20%

Canary math decides how long to wait. Say the app serves 20,000 requests a day and the canary gets 5%, so 1,000 requests a day against 19,000 in control, with a 4% thumbs-down baseline. The standard error of the difference after one day is sqrt(0.04 x 0.96 / 1,000 + 0.04 x 0.96 / 19,000) ≈ 0.0064, so the 95% interval is about ±1.25 points. One day can't reliably detect a 1-point rise. After three days (3,000 canary requests) the interval shrinks to about ±0.72 points, which can. So either wait three days at 5%, or move to 25% sooner if the tool error and refusal rates look clean. For the logging that feeds these metrics, see my post on guardrails and observability, and for cost per request, the cost and latency tracker.

Rollback#

With a release manifest, rollback is a config change: point the router back at the previous manifest, with no rebuild if the code supports both. I aim for under five minutes and rehearse it once a quarter.

Some things do not roll back by flipping a pointer:

  • A deleted index. If you dropped support-kb-v37 after building v38, there is nothing to roll back to. Keep old indexes until the new release has run clean for a set period.
  • State the new version wrote. If the new version wrote user memories or summaries in a new format, the old version may misread them. Version the stored format and keep old readers working.
  • Side effects. Emails sent, tickets created and refunds issued stay done. Gate tool-using changes harder than chat-only ones.
  • A retired model. If the old manifest points at a retired model ID, rolling back to it fails. Your rollback target has to be a release whose model still answers.

Model deprecation upgrades#

A forced model upgrade is the biggest change most LLM apps go through, and it arrives on the provider's schedule. I run it like any other release, with a longer shadow phase:

  1. When the deprecation notice arrives, create a manifest that changes only model.id and run the full suite. Look at the flipped cases before touching any prompt.
  2. Fix prompts for the new model in separate commits, each gated by the full suite, so you know which change fixed what.
  3. Check token counts and cost, not only quality. Anthropic's pricing page notes that Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text. Same price per token with more tokens per request is a cost increase, and it also eats into context budgets you sized for the old tokenizer.
  4. Shadow for longer than usual, then canary.
  5. Finish well before the retirement date, because the old model is your rollback target only until it is retired. Teams coming from classic ML, where they owned the weights, meet this problem for the first time here; the MLOps vs LLMOps post covers the shift.

Failure modes I plan for#

  • The gate is too slow or too flaky, so people skip it. Keep the smoke run under five minutes and its false-failure rate low. A gate that fails good changes every week gets an "override" label, and then it is decoration.
  • Thresholds tuned on noise. A slice limit tighter than the slice's sampling noise fails good changes at random, as the v2-good run shows. Measure the noise by re-running the baseline against itself with fresh seeds before picking limits.
  • Eval set overfitting. If every prompt tweak is tuned until the smoke set passes, the smoke set stops predicting production. Hold out part of the full set that no one tunes against.
  • Untested data changes. Index rebuilds and new document batches ship without a PR. Trigger the full suite from the data pipeline.
  • Judge drift. An unpinned judge model upgrade changes scores for every version at once. Pin it and re-baseline when you change it.

Decision table#

Situation What I would do
Prompt edit, small wording change Smoke set on the PR, full set nightly, ship with canary
Prompt restructure or new few-shot examples Smoke on PR plus full suite before release
Model version change (planned or forced) Full suite, separate prompt-fix commits, 2-3 days shadow, staged canary
Index rebuild or new embedding model Retrieval evals and full answer evals from the data pipeline, keep old index
New tool or tool schema change Unit tests, trajectory evals, canary with stubbed side effects first
Guardrail rule change Safety slice must pass 100%, plus a false-block check on normal traffic
Pure code refactor, no behaviour change intended Unit tests and smoke set; a smoke diff is a bug signal
Small team, under 100 labelled cases Run everything on every PR, accept wide intervals, grow the set from production failures
Online metric worse during canary Roll back first, investigate after

If you only build one piece this week, build the release manifest and the per-case baseline file. Everything else in this post depends on knowing exactly what shipped and how it scored. For more project ideas in this area, see LLMOps projects that teach production AI.

Sources#