GyaanStack. Blog

Articles / projectsllmopsportfolio

10 LLMOps projects that teach you production AI

Ten projects about running LLM systems, not just building them, from prompt registries and model routers to semantic caches and shadow deployments.

By Raghunath · · 7 min read

Most LLM portfolio projects stop at "it works on my laptop". Interviews for production roles ask what happens after that: how you ship a prompt change safely, what a request costs, what happens when the provider goes down, and how you know quality hasn't dropped.

These ten projects are about that second part. Each one takes a small LLM app you already have (a RAG bot, an extractor, an agent) and adds one piece of the machinery around it.

If you need the app first, 12 AI engineering projects that prove you can ship has ideas. I've left out projects that list already covers, such as an eval suite in CI, a cost tracker and a serving benchmark.

Each project below has:

  • Build: what you make.
  • Proves: what an interviewer learns about you.
  • Stack: one reasonable choice. Swap freely.
  • Difficulty: beginner, intermediate or advanced.
  • Twist: the extra step that makes it stand out.

1. A prompt registry with versions and rollback#

Build: Move every prompt out of your code into versioned files (YAML or a small database table). The app loads prompts by name and version. Every response is logged with the prompt version that made it. A CLI command promotes a version to production or rolls it back, with no redeploy.

Proves: You treat prompts as code and can undo a bad change in seconds.

Stack: Python, YAML or SQLite, a small CLI (Typer or argparse), your existing app.

Difficulty: Beginner

Twist: Run your eval set automatically when a prompt version is promoted, and refuse the promotion if the pass rate drops.

Learn it: Evals & LLMOps

2. A model router with a quality guard#

Build: A layer that sends each request to a small cheap model or a larger one. Start with simple rules (input length, task type), then try a small classifier. Log which model handled each request.

Proves: You can cut cost without guessing about quality.

Stack: Python, two models (for example a small and a large one from the same provider, or a local model and an API), your eval set.

Difficulty: Intermediate

Twist: Run your eval set through the router and through the large model only. Publish both pass rates and costs in one table, so the trade-off is a number, not a claim.

Learn it: Inference Engineering

3. A semantic cache with a staleness rule#

Build: Before calling the model, embed the question and look for a close match among recent answered questions. If one is close enough, return the cached answer. Add a time-to-live and a way to clear entries when the source documents change.

Proves: You understand caching trade-offs: hit rate against the risk of a wrong cached answer.

Stack: An embedding model, a vector store (FAISS, pgvector or Redis), your RAG app.

Difficulty: Intermediate

Twist: Find the similarity threshold with data. Label 100 question pairs as "same answer" or "different answer" and pick the threshold that keeps false hits under a rate you choose.

Learn it: RAG

4. An LLM gateway with retries, limits and a fallback provider#

Build: A small HTTP service that every app calls instead of calling the provider directly. It handles API keys, per-user rate limits, retries with backoff, timeouts, and a switch to a second provider or local model when the first fails.

Proves: You've thought about what happens when the dependency you can't control misbehaves.

Stack: FastAPI, httpx, Redis for rate limits, two model providers or one provider plus Ollama.

Difficulty: Intermediate

Twist: Write a test that simulates the main provider returning errors for 60 seconds, and show in a chart that requests kept succeeding through the fallback.

Learn it: MLOps

5. End-to-end tracing for an agent#

Build: Add a span for every step of an agent run: each model call, each tool call, each retrieval. Record inputs, outputs, latency and token counts. View the traces in a tracing UI.

Proves: You can debug a multi-step system by finding the step that failed, not by rereading prompts.

Stack: OpenTelemetry, Jaeger or an LLM tracing tool (Langfuse, Phoenix), your agent.

Difficulty: Intermediate

Twist: Take five failed runs from your eval set, find the failing step in each trace, and write one paragraph per failure on what went wrong and what you changed.

Learn it: AI Agents

6. Online evals on sampled production traffic#

Build: A daily job that samples a fraction of logged requests, scores them with code checks and an LLM judge (faithfulness, format, refusal rate), and writes the scores to a table and a simple dashboard.

Proves: You know offline evals aren't enough, and you can watch quality in production.

Stack: Python, a scheduler (cron or a workflow tool), SQLite or Postgres, a small dashboard (Streamlit or Grafana).

Difficulty: Advanced

Twist: Check your judge first. Hand-label 50 samples and report how often the judge agrees with you. Only then trust the dashboard.

Learn it: Evals & LLMOps

7. A shadow deployment for a new model#

Build: When you want to try a new model or prompt, send real requests to both the current and the new version. Only the current one's answer reaches the user. Log both and compare them with your eval checks.

Proves: You can test a change on real traffic without risking users.

Stack: Your gateway from project 4 (or a small proxy), async calls, a comparison report.

Difficulty: Advanced

Twist: Add a promotion rule: the new version goes live only if it matches or beats the current one on agreed checks across at least a few hundred requests, and the report says so.

Learn it: MLOps

8. A PII guard with a latency budget#

Build: A filter that finds and masks personal data (names, emails, phone numbers, ID numbers) in user input before it reaches the model, and checks model output before it reaches the user.

Proves: You can add a safety layer and measure what it costs.

Stack: Microsoft Presidio or regex plus a small NER model, your gateway or app.

Difficulty: Intermediate

Twist: Set a latency budget (for example 30 ms at p95) and measure against it. Report the guard's precision and recall on a small labelled set, including Indian formats such as phone numbers and PAN-style IDs.

Learn it: AI Security

9. A batch inference pipeline with checkpoints and a cost cap#

Build: A job that runs an LLM over thousands of records (classify support tickets, extract fields from invoices). It saves progress as it goes, resumes after a crash without redoing finished records, and stops when it reaches a spending limit.

Proves: You can run LLM work at volume without losing money or progress.

Stack: Python, asyncio with a concurrency limit, SQLite for progress, a provider's batch API or a local model.

Difficulty: Intermediate

Twist: Compare three ways to run it (one at a time, concurrent requests, the provider's batch endpoint if it has one) on total time and total cost.

Learn it: Inference Engineering

10. A feedback loop that grows your eval set#

Build: Add thumbs up and down to your app's answers. Store each vote with the full trace. A weekly script pulls the thumbs-down cases into a review queue, and the ones you confirm as real failures become new eval cases.

Proves: You can close the loop from user complaints to regression tests.

Stack: Your app's UI, SQLite or Postgres, a small review script or page, your eval harness.

Difficulty: Beginner to intermediate

Twist: Version the eval set itself, and show a chart of eval set size and pass rate over time as you fix the failures it catches.

Learn it: Evals & LLMOps

How these fit together#

You don't need all ten. Three that connect tell a better story than ten loose demos. Two combinations that work well:

  • Ship safely: prompt registry (1) + shadow deployment (7) + feedback loop (10). Story: "Here's how a change goes from idea to production, and how production failures come back as tests."
  • Run it cheaply and reliably: gateway (4) + router (2) + semantic cache (3). Story: "Here's what a request costs, how I cut it, and what happens when the provider goes down."

What to put in the README#

For each project, write:

  • The problem in two sentences.
  • A diagram of where your piece sits in the request path.
  • The numbers: before and after, with how you measured them.
  • What broke while you built it and how you fixed it.
  • How to run it in one command.

The numbers matter most. "Routing cut cost per request by a measured amount while the eval pass rate stayed within one point" is the kind of line that turns a portfolio project into an interview conversation. Use your own measured numbers, whatever they turn out to be.

For the ideas behind these projects in a learning order, see MLOps vs LLMOps: what changes and what to learn first.