MLOps vs LLMOps: what changes and what to learn first
What carries over from MLOps to LLMOps, what is new (prompts as code, evals, guardrails, token cost, routing) and an order to learn it in.
If you already do MLOps, LLMOps can sound like a rebrand. Some of it is: you still version things, deploy things, monitor things and roll back when they break. But the thing you operate has changed, and that moves where the work is.
In classic MLOps, you train the model, so most of the effort goes into data pipelines, training runs and model registries. In LLMOps, someone else usually trains the model. Your effort moves to what surrounds it: prompts, retrieval, tools, evals, guardrails and the bill.
This post covers what carries over, what's new, and an order to learn it in.
The short comparison#
| MLOps | LLMOps | |
|---|---|---|
| What you change most often | Training data, features, model weights | Prompts, retrieval settings, tools, the model you call |
| Where the model comes from | You train it | Usually an API or an open model you serve |
| Unit of release | A trained model artifact | A prompt + model + retrieval config bundle |
| How you test | Holdout set, accuracy/AUC/RMSE | Golden sets, code checks, LLM-as-judge, human review |
| Output | A number or a class | Free text, JSON, tool calls |
| Main cost driver | Training compute, serving instances | Tokens per request, GPU time if self-hosted |
| Main failure you watch | Data drift, accuracy decay | Wrong or unsafe answers, format breaks, cost spikes, latency |
| Security focus | Data access, model theft | Prompt injection, data leakage through context, tool misuse |
What carries over unchanged#
Most of what makes MLOps work still applies. If you have these skills, you are not starting from zero.
- Versioning everything. Code, data, configs. In LLMOps you add prompts and eval sets to the list.
- CI/CD. Tests on every change, staged rollouts, one-command rollback.
- Monitoring and alerting. Latency, error rates, throughput. The dashboards look familiar.
- Reproducibility. Being able to rebuild exactly what ran in production last Tuesday.
- Containers and infrastructure. Docker, Kubernetes, a cloud platform, infrastructure as code.
- Data pipelines. RAG needs ingestion, cleaning, deduplication and scheduled refresh. That's a data pipeline.
- Experiment tracking. Comparing runs side by side, now for prompts and retrieval settings rather than hyperparameters.
What's new in LLMOps#
Prompts are code#
A prompt change can break production as easily as a code change. Treat prompts the same way: keep them in version control or a prompt registry, review changes, tie each production response to the prompt version that produced it, and be able to roll back one prompt without redeploying the app.
Evals replace a single accuracy number#
There is rarely one metric. You check different things with different tools:
- Code checks: is the JSON valid, was the right tool called, does the answer cite a source.
- Model-graded checks: is the answer faithful to the retrieved text, is it relevant. These need their own validation against human labels.
- Human review: a sample of real traffic, read regularly.
The eval set is a first-class artifact. It grows every time production shows you a new failure.
Guardrails#
Inputs and outputs need checks that classic models didn't: PII redaction, topic limits, prompt injection detection, output format enforcement. Each one adds latency, so they become part of your performance budget.
Token cost is a live metric#
With an API model, every request has a price that depends on prompt length, output length and which model you call. A longer system prompt or a chattier answer shows up on the bill the same day. You track cost per request and per feature, and you set budgets and alerts like you would for latency.
Model routing and fallbacks#
You often use more than one model: a small cheap one for easy requests, a larger one for hard ones, and a second provider for when the first is down or rate-limited. Deciding which request goes where, and checking that quality holds, is new operational work.
Non-determinism#
The same input can give different outputs. Tests need to run more than once and report a pass rate, not a single pass or fail. Regressions show up as a drop from 92% to 85%, not as a red test.
Tracing multi-step systems#
An agent or RAG request is many steps: retrieve, rerank, call the model, call a tool, call the model again. When an answer is wrong, you need a trace of every step with its inputs and outputs to find which step failed. Plain request logs are not enough.
What matters less#
- Training pipelines and feature stores. Unless you fine-tune, you won't run many training jobs.
- Model registries for your own weights. You still track which model version you call, but often that's a string in a config.
- Classic drift detection on input features. You still watch for change in what users ask, but it's closer to topic monitoring than statistical feature drift.
These don't disappear. If you fine-tune or serve your own models, they come back. They just stop being the center of the job.
The order I'd learn it in#
If you come from MLOps, start at step 2. If you come from software engineering, start at step 1.
- Foundations you'd need for either. Git, Docker, CI, one cloud, basic monitoring. Library: MLOps.
- Build one LLM app end to end. A small RAG bot or a tool-using agent. You can't operate something you haven't built. See the LLM engineering roadmap for what to build.
- Evals. Golden set, code checks, an LLM judge you've checked against your own labels, and a command that runs them all. This is the biggest new skill. Library: Evals & LLMOps. A worked example: Evaluating an AI agent in Python.
- Prompt and config versioning. Move prompts out of code into versioned files or a registry, and log the version with every response.
- Tracing and observability. Add traces with per-step inputs, outputs, latency and token counts. Look at real traces every week.
- Cost and latency. Track cost per request, add caching, try routing simple requests to a smaller model and measure what it does to quality.
- Guardrails and security. Input and output checks, prompt injection tests, least-privilege tools. Library: AI Security.
- Serving your own models. Only if your job needs it: inference servers, batching, quantization, GPU sizing. Library: Inference Engineering.
Signs a team needs LLMOps work now#
If two or more of these are true, the LLM feature has outgrown "a prompt in the code":
- Nobody can say which prompt version produced a bad answer a user reported.
- A model upgrade went out without anyone running a test set first.
- The monthly model bill surprised someone.
- Debugging a wrong answer means reading code and guessing, because there are no step-by-step traces.
- The only quality check is someone trying a few questions before release.
Each one maps to a step in the learning order above: versioning, evals, cost tracking, tracing.
Which path should you pick?#
You don't have to pick one. LLMOps work draws on MLOps, data engineering and backend skills, so people from any of those backgrounds already have a large part of it.
A rough guide:
- If your team trains and serves its own models for ranking, forecasting or fraud, MLOps is still the core job, and LLM work is an add-on.
- If your team builds products on top of hosted models, the work is mostly LLMOps: evals, prompts, retrieval, cost, safety.
- If you're starting out, learn the shared base (step 1) and build one LLM app. That covers the overlap and lets you move either way.
The skill that transfers best is the habit of measuring before and after every change. That was true in MLOps, and it matters more in LLMOps, where outputs vary and a change that looks fine on five examples can fail on the sixth.