Articles / llmopsgenai-system-designroadmap
The LLM tech stack in 2026: every layer, the real options, and how to choose
The eight layers of a production LLM app in late 2026, from models and serving to MCP, evals, guardrails and gateways, with current options and how to pick.
Most "LLM tech stack" lists from 2024 and early 2025 read like a pile of logos: a model, LangChain, a vector database, Docker, done. A production app in late 2026 has more layers than that, and the ones that cause outages and surprise bills (the gateway, evals, guardrails) are usually missing from those lists. Some tools have also changed status. Hugging Face's TGI is now in maintenance mode, Microsoft folded Semantic Kernel and AutoGen into a single Agent Framework, and MCP became the default way to connect a model to tools.
This post walks the stack layer by layer. For each one: what the layer does, two to four options that are current as of November 2026, and how to choose between them. Prices and benchmark scores change monthly, so there are none here; follow the links for those.
The stack on one page#
user / client app
|
+----------------------v-----------------------+
| 8. Gateway: routing, fallbacks, keys, budgets |
+----------------------+-----------------------+
|
+----------------------v-----------------------+
| 7. Guardrails: input and output checks |
+----------------------+-----------------------+
|
+----------------------v-----------------------+ +----------------------+
| 4. Orchestration / agent framework |<----->| 5. Protocols |
| (loop, state, tool calls, memory) | | MCP (tools/data) |
+-----------+----------------------+------------+ | A2A (agent-agent) |
| | +----------------------+
+-----------v-----------+ +-------v--------------+
| 3. Retrieval | | 1. Models / APIs |
| parsers, embeddings, | | hosted frontier |
| vector DB, reranker | | or open weights |
+-----------------------+ +-------+--------------+
|
+-------v--------------+
| 2. Serving engine |
| vLLM, SGLang, Ollama|
+----------------------+
6. Evals + observability run across every box above (traces, scores, cost)
The numbers are the order this post covers them, not the order a request flows through. A request goes top to bottom. Evals and tracing sit beside everything because a trace that stops at the gateway tells you nothing about why retrieval returned the wrong chunk.
1. Models and APIs#
What it does. Generates text, calls tools and returns structured output. Everything else in the stack exists to feed this layer good context and to check what comes out.
Options.
- Hosted frontier APIs: OpenAI, Anthropic Claude and Google Gemini. The same models also show up inside cloud platforms such as AWS Bedrock, Azure AI Foundry and Google Vertex AI, which matters when your company already has a cloud contract and a data-residency rule.
- Open-weight families you run yourself: Llama, Qwen, Gemma, Mistral, DeepSeek and OpenAI's gpt-oss. Licences differ between families, so read each model card before you ship.
How to choose. Start with a hosted API for the first version. You learn what the task needs (context length, tool calling quality, latency) before you take on GPUs. Move a workload to open weights when one of three things is true: the data cannot leave your network, a narrow high-volume task (classification, extraction, routing) runs fine on a small model, or per-token cost at your volume beats renting GPUs. Measure that last one with your own traffic, not with a blog's estimate.
The use-versus-fine-tune-versus-train question from older stack guides still applies, but the default answer has moved further toward "use and prompt well". Training from scratch is a foundation-lab activity. Fine-tuning is for tone, format or a narrow domain once prompting and retrieval have stalled; fine-tuning vs RAG vs prompting and the LoRA and QLoRA guide cover that decision. The usual tools there are Hugging Face PEFT and Unsloth.
2. Serving engines for open weights#
What it does. Loads model weights onto hardware and serves requests efficiently: batching many users onto one GPU, managing the KV cache, streaming tokens back. If you only call hosted APIs, you can skip this layer.
Options.
- vLLM: the common default for GPU serving. It exposes an OpenAI-compatible server, and its README now lists an Anthropic Messages API and gRPC as well. For Kubernetes there is the vLLM production stack and the llm-d project.
- SGLang: a fast engine built around prefix caching (RadixAttention), strong on workloads with long shared prefixes such as agents and multi-turn chat. Several RL post-training frameworks use it to generate rollouts.
- Ollama and llama.cpp: local and CPU or Apple-silicon serving with quantized models. Ollama has an OpenAI-compatible endpoint, which makes it easy to develop locally and point the same client at a hosted model later. LM Studio is the desktop GUI option.
- NVIDIA TensorRT-LLM, often behind Triton Inference Server, when you are all-in on NVIDIA hardware and will pay an engineering cost for peak throughput.
What changed since 2025. Text Generation Inference is in maintenance mode. Its README now recommends vLLM, SGLang, llama.cpp or MLX instead. If an older tutorial tells you to start with TGI, start with vLLM.
How to choose. Laptop or a single developer box: Ollama. A shared GPU service: vLLM, then benchmark SGLang on your own prompts if most requests share a long system prompt or tool list. Benchmark with your real prompt lengths and concurrency; throughput numbers from other people's tests rarely transfer. The local RAG with Ollama post has a full working local setup.
3. Retrieval: parsing, embeddings, vector search, reranking#
What it does. Finds the right pieces of your own data and puts them in the prompt. This is where most RAG quality problems live, and the vector database is usually not the cause. Bad parsing and bad chunking are.
Options by sub-layer.
- Parsing: Docling and Unstructured for PDFs, slides and tables. LlamaIndex bundles loaders and chunkers.
- Embeddings: hosted embedding endpoints from the model providers, or open models such as BGE-M3 when you want them in-house or need multilingual retrieval.
- Vector storage: pgvector if you already run Postgres; Qdrant, Weaviate or Milvus for a dedicated open-source engine; Pinecone for fully managed; sqlite-vec, Chroma or FAISS for local work and prototypes.
- Reranking: a cross-encoder or a hosted reranker such as Cohere Rerank on the top 20 to 50 hits, which often fixes more ranking errors than swapping the database.
How to choose. Under a few million vectors, pgvector inside the database you already back up and monitor is hard to beat, because metadata filters and permissions stay as plain SQL. Reach for a dedicated engine when you need hybrid search tuning, multi-tenant isolation at scale, or the vector workload is hurting your transactional database. Pick the embedding model by testing recall on 50 to 100 of your own questions, not by leaderboard rank. The RAG ingestion pipeline design and RAG architectures compared posts go deeper.
4. Orchestration and agent frameworks#
What it does. Runs the loop around the model: build the prompt, call the model, execute tool calls, keep state, retry, hand off to a human. In 2026 this layer is mostly about agents, not chains.
Options.
- LangGraph: explicit graphs with checkpointed state, interrupts for human approval and durable execution. The choice when you need to see and control every step. LangChain itself now sits on top of it for higher-level agents.
- Provider SDKs: the OpenAI Agents SDK, the Claude Agent SDK and Google's Agent Development Kit. Less code to reach a working agent, tighter coupling to one vendor's features.
- Microsoft Agent Framework: the successor that combines AutoGen's agent abstractions with Semantic Kernel's state, middleware and telemetry, plus graph workflows. If a guide tells you to start a new project on Semantic Kernel or AutoGen, look here first.
- Lighter or specialised options: Pydantic AI for type-checked agents, CrewAI for role-based multi-agent setups, LlamaIndex when the app is mostly retrieval.
How to choose. Write the first version as a plain loop over the provider's tool-calling API. It is about 50 lines, and you will learn what state you need. Adopt a framework when you need something it gives you for free: checkpointing and resume (LangGraph), multi-agent handoffs, or tracing hooks. Choose the provider SDK if you are committed to one model vendor; choose a vendor-neutral framework if you expect to route between providers. LangGraph from scratch and multi-agent architecture patterns cover the details.
5. Protocols: MCP and A2A#
What it does. Standard wire formats so tools and agents can be reused across frameworks instead of rewritten for each one.
Options.
- Model Context Protocol (MCP): connects a model host to tools, data and prompts through MCP servers. Anthropic donated MCP to the Agentic AI Foundation under the Linux Foundation, and the current spec revision is 2026-07-28. Every framework in layer 4 can consume MCP servers.
- Agent2Agent (A2A): lets independent agents, possibly from different vendors, discover each other and exchange tasks.
How to choose. Wrap internal tools as MCP servers once you have more than one agent or client that needs them; the payoff is that a new agent gets your CRM or ticket tools on day one. Reach for A2A only when you really have separately owned agents talking across team or company lines. The full comparison, including ANP, is in MCP vs A2A vs ANP, and MCP from zero builds a server and client.
6. Evals and observability#
What it does. Tells you whether the app is right, how fast it is and what it costs, per request and per release. Without it, every prompt change is a guess.
Options.
- Tracing platforms: Langfuse (open source, self-hostable), LangSmith, Arize Phoenix and MLflow's GenAI features. Most of these accept OpenTelemetry, and the OpenTelemetry GenAI semantic conventions define standard attribute names for model calls, tokens and tool use.
- Eval frameworks: promptfoo for config-driven regression tests and red teaming in CI, DeepEval and Ragas for pytest-style and RAG metrics, and the UK AISI's Inspect for agent and safety evals.
How to choose. Get tracing in on day one, because it is cheap and retrofitting it is painful. Pick a platform you can self-host if traces will contain customer data. For evals, start with 30 to 100 real examples and a script that runs on every pull request; the framework matters less than having the dataset. Use LLM-as-a-judge for qualities you cannot check with code, and check the judge against human labels first. See evaluating an AI agent, LLM-as-a-judge and CI/CD for LLM applications.
7. Guardrails#
What it does. Checks inputs and outputs against rules: prompt injection, PII, off-topic requests, unsafe content, malformed JSON, actions the agent should not take without approval.
Options.
- NVIDIA NeMo Guardrails: programmable rails around a conversation, including topical and jailbreak checks.
- Guardrails AI: validators for output structure and content, with a hub of reusable checks.
- Safety classifier models such as Llama Guard 4 and the wider Purple Llama tools, which you can serve on the same vLLM cluster as your main model.
- Plain code: schema validation, allow-lists for tools, regexes for PII. These catch more real incidents than people expect and add almost no latency.
How to choose. Put deterministic checks first and model-based classifiers second, because each classifier adds a model call to your latency budget. Give the agent the fewest tool permissions it needs; a guardrail cannot fix an agent that holds a delete-all API key. Test the rails with attack prompts in CI (red teaming with prompt injection tests), and read the guardrails and observability system design post for where each check sits in the request path.
8. Deployment and gateways#
What it does. The gateway is the single front door to every model: one API shape, provider keys kept server-side, routing, fallbacks, rate limits, per-team budgets and a central log. Below it sits the ordinary deployment layer that runs your app and, if you self-host, the serving engine.
Options.
- Gateways: LiteLLM (open-source proxy with fallbacks and retries and per-key budgets), Portkey and OpenRouter as hosted options. Kubernetes shops can look at the Gateway API Inference Extension, which routes to self-hosted model servers with awareness of model load.
- App and model hosting: containers on Kubernetes with KServe or Ray Serve, serverless GPUs on Modal, or the managed endpoints in Bedrock, Azure AI Foundry and Vertex AI.
How to choose. Add a gateway as soon as more than one service or team calls a model. The first time a provider has an outage, the fallback pays for the setup. Self-host LiteLLM if you need data to stay in your network; use a hosted gateway if you do not want to run another service. For the app itself, a FastAPI service in a container is still the standard shape, and production LLM API with FastAPI and dockerize an AI agent cover streaming, retries, rate limits and packaging.
A toy gateway, to see the routing logic#
The core of a gateway is small. This script picks the cheapest model that meets a task's quality tier, falls back on a provider error, and blocks a team once its budget is used. The models are stubs with no network calls or API keys; the comment in complete() marks where a real provider call goes. The cost_weight values are made-up example weights, not real prices. Fill in real numbers from your providers' pricing pages.
from dataclasses import dataclass, field
@dataclass
class Model:
name: str
tier: int # 1 = small/fast, 2 = mid, 3 = frontier
cost_weight: float # example weight per 1K tokens, NOT a real price
fail_next: int = 0 # how many upcoming calls should fail (to simulate an outage)
def complete(self, prompt: str) -> tuple[str, int]:
# A real gateway would call the provider SDK or an OpenAI-compatible endpoint here.
if self.fail_next > 0:
self.fail_next -= 1
raise RuntimeError(f"{self.name}: 503 from provider")
tokens = len(prompt.split()) * 4 + 200 # crude stand-in for prompt + completion tokens
return f"[{self.name}] answer", tokens
@dataclass
class Gateway:
models: list[Model]
budgets: dict[str, float]
spent: dict[str, float] = field(default_factory=dict)
log: list[str] = field(default_factory=list)
def route(self, team: str, prompt: str, min_tier: int) -> str:
if self.spent.get(team, 0.0) >= self.budgets[team]:
self.log.append(f"{team}: blocked, budget used")
return "BLOCKED: budget exhausted"
candidates = sorted((m for m in self.models if m.tier >= min_tier),
key=lambda m: m.cost_weight)
for m in candidates:
try:
text, tokens = m.complete(prompt)
except RuntimeError as err:
self.log.append(f"{team}: {err}, trying next")
continue
cost = tokens / 1000 * m.cost_weight
self.spent[team] = self.spent.get(team, 0.0) + cost
self.log.append(f"{team}: {m.name} tier={m.tier} tokens={tokens} cost={cost:.2f} "
f"spent={self.spent[team]:.2f}/{self.budgets[team]}")
return text
self.log.append(f"{team}: all candidates failed")
return "ERROR: no model available"
if __name__ == "__main__":
small = Model("small-open-weight", tier=1, cost_weight=0.1)
mid = Model("mid-hosted", tier=2, cost_weight=1.0)
big = Model("frontier-hosted", tier=3, cost_weight=5.0)
gw = Gateway([big, small, mid], budgets={"support": 3.0, "research": 10.0})
gw.route("support", "Classify this ticket: refund not received after 10 days", min_tier=1)
mid.fail_next = 1 # simulate a provider outage on the mid model
gw.route("research", "Summarise these three contracts and list conflicting clauses " * 5, min_tier=2)
for _ in range(3):
gw.route("support", "Draft a reply to an angry customer about a late refund " * 3, min_tier=3)
print("\n".join(gw.log))
Output from python3 gateway.py:
support: small-open-weight tier=1 tokens=236 cost=0.02 spent=0.02/3.0
research: mid-hosted: 503 from provider, trying next
research: frontier-hosted tier=3 tokens=360 cost=1.80 spent=1.80/10.0
support: frontier-hosted tier=3 tokens=332 cost=1.66 spent=1.68/3.0
support: frontier-hosted tier=3 tokens=332 cost=1.66 spent=3.34/3.0
support: blocked, budget used
Three behaviours show up. The classification task goes to the small model because tier 1 is enough. The research task falls back from the failed mid model to the frontier model, and the caller never sees the 503. The support team overshoots its budget (3.34 against 3.0) before it is blocked, because the check runs before the call and the call's cost is only known afterwards. Real gateways have the same gap. Close it by estimating cost from max_tokens and reserving that amount before the call, then settling the difference after.
Two failure modes the toy hides. Falling back to a more expensive tier during an outage can burn a month's budget in an afternoon, so cap fallbacks per minute or fall back sideways (same tier, other provider) first. Different providers also return different tool-call and error shapes, which is the main reason to let a maintained gateway normalise them instead of writing your own.
Where teams get the stack wrong#
- Framework first, eval set never. Teams spend weeks choosing between agent frameworks and have zero labelled examples. The eval set outlives every framework choice.
- Treating the vector DB as the RAG fix. Swapping Pinecone for Qdrant rarely moves answer quality. Parsing tables correctly and adding a reranker usually does.
- No gateway until the first outage. Provider keys end up hardcoded in five services, and nobody can say which team spent what.
- Self-hosting too early. A GPU cluster adds on-call work, driver upgrades and capacity planning. Do it for data residency or proven cost savings at volume, not because it feels more serious.
- Following 2025 defaults. TGI for serving and Semantic Kernel or AutoGen for new agents both have recommended successors now.
A sensible default stack for a new project#
For a team of two to five engineers shipping its first production LLM feature in late 2026:
| Layer | Default | Upgrade when |
|---|---|---|
| Model | One hosted frontier API | Data residency, or a narrow high-volume task |
| Serving | None (hosted) or Ollama for local dev | You self-host: vLLM |
| Retrieval | pgvector + reranker | Hybrid search tuning or multi-tenant scale |
| Orchestration | Plain tool-calling loop | Need checkpoints or approvals: LangGraph |
| Protocols | MCP servers for shared tools | Cross-team agents: A2A |
| Evals / tracing | Langfuse + a pytest eval set in CI | Larger red-team suite: promptfoo or Inspect |
| Guardrails | Schema checks, tool allow-lists | Public users: add a safety classifier |
| Gateway | LiteLLM proxy | Many teams or providers: hosted gateway |
How the operational work differs from classic ML is covered in MLOps vs LLMOps, and the order to learn these layers in is in the LLM engineering roadmap for 2027. A good first project that touches most layers is a PDF question-answering bot with citations behind a FastAPI service, with traces and a 50-question eval set; the PDF Q&A bot post builds one.
Sources#
- vLLM documentation: https://docs.vllm.ai/
- SGLang repository and docs: https://github.com/sgl-project/sglang, https://docs.sglang.ai/
- Text Generation Inference README (maintenance-mode notice): https://github.com/huggingface/text-generation-inference
- Ollama OpenAI compatibility: https://docs.ollama.com/api/openai-compatibility
- llama.cpp: https://github.com/ggml-org/llama.cpp
- NVIDIA TensorRT-LLM: https://github.com/NVIDIA/TensorRT-LLM
- llm-d: https://llm-d.ai/
- Model Context Protocol versioning: https://modelcontextprotocol.io/specification/versioning
- Anthropic, donating MCP to the Agentic AI Foundation: https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation
- Agent2Agent protocol: https://a2a-protocol.org/
- Microsoft Agent Framework overview: https://learn.microsoft.com/en-us/agent-framework/
- LangGraph docs: https://docs.langchain.com/oss/python/langgraph/overview
- OpenAI Agents SDK: https://openai.github.io/openai-agents-python/
- Claude Agent SDK: https://docs.claude.com/en/docs/agent-sdk/overview
- Google Agent Development Kit: https://google.github.io/adk-docs/
- pgvector: https://github.com/pgvector/pgvector
- OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/
- Langfuse: https://langfuse.com/
- Inspect (UK AISI): https://inspect.aisi.org.uk/
- NeMo Guardrails: https://github.com/NVIDIA/NeMo-Guardrails
- Llama Guard 4 model card: https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-4/
- LiteLLM proxy reliability and budgets: https://docs.litellm.ai/docs/proxy/reliability, https://docs.litellm.ai/docs/proxy/users
- Gateway API Inference Extension: https://gateway-api-inference-extension.sigs.k8s.io/