GyaanStack. Blog

Articles / genai-system-designagentsraginterview

Agentic system design interviews: five designs, from RAG pipelines to coding agents

Five AI system design interview questions worked through: RAG ingestion, RAG query, a coding agent, a support agent and a text-to-SQL agent.

By Raghunath · · 14 min read

Classic system design rounds ask you to design a URL shortener or a chat app. AI engineering loops can add a second kind of question: design a system where an LLM sits in the middle and makes decisions. The building blocks you already know still apply (queues, caches, databases, rate limits). What changes is that one component is slow, costs money per call, and is sometimes wrong in ways that look right.

Below are five designs I'd prepare for, worked the way I'd talk through them at a whiteboard. Each one covers what to clarify, the components, how data moves, the decisions that carry the most weight, how you'd measure it, and where it breaks. The five share a lot, so the later ones refer back to the earlier ones.

A frame that works for all five#

I use the same six steps every time, in this order:

  1. Clarify. Who uses it, how many, how fast must it answer, what happens when it's wrong.
  2. Draw the boring parts first. Ingestion, storage, APIs, auth. The LLM goes in last.
  3. Place the model calls. Mark each one with its input size, expected latency and cost.
  4. Name the risky actions. Anything that writes, spends money or reaches a customer gets a check.
  5. Say how you'd measure it. An eval set before launch, metrics after.
  6. Walk one failure. Pick the most likely way it goes wrong and trace it through your design.

If you're short on time, don't drop step 5. Saying how you'd measure the system is what shows you've run one.

1. RAG ingestion pipeline#

The prompt usually sounds like: "Design the system that takes a company's documents and makes them searchable for an LLM assistant."

Clarify first#

  • How many documents, and how fast do they change? 50,000 PDFs that update monthly is a different system from a wiki with 2,000 edits a day.
  • Which formats: PDF, HTML, Confluence, Slack, scanned images?
  • Do users have different permissions? If two employees can see different documents, the index has to know that.
  • How fresh must answers be? Minutes or a day?

Components#

sources ──> connectors ──> raw store (S3) ──> parse/clean ──> chunk ──> embed ──> vector + keyword index
                │                                                   │
                └── change events (webhooks / polling) ─────────────┘
                                         metadata DB: doc_id, version, hash, ACL, source_url
  1. Connectors per source, each emitting "document changed" events into a queue.
  2. Raw store keeps the original file, so you can re-parse later without re-downloading.
  3. Parser turns files into clean text with structure (headings, tables, page numbers).
  4. Chunker splits text and attaches metadata.
  5. Embedder workers, batched, with retries.
  6. Index: a vector store plus a keyword (BM25) index, both keyed by chunk ID.
  7. Metadata DB holds document versions, content hashes and access lists.

The decisions that matter#

Chunking. I start with chunks that follow document structure (a section or a few paragraphs, a few hundred tokens) with a small overlap, and keep the section heading inside each chunk. Fixed-size chunks that cut a table in half are the usual cause of "the answer was in the docs but the bot didn't find it." Tables often deserve their own chunk type.

Refresh and dedupe. Store a content hash per document and per chunk. When a document changes, re-chunk it, compare hashes, and only re-embed chunks that changed. Delete chunks for removed documents in the same transaction that writes the new ones, so the index never holds two versions of the same policy. Near-duplicate documents (the same PDF uploaded to three folders) need a dedupe step, or retrieval returns three copies of one paragraph.

Access control. Copy each document's ACL onto its chunks and filter at query time. Filtering after retrieval (fetch 20, drop the ones the user can't see) can leave you with zero results; filtering inside the vector query is better. When a permission changes, you need to update chunk metadata without re-embedding.

Re-embedding. Changing the embedding model means re-embedding everything. Keep the model name and version on every vector, build the new index alongside the old one, and switch reads over when it's complete.

Measure it#

  • Ingestion lag: time from "document changed" to "searchable", per source.
  • Parse failures per format.
  • A retrieval eval set: 50 to 200 questions with the chunk that should come back. Re-run it after any chunking or embedding change.

Where it breaks#

Scanned PDFs with no text layer, connectors that silently stop syncing, and a re-embed job that half-finishes. For the last one, version the index and only switch when the new version passes the eval set.

2. RAG query system#

"Users ask questions; answer from the documents, with sources." This is the read side of design 1. My RAG cheat sheet covers each stage in more depth.

Clarify first#

  • Queries per second at peak, and the latency target (I'd assume under 3 to 5 seconds to first token for a chat UI).
  • Must every answer cite a source? Usually yes.
  • What should happen when the documents don't contain the answer?

Components#

user ──> API ──> query rewrite ──> hybrid retrieval (vector + BM25, ACL filter) ──> rerank ──> prompt build ──> LLM ──> citation check ──> user
                    │                                                                                          │
                    └──────────────────── semantic cache (normalised query + ACL scope) <───────────────────────┘

The decisions that matter#

Hybrid retrieval. Vector search finds paraphrases; keyword search finds exact terms like error codes, product names and policy numbers. Run both, merge with reciprocal rank fusion, and pass the top 30 to 50 to a reranker. Keep the final context small (5 to 8 chunks) and put the strongest chunk first.

Query rewriting. Chat follow-ups like "what about for contractors?" need the earlier turn folded in before retrieval. A small, fast model can rewrite the query into a standalone question.

Citations. Number the chunks in the prompt, require the model to cite chunk numbers, then check after generation that every cited number exists and that each cited chunk shares key terms with the sentence citing it. If the model can't answer from the context, the prompt should tell it to say so, and the UI should show that as a normal outcome.

Caching. Cache retrieval results and full answers keyed on the normalised query plus the user's permission scope. Without the scope in the key, one user's cached answer can leak another team's document. Invalidate when any source document behind the cached answer changes.

Measure it#

What Metric
Retrieval recall@k and MRR on the eval set
Answer faithfulness (claims supported by cited chunks), answer relevance
Product thumbs down rate, "no answer" rate, time to first token
Cost tokens per answer, cache hit rate

Where it breaks#

The most common failure is retrieval missing the right chunk while the model writes a confident answer from the wrong one. The citation check and the "no answer" path are your defences. The second is latency creep: reranking 100 candidates and a long prompt can blow the budget, so set limits per stage.

3. A coding agent like Claude Code#

"Design an agent that takes a task in a code repository, edits files, runs commands and finishes the task." Public docs for agentic coding tools describe the same core shape: a loop where the model reads context, calls tools (read file, search, edit, run shell command), sees the result, and repeats until the task is done or it needs the user. I'd design from that loop outward and not claim anything about how a particular product works inside.

Clarify first#

  • Does it run on the developer's machine, in CI, or in a hosted sandbox?
  • What can it do without asking? Reading is usually fine; writing files, running commands and pushing code are the questions.
  • How large are the repositories? A 2-million-line monorepo changes the context strategy.
  • How do we know the task is done?

Components#

user task ──> agent loop ──> model call ──> tool calls ──> tool runner (sandbox) ──> results back into context
                 │                              │
                 │                              ├── read / search (grep, file tree, symbol index)
                 │                              ├── edit (diff / patch apply)
                 │                              └── shell (tests, build, linters) under a permission policy
                 │
                 ├── context manager (what stays in the window, summaries of old turns)
                 └── checkpoints (undo any edit)

The decisions that matter#

Context management. The repository doesn't fit in the window, so the agent searches for what it needs instead of loading everything. Keep project instructions (build commands, conventions) in a small file the agent always reads. When a long session fills the window, summarise older turns and keep recent tool results verbatim. Tool output needs limits too: a test run that prints 40,000 lines has to be truncated or filtered before it goes back to the model.

Edits as diffs. Have the model produce targeted edits (replace this exact text with that text, or a patch) rather than rewriting whole files. Diffs are cheaper, easier to review and easier to undo. Reject an edit when the text to replace isn't found exactly once, and send that error back to the model.

Sandboxing and permissions. Run commands in a container or a restricted environment with no production credentials. A permission policy decides per tool and per command: reads allowed, file edits allowed inside the repo, shell commands from an allowlist, everything else needs the user to approve. Network access, git push and anything touching secrets should ask. Treat text inside files and web pages as data: a README that says "ignore your instructions" is a prompt injection attempt, and the permission layer is what stops it from doing damage.

Tests as the verifier. The model saying "done" isn't proof. The loop should run the relevant tests, the type checker or the build, and feed failures back in. In an interview, say that the stopping condition is "tests pass and the diff is reviewed", not "the model stopped calling tools".

Extending the tool set. New tools (issue trackers, databases, internal APIs) can plug in through a standard protocol like MCP, so the agent loop doesn't change when you add one. I wrote a small MCP server and client if you want to see how little code that takes.

Measure it#

  • Task success rate on a fixed set of real repository tasks with tests that decide pass or fail.
  • Steps and tokens per task, and how often the agent asks for permission.
  • Bad-action rate: edits outside the requested scope, commands the policy had to block.

Where it breaks#

Loops that repeat the same failing command, edits that pass the tests by deleting the test, and context that fills with noise until the model forgets the task. Put a step budget on the loop, flag diffs that touch test files for review, and keep the original task at the top of the context.

4. An AI customer support agent#

"Design an agent that answers customer messages and can take actions like refunds." This one combines design 2 (answers from help docs) with tool calls that touch money and customer data.

Clarify first#

  • Channels (chat, email), volume, and languages.
  • Which actions: order lookup, address change, refund, cancellation? Up to what amount?
  • What is the handoff to a human, and how fast must it be?
  • Any regulated data (payment details, health information)?

Components#

message ──> PII redaction ──> intent + risk classifier ──> agent (policy RAG + tools) ──> output guardrails ──> customer
                                     │                          │
                                     │                          ├── order lookup (read)
                                     │                          ├── refund / cancel (write) ──> approval queue above threshold
                                     │                          └── create ticket
                                     └── escalate to human (angry, legal, low confidence, repeated failure)

The decisions that matter#

Ground answers in policy documents. Refund windows and shipping rules come from retrieval over the help centre and internal policies, with citations. The model shouldn't answer policy questions from its own knowledge, because a plausible-sounding wrong refund window costs real money.

Actions behind approval. Read actions run freely. Write actions have rules outside the model: refunds below a set amount for a verified order owner can go straight through; above it, they go to a human approval queue with the agent's summary attached. The tool itself enforces the limit, so a prompt injection in a customer message can't talk the model past it. My CareerPilot write-up has an approve-before-send step built the same way.

Escalation. Escalate on explicit request, on detected anger or legal language, after two failed attempts at the same issue, and when retrieval finds nothing relevant. Pass the conversation summary to the human so the customer doesn't repeat themselves.

PII and guardrails. Redact card numbers and similar data before anything reaches the model or the logs. Check outputs for leaked internal notes, promises the policy doesn't allow ("I'll refund you twice"), and other customers' data.

Measure it#

  • Resolution rate without a human, and the reopen rate (customers who come back within 7 days for the same issue). A high resolution rate with a high reopen rate means the agent is closing tickets it didn't fix.
  • Wrong-action rate on a test set of conversations with known correct actions. Evaluating an agent's tool calls shows how to check tool choice, arguments and order.
  • Escalation rate and handoff time.

Where it breaks#

The agent issues a refund the policy didn't allow, or it can't tell that the customer asking about an order isn't its owner. Ownership checks and amount limits belong in the tools, where the model can't skip them.

5. A text-to-SQL analytics agent#

"Let business users ask questions about company data in plain English." I built a small version with guardrails in Text-to-SQL with guardrails; here's the full-size design.

Clarify first#

  • How many tables? Ten is easy; 3,000 tables across several warehouses is the real problem.
  • Who asks, and do row-level permissions apply (a regional manager sees only their region)?
  • Must answers be exact? Finance questions usually need to match the official dashboard.

Components#

question ──> schema retrieval (tables, columns, descriptions, example queries) ──> SQL generation ──> validator ──> read-only replica (timeout, row limit) ──> result + SQL shown to user
                  │                                                                     │
                  └── semantic layer (metric definitions: "revenue", "active user")     └── on error: one repair attempt with the error message

The decisions that matter#

Schema retrieval. Don't put 3,000 table definitions in the prompt. Index table and column descriptions plus a library of verified example queries, retrieve the 5 to 15 most relevant tables, and include their columns and join keys. Example queries help most, because they show the model how your data is joined.

A semantic layer. "Revenue" might mean net of refunds, in INR, excluding test accounts. Define metrics once in a semantic layer (a metrics file or a tool like dbt's metric definitions) and have the model use those definitions instead of inventing the formula each time. Most wrong answers I'd worry about are SQL that runs fine and computes the wrong thing.

Validation before execution. Parse the SQL and reject anything that isn't a single SELECT, references unknown tables, or touches tables the user can't see. Add a LIMIT if missing, run against a read-only replica with a statement timeout, and apply row-level security in the database rather than trusting the generated WHERE clause.

Show the work. Return the SQL and a plain-language description of what it computed next to the result, so an analyst can check it.

Measure it#

  • Execution accuracy on an eval set of questions with known correct results: compare result sets, not SQL text, because two different queries can both be right.
  • Validator rejection rate and repair success rate.
  • Latency (generation plus query time) and warehouse cost per question.

Where it breaks#

Ambiguous questions ("sales last quarter": fiscal or calendar?), wrong joins that double-count, and expensive queries that scan a whole table. Ask a clarifying question when the semantic layer has more than one matching definition, and cap scanned bytes per query.

The same decisions show up in all five#

Concern Where it appeared
Retrieval quality and an eval set 1, 2, 4, 5
Permissions enforced outside the model 1, 2, 3, 4, 5
Human approval for risky actions 3, 4
Validate model output before acting on it 2 (citations), 3 (diffs, tests), 5 (SQL)
Budgets: tokens, steps, latency, cost all five

What interviewers listen for#

These are the points I'd make sure come up in any of the five answers:

  • You ask about scale, freshness and permissions before drawing anything.
  • You treat the LLM as an unreliable component and put checks around it: validation, tests, citations, limits.
  • You keep risky actions behind rules the model can't override.
  • You name an eval set and the metric you'd watch after launch.
  • You give rough numbers: tokens per request, latency per stage, cost per thousand requests.
  • You walk through one realistic failure and show where your design catches it.

To practise, pick one of the five, set a 35-minute timer, and talk through the six steps out loud. Then check your answer against the "where it breaks" section for that design and see whether your design would have caught it.