GyaanStack. Blog

Articles / raggenai-system-design

Design a RAG ingestion pipeline at scale: connectors, chunking, embeddings and re-indexing

The write side of RAG: connectors, change detection, parsing, chunking, embedding cost math, index memory, blue/green re-indexing and monitoring.

By Raghunath · · 22 min read

When a RAG system gives a wrong answer, people look at the prompt, the retriever and the reranker. In my experience the cause is just as often on the write side: the document was updated last week and the index still holds the old version, a scanned PDF went in as empty text, a deleted policy keeps getting cited, or a chunk lost the access label that should have hidden it from this user.

My RAG architectures comparison covers the query side. This post covers ingestion: getting documents into the index and keeping them correct as sources change. Most write-ups list the stages (load, split, embed, store) and stop. I want to put numbers on each stage for a 10 million document corpus, and end with a small runnable ingester that handles changes and deletes idempotently.

Requirements first#

For an internal knowledge base at a large company, a typical set looks like this.

Functional:

  • Pull from several sources: a document store (SharePoint, Google Drive, S3), a wiki, a ticketing system, a relational database.
  • Accept direct uploads through an API, and report per-document job status (queued, parsing, embedded, indexed, failed).
  • Propagate edits and deletes. A deleted document must stop showing up in answers.
  • Carry access control from the source to every chunk, so the query side can filter.

Non-functional (these are targets I would propose, not industry constants):

  • Freshness: P95 time from "source changed" to "searchable" under 15 minutes for event-driven sources, under 24 hours for polled ones.
  • Backfill: a full re-index of the corpus in under a week, so an embedding model upgrade is a planned project and not a quarter.
  • Idempotency: any stage can be retried, and re-running the whole pipeline on unchanged input changes nothing and costs nothing.
  • Cost: steady-state embedding spend proportional to what changed, not to corpus size.
  • Security: no chunk is ever indexed without an ACL label; unlabeled chunks fail closed.

The architecture#

 Sources                     Control plane
 ┌──────────┐  webhooks/CDC  ┌──────────────────────────────┐
 │ Drive/S3 │───────────────▶│ connector workers            │
 │ wiki     │  polling       │  - list + diff vs sync state │
 │ tickets  │───────────────▶│  - emit change events        │
 │ DB (CDC) │                └──────────────┬───────────────┘
 └──────────┘                               │ {doc_id, op, version, uri}
 upload API ───▶ object storage (raw copy)  ▼
                                    ┌───────────────┐
                                    │ ingest queue  │──▶ dead-letter queue
                                    └──────┬────────┘
                                           ▼
                                    ┌───────────────┐   per page:
                                    │ parse router  │── text PDF ──▶ CPU parser
                                    └──────┬────────┘── scanned ───▶ GPU OCR
                                           │         └─ tables ────▶ layout model
                                           ▼
                          chunker + metadata/ACL + quality checks
                                           │
                                           ▼
                          embed queue ──▶ embedding workers (batched,
                                           │                rate-limited)
                                           ▼
                          indexer: idempotent upsert / delete
                                           │
                          ┌────────────────┴───────────────┐
                          ▼                                ▼
                 vector index (v1 live)          keyword index (BM25)
                 vector index (v2 building)      metadata DB: Document,
                                                 Chunk, Job tables

Every arrow between stages is a queue. Parsing a 300-page scanned contract can take minutes, embedding is fast but rate-limited, and the vector store has its own write limits. Queues let each stage scale and fail on its own, with retries and a dead-letter queue (DLQ) for documents that keep failing.

The raw file lands in object storage first, so a parser or chunker change re-processes from that copy instead of re-crawling every source.

Data model and APIs#

Three tables carry the state. I keep them in a normal relational database, separate from the vector store.

Document(doc_id, source, source_uri, content_hash, acl, version,
         status, deleted_at, last_seen_at, parser_version)
Chunk(chunk_id, doc_id, ord, text_hash, section_path, page,
      acl, embed_model, index_version)
Job(job_id, doc_id, stage, attempt, error, started_at, finished_at)

content_hash decides whether a document changed. text_hash on chunks lets you keep vectors for paragraphs that did not change inside an edited document. embed_model and index_version let you run two indexes at once during a model upgrade. deleted_at is the tombstone.

The upload API stays small:

POST /documents            -> {doc_id, job_id}      (stores raw file, enqueues)
GET  /documents/{id}       -> {status, version, chunks, last_indexed_at}
DELETE /documents/{id}     -> 202                   (writes tombstone, enqueues delete)
GET  /jobs/{id}            -> {stage, attempt, error}

Connectors and change detection#

There are three ways to learn that a source changed, and most real pipelines use all three.

Push (webhooks). Lowest lag, but deliveries get lost, so never rely on it alone.

Change data capture (CDC). For databases, a tool like Debezium reads the write-ahead log and emits insert, update and delete events. Deletes come through as delete events followed by tombstone events, which is exactly what you need downstream.

Polling with a diff. List the source, compare modified times or ETags with your sync state, and fetch only what moved. A nightly full listing catches whatever the push path missed.

Whatever the trigger, the decision to re-process is made on a content hash. Modified times change when someone opens and re-saves a file without edits; a hash of the normalized text (plus the ACL, since a permission change must re-label chunks) does not. Skipping on hash is what makes steady-state cost proportional to real change.

Deletes are the part most pipelines get wrong. A document that disappears from a listing sends no event. So the connector compares "IDs the source listed this run" with "IDs we hold", and for each missing one it writes a tombstone (deleted_at) and enqueues a delete of all its chunks. Keep the tombstone row instead of deleting it: it stops a late, out-of-order update event from resurrecting the document, and it gives you an audit trail for "when did this stop being searchable".

Parsing: route per page, not per file#

Parsing is where quality is won or lost, and file type alone is a poor guide. One PDF can contain clean text pages, a scanned signature page and a table that spans two pages. I route each page:

Page looks like Detect by Send to Relative cost
Born-digital text Text layer present, sensible characters per page Fast CPU text extraction Lowest
Scanned image Little or no text layer, one large image OCR (Tesseract on CPU, or a GPU OCR model) High
Tables, multi-column, forms Layout model detects table or column regions Layout-aware parser such as Docling High
Slides Source is PPTX, or page aspect ratio Per-slide text plus speaker notes Low
Word/HTML Source format Convert to structured text keeping headings Low

OCR and layout models are much slower than plain extraction, and usually only a minority of pages need them. Sending everything through the expensive path wastes GPU hours; sending everything through the cheap path silently drops scanned pages.

Tables go out as Markdown or HTML with their header row, because "Q3 | 4.2" means nothing without the column names. Every parse records parser_version, so after a parser bug fix you can re-process exactly the documents it touched.

Chunking: the trade-offs#

Strategy How it splits Good for Weakness
Fixed tokens with overlap Every N tokens, overlap M Uniform text, quick start Cuts mid-sentence and mid-table; overlap inflates token count
Recursive (paragraph, then sentence) Natural boundaries, capped at N Most prose Ignores document structure
Structure-aware Headings, sections, list items, whole tables Manuals, policies, wikis Needs a good parser; uneven sizes
Semantic Split where adjacent sentence embeddings diverge Long unstructured text Costs an embedding pass just to chunk
Parent-child Embed small chunks, return the larger parent Precise match with enough context Two levels to keep in sync on update

My default is structure-aware with a size cap: split on headings, keep tables whole (or split them by rows while repeating the header), and fall back to recursive splitting for long sections. Each chunk carries its section_path ("Leave policy > Carry-over > Exceptions"), which I prepend to the chunk text before embedding. That cheap prefix fixes many "this chunk says '5 days' but of what?" misses.

The heavier version of the same idea is Anthropic's contextual retrieval: an LLM writes a short context for each chunk from the full document, and that context is prepended before embedding and BM25 indexing. Anthropic reports it cut retrieval failures by 49% when combined with contextual BM25, and by 67% with reranking added, on their test sets. It is also by far the most expensive step in the pipeline, as the math below shows, so I treat it as an upgrade you earn with an eval result.

Metadata and access control on every chunk#

Per chunk I store doc_id, source, source_uri, title, section_path, page, updated_at, language, doc_type, and the ACL as a list of principals (group:eng, user:123).

Three rules for ACLs:

  1. Resolve permissions at ingestion into principals the query side can filter on. Calling the source's permission API per query is too slow.
  2. A permission change is a change. Include the ACL in the hash so the document gets re-labelled even when its text is identical. The example below does exactly this.
  3. Fail closed. A chunk without an ACL never reaches the index. Group membership changes (someone leaves the eng group) are resolved on the query side against the identity provider, so you are not re-indexing on every HR event.

Embedding throughput and cost: the math#

Here is the napkin math for a 10 million document corpus. Every input is an assumption, stated so you can swap in your own.

Documents                      10,000,000
Avg pages per doc              5            (assumption)
Avg tokens per page            500          (assumption)
Document tokens                10M x 5 x 500          = 25 B tokens
Chunk size / overlap           512 / 64 tokens
Chunks per doc                 2,500 / (512-64) ~ 5.6 -> round to 6
Total chunks (vectors)         ~60 M
Tokens sent to embedder        60M x 512              ~ 30.7 B tokens

Overlap and padding mean you embed about 20% more tokens than the corpus holds.

Cost with OpenAI's published prices (launch post: $0.00002 per 1K tokens for text-embedding-3-small, $0.00013 per 1K for text-embedding-3-large; the Batch API gives a 50% discount with a 24-hour completion window):

                         sync          batch (50% off)
3-small: 30.7B x $0.02/M  ~ $614        ~ $307
3-large: 30.7B x $0.13/M  ~ $3,990      ~ $1,995

The embedding bill is small. Time is the harder constraint, capped by your account's rate limits, which vary by tier. With an assumed limit:

Assumed limit           5 M tokens/minute
Backfill time           30.7B / 5M = ~6,140 minutes = ~102 hours = ~4.3 days

That fits the one-week target only if the pipeline runs at the limit, with big batches and several workers. If you self-host an open embedding model, the same math becomes GPU hours. With an assumed 20,000 tokens per second per GPU for a small encoder (measure yours; it depends on model size, sequence length and batch size), 30.7B tokens is about 1.5 million GPU-seconds, roughly 430 GPU-hours, which spread across 8 GPUs is about 2.2 days.

Now compare contextual enrichment. Anthropic's post estimates a one-time cost of $1.02 per million document tokens with prompt caching, under their stated assumptions (800-token chunks, 8K-token documents, at the model prices when the post was written). At 25B document tokens:

25,000 M doc tokens x $1.02 / M   ~ $25,500

That is about 40 times the 3-small embedding bill, and it recurs on every full rebuild. Worth it if your evals show the recall gain, not as a default.

Steady state is what you pay every day. If 1% of documents change daily and chunk-level hashing means only about a third of their chunks actually differ:

100,000 docs x 6 chunks x 1/3 x 512 tokens ~ 102 M tokens/day
3-small: ~ $2/day

Batching, rate limits, retries and idempotent upserts#

The embedding stage is a loop that pulls chunks from the queue, groups them into requests, and handles failure without duplicating work.

  • Batch by tokens, not count. Pack chunks up to the per-request limit; it costs the same per token and uses far fewer requests per minute.
  • Rate-limit on the client. A token bucket sized to your tokens-per-minute and requests-per-minute limits, shared across workers (Redis works), keeps you just under the limit instead of bouncing off it.
  • Retry 429s and 5xx with exponential backoff and jitter. Retry 4xx validation errors zero times; those go to the DLQ with the error attached.
  • Make writes idempotent. The chunk ID is deterministic (doc_id plus a hash of the chunk text), and the indexer upserts by that ID. A retry after a timeout writes the same row again instead of a duplicate. Deletes by ID are idempotent by nature.
  • Order by version. Each event carries the document version. The indexer ignores an event older than what it holds, so a slow retry of version 3 cannot overwrite version 4.

Deterministic chunk IDs also mean that when someone edits one paragraph of a 40-page manual, only that chunk is re-embedded.

Index choice and memory math#

For 60M vectors, raw storage is easy to compute:

float32, 1536 dims (3-small)   60M x 1536 x 4 B  = ~369 GB
float32, 3072 dims (3-large)   60M x 3072 x 4 B  = ~737 GB
float16 (halfvec), 1536 dims                     = ~184 GB
shortened to 512 dims, float32                   = ~123 GB

OpenAI's embeddings guide documents a dimensions parameter for shortening the v3 embeddings; the training idea behind this is Matryoshka representation learning. Shortening trades some quality for memory, so measure it on your eval set.

HNSW adds a graph on top. In pgvector the default is m = 16. Common implementations allow up to 2 x m links per vector on the bottom layer, so as an estimate at 4 bytes per link that is roughly 128 bytes per vector, about 8 GB for 60M vectors (exact overhead varies by engine). HNSW wants the vectors and graph in RAM for good latency, so the 1536-dim float32 case needs on the order of 380 GB of memory across the cluster.

IVF clusters vectors into lists and searches only the nearest few. With product quantization (IVF-PQ, as in FAISS), each vector compresses to a code of tens of bytes. At 64 bytes per vector, 60M vectors take about 4 GB, with a recall cost you recover by re-ranking the top candidates against full vectors stored on disk.

HNSW IVF-Flat IVF-PQ
Memory for 60M x 1536 ~380 GB ~370 GB ~4 GB codes + full vectors on disk
Recall at low latency Highest Good with enough probes Lower; needs re-rank
Build time Slow Fast, but needs training on a sample Needs training
Incremental inserts Good Fine, but clusters drift as data changes Same drift problem
Deletes Usually marked, cleaned up on rebuild or compaction Easier Easier

The pgvector README notes that IVFFlat builds faster and uses less memory than HNSW with a worse speed-recall trade-off, and that indexed vector columns are limited to 2,000 dimensions (4,000 for halfvec). So text-embedding-3-large at 3,072 dimensions needs halfvec or shortening there. Find limits like that during design, not during the backfill.

My rule of thumb: HNSW until memory cost hurts, then quantize (scalar int8 first, PQ if you must), and keep full vectors somewhere for re-scoring.

Versioned indexes and blue/green model upgrades#

Vectors from two embedding models live in different spaces. You cannot mix them in one index or compare a query embedded with model B against chunks embedded with model A. So a model upgrade is a full re-embed, and it needs a deployment plan.

1. Create index v2 (new model, maybe new dims).       v1 serves traffic
2. Backfill v2 from the Chunk table (no re-parse).    ~4 days at the limit above
3. Dual-write: new changes go to v1 and v2.           v1 serves traffic
4. Run the retrieval eval set on v1 and v2.           compare recall@k, MRR
5. Shadow or canary: 5% of queries read v2.           watch answer metrics
6. Swap the alias "kb-live" -> v2.                    instant, reversible
7. Keep v1 for N days, then drop it.

Qdrant collection aliases and Elasticsearch aliases let you build a second index in the background and switch the name atomically. In Postgres, a view or a config row naming the live table does the same job.

Without step 3, every edit made during the four-day backfill is missing from v2 on swap day. Chunker and parser changes follow the same plan.

Data quality checks#

Cheap checks between parse and embed catch most garbage before it costs money:

  • Characters per page below a threshold: probably a scan that skipped OCR.
  • High ratio of non-dictionary tokens or replacement characters: encoding or OCR failure.
  • Same chunk hash in thousands of documents: boilerplate (footers, disclaimers). Index it once or drop it.
  • Chunk below a minimum length: headings split from their body; merge with the next chunk.
  • Missing ACL, title or source URI: reject.
  • Embedding sanity: vector norm and dimension as expected, no all-zero vectors.

Failed checks send the document to the DLQ with a reason.

Monitoring#

Metric Why Alert when
Freshness lag (source change to searchable), P50/P95 The user-facing promise P95 over target
Docs by status (queued, failed, indexed) Backlog and stuck jobs Queue age grows for 30 min
DLQ size by reason Silent loss Any reason spikes
Embedding tokens/day and cost Budget; a bug that defeats hash-skipping shows up here first 2x the trailing week
Source count vs index count, per source Missed deletes, failed connectors Drift over 0.5%
Chunks without ACL Security Any non-zero
Retrieval eval score on a fixed set, nightly Quality regressions from parser or chunker changes Drop beyond noise

The source-versus-index count is the one I would build first. When the two numbers drift, a connector is broken or deletes are not propagating, and no other metric shows it.

For the query-side monitoring that pairs with this, see LLM guardrails and observability.

A small runnable example#

This ingester uses only the Python standard library and an in-memory SQLite database. It hashes each document (text plus ACL), skips unchanged ones, re-chunks changed ones while keeping vectors for unchanged chunks, and tombstones documents that left the source. The embedder is a stub that returns deterministic fake vectors and counts calls, so there is no API key; swap in a real batched call. Upserts use SQLite's ON CONFLICT ... DO UPDATE.

"""Incremental RAG ingester: hash docs, skip unchanged, re-chunk changed, tombstone removed.
Stdlib only. The embedder is a stub (deterministic fake vectors), so no API key is needed."""
import hashlib, json, sqlite3

EMBED_MODEL = "stub-embed-v1"
embed_calls = 0

def sha(text):
    return hashlib.sha256(text.encode()).hexdigest()

def embed(texts):
    """Stub: a real pipeline sends this batch to an embedding API or a GPU worker."""
    global embed_calls
    embed_calls += len(texts)
    return [[int(sha(t)[i:i + 2], 16) / 255 for i in range(0, 8, 2)] for t in texts]

def chunk(text, max_words=12):
    """Split on paragraphs, then cap each chunk at max_words. Real chunkers respect headings and tables."""
    out = []
    for para in [p.strip() for p in text.split("\n\n") if p.strip()]:
        words = para.split()
        for i in range(0, len(words), max_words):
            out.append(" ".join(words[i:i + max_words]))
    return out

def setup(db):
    db.executescript("""
    CREATE TABLE IF NOT EXISTS documents (
        doc_id TEXT PRIMARY KEY, content_hash TEXT, acl TEXT,
        version INTEGER, deleted INTEGER DEFAULT 0);
    CREATE TABLE IF NOT EXISTS chunks (
        chunk_id TEXT PRIMARY KEY, doc_id TEXT, ord INTEGER, text TEXT,
        acl TEXT, model TEXT, vector TEXT);
    """)

def ingest(db, source):
    stats = dict(new=0, changed=0, unchanged=0, deleted=0, chunks_added=0, chunks_removed=0)
    start_calls = embed_calls
    seen = set()
    for doc_id, (text, acl) in source.items():
        seen.add(doc_id)
        h = sha(text + "|" + acl)  # an ACL change must re-label chunks too
        row = db.execute("SELECT content_hash, version, deleted FROM documents WHERE doc_id=?",
                         (doc_id,)).fetchone()
        if row and row[0] == h and not row[2]:
            stats["unchanged"] += 1
            continue
        stats["changed" if row and not row[2] else "new"] += 1
        pieces = chunk(text)
        # Chunk ID = doc + content hash, so an unchanged paragraph keeps its ID and its vector.
        wanted = {f"{doc_id}:{sha(p)[:12]}": (i, p) for i, p in enumerate(pieces)}
        have = {r[0] for r in db.execute("SELECT chunk_id FROM chunks WHERE doc_id=?", (doc_id,))}
        to_embed = [cid for cid in wanted if cid not in have]
        vectors = embed([wanted[c][1] for c in to_embed]) if to_embed else []
        for cid, vec in zip(to_embed, vectors):
            i, p = wanted[cid]
            db.execute("""INSERT INTO chunks VALUES (?,?,?,?,?,?,?)
                          ON CONFLICT(chunk_id) DO UPDATE SET ord=excluded.ord, acl=excluded.acl""",
                       (cid, doc_id, i, p, acl, EMBED_MODEL, json.dumps(vec)))
        for cid in wanted.keys() & have:  # kept chunks: refresh order and ACL label, no re-embed
            db.execute("UPDATE chunks SET ord=?, acl=? WHERE chunk_id=?", (wanted[cid][0], acl, cid))
        stale = have - wanted.keys()
        db.executemany("DELETE FROM chunks WHERE chunk_id=?", [(c,) for c in stale])
        stats["chunks_added"] += len(to_embed)
        stats["chunks_removed"] += len(stale)
        db.execute("""INSERT INTO documents (doc_id, content_hash, acl, version, deleted)
                      VALUES (?,?,?,1,0)
                      ON CONFLICT(doc_id) DO UPDATE SET content_hash=excluded.content_hash,
                      acl=excluded.acl, version=documents.version+1, deleted=0""", (doc_id, h, acl))
    # Tombstones: documents we know about that the source no longer lists.
    for (doc_id,) in db.execute("SELECT doc_id FROM documents WHERE deleted=0").fetchall():
        if doc_id not in seen:
            n = db.execute("DELETE FROM chunks WHERE doc_id=?", (doc_id,)).rowcount
            db.execute("UPDATE documents SET deleted=1 WHERE doc_id=?", (doc_id,))
            stats["deleted"] += 1
            stats["chunks_removed"] += n
    db.commit()
    stats["embed_calls"] = embed_calls - start_calls
    stats["live_chunks"] = db.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
    return stats

RUN1 = {
    "hr/leave-policy.md": ("Employees get 24 days of paid leave per year.\n\n"
                           "Unused leave up to 10 days carries over to the next year.", "group:all"),
    "eng/oncall.md": ("The on-call engineer acknowledges pages within 15 minutes.\n\n"
                      "Escalate to the secondary after 30 minutes without acknowledgement.", "group:eng"),
    "fin/expenses.md": ("Submit expense claims within 30 days with receipts attached.", "group:all"),
}
RUN2 = {
    # Changed: second paragraph edited, first paragraph identical.
    "hr/leave-policy.md": ("Employees get 24 days of paid leave per year.\n\n"
                           "Unused leave up to 5 days carries over to the next year.", "group:all"),
    # Unchanged text, but the ACL tightened: chunks must be re-labelled.
    "eng/oncall.md": ("The on-call engineer acknowledges pages within 15 minutes.\n\n"
                      "Escalate to the secondary after 30 minutes without acknowledgement.", "group:sre"),
    # fin/expenses.md removed from the source. New doc added.
    "it/vpn.md": ("Connect to the VPN before opening internal dashboards.", "group:all"),
}

if __name__ == "__main__":
    db = sqlite3.connect(":memory:")
    setup(db)
    for name, src in [("run 1", RUN1), ("run 2", RUN2), ("run 2 again", RUN2)]:
        print(name, ingest(db, src))
    print("chunks for eng/oncall.md:",
          db.execute("SELECT DISTINCT acl FROM chunks WHERE doc_id='eng/oncall.md'").fetchall())
    print("tombstoned:", db.execute("SELECT doc_id FROM documents WHERE deleted=1").fetchall())

Output:

run 1 {'new': 3, 'changed': 0, 'unchanged': 0, 'deleted': 0, 'chunks_added': 5, 'chunks_removed': 0, 'embed_calls': 5, 'live_chunks': 5}
run 2 {'new': 1, 'changed': 2, 'unchanged': 0, 'deleted': 1, 'chunks_added': 2, 'chunks_removed': 2, 'embed_calls': 2, 'live_chunks': 5}
run 2 again {'new': 0, 'changed': 0, 'unchanged': 3, 'deleted': 0, 'chunks_added': 0, 'chunks_removed': 0, 'embed_calls': 0, 'live_chunks': 5}
chunks for eng/oncall.md: [('group:sre',)]
tombstoned: [('fin/expenses.md',)]

In run 2 the leave policy changed one paragraph, so one chunk was embedded and one removed while the first paragraph kept its vector. The on-call doc changed only its ACL (group:eng to group:sre), so its chunks were re-labelled with zero embedding calls. The VPN doc is new, and the expenses doc was tombstoned. Running the same input again does nothing and calls the embedder zero times.

Two simplifications to fix before production. Two identical paragraphs in one document would get the same chunk ID; add the ordinal or a duplicate counter to the ID. And the stub writes to SQLite only; a real indexer writes to the vector store and the metadata DB, so it needs either a transactional outbox or a reconciliation job to repair the case where one write succeeded and the other did not.

Failure modes#

Failure Symptom Fix
Deleted doc still cited Answers quote a retired policy Listing diff + tombstones; source-vs-index reconciliation
Scanned pages ingested as empty "No information found" for content that exists Characters-per-page check; route to OCR
Tables flattened Wrong numbers in answers Layout-aware parsing; tables as Markdown with headers
ACL lost or stale Users see chunks they should not ACL in hash; fail closed on missing labels; ACL tests
Modified-time triggers re-embed everything Cost spikes with no content change Skip on content hash
Retry creates duplicates Same chunk appears twice in top-k Deterministic chunk IDs; upsert
Out-of-order events Old version overwrites new Version check in the indexer
Mixed embedding models in one index Recall collapses after partial upgrade Versioned indexes; alias swap
Edits lost during backfill v2 missing recent changes on swap day Dual-write during backfill
Boilerplate floods results Footers and disclaimers in top-k Cross-document chunk-hash dedupe
DLQ grows unseen Gradual coverage loss DLQ dashboard and alert by reason

Decision table#

Situation Choose
Under ~100K docs, changes rare Nightly full rebuild is fine; skip CDC
Large corpus, frequent edits Hash-based incremental with chunk-level reuse
Database source CDC (Debezium or the database's native stream)
SaaS source with webhooks Webhooks plus nightly listing diff
Mostly born-digital PDFs CPU extraction, OCR only on flagged pages
Many scans or forms Per-page routing with a GPU OCR/layout pool
Policies, manuals, wikis Structure-aware chunking with section paths
Evals show context-free chunks fail Contextual enrichment, budgeted per rebuild
Vectors fit in RAM budget HNSW
Hundreds of millions of vectors or tight RAM Quantized IVF or HNSW with re-scoring
Changing embedding model, chunker or parser New index version, dual-write, eval, alias swap
Backfill not urgent Batch API at half price

If you are building your first one, the PDF question-answering bot with citations is a small end-to-end version of the same flow, and the RAG cheat sheet covers the retrieval settings that consume what this pipeline produces. For the retry and idempotency patterns in more depth, the long-running agent design uses the same ideas for tool calls.

Sources#