Articles / raggenai-system-design
Design a RAG ingestion pipeline at scale: connectors, chunking, embeddings and re-indexing
The write side of RAG: connectors, change detection, parsing, chunking, embedding cost math, index memory, blue/green re-indexing and monitoring.
When a RAG system gives a wrong answer, people look at the prompt, the retriever and the reranker. In my experience the cause is just as often on the write side: the document was updated last week and the index still holds the old version, a scanned PDF went in as empty text, a deleted policy keeps getting cited, or a chunk lost the access label that should have hidden it from this user.
My RAG architectures comparison covers the query side. This post covers ingestion: getting documents into the index and keeping them correct as sources change. Most write-ups list the stages (load, split, embed, store) and stop. I want to put numbers on each stage for a 10 million document corpus, and end with a small runnable ingester that handles changes and deletes idempotently.
Requirements first#
For an internal knowledge base at a large company, a typical set looks like this.
Functional:
- Pull from several sources: a document store (SharePoint, Google Drive, S3), a wiki, a ticketing system, a relational database.
- Accept direct uploads through an API, and report per-document job status (queued, parsing, embedded, indexed, failed).
- Propagate edits and deletes. A deleted document must stop showing up in answers.
- Carry access control from the source to every chunk, so the query side can filter.
Non-functional (these are targets I would propose, not industry constants):
- Freshness: P95 time from "source changed" to "searchable" under 15 minutes for event-driven sources, under 24 hours for polled ones.
- Backfill: a full re-index of the corpus in under a week, so an embedding model upgrade is a planned project and not a quarter.
- Idempotency: any stage can be retried, and re-running the whole pipeline on unchanged input changes nothing and costs nothing.
- Cost: steady-state embedding spend proportional to what changed, not to corpus size.
- Security: no chunk is ever indexed without an ACL label; unlabeled chunks fail closed.
The architecture#
Sources Control plane
┌──────────┐ webhooks/CDC ┌──────────────────────────────┐
│ Drive/S3 │───────────────▶│ connector workers │
│ wiki │ polling │ - list + diff vs sync state │
│ tickets │───────────────▶│ - emit change events │
│ DB (CDC) │ └──────────────┬───────────────┘
└──────────┘ │ {doc_id, op, version, uri}
upload API ───▶ object storage (raw copy) ▼
┌───────────────┐
│ ingest queue │──▶ dead-letter queue
└──────┬────────┘
▼
┌───────────────┐ per page:
│ parse router │── text PDF ──▶ CPU parser
└──────┬────────┘── scanned ───▶ GPU OCR
│ └─ tables ────▶ layout model
▼
chunker + metadata/ACL + quality checks
│
▼
embed queue ──▶ embedding workers (batched,
│ rate-limited)
▼
indexer: idempotent upsert / delete
│
┌────────────────┴───────────────┐
▼ ▼
vector index (v1 live) keyword index (BM25)
vector index (v2 building) metadata DB: Document,
Chunk, Job tables
Every arrow between stages is a queue. Parsing a 300-page scanned contract can take minutes, embedding is fast but rate-limited, and the vector store has its own write limits. Queues let each stage scale and fail on its own, with retries and a dead-letter queue (DLQ) for documents that keep failing.
The raw file lands in object storage first, so a parser or chunker change re-processes from that copy instead of re-crawling every source.
Data model and APIs#
Three tables carry the state. I keep them in a normal relational database, separate from the vector store.
Document(doc_id, source, source_uri, content_hash, acl, version,
status, deleted_at, last_seen_at, parser_version)
Chunk(chunk_id, doc_id, ord, text_hash, section_path, page,
acl, embed_model, index_version)
Job(job_id, doc_id, stage, attempt, error, started_at, finished_at)
content_hash decides whether a document changed. text_hash on chunks lets you keep vectors for paragraphs that did not change inside an edited document. embed_model and index_version let you run two indexes at once during a model upgrade. deleted_at is the tombstone.
The upload API stays small:
POST /documents -> {doc_id, job_id} (stores raw file, enqueues)
GET /documents/{id} -> {status, version, chunks, last_indexed_at}
DELETE /documents/{id} -> 202 (writes tombstone, enqueues delete)
GET /jobs/{id} -> {stage, attempt, error}
Connectors and change detection#
There are three ways to learn that a source changed, and most real pipelines use all three.
Push (webhooks). Lowest lag, but deliveries get lost, so never rely on it alone.
Change data capture (CDC). For databases, a tool like Debezium reads the write-ahead log and emits insert, update and delete events. Deletes come through as delete events followed by tombstone events, which is exactly what you need downstream.
Polling with a diff. List the source, compare modified times or ETags with your sync state, and fetch only what moved. A nightly full listing catches whatever the push path missed.
Whatever the trigger, the decision to re-process is made on a content hash. Modified times change when someone opens and re-saves a file without edits; a hash of the normalized text (plus the ACL, since a permission change must re-label chunks) does not. Skipping on hash is what makes steady-state cost proportional to real change.
Deletes are the part most pipelines get wrong. A document that disappears from a listing sends no event. So the connector compares "IDs the source listed this run" with "IDs we hold", and for each missing one it writes a tombstone (deleted_at) and enqueues a delete of all its chunks. Keep the tombstone row instead of deleting it: it stops a late, out-of-order update event from resurrecting the document, and it gives you an audit trail for "when did this stop being searchable".
Parsing: route per page, not per file#
Parsing is where quality is won or lost, and file type alone is a poor guide. One PDF can contain clean text pages, a scanned signature page and a table that spans two pages. I route each page:
| Page looks like | Detect by | Send to | Relative cost |
|---|---|---|---|
| Born-digital text | Text layer present, sensible characters per page | Fast CPU text extraction | Lowest |
| Scanned image | Little or no text layer, one large image | OCR (Tesseract on CPU, or a GPU OCR model) | High |
| Tables, multi-column, forms | Layout model detects table or column regions | Layout-aware parser such as Docling | High |
| Slides | Source is PPTX, or page aspect ratio | Per-slide text plus speaker notes | Low |
| Word/HTML | Source format | Convert to structured text keeping headings | Low |
OCR and layout models are much slower than plain extraction, and usually only a minority of pages need them. Sending everything through the expensive path wastes GPU hours; sending everything through the cheap path silently drops scanned pages.
Tables go out as Markdown or HTML with their header row, because "Q3 | 4.2" means nothing without the column names. Every parse records parser_version, so after a parser bug fix you can re-process exactly the documents it touched.
Chunking: the trade-offs#
| Strategy | How it splits | Good for | Weakness |
|---|---|---|---|
| Fixed tokens with overlap | Every N tokens, overlap M | Uniform text, quick start | Cuts mid-sentence and mid-table; overlap inflates token count |
| Recursive (paragraph, then sentence) | Natural boundaries, capped at N | Most prose | Ignores document structure |
| Structure-aware | Headings, sections, list items, whole tables | Manuals, policies, wikis | Needs a good parser; uneven sizes |
| Semantic | Split where adjacent sentence embeddings diverge | Long unstructured text | Costs an embedding pass just to chunk |
| Parent-child | Embed small chunks, return the larger parent | Precise match with enough context | Two levels to keep in sync on update |
My default is structure-aware with a size cap: split on headings, keep tables whole (or split them by rows while repeating the header), and fall back to recursive splitting for long sections. Each chunk carries its section_path ("Leave policy > Carry-over > Exceptions"), which I prepend to the chunk text before embedding. That cheap prefix fixes many "this chunk says '5 days' but of what?" misses.
The heavier version of the same idea is Anthropic's contextual retrieval: an LLM writes a short context for each chunk from the full document, and that context is prepended before embedding and BM25 indexing. Anthropic reports it cut retrieval failures by 49% when combined with contextual BM25, and by 67% with reranking added, on their test sets. It is also by far the most expensive step in the pipeline, as the math below shows, so I treat it as an upgrade you earn with an eval result.
Metadata and access control on every chunk#
Per chunk I store doc_id, source, source_uri, title, section_path, page, updated_at, language, doc_type, and the ACL as a list of principals (group:eng, user:123).
Three rules for ACLs:
- Resolve permissions at ingestion into principals the query side can filter on. Calling the source's permission API per query is too slow.
- A permission change is a change. Include the ACL in the hash so the document gets re-labelled even when its text is identical. The example below does exactly this.
- Fail closed. A chunk without an ACL never reaches the index. Group membership changes (someone leaves the
enggroup) are resolved on the query side against the identity provider, so you are not re-indexing on every HR event.
Embedding throughput and cost: the math#
Here is the napkin math for a 10 million document corpus. Every input is an assumption, stated so you can swap in your own.
Documents 10,000,000
Avg pages per doc 5 (assumption)
Avg tokens per page 500 (assumption)
Document tokens 10M x 5 x 500 = 25 B tokens
Chunk size / overlap 512 / 64 tokens
Chunks per doc 2,500 / (512-64) ~ 5.6 -> round to 6
Total chunks (vectors) ~60 M
Tokens sent to embedder 60M x 512 ~ 30.7 B tokens
Overlap and padding mean you embed about 20% more tokens than the corpus holds.
Cost with OpenAI's published prices (launch post: $0.00002 per 1K tokens for text-embedding-3-small, $0.00013 per 1K for text-embedding-3-large; the Batch API gives a 50% discount with a 24-hour completion window):
sync batch (50% off)
3-small: 30.7B x $0.02/M ~ $614 ~ $307
3-large: 30.7B x $0.13/M ~ $3,990 ~ $1,995
The embedding bill is small. Time is the harder constraint, capped by your account's rate limits, which vary by tier. With an assumed limit:
Assumed limit 5 M tokens/minute
Backfill time 30.7B / 5M = ~6,140 minutes = ~102 hours = ~4.3 days
That fits the one-week target only if the pipeline runs at the limit, with big batches and several workers. If you self-host an open embedding model, the same math becomes GPU hours. With an assumed 20,000 tokens per second per GPU for a small encoder (measure yours; it depends on model size, sequence length and batch size), 30.7B tokens is about 1.5 million GPU-seconds, roughly 430 GPU-hours, which spread across 8 GPUs is about 2.2 days.
Now compare contextual enrichment. Anthropic's post estimates a one-time cost of $1.02 per million document tokens with prompt caching, under their stated assumptions (800-token chunks, 8K-token documents, at the model prices when the post was written). At 25B document tokens:
25,000 M doc tokens x $1.02 / M ~ $25,500
That is about 40 times the 3-small embedding bill, and it recurs on every full rebuild. Worth it if your evals show the recall gain, not as a default.
Steady state is what you pay every day. If 1% of documents change daily and chunk-level hashing means only about a third of their chunks actually differ:
100,000 docs x 6 chunks x 1/3 x 512 tokens ~ 102 M tokens/day
3-small: ~ $2/day
Batching, rate limits, retries and idempotent upserts#
The embedding stage is a loop that pulls chunks from the queue, groups them into requests, and handles failure without duplicating work.
- Batch by tokens, not count. Pack chunks up to the per-request limit; it costs the same per token and uses far fewer requests per minute.
- Rate-limit on the client. A token bucket sized to your tokens-per-minute and requests-per-minute limits, shared across workers (Redis works), keeps you just under the limit instead of bouncing off it.
- Retry 429s and 5xx with exponential backoff and jitter. Retry 4xx validation errors zero times; those go to the DLQ with the error attached.
- Make writes idempotent. The chunk ID is deterministic (
doc_idplus a hash of the chunk text), and the indexer upserts by that ID. A retry after a timeout writes the same row again instead of a duplicate. Deletes by ID are idempotent by nature. - Order by version. Each event carries the document version. The indexer ignores an event older than what it holds, so a slow retry of version 3 cannot overwrite version 4.
Deterministic chunk IDs also mean that when someone edits one paragraph of a 40-page manual, only that chunk is re-embedded.
Index choice and memory math#
For 60M vectors, raw storage is easy to compute:
float32, 1536 dims (3-small) 60M x 1536 x 4 B = ~369 GB
float32, 3072 dims (3-large) 60M x 3072 x 4 B = ~737 GB
float16 (halfvec), 1536 dims = ~184 GB
shortened to 512 dims, float32 = ~123 GB
OpenAI's embeddings guide documents a dimensions parameter for shortening the v3 embeddings; the training idea behind this is Matryoshka representation learning. Shortening trades some quality for memory, so measure it on your eval set.
HNSW adds a graph on top. In pgvector the default is m = 16. Common implementations allow up to 2 x m links per vector on the bottom layer, so as an estimate at 4 bytes per link that is roughly 128 bytes per vector, about 8 GB for 60M vectors (exact overhead varies by engine). HNSW wants the vectors and graph in RAM for good latency, so the 1536-dim float32 case needs on the order of 380 GB of memory across the cluster.
IVF clusters vectors into lists and searches only the nearest few. With product quantization (IVF-PQ, as in FAISS), each vector compresses to a code of tens of bytes. At 64 bytes per vector, 60M vectors take about 4 GB, with a recall cost you recover by re-ranking the top candidates against full vectors stored on disk.
| HNSW | IVF-Flat | IVF-PQ | |
|---|---|---|---|
| Memory for 60M x 1536 | ~380 GB | ~370 GB | ~4 GB codes + full vectors on disk |
| Recall at low latency | Highest | Good with enough probes | Lower; needs re-rank |
| Build time | Slow | Fast, but needs training on a sample | Needs training |
| Incremental inserts | Good | Fine, but clusters drift as data changes | Same drift problem |
| Deletes | Usually marked, cleaned up on rebuild or compaction | Easier | Easier |
The pgvector README notes that IVFFlat builds faster and uses less memory than HNSW with a worse speed-recall trade-off, and that indexed vector columns are limited to 2,000 dimensions (4,000 for halfvec). So text-embedding-3-large at 3,072 dimensions needs halfvec or shortening there. Find limits like that during design, not during the backfill.
My rule of thumb: HNSW until memory cost hurts, then quantize (scalar int8 first, PQ if you must), and keep full vectors somewhere for re-scoring.
Versioned indexes and blue/green model upgrades#
Vectors from two embedding models live in different spaces. You cannot mix them in one index or compare a query embedded with model B against chunks embedded with model A. So a model upgrade is a full re-embed, and it needs a deployment plan.
1. Create index v2 (new model, maybe new dims). v1 serves traffic
2. Backfill v2 from the Chunk table (no re-parse). ~4 days at the limit above
3. Dual-write: new changes go to v1 and v2. v1 serves traffic
4. Run the retrieval eval set on v1 and v2. compare recall@k, MRR
5. Shadow or canary: 5% of queries read v2. watch answer metrics
6. Swap the alias "kb-live" -> v2. instant, reversible
7. Keep v1 for N days, then drop it.
Qdrant collection aliases and Elasticsearch aliases let you build a second index in the background and switch the name atomically. In Postgres, a view or a config row naming the live table does the same job.
Without step 3, every edit made during the four-day backfill is missing from v2 on swap day. Chunker and parser changes follow the same plan.
Data quality checks#
Cheap checks between parse and embed catch most garbage before it costs money:
- Characters per page below a threshold: probably a scan that skipped OCR.
- High ratio of non-dictionary tokens or replacement characters: encoding or OCR failure.
- Same chunk hash in thousands of documents: boilerplate (footers, disclaimers). Index it once or drop it.
- Chunk below a minimum length: headings split from their body; merge with the next chunk.
- Missing ACL, title or source URI: reject.
- Embedding sanity: vector norm and dimension as expected, no all-zero vectors.
Failed checks send the document to the DLQ with a reason.
Monitoring#
| Metric | Why | Alert when |
|---|---|---|
| Freshness lag (source change to searchable), P50/P95 | The user-facing promise | P95 over target |
| Docs by status (queued, failed, indexed) | Backlog and stuck jobs | Queue age grows for 30 min |
| DLQ size by reason | Silent loss | Any reason spikes |
| Embedding tokens/day and cost | Budget; a bug that defeats hash-skipping shows up here first | 2x the trailing week |
| Source count vs index count, per source | Missed deletes, failed connectors | Drift over 0.5% |
| Chunks without ACL | Security | Any non-zero |
| Retrieval eval score on a fixed set, nightly | Quality regressions from parser or chunker changes | Drop beyond noise |
The source-versus-index count is the one I would build first. When the two numbers drift, a connector is broken or deletes are not propagating, and no other metric shows it.
For the query-side monitoring that pairs with this, see LLM guardrails and observability.
A small runnable example#
This ingester uses only the Python standard library and an in-memory SQLite database. It hashes each document (text plus ACL), skips unchanged ones, re-chunks changed ones while keeping vectors for unchanged chunks, and tombstones documents that left the source. The embedder is a stub that returns deterministic fake vectors and counts calls, so there is no API key; swap in a real batched call. Upserts use SQLite's ON CONFLICT ... DO UPDATE.
"""Incremental RAG ingester: hash docs, skip unchanged, re-chunk changed, tombstone removed.
Stdlib only. The embedder is a stub (deterministic fake vectors), so no API key is needed."""
import hashlib, json, sqlite3
EMBED_MODEL = "stub-embed-v1"
embed_calls = 0
def sha(text):
return hashlib.sha256(text.encode()).hexdigest()
def embed(texts):
"""Stub: a real pipeline sends this batch to an embedding API or a GPU worker."""
global embed_calls
embed_calls += len(texts)
return [[int(sha(t)[i:i + 2], 16) / 255 for i in range(0, 8, 2)] for t in texts]
def chunk(text, max_words=12):
"""Split on paragraphs, then cap each chunk at max_words. Real chunkers respect headings and tables."""
out = []
for para in [p.strip() for p in text.split("\n\n") if p.strip()]:
words = para.split()
for i in range(0, len(words), max_words):
out.append(" ".join(words[i:i + max_words]))
return out
def setup(db):
db.executescript("""
CREATE TABLE IF NOT EXISTS documents (
doc_id TEXT PRIMARY KEY, content_hash TEXT, acl TEXT,
version INTEGER, deleted INTEGER DEFAULT 0);
CREATE TABLE IF NOT EXISTS chunks (
chunk_id TEXT PRIMARY KEY, doc_id TEXT, ord INTEGER, text TEXT,
acl TEXT, model TEXT, vector TEXT);
""")
def ingest(db, source):
stats = dict(new=0, changed=0, unchanged=0, deleted=0, chunks_added=0, chunks_removed=0)
start_calls = embed_calls
seen = set()
for doc_id, (text, acl) in source.items():
seen.add(doc_id)
h = sha(text + "|" + acl) # an ACL change must re-label chunks too
row = db.execute("SELECT content_hash, version, deleted FROM documents WHERE doc_id=?",
(doc_id,)).fetchone()
if row and row[0] == h and not row[2]:
stats["unchanged"] += 1
continue
stats["changed" if row and not row[2] else "new"] += 1
pieces = chunk(text)
# Chunk ID = doc + content hash, so an unchanged paragraph keeps its ID and its vector.
wanted = {f"{doc_id}:{sha(p)[:12]}": (i, p) for i, p in enumerate(pieces)}
have = {r[0] for r in db.execute("SELECT chunk_id FROM chunks WHERE doc_id=?", (doc_id,))}
to_embed = [cid for cid in wanted if cid not in have]
vectors = embed([wanted[c][1] for c in to_embed]) if to_embed else []
for cid, vec in zip(to_embed, vectors):
i, p = wanted[cid]
db.execute("""INSERT INTO chunks VALUES (?,?,?,?,?,?,?)
ON CONFLICT(chunk_id) DO UPDATE SET ord=excluded.ord, acl=excluded.acl""",
(cid, doc_id, i, p, acl, EMBED_MODEL, json.dumps(vec)))
for cid in wanted.keys() & have: # kept chunks: refresh order and ACL label, no re-embed
db.execute("UPDATE chunks SET ord=?, acl=? WHERE chunk_id=?", (wanted[cid][0], acl, cid))
stale = have - wanted.keys()
db.executemany("DELETE FROM chunks WHERE chunk_id=?", [(c,) for c in stale])
stats["chunks_added"] += len(to_embed)
stats["chunks_removed"] += len(stale)
db.execute("""INSERT INTO documents (doc_id, content_hash, acl, version, deleted)
VALUES (?,?,?,1,0)
ON CONFLICT(doc_id) DO UPDATE SET content_hash=excluded.content_hash,
acl=excluded.acl, version=documents.version+1, deleted=0""", (doc_id, h, acl))
# Tombstones: documents we know about that the source no longer lists.
for (doc_id,) in db.execute("SELECT doc_id FROM documents WHERE deleted=0").fetchall():
if doc_id not in seen:
n = db.execute("DELETE FROM chunks WHERE doc_id=?", (doc_id,)).rowcount
db.execute("UPDATE documents SET deleted=1 WHERE doc_id=?", (doc_id,))
stats["deleted"] += 1
stats["chunks_removed"] += n
db.commit()
stats["embed_calls"] = embed_calls - start_calls
stats["live_chunks"] = db.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
return stats
RUN1 = {
"hr/leave-policy.md": ("Employees get 24 days of paid leave per year.\n\n"
"Unused leave up to 10 days carries over to the next year.", "group:all"),
"eng/oncall.md": ("The on-call engineer acknowledges pages within 15 minutes.\n\n"
"Escalate to the secondary after 30 minutes without acknowledgement.", "group:eng"),
"fin/expenses.md": ("Submit expense claims within 30 days with receipts attached.", "group:all"),
}
RUN2 = {
# Changed: second paragraph edited, first paragraph identical.
"hr/leave-policy.md": ("Employees get 24 days of paid leave per year.\n\n"
"Unused leave up to 5 days carries over to the next year.", "group:all"),
# Unchanged text, but the ACL tightened: chunks must be re-labelled.
"eng/oncall.md": ("The on-call engineer acknowledges pages within 15 minutes.\n\n"
"Escalate to the secondary after 30 minutes without acknowledgement.", "group:sre"),
# fin/expenses.md removed from the source. New doc added.
"it/vpn.md": ("Connect to the VPN before opening internal dashboards.", "group:all"),
}
if __name__ == "__main__":
db = sqlite3.connect(":memory:")
setup(db)
for name, src in [("run 1", RUN1), ("run 2", RUN2), ("run 2 again", RUN2)]:
print(name, ingest(db, src))
print("chunks for eng/oncall.md:",
db.execute("SELECT DISTINCT acl FROM chunks WHERE doc_id='eng/oncall.md'").fetchall())
print("tombstoned:", db.execute("SELECT doc_id FROM documents WHERE deleted=1").fetchall())
Output:
run 1 {'new': 3, 'changed': 0, 'unchanged': 0, 'deleted': 0, 'chunks_added': 5, 'chunks_removed': 0, 'embed_calls': 5, 'live_chunks': 5}
run 2 {'new': 1, 'changed': 2, 'unchanged': 0, 'deleted': 1, 'chunks_added': 2, 'chunks_removed': 2, 'embed_calls': 2, 'live_chunks': 5}
run 2 again {'new': 0, 'changed': 0, 'unchanged': 3, 'deleted': 0, 'chunks_added': 0, 'chunks_removed': 0, 'embed_calls': 0, 'live_chunks': 5}
chunks for eng/oncall.md: [('group:sre',)]
tombstoned: [('fin/expenses.md',)]
In run 2 the leave policy changed one paragraph, so one chunk was embedded and one removed while the first paragraph kept its vector. The on-call doc changed only its ACL (group:eng to group:sre), so its chunks were re-labelled with zero embedding calls. The VPN doc is new, and the expenses doc was tombstoned. Running the same input again does nothing and calls the embedder zero times.
Two simplifications to fix before production. Two identical paragraphs in one document would get the same chunk ID; add the ordinal or a duplicate counter to the ID. And the stub writes to SQLite only; a real indexer writes to the vector store and the metadata DB, so it needs either a transactional outbox or a reconciliation job to repair the case where one write succeeded and the other did not.
Failure modes#
| Failure | Symptom | Fix |
|---|---|---|
| Deleted doc still cited | Answers quote a retired policy | Listing diff + tombstones; source-vs-index reconciliation |
| Scanned pages ingested as empty | "No information found" for content that exists | Characters-per-page check; route to OCR |
| Tables flattened | Wrong numbers in answers | Layout-aware parsing; tables as Markdown with headers |
| ACL lost or stale | Users see chunks they should not | ACL in hash; fail closed on missing labels; ACL tests |
| Modified-time triggers re-embed everything | Cost spikes with no content change | Skip on content hash |
| Retry creates duplicates | Same chunk appears twice in top-k | Deterministic chunk IDs; upsert |
| Out-of-order events | Old version overwrites new | Version check in the indexer |
| Mixed embedding models in one index | Recall collapses after partial upgrade | Versioned indexes; alias swap |
| Edits lost during backfill | v2 missing recent changes on swap day | Dual-write during backfill |
| Boilerplate floods results | Footers and disclaimers in top-k | Cross-document chunk-hash dedupe |
| DLQ grows unseen | Gradual coverage loss | DLQ dashboard and alert by reason |
Decision table#
| Situation | Choose |
|---|---|
| Under ~100K docs, changes rare | Nightly full rebuild is fine; skip CDC |
| Large corpus, frequent edits | Hash-based incremental with chunk-level reuse |
| Database source | CDC (Debezium or the database's native stream) |
| SaaS source with webhooks | Webhooks plus nightly listing diff |
| Mostly born-digital PDFs | CPU extraction, OCR only on flagged pages |
| Many scans or forms | Per-page routing with a GPU OCR/layout pool |
| Policies, manuals, wikis | Structure-aware chunking with section paths |
| Evals show context-free chunks fail | Contextual enrichment, budgeted per rebuild |
| Vectors fit in RAM budget | HNSW |
| Hundreds of millions of vectors or tight RAM | Quantized IVF or HNSW with re-scoring |
| Changing embedding model, chunker or parser | New index version, dual-write, eval, alias swap |
| Backfill not urgent | Batch API at half price |
If you are building your first one, the PDF question-answering bot with citations is a small end-to-end version of the same flow, and the RAG cheat sheet covers the retrieval settings that consume what this pipeline produces. For the retry and idempotency patterns in more depth, the long-running agent design uses the same ideas for tool calls.
Sources#
- OpenAI: new embedding models and API updates, for
text-embedding-3-smalland-largeprices and dimensions. - OpenAI embeddings guide, for the
dimensionsparameter and the 8,192-token input limit. - OpenAI Batch API, for the 50% discount and 24-hour window.
- OpenAI rate limits, for how token and request limits work by tier.
- Anthropic: Introducing Contextual Retrieval, for the 49% and 67% retrieval-failure reductions and the $1.02 per million document tokens estimate.
- Malkov and Yashunin, HNSW, the graph index used by most vector stores.
- pgvector README, for HNSW defaults (
m = 16,ef_construction = 64), IVFFlat trade-offs and dimension limits. - FAISS: guidelines to choose an index, for IVF and PQ trade-offs.
- Matryoshka Representation Learning, the technique behind shortenable embeddings.
- Qdrant collections and aliases and Elasticsearch aliases, for blue/green index swaps.
- Debezium PostgreSQL connector, for CDC delete and tombstone events.
- Docling and Tesseract, for layout-aware parsing and OCR.
- SQLite UPSERT, used in the example.