Build a RAG app with LlamaIndex: local models, saved index, sources and filters
A local RAG app on LlamaIndex 0.14 with a HuggingFace embedder and Ollama. Saved index, source scores, metadata filters, a refusal cutoff and a retrieval test.
LlamaIndex is a Python framework that does the plumbing of a RAG app for you: it reads files, cuts them into chunks, embeds the chunks, stores the vectors, fetches the closest ones for a question and hands them to a language model with a prompt. The four-line quickstart is easy to find. This post goes further and builds an app you could hand to someone else: it runs on your laptop with no API key, saves the index to disk, shows which file each answer came from, filters out outdated documents, refuses when nothing relevant comes back, and has a retrieval test.
Everything below was run with llama-index-core 0.14.26 (the current release on PyPI when this was written), Python 3.15, the BAAI/bge-small-en-v1.5 embedding model and qwen3:1.7b served by Ollama. The output blocks are copied from real runs.
What happens on each call#
build (once) query (every question)
------------ ----------------------
data/**/*.md "How many leave days?"
| SimpleDirectoryReader + file_metadata |
v v embed the question
Documents (text + team/status metadata) retriever: top_k=3 by cosine,
| SentenceSplitter (Settings.node_parser) metadata filter status=current
v |
Nodes (chunks) SimilarityPostprocessor (cutoff 0.55)
| HuggingFaceEmbedding (Settings.embed_model) | no nodes left -> "Not in the documents."
v v
VectorStoreIndex --persist()--> storage/ --> response synthesizer: prompt + chunks -> LLM
|
v
answer + source_nodes (score, metadata)
The left side runs once and writes about 64 KB of JSON for this corpus. The right side runs per question. Each box is a LlamaIndex class you can swap without touching the others, and that swapability is most of what you get from the framework.
Setup#
Install the core package and the two integrations you need, not the llama-index bundle. The bundle (version 0.14.26) pulls in llama-index-llms-openai and llama-index-embeddings-openai, which you do not want in an app meant to run offline.
python3 -m venv .venv && source .venv/bin/activate
pip install llama-index-core llama-index-llms-ollama llama-index-embeddings-huggingface
# Ollama runs the chat model: https://ollama.com/download
ollama pull qwen3:1.7b
llama-index-core needs Python 3.11 or newer. The HuggingFace integration brings in sentence-transformers and PyTorch, so the venv came to 1.2 GB, 635 MB of it torch. If that is too heavy, Ollama can serve the embeddings too (the local RAG with Ollama post uses nomic-embed-text that way).
The test corpus is four short Markdown files in team folders. Two of them disagree on purpose: an old 2024 leave policy says 18 days, the current one says 24.
data/
hr/leave-policy.md 24 days paid leave, carry over up to 8, sick leave 12, parental leave
hr/leave-policy-2024.md 18 days, carry over 5, marked superseded
it/laptop-and-vpn.md laptop specs, VPN drop fix (client 5.2+, UDP -> TCP), lost laptops
finance/expenses.md file within 30 days, receipts above Rs 500, dinner cap, payout date
The app#
"""A small LlamaIndex RAG app: build once, persist, reload, query with sources and filters."""
import os
import sys
from pathlib import Path
from llama_index.core import (Settings, SimpleDirectoryReader, StorageContext,
VectorStoreIndex, load_index_from_storage)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.postprocessor import SimilarityPostprocessor
from llama_index.core.vector_stores import FilterOperator, MetadataFilter, MetadataFilters
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
PERSIST_DIR = "storage"
def configure(use_ollama: bool) -> None:
# Global defaults. Everything below picks these up unless you pass a model explicitly.
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
Settings.node_parser = SentenceSplitter(chunk_size=128, chunk_overlap=20)
if use_ollama:
from llama_index.llms.ollama import Ollama
Settings.llm = Ollama(model="qwen3:1.7b", request_timeout=120.0,
context_window=8192, thinking=False, temperature=0)
else:
# No model server: MockLLM echoes placeholder text, retrieval still runs for real.
from llama_index.core.llms import MockLLM
Settings.llm = MockLLM(max_tokens=20)
def team_and_status(path: str) -> dict:
p = Path(path)
return {
"file_name": p.name,
"team": p.parent.name, # data/hr/x.md -> "hr"
"status": "superseded" if "2024" in p.name else "current",
}
def get_index() -> VectorStoreIndex:
if os.path.isdir(PERSIST_DIR):
storage = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
return load_index_from_storage(storage)
docs = SimpleDirectoryReader("data", recursive=True, required_exts=[".md"],
file_metadata=team_and_status, filename_as_id=True).load_data()
index = VectorStoreIndex.from_documents(docs)
index.storage_context.persist(persist_dir=PERSIST_DIR)
print(f"indexed {len(docs)} files into {len(index.docstore.docs)} chunks")
return index
def current_only() -> MetadataFilters:
return MetadataFilters(filters=[
MetadataFilter(key="status", value="current", operator=FilterOperator.EQ)])
def ask(index: VectorStoreIndex, question: str, filters=None) -> None:
engine = index.as_query_engine(
similarity_top_k=3, filters=filters,
node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.55)])
response = engine.query(question)
if not response.source_nodes: # every chunk fell below the cutoff
print(f"Q: {question}\nA: Not in the documents.\n")
return
print(f"Q: {question}\nA: {str(response).strip()}")
for sn in response.source_nodes:
md = sn.node.metadata
print(f" {sn.score:.3f} {md['team']}/{md['file_name']} [{md['status']}]")
print()
if __name__ == "__main__":
configure(use_ollama="--mock" not in sys.argv)
index = get_index()
q = "How many days of paid leave do I get, and how many can I carry over?"
ask(index, q)
ask(index, q, filters=current_only())
ask(index, "My VPN keeps dropping. What should I do?", filters=current_only())
ask(index, "What is the dress code?", filters=current_only())
Here is what each piece does and where the defaults would hurt you.
Settings is the global config object that replaced ServiceContext in 0.11 (August 2024). Older tutorials that build a ServiceContext now stop with ValueError: ServiceContext is deprecated; the migration guide maps the old arguments to the new ones. Whatever you set on Settings becomes the default for every index, retriever and query engine. You can still pass embed_model= or llm= to one component to override it locally.
SimpleDirectoryReader picks a reader by file extension. recursive=True walks subfolders, required_exts keeps stray files out, and filename_as_id=True gives each document a stable ID based on its path, which you need later if you want to refresh one file instead of rebuilding everything. The file_metadata callback replaces the default metadata (file path, size, dates) with your own dict. Plain text and Markdown work with llama-index-core alone. PDF, DOCX, PPTX, CSV, EPUB and the rest need pip install llama-index-readers-file; without it the reader logs a warning and those formats are not parsed properly. The reader docs list the supported types.
SentenceSplitter is the node parser. Its chunk_size is in tokens and it tries not to cut through a sentence. At 128 tokens, each of these short files fits in one chunk. To see where it would cut, run it alone at a smaller size:
from llama_index.core import SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
docs = SimpleDirectoryReader(input_files=["data/hr/leave-policy.md"]).load_data()
print(sorted(docs[0].metadata))
for size in (128, 48):
nodes = SentenceSplitter(chunk_size=size, chunk_overlap=10).get_nodes_from_documents(docs)
print(f"chunk_size={size}: {len(nodes)} chunks")
for n in nodes:
print(" |", n.get_content()[:70].replace("\n", " "))
['creation_date', 'file_name', 'file_path', 'file_size', 'file_type', 'last_modified_date']
chunk_size=128: 1 chunks
| # Leave policy (2026) Full-time employees get 24 days of paid leave p
Metadata length (9) is close to chunk size (48). Resulting chunks are less than 50 tokens. Consider increasing the chunk size or decreasing the size of your metadata to avoid this.
chunk_size=48: 4 chunks
| # Leave policy (2026) Full-time employees get 24 days of paid leave p
| Leave accrues at 2 days per month. Up to 8 unused days carry over to t
| Sick leave is separate: 12 days per year, and a doctor's note is neede
| Parental leave is 26 weeks for the primary caregiver and 4 weeks for t
The warning tells you something the docs bury. Metadata counts against the chunk budget, because LlamaIndex prepends metadata to the chunk text when it embeds the chunk and when it sends the chunk to the LLM. With the custom file_metadata above, the embedded text for the old policy starts like this:
team: hr
status: superseded
# Leave policy (2024, superseded)
That is fine for two short fields. If you attach long metadata such as a summary or a list of tags, move those keys into excluded_embed_metadata_keys and excluded_llm_metadata_keys on each document (see Documents), or they eat your chunks.
VectorStoreIndex.from_documents runs the splitter, embeds every node and keeps the vectors in an in-memory store. persist() writes that store plus the docstore and index metadata as JSON files into storage/, and load_index_from_storage reads them back, as the save and load guide describes. The second run of the script skips embedding entirely.
as_query_engine wires a retriever (similarity_top_k=3, optional filters), any node postprocessors, and a response synthesizer that fills a prompt with the chunks and calls Settings.llm. The returned Response has source_nodes, each with the chunk, its metadata and its similarity score.
Run it#
Start with --mock to swap in MockLLM. Retrieval, filters and the cutoff all run for real; only the answer text is fake. This is the mode to use in CI, where you have no model server.
$ python rag_app.py --mock
indexed 4 files into 4 chunks
Q: How many days of paid leave do I get, and how many can I carry over?
A: text text text text text text text text text text text text text text text text text text text text
0.734 hr/leave-policy.md [current]
0.725 hr/leave-policy-2024.md [superseded]
0.659 finance/expenses.md [current]
...
Then with Ollama running, using the index saved by the mock run:
$ python rag_app.py
Q: How many days of paid leave do I get, and how many can I carry over?
A: Full-time employees get 24 days of paid leave per calendar year, with up to 8 unused days carrying over to the next year. Any amount above 8 days lapses on 31 December.
0.734 hr/leave-policy.md [current]
0.725 hr/leave-policy-2024.md [superseded]
0.659 finance/expenses.md [current]
Q: How many days of paid leave do I get, and how many can I carry over?
A: Full-time employees get 24 days of paid leave per calendar year. Up to 8 unused days carry over to the next year; anything above 8 days lapses on 31 December.
0.734 hr/leave-policy.md [current]
0.659 finance/expenses.md [current]
0.617 it/laptop-and-vpn.md [current]
Q: My VPN keeps dropping. What should I do?
A: If your VPN keeps dropping, you should update the client to version 5.2 or later and switch the protocol from UDP to TCP in settings.
0.648 it/laptop-and-vpn.md [current]
Q: What is the dress code?
A: Not in the documents.
All four questions took about 8 seconds of wall time on the laptop used for this post, with the index already on disk. Three things in that output are worth reading closely.
The old policy almost won. Without the filter, the 2024 document scored 0.725 against 0.734 for the current one. A gap of 0.009 is noise. Here the small model picked the right numbers, but with both documents in its context it could have answered "18 days" or mixed the two. The second run, with status == current, never shows it the old file. Filtering on metadata is cheaper and more reliable than hoping the model notices the word "superseded". FilterOperator also has IN, NIN, GT, LT, CONTAINS and a few others, so date ranges and team lists work the same way.
The cutoff turned a weak match into a refusal. The best chunk for "dress code" scored 0.486. Without SimilarityPostprocessor, the engine would have sent the leave policy to the model as context and let it improvise. With a cutoff of 0.55, nothing survives, source_nodes is empty, and the app says so without calling the model. The VPN question also lost its two weaker sources (0.445 and 0.402) to the same cutoff.
The cutoff number is specific to this embedding model. bge-small puts unrelated text around 0.4 to 0.5 and good matches at 0.65 and up on this corpus. Another model spreads scores differently. Pick the cutoff by looking at scores for questions you know should and should not match, never by copying 0.55 from a blog. The node postprocessor docs list rerankers and other filters that slot into the same list.
Change the prompt#
The default question-answering prompt in 0.14.26 is short:
Context information is below.
---------------------
{context_str}
---------------------
Given the context information and not prior knowledge, answer the query.
Query: {query_str}
Answer:
It does not ask for quotes and it does not say what to do when the context lacks the answer. Every query engine exposes its prompts by key, so you can replace one without subclassing anything:
from llama_index.core import PromptTemplate
from rag_app import configure, current_only, get_index
configure(use_ollama=True)
engine = get_index().as_query_engine(similarity_top_k=2, filters=current_only())
print(list(engine.get_prompts()))
strict = PromptTemplate(
"Answer only from the context. Quote the sentence you used in double quotes.\n"
"If the context does not contain the answer, reply exactly: NOT FOUND\n"
"Context:\n{context_str}\n\nQuestion: {query_str}\nAnswer: ")
engine.update_prompts({"response_synthesizer:text_qa_template": strict})
for q in ["How long do I have to file an expense claim?", "Can I work from home on Fridays?"]:
print(q, "->", str(engine.query(q)).strip())
['response_synthesizer:text_qa_template', 'response_synthesizer:refine_template', 'response_synthesizer:chat_content_qa_template', 'response_synthesizer:chat_content_refine_template']
How long do I have to file an expense claim? -> "Claims must be filed within 30 days of the expense."
Can I work from home on Fridays? -> NOT FOUND
There is no similarity cutoff in this engine, so the work-from-home question did reach the model with two unrelated chunks, and the prompt alone made it refuse. Use both: the cutoff saves a model call on obvious misses, and the prompt handles the borderline ones. For page-level citations and exact quote checking, the PDF question-answering bot post goes deeper.
Test retrieval before you judge answers#
When an answer is wrong, first check whether the right chunk was retrieved at all. This test needs no LLM, runs in seconds, and fails loudly if a change to chunk size, embedding model or metadata breaks retrieval:
"""Retrieval check: does the right file come back first? No LLM involved."""
from rag_app import configure, current_only, get_index
CASES = [ # (question, file that must rank first)
("How many paid leave days per year?", "leave-policy.md"),
("Do I need a doctor's note for sick leave?", "leave-policy.md"),
("How long is parental leave?", "leave-policy.md"),
("Can I get a laptop with more RAM?", "laptop-and-vpn.md"),
("I lost my laptop, who do I tell?", "laptop-and-vpn.md"),
("What is the deadline for filing an expense claim?", "expenses.md"),
("When do reimbursements get paid?", "expenses.md"),
("Is there a limit on client dinner spending?", "expenses.md"),
]
configure(use_ollama=False)
retriever = get_index().as_retriever(similarity_top_k=1, filters=current_only())
hits = 0
for question, expected in CASES:
got = retriever.retrieve(question)[0].node.metadata["file_name"]
hits += got == expected
print(f"{'ok ' if got == expected else 'MISS'} {got:<20} {question}")
print(f"hit@1 = {hits}/{len(CASES)}")
assert hits == len(CASES)
ok leave-policy.md How many paid leave days per year?
ok leave-policy.md Do I need a doctor's note for sick leave?
ok leave-policy.md How long is parental leave?
ok laptop-and-vpn.md Can I get a laptop with more RAM?
ok laptop-and-vpn.md I lost my laptop, who do I tell?
ok expenses.md What is the deadline for filing an expense claim?
ok expenses.md When do reimbursements get paid?
ok expenses.md Is there a limit on client dinner spending?
hit@1 = 8/8
Eight cases over four documents is a smoke test, not an evaluation. With a real corpus you want 50 or more labelled questions, recall@k and MRR, and a separate score for the generated answers. The RAG evaluation pipeline post builds that.
Failure modes that bite on real projects#
- The index forgets which embedding model built it. Load a saved index without setting
Settings.embed_modelfirst and LlamaIndex falls back to its OpenAI default. With onlyllama-index-coreinstalled,load_index_from_storageitself failed withImportError: llama-index-embeddings-openai package not found. Always call your config function before loading. - Stale index.
storage/does not notice thatdata/changed, and it does not notice a new chunk size or embedding model either. Delete it after any of those changes, or useindex.refresh_ref_docs(docs)with stablefilename_as_idIDs to re-embed only changed files. - JSON storage does not scale. The default store keeps every vector in memory and on disk as JSON. That is fine for a few thousand chunks. Past that, point a
StorageContextat a real vector store (Chroma, Qdrant, pgvector and others have integrations); the rest of the code stays the same, and metadata filters run inside the database. - PDFs that come out empty or garbled. With only
llama-index-coreinstalled there is no PDF reader. Installllama-index-readers-fileand print the first loaded document before you index anything; scanned PDFs need OCR on top of that. - Top-k too small for multi-part questions. "Leave days and carry-over" sits in one chunk here. If the facts are split across files,
similarity_top_k=1will only ever see half the answer.
LlamaIndex or your own 80 lines?#
The local RAG with Ollama post builds the same kind of app by hand with sqlite-vec, in roughly the same amount of code. So the choice is not about line count.
LlamaIndex pays off when:
- You read many file types. The file readers package covers PDF, DOCX, PPTX, CSV, EPUB and notebooks behind the same
SimpleDirectoryReadercall, and separate reader packages cover sources such as web pages and databases. - You expect to change parts. Swapping the vector store, the LLM or the embedding model is a one-line change, and the retrieval test above keeps running unchanged.
- You want the extras ready-made: metadata filters that translate to each vector database, rerankers, sub-question and router query engines, chat engines with memory, and tracing hooks.
Your own code is the better choice when:
- The corpus is small and fixed, and the pipeline is "embed, cosine, prompt". You can read every line of it, and nothing changes underneath you on a minor release.
- You need exact control of the prompt, the token budget and the request to the model. With LlamaIndex that control sits behind a few layers you have to find first, as the prompt-key lookup above shows.
- Dependency weight matters. Most of this venv's 1.2 GB came from the HuggingFace embedder, not LlamaIndex itself, but every integration adds packages you have to keep upgraded.
- You are learning RAG. Write it by hand once so you know what
as_query_enginehides, then switch.
A sensible middle path is to use LlamaIndex for loading and chunking, which is the tedious part, and keep retrieval and the model call in code you own. For a one-page view of every stage and the knobs on each, keep the RAG cheat sheet open next to the code.
Sources#
- llama-index-core on PyPI
- llama-index-llms-ollama on PyPI
- llama-index-embeddings-huggingface on PyPI
- llama-index-readers-file on PyPI
- LlamaIndex changelog (0.11.0 removes ServiceContext)
- LlamaIndex: ServiceContext migration
- LlamaIndex on GitHub
- LlamaIndex: RAG overview
- LlamaIndex: Settings
- LlamaIndex: SimpleDirectoryReader
- LlamaIndex: node parser modules
- LlamaIndex: using documents and metadata
- LlamaIndex: VectorStoreIndex
- LlamaIndex: persisting and loading
- LlamaIndex: node postprocessors
- BAAI/bge-small-en-v1.5 model card
- qwen3 in the Ollama library
- Ollama download