Articles / raggenai-system-design
Multilingual RAG for Indian languages: Hindi, Hinglish and Tamil in one pipeline
Build RAG where documents and questions are in different languages. Real retrieval and token numbers for Hindi, Hinglish and Tamil, plus answer-language checks.
Picture an HR helpdesk at a company with offices in Pune and Chennai. The leave policy was written in Hindi, the insurance policy in Tamil, the travel policy in English. An employee types saal mein kitni paid leave milti hai? into the chat box. That question is Hindi grammar in Roman letters with English nouns mixed in, and the answer sits in a Devanagari document. A standard English RAG stack will usually fetch the wrong passage, and if it does find the right one, the model may reply in a script the user did not type.
This post builds that pipeline and measures it. Everything below ran on a laptop with no API key: three real embedding models, a real tokenizer comparison, a per-language retrieval eval, and an answer-language check with the LLM stubbed behind a small interface. The numbers come from a tiny test set (6 questions in 4 languages), so treat them as a smoke test that shows where things break, not as a benchmark.
Two ways to cross the language gap#
There are two designs. Most production systems end up with some mix of both.
A. Multilingual embeddings (one shared vector space)
question (any language) --> multilingual encoder --> vector --+
|--> top-k --> LLM --> answer
documents (any language) --> multilingual encoder --> index --+ (reply in the user's language)
B. Translate, then retrieve
question --> MT to English --> English encoder --> vector --+
|--> top-k --> LLM --> answer
documents --> MT to English --> English encoder --> index --+ (translate back, or prompt for it)
Design A depends on an encoder trained so that a Hindi sentence and its English translation land near each other. The sentence-transformers multilingual models were built this way: a student model learns to put translated sentence pairs where an English teacher model puts the English side (Reimers and Gurevych, 2020). The E5 and BGE-M3 multilingual families train on large multilingual retrieval data instead (multilingual E5 report, M3-Embedding).
Design B uses an English encoder, which is often the strongest one available, and pushes the language problem into machine translation. For Indian languages, IndicTrans2 from AI4Bharat is the usual open model. You pay for it in three places: translating the whole corpus at ingest (and again when documents change), an extra model call on every query, and translation errors that retrieval cannot recover from. Names, policy terms and numbers are where those errors hurt.
The test set#
Six policy documents, each in one language only, so that most questions have to cross a language boundary:
| Doc | Language | Content |
|---|---|---|
| leave | Hindi | 20 days paid leave, apply two weeks ahead |
| it | Tamil | Support 24 hours, call the emergency number for urgent issues |
| travel | English | Reimbursed within 15 working days after approval |
| wfh | Hindi | Two days a week from home, office on Tuesdays |
| insurance | Tamil | Cover up to 5 lakh rupees, parents can be added |
| laptop | English | Report loss within 24 hours with a police complaint copy |
Each of the six questions is written four ways: English, Hindi in Devanagari, Hinglish (romanised and code-mixed) and Tamil. A few examples from data.py:
QUERIES = {
"leave": {
"en": "How many paid leave days do I get in a year?",
"hi": "मुझे साल में कितनी छुट्टियाँ मिलती हैं?",
"hinglish": "saal mein kitni paid leave milti hai?",
"ta": "ஒரு வருடத்தில் எத்தனை நாள் சம்பளத்துடன் கூடிய விடுப்பு கிடைக்கும்?",
},
"insurance": {
"en": "Does the health insurance cover my parents?",
"hi": "क्या मेडिकल इंश्योरेंस में मेरे माता-पिता शामिल हैं?",
"hinglish": "kya health insurance mein parents cover hote hain?",
"ta": "மருத்துவக் காப்பீட்டில் என் பெற்றோரைச் சேர்க்க முடியுமா?",
},
# ... it, travel, wfh, laptop
}
Setup is a virtual environment with three packages. The versions used here were sentence-transformers 6.3.0 and tiktoken 0.14.0.
python3 -m venv .venv
.venv/bin/pip install sentence-transformers tiktoken numpy
Tokens cost more outside English#
Before retrieval, check the cost. The same sentence (the leave policy) was written in four ways and counted with OpenAI's tiktoken encodings cl100k_base (the GPT-4 and GPT-3.5 era tokenizer) and o200k_base (the GPT-4o era tokenizer), plus the XLM-R tokenizer inside the multilingual MiniLM embedding model.
import tiktoken
from transformers import AutoTokenizer
encs = {name: tiktoken.get_encoding(name) for name in ["cl100k_base", "o200k_base"]}
xlmr = AutoTokenizer.from_pretrained("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
base = len(encs["o200k_base"].encode(same["en"]))
for lang, text in same.items():
c = len(encs["cl100k_base"].encode(text))
o = len(encs["o200k_base"].encode(text))
x = len(xlmr(text, add_special_tokens=False)["input_ids"])
print(f"{lang:9} {len(text):5} {c:7} {o:6} {x:5} {o/base:11.2f}x")
Output:
lang chars cl100k o200k xlmr o200k vs en
en 102 22 22 25 1.00x
hinglish 111 36 33 32 1.50x
hi 108 102 37 29 1.68x
ta 150 191 43 29 1.95x
The character counts are close, but the token counts are not. On cl100k_base, the Hindi sentence takes 4.6 times the tokens of the English one and the Tamil sentence takes 8.7 times. The newer o200k_base vocabulary closes most of that gap, but Tamil still costs about twice what English does. Hinglish costs 1.5 times English even though it is written in Latin letters, because words like milti and hafte are not common English subwords. Petrov et al. measured the same effect across many languages and tokenizers.
Three practical consequences:
- Prompt cost. If your context passages are Tamil, you pay roughly twice per retrieved chunk on an
o200k-class tokenizer, and far more on older ones. The tokenizer your LLM provider uses decides this, so count with that tokenizer, not with tiktoken, if you are not on an OpenAI model. The post on saving Claude tokens covers trimming context once you know the count. - Chunk size. A chunker that splits on a fixed token count produces chunks with much less text in Tamil than in English. Split on characters or sentences, then check token counts per language.
- Embedding truncation. The XLM-R tokenizer is fair across these scripts (25 to 32 tokens), but
paraphrase-multilingual-MiniLM-L12-v2truncates input at 128 tokens (model.max_seq_lengthprinted 128 in the run). A long chunk is cut silently and the tail never gets embedded.multilingual-e5-smallreported 512.
Retrieval, measured per language#
The index is a brute-force cosine search over normalised vectors. Six documents need nothing more, and the same class shape works if you swap in Chroma, pgvector or Qdrant later.
import numpy as np
from sentence_transformers import SentenceTransformer
class Index:
"""Brute-force cosine search. Swap for Chroma/pgvector/Qdrant in production."""
def __init__(self, model, docs: dict[str, str]):
self.model, self.ids = model, list(docs)
self.vecs = model.encode(list(docs.values()), normalize_embeddings=True)
def search(self, query: str, k: int = 3) -> list[tuple[str, float]]:
q = self.model.encode([query], normalize_embeddings=True)[0]
scores = self.vecs @ q
top = np.argsort(-scores)[:k]
return [(self.ids[i], float(scores[i])) for i in top]
The eval reports hit@1 (was the right document ranked first) and MRR@3 (reciprocal rank of the right document within the top three, zero if absent) separately for each query language:
def evaluate(index: Index, queries_for) -> dict[str, tuple[float, float]]:
"""Per-language hit@1 and MRR@3. queries_for(topic, lang) -> query text."""
out = {}
for lang in LANGS:
hits, rr = 0, 0.0
for topic in QUERIES:
ranked = [doc_id for doc_id, _ in index.search(queries_for(topic, lang))]
hits += ranked[0] == topic
rr += 1 / (ranked.index(topic) + 1) if topic in ranked else 0
n = len(QUERIES)
out[lang] = (hits / n, rr / n)
return out
Four configurations were run. A is an English-only model (all-MiniLM-L6-v2) on the native-language documents, as a baseline. B is paraphrase-multilingual-MiniLM-L12-v2, the model most tutorials reach for. C simulates a perfect translate-then-retrieve: English versions of every document, the English question for every column. It is a ceiling that real MT will not reach, not a real MT system. D is intfloat/multilingual-e5-small, which needs a query: or passage: prefix on every input according to its model card; the wrapper in e5.py adds them.
hit@1 / MRR@3 per query language (6 questions each)
A english-only model, native docs en=0.33/0.42 hi=0.17/0.31 hinglish=0.17/0.36 ta=0.33/0.39
B multilingual model, native docs en=0.83/0.83 hi=0.83/0.83 hinglish=0.50/0.67 ta=0.17/0.42
C oracle translate, english model en=1.00/1.00 hi=1.00/1.00 hinglish=1.00/1.00 ta=1.00/1.00
D multilingual-e5-small, native en=1.00/1.00 hi=1.00/1.00 hinglish=0.83/0.92 ta=0.67/0.83
What the table says:
The English-only model fails as expected. Even English questions hit only 2 of 6, because 4 of the 6 answers are in Hindi or Tamil.
The popular multilingual MiniLM fails on Tamil. Tamil questions hit 1 of 6. The reason is on the model card: its language list includes Hindi, Marathi, Gujarati and Urdu, but not Tamil. The tokenizer can split Tamil text into pieces, so nothing errors, but the model was never trained to align Tamil meaning with other languages. In the run, all six Tamil questions put the two Tamil documents (insurance and IT) in first and second place, whatever they asked. The Tamil leave question ranked the Tamil insurance document first at 0.83, with the Hindi leave document nowhere in the top three. That is script matching, not meaning matching. Read the language list on the model card before you pick an encoder.
Hinglish is the hardest column for both multilingual models. Romanised Hindi is under-represented in parallel training data. With MiniLM, kya health insurance mein parents cover hote hain? found the right document with a score of 0.152 and the travel policy right behind at 0.148, which is a coin flip.
multilingual-e5-small is close to the ceiling. It is built on XLM-R's 100 languages (the card warns that low-resource languages may degrade). Its three misses were all near ties at rank 2:
ta want=travel got=[('leave', 0.823), ('travel', 0.817)]
hinglish want=laptop got=[('it', 0.803), ('laptop', 0.800)]
ta want=laptop got=[('it', 0.798), ('laptop', 0.793)]
Retrieving k=3 and letting the LLM read all three recovers those. A cross-encoder reranker would also help here; the RAG architectures comparison covers where reranking fits.
The E5 scores also sit in a narrow band (0.77 to 0.88 for the top three in every query printed above) while MiniLM's spread from 0.07 to 0.83. A similarity threshold such as "drop anything under 0.5" means something different for each model, so tune any cutoff per model and per language, or skip it and rely on top-k.
If you can afford a larger model, BGE-M3 lists support for more than 100 languages and inputs up to 8,192 tokens. It was not run here, so no numbers are given for it.
When translate-then-retrieve is still worth it#
Row C hits every question, but only because the translations were written by hand. Choose design B when your best domain-tuned encoder is English-only, when the corpus is mostly English and only queries vary, or when you need English text for downstream tools anyway. A cheap middle ground is to keep a multilingual index and also run the translated English query against it, then merge the two result lists. That costs one MT call per query and no re-ingest.
Controlling the answer language#
Retrieval brings back a Hindi passage for a Hinglish question. Left alone, many models will answer in Devanagari because the context is in Devanagari. Hinglish users usually want a Romanised reply. So detect the language of the question, state the reply language in the prompt, and check the reply.
Detection here is deliberately cheap: Unicode ranges for script, then a small marker-word list for romanised Hindi.
HINGLISH_MARKERS = {"kya", "hai", "hain", "mein", "ka", "ki", "ke", "ko", "kitne", "kitni",
"kaise", "karna", "karun", "nahi", "toh", "mera", "meri", "ab", "ho", "gaya"}
def detect_lang(text: str) -> str:
"""Script first, then a cheap word-list check for romanised Hindi."""
if re.search(r"[-]", text):
return "ta"
if re.search(r"[ऀ-ॿ]", text):
return "hi"
words = re.findall(r"[a-z]+", text.lower())
if sum(w in HINGLISH_MARKERS for w in words) >= 2:
return "hinglish"
return "en"
REPLY_RULE = {
"en": "Reply in English.",
"hi": "Reply in Hindi, in Devanagari script.",
"hinglish": "Reply in Hinglish: Hindi words in Roman script, mixed with English the way the user wrote. Do not use Devanagari.",
"ta": "Reply in Tamil, in Tamil script. Keep product names like IT or laptop in English.",
}
It labelled all 24 test questions correctly. On real traffic, a word list will miss Hinglish that avoids the common function words, and it will mislabel English sentences that contain ho or ab. Log the detected language and review a sample each week. A trained language-ID model is the upgrade when the word list stops being enough.
The prompt carries the rule and tells the model that the context may be in another language:
Answer using only the context. The context may be in a different language from the question.
Cite the [doc_id] you used. If the context does not answer it, say so.
Reply in Hinglish: Hindi words in Roman script, mixed with English the way the user wrote. Do not use Devanagari.
Context:
[leave] कर्मचारियों को हर साल 20 दिन की सवेतन छुट्टी मिलती है। छुट्टी के लिए कम से कम दो हफ्ते पहले आवेदन करना होगा।
Question: saal mein kitni paid leave milti hai?
Instructions are not guarantees, so a post-check runs on the reply, with one retry if the script is wrong. The LLM is a stub here because no API key is used; replace complete() with your provider call.
class StubLLM:
"""No API key in this demo. Replace complete() with your provider call,
e.g. client.messages.create(...) or an Ollama/vLLM endpoint."""
def __init__(self, canned: str):
self.canned = canned
def complete(self, prompt: str) -> str:
return self.canned
def answer_lang_ok(answer: str, want: str) -> bool:
"""Post-check: does the reply use the script we asked for?"""
got = detect_lang(answer)
if want == "hinglish":
return got in ("hinglish", "en") # romanised either way; Devanagari would fail
return got == want
def answer(question, passages, llm: LLM, retries: int = 1) -> str:
lang, prompt = build_prompt(question, passages)
reply = llm.complete(prompt)
for _ in range(retries):
if answer_lang_ok(reply, lang):
break
reply = llm.complete(prompt + f"\n\nYour last reply was in the wrong language. {REPLY_RULE[lang]}")
return reply
With two canned replies, the check behaves as intended:
devanagari reply ok for hinglish user? False
romanised reply ok for hinglish user? True
The check covers script only. Whether the reply says "20 days" and not "15 days" is a faithfulness question, and the hallucination detection post covers it.
Evaluating per language#
The single most useful habit from this exercise is splitting every metric by query language. Averaged over all 24 questions, MiniLM's hit@1 is 0.58, which looks like a mediocre model. Split by language, it is a decent model for English and Hindi with a Tamil column that is close to useless. An average hides a broken language, and the users of that language are the ones who complain.
A per-language eval set for production:
- Write each question in every language you serve, including the code-mixed form people really type. Collect real queries from logs when you have them; hand-written Hinglish is cleaner than what users send.
- Keep the gold document fixed across the language variants so the only variable is the query language, as in the
QUERIESdict above. - Report retrieval per language (hit@k, MRR, recall@k) and fail the build if any language drops below its own threshold, not the average.
- Report answer checks per language: reply-script correctness from
answer_lang_ok, plus a faithfulness judge. Run the judge prompt in English but show it the question, context and answer in their original languages, and spot-check its verdicts with a native speaker. The RAG evaluation pipeline post has the scoring code; add alangcolumn to it. - Track token cost per language using the counting code above, since Tamil traffic costs more per request.
Public benchmarks help with model selection before you have your own set. The multilingual E5 card reports results on MIRACL, a retrieval benchmark across 18 languages that include Hindi and Telugu but not Tamil. Your domain terms and your users' Hinglish will still differ from any benchmark.
Failure modes to plan for#
- Unsupported language, no error. Covered above. Every tokenizer will split any Unicode text, so a model without Tamil training runs fine and retrieves by script.
- Same-script pull. Even a good model gives a bonus to documents in the query's script. If one language dominates your corpus, its documents will crowd the top-k for its speakers. Check whether the right document's language matches the query's language in your misses.
- Romanised Tamil and other mixes. Tanglish (Tamil in Roman script) exists too and was not tested here. Add it to the eval if your users write it.
- Numbers and units. "5 lakh" in a Tamil document and "500,000" in an English question share no tokens. Normalise numbers at ingest if the answers depend on them.
- Silent truncation. A 128-token encoder limit plus a token-hungry script means long chunks lose their ending. Check
model.max_seq_lengthand the token count of your longest chunk per language. The RAG ingestion pipeline design covers chunking strategy in more depth.
What to use#
For Indian-language RAG on a budget, start with multilingual-e5-small, keep top-k at 3 or more, set the reply language from the question and check it, and split every metric by language from the first day. Move to a larger encoder or add translated-query retrieval only after the per-language eval shows which column is weak. Before adopting any model, check that every language you serve is on its model card's list.
The full scripts (data.py, tokens.py, retrieve.py, e5.py, answer.py) are the ones shown above; run retrieve.py and e5.py to reproduce the table. The first run downloads the three models from Hugging Face.
Sources#
- Reimers and Gurevych, Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation, 2020.
- Wang et al., Multilingual E5 Text Embeddings: A Technical Report, 2024.
- Chen et al., M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation, 2024.
- Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, 2023.
- Hugging Face model cards: paraphrase-multilingual-MiniLM-L12-v2 (language list), multilingual-e5-small (prefixes, 100 languages), all-MiniLM-L6-v2, BGE-M3; checked 22 Nov 2026.
- OpenAI, tiktoken.
- Sentence Transformers documentation.
- AI4Bharat, IndicTrans2.
- MIRACL dataset.