Aragog: The Medical RAG That Refuses to Answer When the Evidence Is Too Thin
A hybrid FAISS + BM25 assistant that treats retrieval confidence as a safety boundary, not just a ranking signal.
- Aragog uses retrieval distance as a safety policy, so off-topic questions fail before the model can improvise an answer.
- Its hybrid retriever pairs semantic search with keyword matching, which matters when medical language includes both concepts and exact drug or symptom terms.
- The repo is engineered to run lean, with memory-conscious choices like reranker removal, chunk pruning, and precomputed indices.
- The broader lesson is simple: in narrow, high-stakes domains, the best guardrail may live inside retrieval rather than outside it.
Why this assistant says “I can’t answer that”
Most chat assistants are built to keep talking. Aragog is built to stop talking when the question is out of bounds. That is the interesting part: it does not merely rank medical answers, it first decides whether the prompt belongs anywhere near medical knowledge.
That makes the repo feel less like a chatbot and more like a triage layer. The goal is not universal helpfulness. The goal is controlled answerability, where the retrieval system becomes the safety boundary.
The medical gate is hidden inside retrieval
Aragog’s guardrail lives in `is_relevant_context`. The system embeds the user query, compares it to retrieved context, and hard-rejects the prompt if the best similarity score stays below a threshold. In the research notes, that cutoff is around 0.45.
FAISS + BM25 is the real engine
The hybrid retriever is the practical heart of the project. FAISS gives Aragog semantic recall, so questions can match conceptually related medical content even when the wording differs. BM25 patches the opposite failure mode, pulling in exact terms that matter in medicine, like medication names, symptom phrasing, and spelling-sensitive expressions.
| Layer | What it catches | Where it helps | Trade-off |
|---|---|---|---|
| FAISS | Semantic similarity | Paraphrases, concept matches, related medical context | Can miss exact terminology |
| BM25 | Exact lexical overlap | Drug names, rare symptoms, precise phrases | Can miss broader meaning |
| Hybrid merge | Both signals together | Medical retrieval with better recall and precision | More moving parts, but better coverage |
| Threshold gate | Whether the query is close enough | Refusal when the prompt is too far from domain data | Less helpful on borderline prompts, by design |
That merge step matters because medicine is unforgiving about vocabulary. A question can be semantically close and still fail on exact wording. Aragog avoids betting on only one retrieval style.
def retrieve_candidates(query):
faiss_hits = faiss_search(query)
bm25_hits = bm25_search(query)
return merge_results(faiss_hits, bm25_hits)
def is_relevant_context(query, context):
score = cosine_similarity(embed(query), embed(context))
return score >= 0.45Why it is built to run lean
The repo’s engineering choices are not ornamental. The notes point to reranker removal to save memory, chunk truncation to keep prompts focused, and precomputed indices so the system can start without expensive setup work. That is a product decision, not an apology.
| Choice | What it buys | What it gives up |
|---|---|---|
| Remove reranker | Lower memory use and simpler inference | Some ranking quality |
| Truncate chunks | Smaller prompts and faster retrieval | Less surrounding context |
| Precompute FAISS index | Faster startup and predictable deployment | Less flexibility at runtime |
| Lazy loading | Reduced idle resource use | More complexity in initialization |
Taken together, these choices point to a system designed for constrained environments. The repo is not trying to win a benchmark on maximal sophistication. It is trying to stay small, responsive, and safe enough to be useful.
From prototype to application
Aragog is not just a retrieval script. The backend includes authentication, persistent user storage, and a React frontend, which makes it a real multi-user application rather than a notebook demo. That matters because domain gates are only useful if people can actually rely on them in a stable interface.
The structure also suggests deployment intent. FastAPI handles the service layer, MongoDB stores user data, and the frontend is separated cleanly from the retrieval logic. The result is a narrow tool with an application wrapper around it, which is exactly where this kind of domain-specific assistant belongs.
Where Aragog sits among alternatives
Aragog does not compete by being the most general assistant. It competes by choosing a different objective function. Compared with a generic chat-first system, or even a single-retriever RAG stack, it is more selective, more explicit about refusal, and more willing to trade breadth for domain control.
| System type | Domain control | Exact term recall | Semantic recall | Refusal behavior | Memory footprint | Best use case |
|---|---|---|---|---|---|---|
| Generic chat-first assistant | Low | Inconsistent | Broad but loose | Usually answers anyway | Variable | Open-ended conversation |
| Single-retriever RAG | Medium | Depends on retriever | Good if embeddings fit | Often weak | Moderate | General knowledge lookup |
| Aragog | High | Strong via BM25 | Strong via FAISS | Built-in hard refusal | Lean | Narrow, high-stakes medical Q&A |
That is the niche. Aragog is interesting because it treats retrieval as a policy layer. In a domain where wrong answers are expensive, the ability to say no is not a limitation. It is the feature.
What this repo is really teaching
The clean lesson here is architectural, not medical. If a domain is narrow and the cost of hallucination is high, you do not need a smarter liar. You need a better gate. Aragog shows how far you can get when the retrieval layer is allowed to enforce that boundary itself.