Adaptive RAG — a stateful retrieval agent at production scale
The retrieval and evaluation infrastructure behind a stateful interviewer agent — 0.95 Recall@5 over 10K+ docs, 68% latency cut for 500+ concurrent users.
I architected a 3-path RAG pipeline with hybrid keyword + dense search and cross-encoder re-ranking, behind a stateful LangGraph agent that adapts question difficulty in real time. Latency dropped from 2.5s to 0.8s, domain-query F1 rose 42% over the single-path retriever it replaced, and the system serves 500+ users at 99.9% uptime.
The system
The product is an AI interviewer that runs a real conversation: it scores the candidate's answer, adapts the difficulty of the next question, and follows up with targeted probes. Doing that credibly requires retrieving the right context — fast — from a large, heterogeneous knowledge corpus. A wrong retrieval here is not a search-engine miss; it is the interviewer asking about the wrong topic in front of a candidate.
What I built
1. The adaptive-intelligence core (LangGraph)
A stateful LangGraph agent where each node is a reasoned step: scoring the response, deciding difficulty scaling, and retrieval-conditioning each counter and follow-up probe so the agent argues like an interviewer, not a chatbot. State is carried across turns so the conversation stays coherent over long sessions.
2. Three parallel retrieval paths
A single dense index wasn't enough — the corpus has very different shapes. Three retrieval paths run in parallel and fuse their results:
- Conversation embeddings — semantic match over the live dialogue for short-term context.
- MongoDB lookup — structured / relational signals (candidate state, rubric metadata).
- Knowledge vault — a deep domain index over 10K+ documents using 512-d embeddings.
Each path contributes candidates to a hybrid keyword + dense ranking, then a cross-encoder re-ranker re-scores the top-K with full query-document attention. The geometry behind hybrid dense + keyword ranking is written up separately — the same cosine/L2 and precision-recall arithmetic every dense retriever rests on.
3. Latency: 2.5s → 0.8s
Three retrieval paths sounds expensive. It isn't, when orchestrated well:
- Async orchestration — the three paths run concurrently, not sequentially; we wait on the slowest, not the sum.
- Context fusion — candidates are deduped, re-ranked once, and only the fused top-K is passed to the LLM, shrinking the prompt.
- Right-sized embeddings — 512-d gave us the recall we needed without the cost of 1K+ vectors.
4. Evaluation & guardrails
I don't trust LLM output without measurement. The eval harness scores every change on:
- Recall@k — is the right context being retrieved?
- Faithfulness — is the answer grounded in retrieved context, not invented?
- Answer-relevance — does it actually answer the question?
Plus LLM guardrails with citations so every claim is traceable to a source — safe, grounded outputs by construction, not by hope. The harness is the source of truth: every optimization above shipped only after it moved a measured number.
Measured outcome
All numbers from the eval harness, measured on the same corpus and query set before and after the 3-path architecture replaced the single-path dense retriever:
| Metric | Single-path (before) | 3-path hybrid (after) |
|---|---|---|
| Recall@5, 10K+ doc vault | — | 0.95 |
| Query latency, p50 | 2.5 s | 0.8 s (−68%) |
| Domain-query F1 | baseline | +42% |
| Concurrent users served | — | 500+ at 99.9% uptime |
The single-path system predates the eval harness, so its Recall@5 was never measured under the same protocol — which is itself the lesson. The deltas that are comparable (latency, F1) were measured on identical query sets.
What I learned
Retrieval quality is a system property, not an embedding property. The wins came from the topology — parallel paths, hybrid search, cross-encoder re-ranking — and from making the eval harness the source of truth so every optimization was a measured bet, not a guess.