All blogs Hybrid Search · Part 2 of 3

Hybrid search with reciprocal rank fusion: semantic, keyword, filename, and vision lanes.

ipic's search box doesn't know what kind of query is coming — a meaning, an exact identifier, a half-remembered filename, or "that beach photo." So it runs four retrievers at once and fuses their rankings with one line of math. This part derives that line and computes it on a worked example.

Lanessemantic · vision · keyword · filename
Fusionweighted RRF · k = 60
Cost4–10 ms, all lanes
TL;DR

Every retrieval signal fails differently: semantic search misses exact strings, keyword search misses meaning, filename search is trivial but sparse, and only a vision lane can find a photo by its pixels. ipic runs all four per query and fuses their rankings — never their raw scores — with weighted reciprocal rank fusion: score(d) = Σ wlane / (60 + rank), weights 1.0 · 0.9 · 0.7 · 0.5. The computed example below shows the payoff twice: an exact-identifier doc the semantic lane buried at rank 8, and a beach photo it buried at rank 25 — both rescued by the lanes that could actually see them.

What this post is about

Search feels like one problem but is at least four. Consider looking for things on your own disk:

  • "the lecture where she explains radiation shielding" — a meaning. No keyword matches; you need dense vectors.
  • "q3_roadmap_final_v2" — an exact string. Dense vectors are fuzzy precisely where you need zero fuzz; BM25 nails it.
  • "that invoice… something with may" — a filename fragment. The file's name is the signal; its content is irrelevant.
  • "the beach photo with the orange sunset" — pixels. No text anywhere in the pipeline describes those; only an image embedder can see them.

No single retriever wins all four races. The engineering question: when four retrievers each return a ranking of the same files, how do you make one ranking out of them? This post derives ipic's answer from requirements, computes it on a worked example, and shows how the implementation keeps it in single-digit milliseconds. Prerequisites: none beyond "a ranking is an ordered list." Vector background: my image-search post (short version: after L2 normalization, cosine and L2 give the same ranking — a two-line proof). Series context: Part 1 built the engine; this part is the ranking math; Part 3 scales it.

The vocabulary you'll need

  • Ranking / rank: an ordered list and a document's position in it (rank 1 = best). Fusion combines rankings.
  • Semantic lane (w = 1.0): the query is embedded (bge-small, int8 ONNX) and matched against 384-d chunk vectors by integer dot product over the mmap'd i8 store.
  • Vision lane (w = 0.9): CLIP ViT-B/32 embeds the query text and every indexed image into the same 512-d space — "sunset over water" matches photographs by content, no filename hints.
  • Keyword lane (w = 0.7): BM25 via SQLite FTS5. The query becomes up to 12 quoted words (quoting disables operator injection), and FTS5 returns highlighted snippets that become the result previews.
  • Filename lane (w = 0.5): substring match over names and paths. Trivial, unbeatable when it applies, useless otherwise.
  • Score vs rank: lanes' raw scores live on incommensurable scales (a dot product, a BM25 value, a match flag). This distinction drives the whole derivation.

Deriving the formula from requirements

Suppose four lanes each hand you a ranking. What must a fusion function satisfy?

  1. Never compare raw scores across lanes. Only positions are comparable.
  2. Monotone: a better rank in any lane must never hurt the fused score.
  3. Bounded contributions: rank 1 should matter more than rank 2, but not infinitely more; a document absent from a lane (rank → ∞) contributes nothing.
  4. Weightable: trust must be expressible per lane.

The simplest function satisfying all four is the reciprocal of rank-plus-constant, summed with weights — reciprocal rank fusion (Cormack et al., 2009):

RRFk(d) = Σlane wlane · 1 / (k + ranklane(d))

Check the requirements: ranks only (1); each term decreases in rank (2); bounded between 0 and w/k with missing documents contributing zero (3); weights inline (4). ipic runs k = 60 — the original paper's empirically robust value — with weights semantic 1.0, vision 0.9, keyword 0.7, filename 0.5. Each file's score uses its best rank per lane, so a chunk that matches five times counts once.

The computed example

Seven files, three lanes shown (filename omitted for clarity), rankings chosen to disagree the way real lanes do:

Left: horizontal bars of fused reciprocal rank fusion scores for seven documents annotated with per-lane ranks, with a beach photo rescued by the vision lane and an exact-match document rescued by the keyword lane highlighted. Right: curves of points contributed versus rank for several values of the constant k.
How to read this: left — seven files sorted by fused score (computed, not sketched); each bar's label shows the file's rank in each lane. The accent bars are the two rescues: D7, a beach photo ranked 25th by semantic (text search is blind to pixels) and 1st by vision — fusing to 2nd overall. D4, an exact identifier ranked 8th semantically and 1st by keyword — fusing to 4th, ahead of D3 which semantically ranked 3rd. Right — the kernel 1/(k + r): small k makes fusion winner-take-most, large k flattens it; k = 60 is the robust middle.

Both rescues are the whole story, computed. You searched something containing an exact identifier: the semantic lane — which has never seen that string in a training-strength context — ranked the right document 8th; keyword ranked it 1st; the fused sum of evidence (1/(60+8) + 0.7/(60+1) = 0.0262) jumps it over the semantically plausible but keyword-silent D3 (0.0252). And you searched "beach photo orange sunset": semantic ranked the actual photo 25th because no text in the index describes the pixels — CLIP ranked it 1st, and the fusion (1/(60+25) + 0.9/(60+1) = 0.0265) lifts it to 2nd overall. One query box, four intents, all served, no mode picker.

The knobs: k and the weights

k controls top-heaviness (right panel). At k = 1 a rank-1 hit is worth 13.5× a rank-26 hit (½ vs 1/27 — and a pure 26× only at k = 0) — fusion becomes near-winner-take-all, and any lane's champion dominates. At k = 600 the same pair differs by 4% — fusion degenerates toward "count which lanes listed you." At k = 60, a rank-1 hit is worth ~2.6× a rank-100 hit: strong, not tyrannical.

The weights express trust ordered by how often each lane answers a natural-language query best: semantic, then vision (pixels are un-fakeable evidence when they match), then keyword (the exact-string safety net), then filename. They are a sensible prior plus hands-on tuning on my own disk, not a hyperparameter study — typical of shipped fusion weights, and tolerable because RRF is not very sensitive to them (perturb the example by ±0.1 and watch the order barely move; that flatness is requirement 3's taper doing its job).

Keeping it at 4–10 ms

The fusion itself is a HashMap pass; the engineering is in what surrounds it. Candidates are collected per shard in parallel (store locks held only for the vector scans), then each lane's candidates are sorted globally across shards so ranks are comparable, and — the detail I most appreciate — hydration happens after fusion: file paths and metadata are fetched from SQLite only for the final top-k, not for the hundreds of candidates each lane produced. The filename lane, which has no scores at all, is merged across shards round-robin so no single root monopolizes it. Total lane time: 4–10 ms typical, with the vector scan itself sub-millisecond — the query's ~30 ms end-to-end is dominated by embedding the query text, not by searching.

And deliberately: no re-ranking pass. A cross-encoder re-ranker is the standard upgrade when you can afford it — the production RAG system I run at Griphic uses one over hybrid keyword + dense retrieval at 500-user scale. On a laptop with a millisecond budget, RRF over four lanes is the right weight class: the whole point of this lane design is that each lane's champion is the evidence.

The fifth signal: query by voice, match by sound

A spoken query is transcribed locally (whisper) and runs the same four lanes over the transcript. But the audio itself also matches against indexed recordings through the acoustic-fingerprint store (Part 1's 128-dim spectral-shape + flux vectors): matches are added to the fused hits by path with acoustic weight 0.9 — the same RRF kernel, applied where a file path rather than a chunk carries the match. "Find me that talk again" works even when your spoken query words differ from the talk's words.

Reproduce

git clone https://github.com/anubhavagr/ipic
cd ipic && cargo build --release
./target/release/ipic-cli scan
./target/release/ipic-cli search "sunset over water"

The fusion table recomputes in five lines of anything: score = 1.0/(60+sem) + 0.9/(60+vis) + 0.7/(60+kw). Change k to 1, then to 600, and watch the ordering — that exercise teaches more about RRF than any diagram.

Part 3: what 350,000 files did to the design → ← Part 1: the v1 engine Repo on GitHub