On-device semantic file search in Rust: the ipic engine architecture.

What this post is about

You have thousands of files. Some you can search by name; most you can't — the thing you remember about a lecture is what was said in it, not its filename. Cloud services solve this by uploading your life to a server. On-device RAG (retrieval-augmented search, running locally) solves it without that: an indexing process reads your files, converts their content into searchable representations, and answers natural-language queries from those representations — all on your machine.

ipic is my from-scratch build of that, in Rust, under three hard constraints: fast (millisecond queries), small RAM (it's a file manager, not a datacenter), and never a network call after first-run model download. The series is the honest build log:

1. this post — the v1 engine: lanes, vector store, whisper, crash-safety;
2. Part 2 — the ranking math: reciprocal rank fusion, derived and computed;
3. Part 3 — what 350,000 files did to the design: the sharded v2 rewrite.

Prerequisites: general programming comfort. mmap, WAL, and quantization are defined below; the deeper INT8 math lives in my quantization post.

The vocabulary you'll need

• Embedding model: a small neural network (ipic uses bge-small-en-v1.5, the int8-quantized ONNX build, 384-d) that turns text into a vector such that similar meanings land close together. If the model can't even load — no network, fresh machine — a deterministic feature-hashing embedder (512-d, signed FNV hashing of word unigrams and bigrams) takes over, so search never dies.
• FTS5 / BM25: SQLite's full-text search and the classic ranking formula behind it — keyword search, exact and fast.
• whisper: open-source speech-to-text, compiled locally (whisper.cpp bindings). ipic ships q5_1-quantized models and beam-search decoding (beam 5).
• mmap: memory-mapped files — the OS pages data in on demand, so a large index costs RAM only for what you touch.
• WAL: SQLite's write-ahead log — writers append instead of rewriting pages, so batched indexing never blocks readers.

The week it was built

The repository's 23 commits tell the story with unusual clarity, because the whole thing happened in two sprints. The first: initial commit on Aug 20, then on Aug 22 the two commits that made it a product — "on-device multimodal RAG file manager" and "unified multimodal search, full file-manager actions, lean codebase."

Then the commit messages get humble, which is my favorite part of the log: "file rows were not click-sensing — open/context menu never triggered," followed by "rows never received clicks — interaction now hangs off widget responses," followed by a full "workflow bug sweep across search, navigation, file ops and toolbar." A retrieval engine can be mathematically beautiful and still lose to an event loop that never delivered the clicks. The same day: "tests no longer spawn Finder windows" and "test runs no longer touch the user's real config" — hermeticity fixes, because the first test suite was opening real windows on my desktop while it ran.

The sprint closed with two feature commits that rounded out v1: "audio-to-audio matching, stronger STT" (acoustic fingerprints — more below) and "finder-grade UX" — multi-select, icons, footer progress. Total: a working multimodal RAG file manager in roughly 72 hours of calendar, ~5k lines of Rust at that point.

The v1 engine

The pipeline, left to right. A walker fans across all cores and batch-upserts file metadata into SQLite (WAL mode). Extraction workers claim jobs: text and PDF take a fast lane; audio and video queue for whisper, gated by a semaphore so transcription parallelism matches what the machine can feed. Extracted text is chunked — 320 words (≈512 tokens) with a 24-word trailing overlap so context at boundaries survives, at most 8 chunks per file — then batch-embedded and written to both stores transactionally, and the file flips to Done.

Crash-safety is a job-state machine, not a ceremony: every unit of work carries a state; anything interrupted mid-flight resets to Pending on the next launch and is re-claimed. Kill the process during a transcription marathon, restart, and nothing completes twice and nothing is lost.

The vector store: 396 bytes at a time

The piece I'm proudest of in v1 is the custom vector store, because it's where the constraints compound. Dense retrieval needs a 384-float vector per chunk; storing f32 costs 1,536 bytes each. ipic's store writes fixed 396-byte records: [chunk_rowid u64][scale f32][384 × i8] — one byte per component, symmetric quantization with a per-vector scale, a 64-byte file header carrying the dimension and an FNV-1a hash of the model identity (change the embedder and the store refuses to serve stale vectors — derived data is wiped and files re-queued). Freed slots recycle through a free-list; freed middles are tombstoned so scans skip them; freed tails truncate straight off the file.

Queries scan the file through an mmap (prefaulted with MADV_WILLNEED, so the first query hits resident pages) and compute top-k as integer i8×i8 dot products — the kind of loop the compiler auto-vectorizes to NEON — parallelized by rayon over 4,096-record shards with a partial top-k per shard, then merged. Why this is safe is measured, not assumed: at one million chunks, the documented benchmark shows recall within 1% of f32 at rank 10, for a scan that moves ~400 MB of i8 data through the page cache. The quantization math behind that tolerance is the same affine map and s/2 bound I derive in the INT8 post — retrieval is kinder to quantization than almost any workload, because a dot product over 384 components averages the per-component noise away.

Whisper and the GPU trap

The most instructive v1 decision: whisper runs on CPU by default, even though the GPU is faster. The whisper context keeps mutable decode state, and the Metal path is only safe with a single concurrent state — one stream of transcription. The CPU path (Apple's Accelerate BLAS) transcribes at ~35× realtime with beam-5 decoding — and parallelizes across workers. For a background indexer, twelve 35×-realtime lanes beat one faster serialized lane every time. "Fastest hardware" and "fastest system" are different claims; the semaphore in the pipeline exists to enforce the second one.

Audio-to-audio: the fourth signal, early

Text embeddings can't hum. For audio files (and spoken queries), v1 already shipped acoustic fingerprints: a 128-dim vector per recording — 64 frequency bands × two statistic halves, spectral shape (mean log-energy per band) and spectral flux (frame-to-frame change), each half unit-normalized so timbre and dynamics weigh equally. Two encodes of the same recording land close; a cosine over 128 dims costs microseconds. It's "find me that talk again," not "find music that feels like this" — the honest limit, stated in the README, not hidden.

What v1 couldn't do

• One catalog for every root — a single SQLite database and a single vector file meant one writer's lock domain and one scan stream, fine at thousands of files, suspect at hundreds of thousands.
• Images were filename-only — indexed by name and folder context, not pixels.
• No compute budget — the walker, extraction pool, ONNX runtime, and whisper each sized themselves; pointed at a big enough disk, they'd fight each other for cores mid-index while a query waited.
• No honest progress — "indexing…" with no rate or ETA is a trust problem in a tool that lives in your menu bar.

Pointing v1 at a 350,000-file home directory turned all four into blockers at once. That story — per-root shards, a CLIP vision lane that nearly starved to death, and the measured results — is Part 3. First, the math that makes multi-lane ranking coherent: Part 2.

Reproduce

git clone https://github.com/anubhavagr/ipic
cd ipic
cargo build --release          # rustup + cmake; builds whisper.cpp locally

./target/release/ipic          # GUI
./target/release/ipic-cli scan
./target/release/ipic-cli search "radiation shielding on mars"
./target/release/ipic-cli status

First run downloads the local models once (~250 MB); everything after is offline.

Part 2: four retrieval lanes, one formula → Repo on GitHub Same philosophy at work: RAG case study