# Semantic search with no server in the loop

> Sentence embeddings, int8 vectors and a brute-force cosine scan let Stashio search your saved links by meaning entirely on the phone.

2026-08-11 · 19 min read · stashio, math, engineering · by ntan (ntan) for uranashel

Canonical HTML: https://uranashel.com/blog/on-device-embeddings.html

---

Three weeks ago you saved an article about hydrostatic pressure in parking garages. Today you need it, and all you can recall is "the water pressure thing". What do you type? Keyword search fails here because the query and the document share no words, a failure mode retrieval people call vocabulary mismatch. Stashio ships semantic search for exactly this case. The product story lives in [the second-brain post](https://uranashel.com/blog/stashio-second-brain.html): my pre-Stashio Apple Notes held roughly 2,400 unsorted links, and after three months of daily dogfooding about 94% of my retrieval attempts succeed in under 10 s. This post is the machinery behind the semantic half of that number.

The constraint, as always at this studio, is that the machinery runs on the phone. No embedding API, no vector database in someone's data center, no query leaving the device. That rules out most of the standard retrieval stack, and, it turns out, none of the useful parts.

## From sentences to vectors

A sentence embedding model reads a piece of text and outputs a fixed-length vector. Ours outputs 384 numbers. Training pushes texts with similar meaning toward nearby points in that 384-dimensional space, so "hydrostatic pressure in a basement garage" and "water pressure underground parking" land close together even though they share one word. The geometry does the matching that the vocabulary cannot.

Stashio uses a distilled MiniLM-class sentence encoder with 384-dimensional output, weights quantized to int8, about 25 MB inside the app bundle. It is a multilingual model because our users mix Vietnamese and English in the same library, sometimes in the same query. A multilingual embedding space puts "áp suất thủy tĩnh" near "hydrostatic pressure", which means a Vietnamese query finds an English article without any translation step. On iOS the encoder runs through Core ML, on Android through LiteRT. Studio bench note: median encode time for a title-plus-excerpt input is 31 ms on an iPhone 15 with the Neural Engine and 54 ms on a Pixel 8 on CPU, 200 runs per device, release builds, batteries above 50%.

## Cosine similarity, normalized early

Two vectors are compared by the angle between them: `cos θ = a·b / (|a||b|)`. Identical direction scores 1, unrelated directions score near 0. Raw vector magnitude mostly encodes text length and token frequency artifacts, so we throw it away by L2-normalizing every vector once, at save time. After normalization `|a| = |b| = 1` and cosine collapses to a plain dot product, 384 multiplies and adds. That is the whole trick. For unit vectors Euclidean distance ranks identically anyway, since `|a − b|² = 2 − 2·(a·b)`, so nothing is lost by picking the cheaper formula.

## Brute force is fine below 10k items

Search is a loop. Embed the query, dot it against every stored vector, keep the top 20 in a small heap. Over 10,000 items that is 3.84 million multiply-adds per query, which sounds expensive until you measure it. Studio bench note: 10,000 synthetic 384-dim vectors, 200 queries, medians. The float32 scan takes 1.8 ms on an iPhone 15 with vDSP and 3.4 ms on a Pixel 8 with NEON intrinsics. The int8 scan takes 0.9 ms and 1.6 ms on the same two devices. The encoder needs 31 ms just to embed the query. Next to the encoder, the scan barely registers.

Approximate indexes like HNSW and IVF exist for a real reason, and that reason is a hundred million vectors in a server rack. On a phone they charge rent: roughly 1.5× memory for graph links, insert work on every share-sheet save, tombstone bookkeeping when you delete, recall that drops below 1.0, and three tuning parameters per index. The exact scan has recall 1.0 by construction, zero tuning, and delete is a row delete. On our own numbers the crossover where an approximate index starts paying for itself sits somewhere above 50,000 vectors. My whole library after four years of hoarding is 2,400 items, about 7,900 vectors once the PDFs are chunked. There is an HNSW branch in the repo. It has never been merged.

## Int8 and the price of a byte

A 384-dim float32 vector costs 1,536 bytes. At 10,000 items the index weighs 15.4 MB, which a modern phone shrugs at in storage but feels in memory bandwidth during a scan. Quantizing each vector to int8 cuts it to 384 bytes plus one 4-byte scale factor, 3.9 MB for the same 10,000 items, and the narrower reads are the main reason the int8 scan above runs twice as fast.

The scheme is symmetric per-vector quantization. Take the normalized vector, compute `s = max|x_i| / 127`, store `q_i = round(x_i / s)` together with `s` itself. A dot product then runs in int32 accumulators and rescales once at the end: `a·b ≈ s_a·s_b·Σ q_a[i]·q_b[i]`. ARM's sdot instruction eats four of those multiply-adds per lane per cycle.

```
// per query: embed, normalize, quantize, then scan
fun topK(q: QVec, items: List<QVec>, k: Int): List<Hit> {
    val heap = BoundedMinHeap<Hit>(k)
    for (item in items) {
        var acc = 0                     // int32
        for (i in 0 until 384) {        // NEON sdot in practice
            acc += q.q[i] * item.q[i]
        }
        val score = acc * q.scale * item.scale
        heap.offer(Hit(item.id, score))
    }
    return heap.sortedByDescending { it.score }
}
```

Quantization costs accuracy, so we measured the cost. Against exact float32 top-10 lists on my dogfood library, replaying 500 real queries from my own local search history, int8 reproduced 98.4% of the results. Every disagreement sat in ranks 8 through 10, where neighboring scores differ by less than 0.004 and the ordering is honestly arbitrary. For a bookmark app that trade is free money. One detail matters: quantize the normalized vector. Quantizing first and normalizing after wastes int8 range on magnitude you were about to divide out.

## Chunking long PDFs

One vector per item works for links and short notes. It fails for a 68-page motor datasheet, because the encoder reads about 256 tokens and a single vector for 68 pages averages everything into mud. Stashio splits extracted PDF text into windows of roughly 180 words with a one-sentence overlap, so no idea gets cut mid-thought. Each window becomes its own vector, tagged with the item id and the page number. That datasheet becomes 214 chunks, about 83 KB of int8 index; opening the result deep-links to the matching page.

An item's score is the maximum over its chunk scores. We tried mean pooling first and discarded it, because averaging punishes long documents: one perfect page drowns among 213 mediocre ones. Max pooling scores a document by its best page, which matches how people remember PDFs anyway, by the one diagram they need.

## Hybrid ranking, because keywords still win exact matches

Embeddings are terrible at identifiers. The query "E9" should return the screenshot of pillar E9 on floor B2 from my garage logs, and no 384-dim vector reliably separates E9 from E7. Part numbers, error codes, ISO standard names, all the same story. So keyword search stays: SQLite FTS5 with BM25 scoring over titles, tags and extracted text. Each engine covers the other's blind spot, paraphrase on one side and exact tokens on the other.

Merging two ranked lists is its own small science. Our first attempt was a weighted sum of normalized scores, discarded after a week, because BM25 scores and cosine scores live on unrelated scales and every normalization we tried was fragile against outlier queries. What shipped is reciprocal rank fusion: `score(d) = Σ 1/(60 + rank_i(d))`, summed over both lists. Only ranks matter, the scales cancel, and the constant 60 keeps a single first-place vote from steamrolling an item both engines ranked fifth. It is one line of code. It has survived every query type we have thrown at it.

## Syncing an index the server cannot read

Published inversion results keep showing that sentence embeddings can be decoded back into much of their source text. A vector is content, not metadata, and Stashio treats it exactly like the bookmarks themselves. When you enable the optional account, the index syncs through atuan's backend as encrypted blobs: shards of a few hundred vectors each, encrypted on the device before upload, with keys that never leave your hardware. The server stores ciphertext and a version counter per shard. It can tell how big your library is and when it changed. It cannot search it, and neither can we.

Merging across devices happens locally after download, per item id, newest write wins. Each vector also carries the encoder model version, so when we ship a better model the app re-embeds stale items lazily in the background and a mixed-version library stays searchable the whole time. Because the index is built on the phone before any of this, search works in airplane mode, in a basement garage, on an install with no account at all. The reasoning behind that default is the studio's [on-device-first post](https://uranashel.com/blog/on-device-first.html).

## The latency budget, honestly

Add it up: 31 ms to embed the query, about 1 ms to scan, under 1 ms for FTS5 plus fusion over a few thousand rows. Call it 40 ms from keystroke to ranked results, or 70 ms on the slower bench phone. The 94%-of-retrievals-under-10 s figure from the top of this post was never limited by compute; the other 9.9 s is a human remembering that the thing they want had something to do with water pressure. The phone's job is to be ready the moment the words arrive, with no server in the loop and nothing to be down. The short pitch, screenshots included, is on [the Stashio app page](https://uranashel.com/apps/stashio.html).

---

uranashel · [Home](https://uranashel.com/) · [Apps](https://uranashel.com/apps.html) · [Lab](https://uranashel.com/lab.html) · [About](https://uranashel.com/about.html) · [Blog](https://uranashel.com/blog/) · [Developers](https://uranashel.com/developers/) · [API docs](https://uranashel.com/docs/) · [Privacy](https://uranashel.com/privacy.html)

Machine-readable: [llms.txt](https://uranashel.com/llms.txt) · [sitemap.xml](https://uranashel.com/sitemap.xml) · [openapi.json](https://uranashel.com/openapi.json) · [API](https://uranashel.com/api/v1/)