embedcache

Overview

embedcache begins from a cost observation rather than a performance one. In a retrieval pipeline the same text is embedded repeatedly: during ingestion, again after a re-index, again when a chunking parameter changes, again when someone re-runs an evaluation. Against a hosted embedding API every one of those repetitions is billed, rate-limited and network-bound, and none of them produce a different vector.

The project replaces the hosted embedder rather than memoising it, which is the distinction that matters. Embeddings are generated locally with FastEmbed across more than twenty models — the BGE family, MiniLM, Nomic, multilingual E5 — and cached in SQLite keyed by a content hash together with the model identifier. Including the model id in the key is not a detail: a cache that returns a BGE vector for a MiniLM request is silently corrupting a search index, and the failure surfaces much later as poor relevance rather than as an error.

The consequences of running locally are the interesting part. There is no per-token bill and no rate limit, so re-indexing stops being a budgeted operation. Text never leaves the process, which removes an entire class of question from a privacy review. And latency becomes a function of local CPU rather than of a network round trip, which changes the shape of the ingestion pipeline: batching to amortise network cost stops being necessary.

It ships as both a Rust library and a REST API, so it can be embedded in a Rust service or run alongside one written in something else. Rust is the right choice here because the workload is a tight loop over tensors with a persistent cache attached, and predictable memory behaviour matters more than developer convenience for a component that runs on every ingest.

Where it loses: if you need a proprietary embedding model available only through a hosted API, local generation is not an option and the question is closed. Local inference also consumes CPU you were previously renting, which is a real trade on a small instance. And a semantic cache that returns a near-match for a similar-but-different query is solving a different problem — embedcache is an exact-match cache by design, because an approximate hit in an index build is a correctness bug rather than an optimisation.

Within the portfolio, embedcache produces vectors and memista can store and search them; polymathy delegates chunking and embedding to an external content processor over HTTP. They are separable on purpose, and each states where its responsibility ends.

One operational caution is worth stating. A cache keyed on content and model identifier is correct only while the model binary behind that identifier is stable. Upgrading a FastEmbed model in place without changing the key would serve old vectors against a new index, which is the one failure this design must not have, so model version belongs in the key and a deliberate cache invalidation belongs in any upgrade procedure. We would rather say that here than have somebody discover it as a relevance regression three weeks later.