polymathy

Overview

polymathy asks what has to exist between a search engine and an answer, and it answers by building that middle layer and nothing else. Retrieval-augmented generation is usually presented as a single capability, which obscures that it is a pipeline of separable steps: query a source, fetch the results, extract the useful text, chunk it sensibly, embed the chunks, and return them with their provenance intact.

The service sits in front of a SearxNG instance, fetches the top result URLs, chunks and embeds their content, and returns a JSON map from chunk identifier to source URL and text. One endpoint. No model call, no answer synthesis, no prompt. What comes back is evidence with addresses attached, and what a caller does with it is a separate decision made somewhere else.

That scope is the argument. A pipeline that goes from query to generated answer in one call cannot be inspected in the middle, which is precisely where retrieval systems fail: the chunking was wrong, the extraction picked up navigation furniture, the source was irrelevant. Returning the intermediate artefact makes those failures visible instead of manifesting as a confidently wrong answer with a citation attached.

Rust with async I/O suits the workload because it is almost entirely concurrent network fetching with bursts of CPU-bound embedding work, and the combination is where a runtime with cheap tasks and no garbage collector earns its keep. The practical benefit is that fetching thirty pages concurrently does not require tuning a thread pool.

Where it loses: Haystack, LangChain and LlamaIndex offer far more — document loaders for dozens of formats, evaluation harnesses, agent integrations, a large community answering questions — and for a Python team that wants a working RAG system this afternoon they remain the right answer. polymathy is a component for people who have decided they want to own the pipeline and need the retrieval half to be fast and inspectable.

It also inherits its recall from SearxNG, which means the quality ceiling is set by the search backends configured behind it. That dependency is stated rather than hidden, because a retrieval component that silently caps your recall is worse than one that tells you where its input comes from.

Within the portfolio, polymathy is the harness between a search engine and an answer: it hands chunking and embedding to whatever content processor its PROCESSOR_URL points at and keeps its own USearch index, rather than depending on embedcache or memista. The three are separable on purpose — a different embedder, a different index — because the interfaces between them are ordinary HTTP ones rather than a shared internal abstraction.