Why We Write AI Infrastructure in Rust (and Zig, and Go)

There is no shortage of “Why we rewrote X in Rust” posts on the internet. This is not one of those. We are not here to tell you that Rust is fast, or that its type system is elegant, or that the community is friendly. All of those things may be true, but none of them are why we chose Rust — or Zig, or Go — for the systems that underpin our AI research.

This is a post about methodology. Specifically, it is about why language choice is a methodological decision when you are building infrastructure whose behaviour you intend to make falsifiable claims about.

The Problem with Benchmarking on Quicksand

Consider a common scenario in systems research. You build a prototype of a new scheduling algorithm, a caching layer, or a storage engine. You benchmark it. You report that your system achieves X throughput at Y latency under Z workload. You submit the paper.

But buried in the results are confounding variables that have nothing to do with your contribution. A garbage collector pause added 14 milliseconds to your p99 tail latency. A buffer overflow in your comparison baseline inflated its error rate. A data race in your concurrent hash map produced silently wrong results that happened to look like correct results during the evaluation window.

These are not hypothetical problems. They are the daily reality of systems research conducted in languages that leave memory management, concurrency control, and performance determinism as exercises for the programmer.

When we set out to build the infrastructure behind projects like numaperf, embedcache, and memista, we made a deliberate decision: the language itself should eliminate as many confounding variables as possible. Not because we are language partisans, but because we want our experimental claims to be about the thing we are studying — topology-aware scheduling, embedding cache eviction, vector similarity search — rather than about whether we remembered to free a buffer. That is a statement about what a measurement would mean when we make one; where a project has not yet published a benchmark, we say so on the project.

Rust: The Ownership Model as Experimental Control

Rust is our default choice for projects where we need to make precise claims about memory behaviour and performance characteristics. numaperf gives latency-critical services explicit control over NUMA topology: thread pinning, memory placement and per-node scheduling. embedcache generates embeddings locally and caches them in SQLite, keyed by content hash and model identifier. memista pairs SQLite metadata with a USearch HNSW index for approximate nearest-neighbour search. All three are written in Rust.

The ownership model is the key. When we write NUMA-aware memory placement in Rust, we know at compile time that every allocation has exactly one owner, that borrows are checked statically, and that there are no use-after-free bugs lurking in the harness. This is not a convenience — it is an experimental control. It means that when someone measures the difference between NUMA-local and NUMA-remote placement on their own service using numaperf, the difference can be attributed to topology effects rather than memory corruption artefacts. numaperf publishes no benchmark of its own, deliberately: whether placement moves a service’s tail latency has to be measured on that service.

The borrow checker also eliminates data races in safe code. When embedcache serves concurrent lookups and inserts, Rust’s type system guarantees that shared mutable state is accessed through proper synchronisation primitives. For a cache whose one unacceptable failure is returning the wrong vector for a key, that guarantee matters more than speed.

There are real costs. Rust’s compile times are slow — significantly slower than C, Go, or Zig. The learning curve is steep, particularly around lifetime annotations and trait bounds. Some patterns that are trivial in other languages (self-referential structs, certain graph structures) require unsafe code or architectural workarounds. And the borrow checker occasionally rejects programs that are, in fact, correct. These are genuine productivity taxes.

We pay them because the alternative — spending weeks hunting a subtle memory bug that turns out to invalidate an entire evaluation — is worse.

Zig: Comptime as Policy Compiler

zviz is our container runtime for executing untrusted and AI-generated code. It is OCI-compatible and contains a process by selective denial — namespaces, all 41 Linux capabilities dropped, Landlock, seccomp-BPF and cgroups v2 — rather than by emulating the kernel the way gVisor does. It is written in Zig, not Rust.

The reason is comptime. Zig’s compile-time execution model lets us build sandbox policy — seccomp filters and capability sets — during compilation, so the policy a binary enforces is fixed, deterministic and reviewable as code rather than assembled from configuration at runtime. In Rust, achieving the same effect requires proc macros — a separate, complex sublanguage that operates on token streams. In Zig, comptime is just Zig. You write ordinary functions, and the compiler evaluates them at compile time when the inputs are known.

For zviz, this matters concretely. The runtime installs a seccomp-BPF filter, which the kernel itself evaluates at syscall entry: 132 syscalls pass through to the host kernel natively, 24 are denied before any kernel code runs, and socket is filtered on its arguments. Nothing is trapped into userspace or emulated. The set of rules varies by profile, and comptime lets the filter for each profile be constructed by the compiler rather than by code that runs on every container start. We have not yet published overhead figures for zviz; the architectural point is that allowed syscalls take no detour through a second process.

Zig also gives us explicit control over allocators. In a runtime’s security-critical setup path, you want to know exactly where every byte of memory comes from and goes to; the sandboxed workload’s own memory is bounded separately, by cgroups. Zig’s allocator interface makes this the default, not an afterthought. Every function that allocates takes an allocator parameter. There is no hidden global allocator.

The downsides are real too. Zig’s ecosystem is young. The standard library is still stabilising. IDE support is improving but lags behind Rust. Documentation is sparse. We accept these costs for zviz because comptime is not a nice-to-have for the project — it is architecturally central.

Go: Concurrency Without Ceremony

route-switch, our LLM routing and prompt-tuning system, is written in Go. This might seem contradictory for a lab that just spent several paragraphs arguing for deterministic performance. route-switch is an OpenAI-compatible gateway that routes requests across providers by a configured strategy, logs per-prompt success, cost and latency to DuckDB, and reruns MIPROv2 prompt optimisation against the captured traces. The latency-critical path is the network call to the LLM provider, which takes tens to hundreds of milliseconds. A garbage collector pause of a few hundred microseconds is irrelevant at this timescale.

What matters for route-switch is the ability to manage many concurrent connections to different LLM providers, balance load across them, and run MIPROv2-based prompt tuning with minimal boilerplate. Go’s goroutines and channels make concurrent network programming straightforward. The standard library’s HTTP server and client are production-grade. The build toolchain produces statically linked binaries with no fuss.

Go’s GC is a confounding variable we can tolerate in route-switch because the questions it studies are about routing and prompting decisions (which model to call, with what prompt), not about sub-millisecond latency. If we were building route-switch to study microsecond-level scheduling, we would use Rust.

This is the core of our argument: language choice should follow from what you are trying to measure. A GC is a confounding variable when you study tail latency. It is irrelevant when you study routing policy.

The Meta-Principle: Languages as Methodology

We do not have a “one language to rule them all” policy. We have a methodological principle: the language should eliminate confounding variables relevant to the claims you intend to make.

This means:

• When measuring memory access patterns (numaperf): Use a language with no GC and explicit memory control. Rust.
• When sandbox policy should be fixed at compile time (zviz): Use a language where compile-time execution is a first-class feature. Zig.
• When managing concurrent I/O at millisecond timescales (route-switch): Use a language with lightweight concurrency and a mature networking stack. Go.
• When studying cache behaviour (embedcache) or vector index behaviour (memista): Use a language where you control allocation and deallocation. Rust.

This is not about which language is “best.” It is about which language produces the most trustworthy experimental results for a given research question.

What This Means for Reproducibility

There is a secondary benefit to this approach that we did not initially anticipate: reproducibility. When a system is written in Rust, another researcher can build it on a different machine and get the same performance characteristics (modulo hardware differences). There is no “well, it depends on your GC tuning” or “you need to set the JVM heap to exactly 8 GB.” The performance is determined by the code, the compiler flags, and the hardware. Full stop.

This is not a universal truth — Rust programs can still exhibit non-deterministic behaviour through thread scheduling, I/O ordering, or LLVM optimisation decisions. But the surface area for non-determinism is dramatically smaller than in garbage-collected or manually-managed languages.

Conclusion

We write AI infrastructure in Rust, Zig, and Go not because we think these languages are the future of all programming, but because we think language choice is part of the scientific method. When you make a claim about system performance, the tools you used to build and measure that system are part of the claim. Choosing tools that minimise confounding variables is not engineering preference — it is experimental hygiene.

If you are building AI infrastructure and intend to make empirical claims about its behaviour, consider what your language is and is not controlling for. The answer might surprise you.