numaperf

Overview

numaperf addresses a performance problem that is invisible in the mean and dominant in the tail. On a multi-socket machine, memory is attached to sockets, and a thread reading from a node it is not running on pays a latency penalty on every access. For throughput workloads this averages out. For latency-critical inference serving it does not: it shows up at the ninety-ninth percentile, which is the number an SLA is written against.

The library gives a Rust application explicit control over the things the allocator and scheduler otherwise decide for it. Topology is discovered at startup. Threads are pinned through an RAII guard, so a pin is released when its scope ends rather than leaking across a thread pool. Memory placement is stated explicitly — bind to a node, prefer a node, interleave across nodes, or allocate local — rather than inherited from whichever thread happened to touch the page first. Work can be scheduled per node, data structures sharded per node, and device locality taken into account so a GPU's memory traffic does not cross a socket boundary unnecessarily.

Observability is part of the design rather than an afterthought. NUMA problems are diagnosed by seeing which allocations landed where, and a library that places memory without reporting placement leaves an engineer guessing. The measurement surface is what makes a claim about locality falsifiable.

Platform support is stated honestly: Linux on x86_64 and aarch64 is the target, with graceful degradation on macOS so a developer can build and test on a laptop without the code paths diverging. The minimum supported Rust version is 1.70.

Where it loses: most applications should not use this. If the workload is not memory-bound, or runs on a single-socket machine, or is dominated by network latency, explicit NUMA placement adds complexity and buys nothing measurable. hwloc and libnuma are also mature, well-understood and callable from any language, and for a C or C++ codebase they remain the sensible choice. numaperf is for Rust services where tail latency is the product requirement and the hardware has more than one socket.

The open question is portability of the tuning. A placement strategy tuned for one topology is not automatically right for another, and the second machine a service is deployed on rarely has the same shape as the first.

The diagnostic sequence we would suggest before adopting anything here: confirm the machine has more than one NUMA node, confirm the workload is memory-bound rather than compute- or network-bound, and confirm the problem appears in the tail rather than the mean. If any of those three is false, explicit placement will not help and the complexity is unjustified. Skipping that check is how NUMA tuning acquires its reputation for being folklore; the technique works, and it works on a narrower set of workloads than its advocates imply.