Systems validation

Two questions, one discipline

This offer covers two problems that look unrelated but share a method. In both, the system appears to work, the usual check passes, and the failure only shows up under conditions that ordinary testing does not reach: an awkward tensor shape, or a thread that happened to be scheduled on the far socket. In both, the job is to construct those conditions on purpose, measure carefully, and hand back something a team can rerun.

Both strands are narrow. We take them on because Skelf has built and published the instruments behind them. We do not offer general GPU performance consulting, compiler development, or whole-system performance audits, and if your question sits outside what is described here we will say so in the first conversation rather than stretch the scope.

Strand A: GPU kernel correctness campaigns

For: inference vendors, kernel and compiler teams, and anyone shipping hand-optimised, autotuned or LLM-generated CUDA or Triton kernels.

The question: are these kernels silently wrong?

A kernel that crashes gets fixed. A kernel that returns plausible numbers that are slightly wrong, on some shapes, ships. The usual check, comparing output against a reference with allclose on one or two convenient shapes, is blind to whole classes of bug. Tail-mask errors appear only when a dimension is not a multiple of the block size. Accumulator scaling and online-softmax rescaling errors may stay inside a loose tolerance on small inputs and leave it on large ones. A single fixed tolerance is either too loose for well-conditioned operations or too tight for reduced-precision ones.

What the published work shows, and where it stops

The method rests on three studies from GPUEmu, each recorded in the evidence register with its scope:

Study	Measured result	Boundary
Correctness corpus (P1)	On a 26-op corpus across five GPU classes, a standard one-shape check accepted all ten deliberately buggy LLM-style kernels; the seeded fp64 oracle caught ten of ten with no false positives on sixteen correct controls. Nine of nine on the 24-op single-GPU corpus.	A small, deliberately constructed corpus. Not an estimate of recall on arbitrary kernels, and silent on bug classes it does not contain.
Tolerance calibration (P2)	Per-op calibrated tolerances raised bug recall from 65% to 82% against a single fixed atol/rtol, with no loss of precision.	Calibration is fitted to control kernels; a workload with different numerical regimes needs its own controls.
Input generation (P3)	Adversarial sampling reached 93% recall in a seven-strategy ablation; regular shapes only missed every tail-mask bug.	Measured on the research corpus with kernel and oracle held fixed.

All three are about correctness. None says anything about kernel performance. The corpus write-up has the detail.

How a campaign runs

1. Scope the op list. We agree which operations matter most, usually those on the hot path of your production models or those most recently rewritten or generated.
2. Build fp64 references for each op, on the CPU, so the truth does not depend on the hardware under test.
3. Generate adversarial inputs that respect each op’s schema: non-multiple dimensions, extreme magnitudes, degenerate shapes, the edges where masking and accumulation go wrong.
4. Calibrate tolerances per op from known-correct controls, so a reduced-precision op is not failed for being reduced-precision, and a full-precision op is not passed for being nearly right.
5. Reproduce every failure from a recorded seed, so your engineers can step through it.
6. Hand over a CI harness that reruns the campaign without a GPU on every kernel change.

The kernel validation guide sets out the decision in more depth. If your existing test infrastructure can host the oracle and inputs, we build there rather than install new tooling.

Strand B: NUMA tail-latency investigations

For: performance and SRE leads running latency-critical services on multi-socket or multi-node hosts.

The question: is NUMA placement behind our p99?

On a NUMA machine, memory attached to another socket costs more to reach than local memory. A service whose threads and data drift apart can show a healthy median and an erratic tail. But many tail-latency problems that look like NUMA are something else: lock contention, allocator behaviour, garbage collection, noisy neighbours, the network. Changing placement on a hunch can make things no better and harder to reason about.

What we do

• Topology audit. What the hardware and firmware actually expose: node count, distances, which cores and devices (NICs, NVMe, GPUs) sit on which node, and firmware settings such as node interleaving that change the picture.
• Baseline under representative load, at p50, p99 and beyond, on the hosts you run in production or a faithful copy.
• Controlled placement experiments. Pin the hot path to one node, bind its memory locally, then vary one thing at a time. Each experiment is recorded so it can be rerun.
• Cross-node traffic attribution with numastat and hardware counters, so a change in tail latency can be tied to a change in remote memory access, or shown not to be.
• A recommendation: which placement change, if any, is justified, and the evidence for it.

NumaPerf is a Rust runtime offering topology discovery, RAII thread pinning, explicit placement policies, per-node scheduling and locality observability (claim record). It publishes no benchmarks of its own, and we will not quote any; whether placement moves a service’s p99 has to be measured on that service. Most investigations need only standard Linux tooling. NumaPerf becomes relevant only when the fix requires placement control inside a Rust codebase. The NumaPerf essay on when NUMA actually matters describes the filtering we start with.

A finding that NUMA is not your problem is a valid outcome. It comes with the experiments that ruled placement out, which tells your team where not to spend the next month.

What we need from you

	Kernel correctness	NUMA investigation
Code	Kernels under test and the reference semantics for each op	The service, or a build of it we can run under load
Hardware	Access to the GPU classes you ship on, for confirmation	Production-class hosts, or a copy with the same topology
Load	Representative shapes from your models	A load generator or replayed traffic that reproduces the tail
People	A kernel engineer to triage findings	An engineer who can change pinning and deploy settings

How it typically runs

A technical diagnostic (£2,500–£5,000) scopes the op list or audits the topology, reviews existing tests or latency data, and proposes the campaign or experiment plan. For NUMA, the diagnostic can settle the matter: a single-node box, or a throughput-bound workload, rarely needs more.

A sprint (£8,000–£20,000) runs the campaign or the placement experiments and delivers the harness, the findings and the recommendation. Recurring re-evaluation suits kernel teams whose op library changes often, rerunning the correctness campaign on new or regenerated kernels. Our methods and procurement pages cover reporting standards, confidentiality and contracting.