Building mullama: What We Learned Replacing Ollama from Scratch

This article was rewritten in October 2026 to describe mullama’s actual architecture and to withdraw a benchmark figure that had not been published.

We built mullama because the way most people run local models — a daemon on one port, spoken to over HTTP — is not the only shape the problem can take, and for a lot of applications it is not the right one. That is not a criticism of Ollama. Ollama is good software that solves a real problem. It solves a slightly different problem from the one we cared about.

This post covers why we started, what mullama is, the parts that are genuinely hard, and where we think it does and does not belong.

Why Not Just Ollama

Ollama makes it easy to run open-weight models locally. Pull a model, run ollama serve, hit the API. For most use cases this is exactly right, and if it is right for you, you should keep using it.

The friction appears when the model is a component of an application rather than a service next to it:

Inference behind a network hop. If your application is in Node.js, Go or PHP, the usual options are to run a separate daemon and talk to it over HTTP, or to spawn a Python process and talk to that. Both add a process to supervise, a port to manage and a serialisation boundary on every call. For a desktop app, an edge deployment or a CLI tool, that is a lot of machinery for a function call.

One integration per language. The llama.cpp engine is C++, and every language that wants to call it in-process ends up with its own hand-written FFI layer of varying quality. We wanted one safe core with thin, consistent bindings rather than a separate wrapper per ecosystem.

Migration cost. People have Ollama commands in scripts, Modelfiles in repositories and clients pointed at localhost:11434. A replacement that requires rewriting any of that will not be adopted, however good its internals.

So we built mullama to be both things at once: a library you can embed in-process, and a server that behaves like the one people already use.

What mullama Is

mullama is a local LLM runtime built on llama.cpp that runs GGUF models — Llama, Qwen, DeepSeek R1, Mistral, Phi, Gemma, LLaVA, and any other GGUF file from Hugging Face. The core is a safe Rust API over the llama.cpp C++ library, with native bindings on top of it for Python, Node.js, Go, PHP and C/C++.

It has two modes of use.

As a library, inference runs inside your process with no daemon and no HTTP:

from mullama import Model, Context
model = Model.load("llama3.2-1b.gguf", n_gpu_layers=32)
ctx = Context(model, n_ctx=4096)
print(ctx.generate("Hello, AI!"))

The Rust, Node.js, Go and PHP bindings have the same shape: load a model, create a context, generate.

As a server, it exposes the same CLI verbs as Ollama — run, pull, serve, list, ps, create, show, rm, cp — reads the same Modelfile format, resolves the same short model names, and listens on the same port:

# Run a model; the daemon starts in the background
mullama run llama3.2:1b "What is the capital of France?"

# Start an OpenAI-compatible HTTP server on port 11434
mullama serve --model llama3.2:1b

The HTTP server speaks the OpenAI chat-completions API, so existing OpenAI SDK, LangChain and LlamaIndex clients work against it, and it also provides an Anthropic-compatible API, which Ollama does not.

Around the engine sit the parts a serving layer needs: per-model resource limits, least-recently-used model eviction, persistent statistics, Prometheus metrics, a built-in web UI and TUI, multimodal input (vision models, and real-time audio with voice activity detection), embeddings, grammar-constrained JSON output via llama.cpp’s grammar feature, and work-stealing parallel inference behind a parallel feature flag. GPU acceleration is chosen at build time, with backends for CUDA, Metal, ROCm, Vulkan, OpenCL, SYCL and llama.cpp’s RPC mode.

The Hard Parts

Building on llama.cpp from a safe language is mostly a matter of getting a small number of things exactly right.

Ownership across the FFI boundary. llama.cpp manages its own memory for weights, KV caches and scratch buffers. A context must not outlive the model it was created from, and nothing may be freed while the engine still holds a pointer into it. In C this is a convention; in Rust it can be a type, which is the main reason the core is Rust rather than a scripting language. Each binding then inherits those lifetimes instead of reinventing them.

Version churn. llama.cpp is under very active development, and its C API changes across releases. mullama pins the engine as a git submodule and updates it deliberately. Every update means reviewing the API changes and adjusting the bindings, and that is an ongoing maintenance cost of building on a fast-moving dependency rather than a one-off.

Consistency across six bindings. It is easy for a Python binding to grow a convenience the Go binding lacks, and for the documentation to describe the union of all of them. Keeping the bindings thin over one C ABI is what keeps them honest.

Compatibility as a contract. Being a drop-in replacement means matching behaviour, not just endpoint names: model aliases, Modelfile semantics and the port. Every divergence becomes somebody’s broken script.

Model Lifecycle

The model lifecycle in a multi-model server is more involved than “load, serve, unload”:

Resolution. Turn a short name such as llama3.2:1b, an hf:owner/repo reference or a path into a concrete GGUF file and quantisation.
Loading. Allocate memory, read weights, initialise the context; on a GPU, allocate VRAM and transfer weights.
Serving. Handle requests within the model’s resource limits.
Eviction. When a request arrives for a model that is not loaded and memory is short, something has to go. mullama evicts the least recently used model.
Unloading. Release everything in order, after in-flight work completes or is cancelled.

Eviction is the interesting step. LRU is a sensible default and a poor fit for some workloads — one that alternates between two models that cannot both fit will thrash under any policy that only looks backwards. We have not published measurements of load time, eviction behaviour or swap latency, and we do not quote any here. If you are evaluating mullama for a multi-model workload, those are the numbers to measure on your own hardware and models: time to load each model from cold, time to first token after an eviction, and how often your request pattern forces one.

What We Think Worked

Two front doors on one engine. The same runtime serving as an embedded library and as an Ollama-compatible daemon means an application can start against the server during development and move inference in-process later without changing models or behaviour.

Compatibility over novelty. Matching Ollama’s CLI, Modelfiles and port, and the OpenAI API, removed most of the adoption cost. Nobody has to learn a new tool to try it.

Observability as a feature. Prometheus metrics and persistent statistics make the server’s behaviour something you can look at rather than infer, which matters more for a local server than people expect: local deployments are where nobody is watching.

Trade-offs

mullama is not a high-throughput GPU serving system. If you are serving a high-QPS production workload on data-centre GPUs, vLLM is designed for that and mullama is not. And if you want the simplest path to a local model with the largest community behind it, Ollama remains the default for good reason.

mullama’s niche is narrower: applications that want local inference as a library in the language they already use, teams that want an Ollama-compatible server without the daemon being the only option, and anyone who needs the Anthropic-compatible API locally.

Lessons

Building mullama taught us that the distance between “run a model locally” and “make a local model a dependency of an application” is larger than it looks. The existing tools are very good at the first. The second is mostly about packaging, lifetimes and compatibility — unglamorous problems that decide whether anybody can actually use the thing.

If you need a local model server for application development, Ollama is excellent and you should feel no pressure to switch. If you want the same models in-process, from Rust, Python, Node.js, Go, PHP or C, without running a daemon, that is the gap mullama fills.