Rapid-MLX: An Apple Silicon LLM Inference Server Built for Coding Agents

Apple Silicon has long held an awkward position in local LLM work. Unified memory lets a single Mac mini hold a quantized model with billions of parameters, yet the serving layer around it has lagged well behind what CUDA users take for granted. Rapid-MLX aims at that gap. It is an open-source inference server, licensed Apache 2.0 and built on Apple's MLX framework. It exposes both OpenAI-style and Anthropic-style endpoints, namely /v1/chat/completions, /v1/responses and /v1/messages. The stated focus is plain: reliable tool calling for coding agents. That choice of focus is telling. The project does not treat raw speed as the only goal. It gives equal weight to whether an agent can call tools without breaking. Installation is deliberately ordinary too. The package is on PyPI and in Homebrew core, so one command, brew install rapid-mlx, gets a server running, and a desktop app and a model mirror are offered for people who prefer not to live in a terminal.

Speed is the headline claim. According to the project, decode speed reaches up to four times that of Apple's own mlx-lm, and about 1.5 times on a typical task, with the same model and the same weights. The figures come from Qwen3.5-9B at 4-bit on a Mac mini M4 Pro, using greedy decoding, with both servers left at their defaults. Two cautions apply. First, the gap between "up to 4x" and "1.5x typical" shows that the gain depends heavily on the shape of the workload. Scenarios that hit the cache or serve several requests at once benefit most. Second, the test covers one model, one machine and one decoding mode, and the publisher is the project itself. Treat the numbers as directional evidence and rerun them on your own models and tasks before deciding. To its credit, the project publishes per-task results, the method and the raw data, which lets outsiders check the work instead of trusting a single marketing figure.

Several mechanisms work together behind those numbers. The first is continuous batching, which lets many requests advance inside one decoding loop so that a long request does not starve the others. The second is prefix caching. Rapid-MLX keeps a radix-tree prefix cache in memory, stores state snapshots for hybrid-architecture models, writes the cache to disk at shutdown and restores it at startup. This matters a great deal for coding agents. On every turn an agent resends a long system prompt, the tool definitions and the whole conversation so far, and most of that prefix is identical to the previous turn. Reusing the cache lets the server skip most of the prefill work. The third mechanism is a quantized KV cache, which trades some precision for memory so that limited unified memory can hold longer contexts or more concurrent sessions. Together these target the statistical profile of agent traffic, which is long prefixes, many turns and repeated calls, rather than the peak throughput of a single question.

Tool-call parsing is the second main thread, and it separates this project from many peers. Model families emit tool calls in different text formats. Some use XML-style tags, some use JSON fragments, and some interleave calls with reasoning text. If the server parses them badly, the agent receives malformed arguments and stops halfway through a task. Rapid-MLX ships 27 parser modules, with an auto-detect fallback, so that users do not need to adapt by hand when they switch models. By contrast, the project's own comparison table says mlx-lm's server offers tool calls only for models whose tokenizer declares support, while Ollama and LM Studio each maintain their own tool-calling documentation and implementation. A caution is in order: the number of parsers is not the same as reliability. The real test is whether a valid call can still be recovered from a long conversation, nested arguments, or a stream cut off mid-call. Readers should check that with their own agent framework.

Set against the wider field, Rapid-MLX has plenty of competition. oMLX is also built on MLX and uses a tiered KV cache with a hot in-memory tier and a cold SSD tier. Ollama defaults to one parallel request per model and has added an MLX engine for safetensors models. LM Studio ships both a llama.cpp engine and an MLX engine. The server bundled with mlx-lm handles one request at a time when a quantized KV cache is in use. The comparison table, written by the project and dated 26 September 2026, itself shows how fast this category is converging on a common feature set. Differentiation will have to come from engineering detail and measured behavior. A practical reader whose workload is a local coding agent on a Mac should run Rapid-MLX against the current setup, with the same model and the same task. Watch three numbers: time to first token, cache hit rate over a long session, and the rate of failed tool calls. Those say more about fit than any speed multiple.