engine-mlx

A small, honest LLM inference engine for Apple Silicon — in Rust, on Apple MLX. Token-exact with mlx_lm, OpenAI-compatible, fully local.

Runs entirely on your Mac — private, offline, no data leaves the machine. Open source under the MIT license.

What it is

engine-mlx loads a quantized model, runs prefill and decode, and serves an OpenAI-compatible HTTP API — so any OpenAI client, agent or harness can talk to it locally. It targets the Qwen3 family in MLX format and links Metal directly through Apple's MLX C API. Correctness first: it generates text that matches mlx_lm token-for-token at temperature 0, and ships with a reproducible benchmark so performance is verified, not trusted.

Token-exact

Matches mlx_lm token-for-token at temperature 0 — verified by tests, not by eye.

Static KV cache

Pre-allocated buffers + masked SDPA, so decode throughput doesn't collapse as context grows.

Stable under load

Each request rebuilds fresh state; no buffer accumulation across sustained runs.

How a request runs

One OpenAI-compatible request, traced on the real source code: HTTP handler → single GPU worker → tokenize → prefill (static KV cache) → decode loop on Metal → detokenize → SSE. Interactive — hover the nodes.

Benchmarks

Measured on a MacBook Air M1, 16 GB — quantized Qwen3 in MLX 4-bit format, decode throughput from the engine's own reproducible benchmark. These are honest local numbers, not a speed record: absolute throughput still trails mlx_lm, and the gap is kernel efficiency.

Model (4-bit)

Scenario

Decode throughput

Qwen3-0.6B

100-token decode

~91 tok/s (peak)

Qwen3-1.7B

100-token decode

~42 tok/s

Qwen3-1.7B

500-token decode (int8 KV)

~37 tok/s

Tech details

Language

Rust (workspace of 8 crates)

Backend

Apple MLX via mlx-c (links Metal directly)

API

OpenAI-compatible: /v1/chat/completions, /v1/models, /health, SSE streaming

Models

Qwen3 family, MLX 4-bit quantized format

Pipeline

BF16, pipelined async_eval, in-graph greedy argmax, static KV cache + masked SDPA

Tokenizer

Pure-Rust multi-format (BPE / WordPiece / Unigram), HF-parity tested

Platform

macOS on Apple Silicon (M-series); non-MLX crates build on Linux for CI

License

MIT

Honest limitations

This is a reference-quality baseline, not a speed record. Absolute throughput still trails mlx_lm, and the gap is kernel efficiency, not graph overhead. It targets the Qwen3 family and requires macOS on Apple Silicon plus the MLX C API. The point isn't to beat the fastest runtime — it's to be readable, auditable, and token-exact, so you can trust exactly what it does.

Why it exists

engine-mlx is proof of competence: writing an inference engine from the tokenizer up — prefill, decode, KV cache, Metal via the MLX C API — forces you to understand every layer instead of trusting a black box. It's also a statement about where NXM stands: local-first and sovereign. The model runs on your machine, the code is open and auditable, and nothing you type leaves your Mac.

Created by Daniele Oddo — AI Engineering Architect

Project: engine-mlx · github.com/dangranaz/engine-mlx

View on GitHub →