A 45-Million Parameter Model in 14MB, Aimed at Wearables and Robots

cactus-compute/needle picked up 662 stars in a day. Needle 2 is a 45-million parameter foundation model compressed into a single 14MB binary, targeting phones, wearables, smart home devices and robots. A full session needs roughly 28MB of RAM.

Those numbers in context

Fourteen megabytes is smaller than most mobile apps. Twenty-eight megabytes of working memory fits hardware that has no business running a language model at all — a smartwatch, a sensor hub, a domestic robot's controller.

The architecture is described as a Simple Attention Network, using Hadamard MLPs, grouped query attention, engram key-value memory and multi-lane hyper-connections, running a 256-token sliding window with bounded memory overhead.

On performance, the team claims Needle 2 trades wins with FunctionGemma 270M, LFM2.5 230M and Apple FM while being 5x to 70x smaller, and quantised to 2 bits against their f16. That is a vendor claim on a young repo — a reason to test, not a result.

The design decision worth stealing

The capability list is deliberately narrow: tool calling and structured data extraction using byte-level grammar constraints, tool retrieval that surfaces the top five options from a larger catalogue, and confidence scoring. This is not a model built to converse. It is built to be the routing and extraction layer in front of something else.

The confidence head is the part worth pointing at. Every response carries a calibrated confidence score from a learned head — set a threshold, act above it, escalate below it.

That one mechanism is what makes a small on-device model deployable when a general chat model would not be. The failure mode of tiny models is confident nonsense, and a calibrated threshold converts that from a correctness problem into a routing problem: handle the easy 80% locally, escalate the rest to a larger model in the cloud. Latency, privacy and cost all improve, and the quality floor gets set by the escalation path rather than by the small model itself.

Getting at it

pip install cactus-needle

That gives you Python inference, LoRA fine-tuning, model export, a browser playground, and GPU acceleration on NVIDIA and Apple Silicon. MIT licensed.

Why this direction matters

Most of the industry's attention and capital points at the frontier. The economically larger question may be how small a model can get while remaining useful, because that number decides whether inference happens in a data centre or on the device in someone's pocket.

Every workload that fits on-device is a workload that needs no GPU cluster, sends no data anywhere, and carries no per-token cost. For anyone thinking about AI unit economics at scale, that is the more interesting frontier.

Source: cactus-compute/needle — GitHub