What is Liquid AI’s Pipette, the free open source on device AI benchmarking platform released on August 24 by the startup founded by former MIT computer scientists in partnership with Artificial Analysis, what devices, operating systems, AI models, quantization formats, context l

Pipette is Liquid AI’s free, open-source benchmark suite for measuring an actual on-device deployment—not merely a model—across the combination of model, quantization, runtime, device, and workload. Released August 24 with Artificial Analysis, it pairs Liquid’s inference measurements with Artificial Analysis’s mobile-model intelligence evaluation. 119

What it measures and supports

• The public release includes a lab-verified dataset of more than 1,000 configurations and five on-device performance metrics; its leaderboard covers 30-plus models, seven quantization options, and four initial devices. 114
• It supports iOS and Android clients and tests locally downloaded models rather than a cloud endpoint. The initially covered model set includes Liquid’s LFM family and third-party small models such as Gemma and Nanbeige; the mobile-intelligence comparison focused on models that fit within 8 GB at 4-bit quantization. 108
• The published mobile intelligence tests use a 16K-token context limit. Liquid’s LFM catalogue itself includes 32K contexts for many models and 128K for LFM2.5-8B-A1B, but those are model capabilities, not necessarily the benchmark’s initial standardized workload. 82
• Reported metrics span prompt/prefill and generation/decode performance, time-to-first-token, end-to-end response time, and memory usage—so a fast decode rate is not treated as sufficient by itself. 411

Reproducibility

Pipette’s useful contribution is methodological: results are tied to an explicit full configuration, come from verified lab runs, and are accompanied by published protocols. It also pins software dependencies—for example, current public performance results require specified llama.cpp builds—rather than allowing runtime-version drift to be hidden. 4116

That is materially better than comparing a vendor’s isolated “tokens per second” number, but it does not eliminate variance from thermals, OS state, device SKU/RAM, and backend choice. Those fields need to match before results can be called comparable.

Initial results—and the important iOS/Android caveat

• The reported iPhone 17 Pro result of 48.37 decode tokens/s for 4-bit Gemma 4 E2B through Apple MLX is an impressive result for that specific configuration. It should be read as “Gemma 4 E2B + 4-bit + MLX/Metal + iPhone 17 Pro,” not as a universal Gemma or phone score. 45
• In the 16K-context mobile intelligence assessment, Nanbeige4.2-3B and LFM2.5-2.6B jointly led the average score, at 63. 98
• iOS can use GPU computation through Metal/MLX, whereas the initial Android client is CPU-only. 10
• Therefore, today’s iPhone-versus-Android throughput figures are not clean silicon-to-silicon comparisons: Apple results can include GPU acceleration while Android results reflect CPU inference. Android figures may be useful for CPU performance and user experience under that backend, but they cannot establish that one phone’s total local-AI hardware is inherently faster. 104
• NPU-enabled and equivalently accelerated Android backends are the necessary next step for fair cross-platform comparisons.

Relation to the mobile-chip race

Pipette arrives as chip vendors emphasize faster local/agentic AI workloads. Qualcomm says its Snapdragon 8 Elite Gen 5’s third-generation Oryon CPU reaches 4.74 GHz; Qualcomm also claims 20% CPU-performance and 35% CPU-power-efficiency improvements for that platform. 14

Separately, Qualcomm has previewed a subsequent flagship Snapdragon with an Oryon CPU targeting 5 GHz and a FlexCache design that dynamically shares cache among cores. Those announcements signal a push for faster and more responsive local workloads, but clock speed alone does not predict LLM inference: runtime, memory bandwidth, cache behavior, quantization, GPU/NPU backend, thermals, and sustained-power limits matter. 913

Liquid’s stated direction—more device coverage and NPU-accelerated testing—would make Pipette more valuable precisely because it could distinguish CPU-only gains from GPU and NPU gains under common workloads. Until that arrives, the most defensible interpretation of Pipette is a transparent deployment benchmark, not a final cross-platform hardware ranking. 64