Liquid AI Pipette Measures the Full On-Device AI Stack

Pipette is Liquid AI’s open-source benchmark suite for evaluating AI models where they actually run: on phones, laptops, PCs and other edge hardware. Released in partnership with Artificial Analysis on August 24, it treats the deployment—not just the model—as the unit of measurement. That means results are tied to the model, quantization, runtime, device and workload together. 31

Why Pipette is different from a model benchmark

Many AI leaderboards focus on capability scores or report a single inference-speed number. Pipette is designed to show how the complete stack behaves in a real deployment. A change in quantization, runtime, backend, memory pressure or hardware can alter the result, even when the model weights remain the same.

The launch included a public dataset of lab-verified results covering more than 1,000 model, quantization, runtime, device and context configurations. The initial release spans more than 30 models, multiple quantization formats and llama.cpp builds for macOS, iOS, Windows and Android. Its standardized performance data includes context lengths from 256 to 8,192 tokens. 3

Pipette’s inference measurements are paired with Artificial Analysis’s evaluation of mobile-model intelligence. The collaboration therefore looks at both how well a model performs and how quickly and efficiently a particular deployment runs. 33

What Pipette supports

Devices and operating systems

Pipette provides native iOS and Android clients for running models locally on a smartphone. Its broader benchmark coverage also includes macOS and Windows through supported runtime builds. Public launch materials identify reference hardware including the iPhone 17 Pro, Galaxy S26 Ultra and MacBook Pro with an M5 Max chip. 338

The app requires models to be downloaded to the device before testing. It is therefore measuring local inference rather than the latency of a cloud API. 4150

Models and quantization

The leaderboard includes Liquid AI’s LFM2.5 family alongside third-party models. Listed examples include Gemma, Nanbeige, Granite, Qwen and other small or compressed model variants. 941

The benchmark supports multiple quantization formats. The mobile-intelligence comparison concentrated on models that fit within 8 GB when quantized to 4 bits, making the test relevant to phones with limited memory. 341

For local deployment, Liquid’s documentation distinguishes between GGUF, commonly used with llama.cpp across CPU and GPU targets, and MLX formats designed for Apple Silicon deployments. 47 A quantized model’s file size is only one part of the result: the runtime and hardware backend determine how that model is actually processed.

Context lengths and workloads

The launch dataset reports standardized inference configurations across context lengths from 256 to 8,192 tokens. 3 The separate Artificial Analysis mobile-intelligence evaluation uses a 16K-token context limit. 33

Those figures should not be confused with a model’s maximum advertised context window. Liquid’s model documentation lists 32K-token contexts for many LFM models and 128K for LFM2.5-8B-A1B, but a model capability does not mean every phone benchmark runs that full length. 47

Which metrics does Pipette measure?

Pipette combines several dimensions of on-device performance instead of reducing the result to decode speed. Its five performance metrics cover prompt or prefill processing, generation or decode throughput, time to first token, complete-response time and memory use. 1314

That combination matters because a deployment can generate tokens quickly while still taking a long time to start, consuming too much memory or slowing down as the context grows. End-to-end response time is especially useful for judging the experience a person actually gets from a local assistant.

Artificial Analysis supplies the quality side of the mobile comparison through tests including instruction following, tool use, reasoning and related capability evaluations. 33

How Pipette tries to make results reproducible

Pipette ties each result to an explicit configuration and publishes the protocols used to produce its verified data. The public dashboard separates leaderboard comparisons from detailed results and submissions, making it possible to inspect the underlying benchmark rows rather than relying only on a headline ranking. 1

The methodology also records runtime dependencies. For example, Liquid says that current public performance results require specified llama.cpp builds, including builds b10216 and b10516. 2 Pinning the runtime helps prevent software-version changes from being mistaken for hardware or model improvements.

These safeguards improve comparability, but they do not remove every source of variation. Thermal throttling, operating-system state, device memory, firmware, backend selection and sustained power limits can all affect a phone’s result. A score is most meaningful when those variables match.

What the first results show

The launch results illustrate why Pipette reports complete configurations rather than universal model rankings.

• A reported iPhone 17 Pro result of 48.37 decode tokens per second for 4-bit Gemma 4 E2B using Apple MLX is a result for that precise Gemma, quantization, MLX/Metal and iPhone configuration—not a score that applies to every Gemma deployment or every phone. 45
• In the 16K-context mobile-intelligence assessment, Nanbeige4.2-3B and LFM2.5-2.6B jointly led the average score at 63. 33
• Artificial Analysis also reported that LFM2.5-2.6B answered a standard 1,024-token prompt on an iPhone 17 Pro in 8.0 seconds while using 2.3 GB of memory. That result describes a particular test configuration and should not be generalized to all devices. 33

The key lesson is that quality, speed, latency and memory can point in different directions. The fastest model is not automatically the most capable, and the highest capability score is not automatically the best phone deployment.

Why iOS and Android results are not yet directly comparable

The most important limitation in the initial mobile release is the difference between the two clients. The iOS version can use GPU computation through Apple’s Metal ecosystem, including MLX. The Android version was initially limited to CPU inference. 4150

As a result, an iPhone result using MLX/Metal and an Android result produced by CPU-only inference are not clean comparisons of the phones’ total AI hardware. The iOS measurement may benefit from GPU acceleration, while the Android measurement reflects the performance of the CPU path exposed by the current client.

Android results remain useful for understanding CPU-based local inference and the experience delivered by that implementation. They should not, however, be used to declare one phone’s complete AI silicon faster than another’s until equivalent GPU or NPU backends are tested under the same workload.

How Pipette fits into the mobile-chip race

Pipette arrives as chipmakers promote faster local and agentic AI processing. Qualcomm’s Snapdragon 8 Elite Gen 5 platform uses a third-generation Oryon CPU with clock speeds up to 4.74 GHz and claims a 20% CPU-performance improvement alongside a 35% improvement in CPU power efficiency. 24

Qualcomm has also previewed a subsequent flagship Snapdragon platform with an Oryon CPU targeting 5 GHz and a FlexCache architecture that allows heterogeneous CPU cores to share a dynamically allocated cache pool. 23

Those specifications show why on-device AI is becoming a hardware competition, but clock speed alone does not predict language-model performance. Quantization, memory bandwidth, cache behavior, runtime implementation, GPU or NPU use, thermals and sustained power can matter just as much.

That is where Pipette’s configuration-based approach is useful. Its strongest long-term role will be to show whether a claimed chip improvement translates into faster, more responsive and more memory-efficient local AI under a reproducible workload. For now, Pipette is best understood as a transparent deployment benchmark—not a definitive ranking of iPhone versus Android hardware.