From the source
Today, in partnership with Artificial Analysis , we release Pipette , an open-source platform for benchmarking foundation models on edge devices.
Pipette is built around a simple premise: on-device behavior is a property of the deployed system, not the model in isolation.
It turns edge deployment into an empirical systems problem, making the interactions that determine real-world behavior visible and testable.
What is being released today: A public dataset of lab-verified results generated under published reproducibility protocols.
It contains five on-device performance metrics for more than 1,000 model × quantization × runtime × device × context configurations.
The current data spans 30+ models, multiple quantization formats, llama.cpp builds for macOS, iOS, Windows, and Android, and context lengths from 256 to 8,192 tokens.
Initial published results cover MacBook Pro with M5 Max, iPhone 17 Pro, and Galaxy S26 Ultra, with AMD Ryzen AI Max+ 395 and Radeon 8060S results coming soon.
Explore the coverage details here .
Open-source benchmark clients for macOS, Windows, iOS, and Android.
…
