From the source
- AWS added a Model Leaderboard to SageMaker JumpStart, a view in SageMaker Studio that ranks open-weight models side by side on quality, speed and cost.
- It answers a practical question for JumpStart users: comparing models otherwise means reading model cards one at a time and hunting for benchmark scores published under different conditions.
- Quality scores come from three reasoning benchmarks, BIG-Bench Hard, GPQA Diamond and MMLU-Pro, which AWS runs itself with the same evaluation setup for every model.
- Performance and cost metrics come from sending a standardized test workload to an endpoint using each JumpStart deployment configuration, so the numbers are like-for-like.
- At launch the leaderboard covers 70+ models from providers including
Meta, Google, Qwen, IBM and Arcee, with more added over time.
- Estimated cost reflects SageMaker instance cost only, based on the on-demand hourly rate for the configuration's default instance type, and excludes data transfer, storage, inference component costs and discounts such as Reserved Instances or AWS Savings Plans.






