Google updates Android Bench with new LLMs, but Gemini still lags behind
Ars Technica AI · Ryan Whitwam · 2026-07-08
Google updates Android Bench, its LLM benchmark for Android app development across 100 tasks, by adding eight new frontier models including Claude Fable 5 and Qwen 3.7, while Gemini continues to lag behind competing models on Google's own leaderboard.
Appears in
Extraction
Topics: ai-benchmarksandroid-developmentcode-generationllm-evaluationgoogle
Claims
- Google updated Android Bench with eight new models: Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, and Qwen 3.7 Max.
- Android Bench now includes cost and efficiency metrics alongside accuracy, and has expanded to cover open-weight models since its March launch.
- Google adopted a new framework for Android Bench intended to be easier for developers to use and for the community to submit evaluation results.
- Gemini lags behind competing models on Android Bench despite the benchmark being created and maintained by Google.
- Code generation has emerged as one of the most popular LLM applications, creating demand for domain-specific development benchmarks that separate high-quality outputs from low-quality ones.
Key quotes
Separating the useful outputs from straight-up slop means choosing the right tool.