Android Bench
2026-10-06 · Mobile
2.0 long-horizon
- What it measures
- Long Android changes across modules: a whole feature, not a single edit.
- How
- Long-horizon task results on developer.android.com/bench. Each row is a model plus its agent harness. Columns are pass rate, CI, completion, average latency, and average cost.
- What the score means
- The percent is the average pass rate across 30 tasks. Completion is how close a failed run got. Latency is in hours. Cost is one full run.
- Caveats
- Pass rates are low and intervals are wide. First place, 18.7–47.3, overlaps second place, 13.3–42.0. Version 1.0 stays complete on this page as an archived version, and is no longer the scenario's current column.
- Updates
- The page's newest agent results are dated 2026-10-06. This page checked all 10 rows against #long-horizon-task-results on 2026-10-10.
Full board
10
All
| # | Model | Agent | Pass rate % | CI | Completion % | Avg latency h | Avg cost $ |
|---|
| 1 | Claude Opus 5.5 | claude-code | 32.7 | 18.7–47.3 | 84.7 | 15.2 | $215.4 |
| 2 | GPT 6 Astra | codex | 28.0 | 13.3–42.0 | 82.2 | 7.9 | $375.7 |
| 3 | Claude Fable 5.1 | claude-code | 22.7 | 10.7–36.0 | 82.4 | 22.2 | $492.6 |
| 4 | GPT 6 Sol | codex | 18.0 | 7.3–29.3 | 70.2 | 6.0 | $93.4 |
| 5 | Qwen3.8 Max | qwen-coder | 14.0 | 4.7–24.0 | 74.3 | 47.2 | $260.2 |
| 6 | Kimi K3 | kimi-code | 12.0 | 3.3–22.0 | 72.8 | 66.2 | $418.3 |
| 7 | GPT 6 Luna | codex | 10.7 | 4.0–17.3 | 53.2 | 6.2 | $5.3 |
| 8 | Gemini 3.8 Flash | antigravity-sdk | 8.0 | 3.3–13.3 | 47.4 | 12.1 | $34.5 |
| 9 | Gemini 3.7 Flash | antigravity-sdk | 7.3 | 1.3–14.7 | 50.1 | 9.9 | $26.0 |
| 10 | Claude Sonnet 5 | claude-code | 6.7 | 1.3–13.3 | 59.8 | 14.7 | $283.7 |