SWE-bench Verified
Autonomous resolution of real-world GitHub issues across popular open-source Python repositories, hand-verified by human software engineers.
Standardized evaluations, empirical telemetry, and reproducible agent benchmarks. Every score is captured under isolated execution harnesses without prompt leakage.
Autonomous resolution of real-world GitHub issues across popular open-source Python repositories, hand-verified by human software engineers.
Autonomous command-line navigation, multi-step debugging, environment triage, and terminal state evaluation in isolated Linux sandboxes.
Algorithmic problem solving, complex mathematical reasoning, and edge-case handling on fresh competitive programming problems published post training cutoff.
Multi-language code editing and refactoring across diverse programming paradigms, evaluating SEARCH/REPLACE diff format compliance and unit test verification.
Graduate-level physics, chemistry, and biology problems written and validated by domain experts to withstand search engine retrieval.
Complex scientific calculations, numerical simulations, and differential equations requiring multi-step mathematical programming in Python.
| RANK | MODEL & PROVIDER | VERIFIED SCORE | TEST HARNESS | COST / TASK | RUN DATE | PROVENANCE & SOURCE |
|---|---|---|---|---|---|---|
| 01 |
GPT-6 Astra (max)
OpenAI
|
55.0% | OpenAI Sandbox v4 | $3.26 | 2026-09-01 | ★ In-House Test |
| 02 |
Gemini 4 Argon (high)
Google
|
55.0% | default | — | 2026-10-21 | ★ In-House Test |
| 03 |
Claude Fable 5.1
Anthropic
|
54.6% | SWE-agent v2.1 | $4.12 | 2026-09-01 | ★ In-House Test |
| 04 |
Claude Opus 5.5
Anthropic
|
53.8% | default | — | 2026-10-21 | ★ In-House Test |
| 05 |
Muse Wonder
Muse Wonder
|
51.5% | default | — | 2026-10-21 | ★ In-House Test |
| 06 |
Claude Sonnet 5.5
Anthropic
|
50.5% | default | — | 2026-10-21 | ★ In-House Test |
| 07 |
GPT-6 Sol
OpenAI
|
49.5% | default | — | 2026-10-21 | ★ In-House Test |
| 08 |
Mistral Large 4 (preview)
Mistral
|
46.5% | default | — | 2026-10-21 | ★ In-House Test |
| 09 |
Qwen3.8 Max (0902)
Alibaba
|
46.0% | Qwen-Agent Loop | $2.10 | 2026-09-01 | ★ In-House Test |
| 10 |
Kimi K3 (max)
Kimi
|
44.8% | Kimi Agent | $2.00 | 2026-09-01 | ★ In-House Test |
| 11 |
Step 5 Preview (preview)
StepFun
|
44.6% | StepFun Code Harness | $0.71 | 2026-09-01 | ★ In-House Test |
| 12 |
Grok 4.7
xAI
|
44.0% | default | — | 2026-10-21 | ★ In-House Test |
| 13 | 43.5% | default | — | 2026-10-21 | ★ In-House Test | |
| 14 |
Muse Spark Open (816B MoE)
Muse Spark Open 816b Moe
|
43.0% | default | — | 2026-10-21 | ★ In-House Test |
| 15 |
Gemini 3.8 Flash (high)
Google
|
41.8% | Google Code Harness | $1.24 | 2026-09-01 | ★ In-House Test |
| 16 |
Qwen3.8-Flash-Next
Alibaba
|
40.6% | Qwen-Agent Open | $0.37 | 2026-09-01 | ★ In-House Test |
| 17 |
GLM-5.3 Fast (highspeed)
Zhipu / Z.ai
|
40.2% | default | — | 2026-10-21 | ★ In-House Test |
| 18 |
GPT-6 Luna
OpenAI
|
39.5% | default | — | 2026-10-21 | ★ In-House Test |
| 19 |
Claude Haiku 5.5
Anthropic
|
39.0% | default | — | 2026-10-21 | ★ In-House Test |
| 20 |
MiniMax M3.1 Flash (max)
MiniMax
|
37.5% | default | — | 2026-10-21 | ★ In-House Test |
| 21 |
DeepSeek V4 Pro (0813 max)
DeepSeek
|
36.8% | SWE-agent v2.1 | $0.67 | 2026-09-01 | ★ In-House Test |
| 22 |
Beam 1.0 501B (A23B MoE)
Beam 10 501b A23b Moe
|
36.0% | default | — | 2026-10-21 | ★ In-House Test |
| 23 |
DeepSeek V4 Flash Vision (max)
DeepSeek
|
35.6% | OpenHands | $0.31 | 2026-09-01 | ★ In-House Test |
| 24 |
Qwen3.8 27B (xhigh)
Alibaba
|
34.6% | Qwen-Agent Open | $0.82 | 2026-09-01 | ★ In-House Test |
| 25 | 34.5% | default | — | 2026-10-21 | ★ In-House Test | |
| 26 |
K2 Horizon 375B (A23B)
Thinking Machines
|
31.8% | IFM Agent Loop | $1.20 | 2026-09-01 | ★ In-House Test |
| 27 |
Gemini 3.1 Pro (preview)
Google
|
30.2% | Google Code Harness | $0.67 | 2026-09-01 | ★ In-House Test |
| 28 |
LFM2.5 24B A2B (on-device MoE)
Lfm25 24b A2b On Device Moe
|
29.0% | default | — | 2026-10-21 | ★ In-House Test |
| 29 |
Inkling
Inkling
|
26.6% | Thinking Machines | $0.61 | 2026-09-01 | ★ In-House Test |
| 30 |
Gemini 3.5 Flash-Lite
Google
|
23.2% | Google Code Harness | $0.12 | 2026-09-01 | ★ In-House Test |
| RANK | MODEL & PROVIDER | VERIFIED SCORE | TEST HARNESS | COST / TASK | RUN DATE | PROVENANCE & SOURCE |
|---|---|---|---|---|---|---|
| 01 |
Claude Sonnet 5.5
Anthropic
|
70.6% | default | — | 2026-10-21 | ★ In-House Test |
| 02 |
Claude Opus 5.5
Anthropic
|
66.4% | default | — | 2026-10-21 | ★ In-House Test |
| 03 |
GPT-6 Astra (max)
OpenAI
|
57.9% | default | — | 2026-10-21 | ★ In-House Test |
| 04 |
Gemini 4 Argon (high)
Google
|
57.4% | default | — | 2026-10-21 | ★ In-House Test |
| 05 |
Muse Wonder
Muse Wonder
|
53.0% | default | — | 2026-10-21 | ★ In-House Test |
| 06 |
Claude Fable 5.1
Anthropic
|
52.0% | default | — | 2026-10-21 | ★ In-House Test |
| 07 |
GPT-6 Sol
OpenAI
|
51.0% | default | — | 2026-10-21 | ★ In-House Test |
| 08 |
Mistral Large 4 (preview)
Mistral
|
45.0% | default | — | 2026-10-21 | ★ In-House Test |
| 09 |
Qwen3.8 Max (0902)
Alibaba
|
44.5% | Docker Ubuntu 24.04 sandbox with mock CLI | — | 2026-09-01 | ★ In-House Test |
| 10 |
Step 5 Preview (preview)
StepFun
|
44.0% | Docker Ubuntu 24.04 sandbox with mock CLI | — | 2026-09-01 | ★ In-House Test |
| 11 |
Kimi K3 (max)
Kimi
|
43.8% | Docker Ubuntu 24.04 sandbox with mock CLI | — | 2026-09-01 | ★ In-House Test |
| 12 |
Grok 4.7
xAI
|
43.5% | default | — | 2026-10-21 | ★ In-House Test |
| 13 | 42.0% | default | — | 2026-10-21 | ★ In-House Test | |
| 14 |
Muse Spark Open (816B MoE)
Muse Spark Open 816b Moe
|
42.0% | default | — | 2026-10-21 | ★ In-House Test |
| 15 |
Gemini 3.8 Flash (high)
Google
|
40.5% | Docker Ubuntu 24.04 sandbox with mock CLI | — | 2026-09-01 | ★ In-House Test |
| 16 |
Qwen3.8-Flash-Next
Alibaba
|
40.0% | Docker Ubuntu 24.04 sandbox with mock CLI | — | 2026-09-01 | ★ In-House Test |
| 17 |
GLM-5.3 Fast (highspeed)
Zhipu / Z.ai
|
39.5% | default | — | 2026-10-21 | ★ In-House Test |
| 18 |
GPT-6 Luna
OpenAI
|
38.8% | default | — | 2026-10-21 | ★ In-House Test |
| 19 |
Claude Haiku 5.5
Anthropic
|
38.5% | default | — | 2026-10-21 | ★ In-House Test |
| 20 |
MiniMax M3.1 Flash (max)
MiniMax
|
37.0% | default | — | 2026-10-21 | ★ In-House Test |
| 21 |
DeepSeek V4 Pro (0813 max)
DeepSeek
|
35.5% | default | — | 2026-10-21 | ★ In-House Test |
| 22 |
Beam 1.0 501B (A23B MoE)
Beam 10 501b A23b Moe
|
35.0% | default | — | 2026-10-21 | ★ In-House Test |
| 23 |
DeepSeek V4 Flash Vision (max)
DeepSeek
|
34.5% | default | — | 2026-10-21 | ★ In-House Test |
| 24 | 34.0% | default | — | 2026-10-21 | ★ In-House Test | |
| 25 |
Qwen3.8 27B (xhigh)
Alibaba
|
33.5% | Docker Ubuntu 24.04 sandbox with mock CLI | — | 2026-09-01 | ★ In-House Test |
| 26 |
K2 Horizon 375B (A23B)
Thinking Machines
|
30.5% | default | — | 2026-10-21 | ★ In-House Test |
| 27 |
Gemini 3.1 Pro (preview)
Google
|
30.2% | default | — | 2026-10-21 | ★ In-House Test |
| 28 |
LFM2.5 24B A2B (on-device MoE)
Lfm25 24b A2b On Device Moe
|
28.5% | default | — | 2026-10-21 | ★ In-House Test |
| 29 |
Inkling
Inkling
|
25.5% | default | — | 2026-10-21 | ★ In-House Test |
| 30 |
Gemini 3.5 Flash-Lite
Google
|
23.2% | default | — | 2026-10-21 | ★ In-House Test |
| RANK | MODEL & PROVIDER | VERIFIED SCORE | TEST HARNESS | COST / TASK | RUN DATE | PROVENANCE & SOURCE |
|---|---|---|---|---|---|---|
| 01 |
Claude Opus 5.5
Anthropic
|
51.2% | default | — | 2026-10-21 | ★ In-House Test |
| 02 |
GPT-6 Astra (max)
OpenAI
|
50.5% | default | — | 2026-10-21 | ★ In-House Test |
| 03 |
Claude Fable 5.1
Anthropic
|
50.0% | default | — | 2026-10-21 | ★ In-House Test |
| 04 |
Muse Wonder
Muse Wonder
|
49.0% | default | — | 2026-10-21 | ★ In-House Test |
| 05 |
Gemini 4 Argon (high)
Google
|
47.5% | default | — | 2026-10-21 | ★ In-House Test |
| 06 |
Claude Sonnet 5.5
Anthropic
|
47.0% | default | — | 2026-10-21 | ★ In-House Test |
| 07 |
GPT-6 Sol
OpenAI
|
45.0% | default | — | 2026-10-21 | ★ In-House Test |
| 08 |
Mistral Large 4 (preview)
Mistral
|
43.5% | default | — | 2026-10-21 | ★ In-House Test |
| 09 |
Qwen3.8 Max (0902)
Alibaba
|
43.0% | default | — | 2026-10-21 | ★ In-House Test |
| 10 |
Kimi K3 (max)
Kimi
|
42.5% | default | — | 2026-10-21 | ★ In-House Test |
| 11 |
Grok 4.7
xAI
|
42.0% | default | — | 2026-10-21 | ★ In-House Test |
| 12 |
Step 5 Preview (preview)
StepFun
|
42.0% | default | — | 2026-10-21 | ★ In-House Test |
| 13 |
Muse Spark Open (816B MoE)
Muse Spark Open 816b Moe
|
41.0% | default | — | 2026-10-21 | ★ In-House Test |
| 14 | 40.5% | default | — | 2026-10-21 | ★ In-House Test | |
| 15 |
Gemini 3.8 Flash (high)
Google
|
39.5% | default | — | 2026-10-21 | ★ In-House Test |
| 16 |
Qwen3.8-Flash-Next
Alibaba
|
38.5% | default | — | 2026-10-21 | ★ In-House Test |
| 17 |
GLM-5.3 Fast (highspeed)
Zhipu / Z.ai
|
38.0% | default | — | 2026-10-21 | ★ In-House Test |
| 18 |
GPT-6 Luna
OpenAI
|
37.5% | default | — | 2026-10-21 | ★ In-House Test |
| 19 |
Claude Haiku 5.5
Anthropic
|
37.0% | default | — | 2026-10-21 | ★ In-House Test |
| 20 |
MiniMax M3.1 Flash (max)
MiniMax
|
35.5% | default | — | 2026-10-21 | ★ In-House Test |
| 21 |
DeepSeek V4 Pro (0813 max)
DeepSeek
|
35.0% | default | — | 2026-10-21 | ★ In-House Test |
| 22 |
Beam 1.0 501B (A23B MoE)
Beam 10 501b A23b Moe
|
34.0% | default | — | 2026-10-21 | ★ In-House Test |
| 23 |
DeepSeek V4 Flash Vision (max)
DeepSeek
|
33.5% | default | — | 2026-10-21 | ★ In-House Test |
| 24 | 32.5% | default | — | 2026-10-21 | ★ In-House Test | |
| 25 |
Qwen3.8 27B (xhigh)
Alibaba
|
32.0% | LiveCodeBench test runner with fresh problem sets | — | 2026-09-01 | ★ In-House Test |
| 26 |
K2 Horizon 375B (A23B)
Thinking Machines
|
30.0% | default | — | 2026-10-21 | ★ In-House Test |
| 27 |
Gemini 3.1 Pro (preview)
Google
|
29.0% | default | — | 2026-10-21 | ★ In-House Test |
| 28 |
LFM2.5 24B A2B (on-device MoE)
Lfm25 24b A2b On Device Moe
|
27.5% | default | — | 2026-10-21 | ★ In-House Test |
| 29 |
Inkling
Inkling
|
25.0% | default | — | 2026-10-21 | ★ In-House Test |
| 30 |
Gemini 3.5 Flash-Lite
Google
|
22.0% | default | — | 2026-10-21 | ★ In-House Test |
| RANK | MODEL & PROVIDER | VERIFIED SCORE | TEST HARNESS | COST / TASK | RUN DATE | PROVENANCE & SOURCE |
|---|---|---|---|---|---|---|
| 01 |
Claude Fable 5.1
Anthropic
|
54.0% | default | — | 2026-10-21 | ★ In-House Test |
| 02 |
Claude Opus 5.5
Anthropic
|
54.0% | default | — | 2026-10-21 | ★ In-House Test |
| 03 |
GPT-6 Astra (max)
OpenAI
|
53.5% | default | — | 2026-10-21 | ★ In-House Test |
| 04 |
Claude Sonnet 5.5
Anthropic
|
52.0% | default | — | 2026-10-21 | ★ In-House Test |
| 05 |
Gemini 4 Argon (high)
Google
|
51.0% | default | — | 2026-10-21 | ★ In-House Test |
| 06 |
GPT-6 Sol
OpenAI
|
49.0% | default | — | 2026-10-21 | ★ In-House Test |
| 07 |
Muse Wonder
Muse Wonder
|
48.0% | default | — | 2026-10-21 | ★ In-House Test |
| 08 |
Qwen3.8 Max (0902)
Alibaba
|
47.0% | default | — | 2026-10-21 | ★ In-House Test |
| 09 |
Mistral Large 4 (preview)
Mistral
|
46.0% | default | — | 2026-10-21 | ★ In-House Test |
| 10 |
Step 5 Preview (preview)
StepFun
|
45.2% | Aider CLI with benchmark harness | — | 2026-09-01 | ★ In-House Test |
| 11 |
Kimi K3 (max)
Kimi
|
45.0% | default | — | 2026-10-21 | ★ In-House Test |
| 12 |
Grok 4.7
xAI
|
44.0% | default | — | 2026-10-21 | ★ In-House Test |
| 13 |
Muse Spark Open (816B MoE)
Muse Spark Open 816b Moe
|
43.5% | default | — | 2026-10-21 | ★ In-House Test |
| 14 | 43.0% | default | — | 2026-10-21 | ★ In-House Test | |
| 15 |
Gemini 3.8 Flash (high)
Google
|
42.0% | Aider CLI with benchmark harness | — | 2026-09-01 | ★ In-House Test |
| 16 |
Qwen3.8-Flash-Next
Alibaba
|
41.0% | default | — | 2026-10-21 | ★ In-House Test |
| 17 |
GLM-5.3 Fast (highspeed)
Zhipu / Z.ai
|
40.5% | default | — | 2026-10-21 | ★ In-House Test |
| 18 |
GPT-6 Luna
OpenAI
|
39.5% | default | — | 2026-10-21 | ★ In-House Test |
| 19 |
Claude Haiku 5.5
Anthropic
|
39.0% | default | — | 2026-10-21 | ★ In-House Test |
| 20 |
MiniMax M3.1 Flash (max)
MiniMax
|
38.0% | default | — | 2026-10-21 | ★ In-House Test |
| 21 |
DeepSeek V4 Pro (0813 max)
DeepSeek
|
37.0% | default | — | 2026-10-21 | ★ In-House Test |
| 22 |
Beam 1.0 501B (A23B MoE)
Beam 10 501b A23b Moe
|
36.5% | default | — | 2026-10-21 | ★ In-House Test |
| 23 |
DeepSeek V4 Flash Vision (max)
DeepSeek
|
36.0% | default | — | 2026-10-21 | ★ In-House Test |
| 24 |
Qwen3.8 27B (xhigh)
Alibaba
|
35.0% | Aider CLI with benchmark harness | — | 2026-09-01 | ★ In-House Test |
| 25 | 35.0% | default | — | 2026-10-21 | ★ In-House Test | |
| 26 |
K2 Horizon 375B (A23B)
Thinking Machines
|
32.0% | default | — | 2026-10-21 | ★ In-House Test |
| 27 |
Gemini 3.1 Pro (preview)
Google
|
31.0% | default | — | 2026-10-21 | ★ In-House Test |
| 28 |
LFM2.5 24B A2B (on-device MoE)
Lfm25 24b A2b On Device Moe
|
29.5% | default | — | 2026-10-21 | ★ In-House Test |
| 29 |
Inkling
Inkling
|
27.0% | default | — | 2026-10-21 | ★ In-House Test |
| 30 |
Gemini 3.5 Flash-Lite
Google
|
23.8% | Aider CLI with benchmark harness | — | 2026-09-01 | ★ In-House Test |
| RANK | MODEL & PROVIDER | VERIFIED SCORE | TEST HARNESS | COST / TASK | RUN DATE | PROVENANCE & SOURCE |
|---|---|---|---|---|---|---|
| 01 |
Gemini 4 Argon (high)
Google
|
96.5% | default | — | 2026-10-21 | ★ In-House Test |
| 02 |
GPT-6 Astra (max)
OpenAI
|
96.0% | default | — | 2026-10-21 | ★ In-House Test |
| 03 |
Claude Sonnet 5.5
Anthropic
|
93.0% | default | — | 2026-10-21 | ★ In-House Test |
| 04 |
GPT-6 Sol
OpenAI
|
91.5% | default | — | 2026-10-21 | ★ In-House Test |
| 05 |
Claude Opus 5.5
Anthropic
|
90.6% | default | — | 2026-10-21 | ★ In-House Test |
| 06 |
Muse Wonder
Muse Wonder
|
88.5% | default | — | 2026-10-21 | ★ In-House Test |
| 07 |
Mistral Large 4 (preview)
Mistral
|
87.0% | default | — | 2026-10-21 | ★ In-House Test |
| 08 |
Grok 4.7
xAI
|
86.5% | default | — | 2026-10-21 | ★ In-House Test |
| 09 |
Qwen3.8 Max (0902)
Alibaba
|
86.0% | default | — | 2026-10-21 | ★ In-House Test |
| 10 |
Kimi K3 (max)
Kimi
|
85.0% | default | — | 2026-10-21 | ★ In-House Test |
| 11 |
Step 5 Preview (preview)
StepFun
|
84.5% | default | — | 2026-10-21 | ★ In-House Test |
| 12 | 84.0% | default | — | 2026-10-21 | ★ In-House Test | |
| 13 |
Muse Spark Open (816B MoE)
Muse Spark Open 816b Moe
|
83.0% | default | — | 2026-10-21 | ★ In-House Test |
| 14 |
GPT-6 Luna
OpenAI
|
81.0% | default | — | 2026-10-21 | ★ In-House Test |
| 15 |
Gemini 3.8 Flash (high)
Google
|
81.0% | default | — | 2026-10-21 | ★ In-House Test |
| 16 |
Claude Haiku 5.5
Anthropic
|
80.5% | default | — | 2026-10-21 | ★ In-House Test |
| 17 |
Qwen3.8-Flash-Next
Alibaba
|
80.0% | default | — | 2026-10-21 | ★ In-House Test |
| 18 |
GLM-5.3 Fast (highspeed)
Zhipu / Z.ai
|
79.5% | default | — | 2026-10-21 | ★ In-House Test |
| 19 |
MiniMax M3.1 Flash (max)
MiniMax
|
78.0% | default | — | 2026-10-21 | ★ In-House Test |
| 20 |
DeepSeek V4 Pro (0813 max)
DeepSeek
|
77.0% | default | — | 2026-10-21 | ★ In-House Test |
| 21 |
Beam 1.0 501B (A23B MoE)
Beam 10 501b A23b Moe
|
76.0% | default | — | 2026-10-21 | ★ In-House Test |
| 22 |
DeepSeek V4 Flash Vision (max)
DeepSeek
|
75.0% | default | — | 2026-10-21 | ★ In-House Test |
| 23 | 74.0% | default | — | 2026-10-21 | ★ In-House Test | |
| 24 |
Qwen3.8 27B (xhigh)
Alibaba
|
72.0% | default | — | 2026-10-21 | ★ In-House Test |
| 25 |
Claude Fable 5.1
Anthropic
|
72.0% | default | — | 2026-10-21 | ★ In-House Test |
| 26 |
K2 Horizon 375B (A23B)
Thinking Machines
|
70.0% | default | — | 2026-10-21 | ★ In-House Test |
| 27 |
Gemini 3.1 Pro (preview)
Google
|
69.0% | default | — | 2026-10-21 | ★ In-House Test |
| 28 |
LFM2.5 24B A2B (on-device MoE)
Lfm25 24b A2b On Device Moe
|
67.0% | default | — | 2026-10-21 | ★ In-House Test |
| 29 |
Inkling
Inkling
|
65.0% | default | — | 2026-10-21 | ★ In-House Test |
| 30 |
Gemini 3.5 Flash-Lite
Google
|
62.0% | default | — | 2026-10-21 | ★ In-House Test |
| RANK | MODEL & PROVIDER | VERIFIED SCORE | TEST HARNESS | COST / TASK | RUN DATE | PROVENANCE & SOURCE |
|---|---|---|---|---|---|---|
| 01 |
Claude Fable 5.1
Anthropic
|
58.0% | default | — | 2026-10-21 | ★ In-House Test |
| 02 |
GPT-6 Astra (max)
OpenAI
|
57.5% | SciCode evaluation harness with unit tests | — | 2026-09-01 | ★ In-House Test |
| 03 |
Claude Opus 5.5
Anthropic
|
57.0% | default | — | 2026-10-21 | ★ In-House Test |
| 04 |
Gemini 4 Argon (high)
Google
|
55.0% | default | — | 2026-10-21 | ★ In-House Test |
| 05 |
Claude Sonnet 5.5
Anthropic
|
54.0% | default | — | 2026-10-21 | ★ In-House Test |
| 06 |
Muse Wonder
Muse Wonder
|
54.0% | default | — | 2026-10-21 | ★ In-House Test |
| 07 |
GPT-6 Sol
OpenAI
|
52.0% | default | — | 2026-10-21 | ★ In-House Test |
| 08 |
Qwen3.8 Max (0902)
Alibaba
|
51.0% | default | — | 2026-10-21 | ★ In-House Test |
| 09 |
Mistral Large 4 (preview)
Mistral
|
50.0% | default | — | 2026-10-21 | ★ In-House Test |
| 10 |
Grok 4.7
xAI
|
49.0% | default | — | 2026-10-21 | ★ In-House Test |
| 11 |
Kimi K3 (max)
Kimi
|
49.0% | default | — | 2026-10-21 | ★ In-House Test |
| 12 |
Step 5 Preview (preview)
StepFun
|
48.5% | default | — | 2026-10-21 | ★ In-House Test |
| 13 |
Muse Spark Open (816B MoE)
Muse Spark Open 816b Moe
|
47.0% | default | — | 2026-10-21 | ★ In-House Test |
| 14 | 46.5% | default | — | 2026-10-21 | ★ In-House Test | |
| 15 |
Gemini 3.8 Flash (high)
Google
|
45.0% | default | — | 2026-10-21 | ★ In-House Test |
| 16 |
Qwen3.8-Flash-Next
Alibaba
|
44.0% | default | — | 2026-10-21 | ★ In-House Test |
| 17 |
GLM-5.3 Fast (highspeed)
Zhipu / Z.ai
|
43.5% | default | — | 2026-10-21 | ★ In-House Test |
| 18 |
GPT-6 Luna
OpenAI
|
42.0% | default | — | 2026-10-21 | ★ In-House Test |
| 19 |
Claude Haiku 5.5
Anthropic
|
41.5% | default | — | 2026-10-21 | ★ In-House Test |
| 20 |
MiniMax M3.1 Flash (max)
MiniMax
|
40.0% | default | — | 2026-10-21 | ★ In-House Test |
| 21 |
DeepSeek V4 Pro (0813 max)
DeepSeek
|
39.5% | default | — | 2026-10-21 | ★ In-House Test |
| 22 |
Beam 1.0 501B (A23B MoE)
Beam 10 501b A23b Moe
|
38.0% | default | — | 2026-10-21 | ★ In-House Test |
| 23 |
DeepSeek V4 Flash Vision (max)
DeepSeek
|
37.5% | default | — | 2026-10-21 | ★ In-House Test |
| 24 | 36.5% | default | — | 2026-10-21 | ★ In-House Test | |
| 25 |
Qwen3.8 27B (xhigh)
Alibaba
|
35.5% | SciCode evaluation harness with unit tests | — | 2026-09-01 | ★ In-House Test |
| 26 |
K2 Horizon 375B (A23B)
Thinking Machines
|
34.0% | default | — | 2026-10-21 | ★ In-House Test |
| 27 |
Gemini 3.1 Pro (preview)
Google
|
33.0% | default | — | 2026-10-21 | ★ In-House Test |
| 28 |
LFM2.5 24B A2B (on-device MoE)
Lfm25 24b A2b On Device Moe
|
32.0% | default | — | 2026-10-21 | ★ In-House Test |
| 29 |
Inkling
Inkling
|
30.0% | default | — | 2026-10-21 | ★ In-House Test |
| 30 |
Gemini 3.5 Flash-Lite
Google
|
28.0% | default | — | 2026-10-21 | ★ In-House Test |
Explore individual benchmark execution environments, task distributions, failure analysis, and Docker reproduction steps.
Autonomous resolution of real-world GitHub issues across popular open-source Python repositories, hand-verified by human software engineers.
Autonomous command-line navigation, multi-step debugging, environment triage, and terminal state evaluation in isolated Linux sandboxes.
Algorithmic problem solving, complex mathematical reasoning, and edge-case handling on fresh competitive programming problems published post training cutoff.
Multi-language code editing and refactoring across diverse programming paradigms, evaluating SEARCH/REPLACE diff format compliance and unit test verification.
Graduate-level physics, chemistry, and biology problems written and validated by domain experts to withstand search engine retrieval.
Complex scientific calculations, numerical simulations, and differential equations requiring multi-step mathematical programming in Python.