BENCHMARK SUITE • EMPIRICAL TELEMETRY Isolated Docker Harnesses

Vibe Coder Journal Mascot AI Coding Benchmarks Hub

Standardized evaluations, empirical telemetry, and reproducible agent benchmarks. Every score is captured under isolated execution harnesses without prompt leakage.

6 Canonical Suites ★ Verified Test Harnesses Updated Weekly Zero Data Fabrication
🛠️ PRIMARY AGENTIC BENCHMARK

SWE-bench Verified

500 Tasks • Python Repositories
Leader:
OpenAI logo GPT-6 Astra (max) 55.0%

Autonomous resolution of real-world GitHub issues across popular open-source Python repositories, hand-verified by human software engineers.

💻 COMMAND-LINE AGENT EXECUTION

Terminal-Bench 2.0

89 Tasks • Linux Sandboxes
Leader:
Anthropic logo Claude Sonnet 5.5 70.6%

Autonomous command-line navigation, multi-step debugging, environment triage, and terminal state evaluation in isolated Linux sandboxes.

⚡ CONTAMINATION-FREE ALGORITHMS

LiveCodeBench

400 Tasks • Competitive Coding
Leader:
Anthropic logo Claude Opus 5.5 51.2%

Algorithmic problem solving, complex mathematical reasoning, and edge-case handling on fresh competitive programming problems published post training cutoff.

🌐 MULTI-LANGUAGE CODE EDITING

Aider Polyglot Benchmark

225 Tasks • 6 Languages
Leader:
Anthropic logo Claude Fable 5.1 54.0%

Multi-language code editing and refactoring across diverse programming paradigms, evaluating SEARCH/REPLACE diff format compliance and unit test verification.

🔬 EXPERT SCIENTIFIC REASONING

GPQA Diamond

198 Tasks • Graduate Level
Leader:
Google logo Gemini 4 Argon (high) 96.5%

Graduate-level physics, chemistry, and biology problems written and validated by domain experts to withstand search engine retrieval.

📐 SCIENTIFIC COMPUTING & PHYSICS

SciCode

338 Sub-problems • Numerical Methods
Leader:
Anthropic logo Claude Fable 5.1 58.0%

Complex scientific calculations, numerical simulations, and differential equations requiring multi-step mathematical programming in Python.

PRIMARY AGENTIC BENCHMARK • 500 Tasks • Python Repositories

SWE-bench Verified Leaderboard

Test Scaffolding: SWE-agent v2.1 in isolated Docker sandboxes • Evaluated without prompt leakage.

Top 15 Verified Models on SWE-bench Verified

Horizontal performance bars with verified provider branding • Higher is better
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com
RANK MODEL & PROVIDER VERIFIED SCORE TEST HARNESS COST / TASK RUN DATE PROVENANCE & SOURCE
01 55.0% OpenAI Sandbox v4 $3.26 2026-09-01 ★ In-House Test
02 55.0% default — 2026-10-21 ★ In-House Test
03 54.6% SWE-agent v2.1 $4.12 2026-09-01 ★ In-House Test
04 53.8% default — 2026-10-21 ★ In-House Test
05
Muse Wonder logo
Muse Wonder Muse Wonder
51.5% default — 2026-10-21 ★ In-House Test
06 50.5% default — 2026-10-21 ★ In-House Test
07 49.5% default — 2026-10-21 ★ In-House Test
08 46.5% default — 2026-10-21 ★ In-House Test
09 46.0% Qwen-Agent Loop $2.10 2026-09-01 ★ In-House Test
10 44.8% Kimi Agent $2.00 2026-09-01 ★ In-House Test
11 44.6% StepFun Code Harness $0.71 2026-09-01 ★ In-House Test
12 44.0% default — 2026-10-21 ★ In-House Test
13 43.5% default — 2026-10-21 ★ In-House Test
14
Muse Spark Open 816b Moe logo
Muse Spark Open (816B MoE) Muse Spark Open 816b Moe
43.0% default — 2026-10-21 ★ In-House Test
15 41.8% Google Code Harness $1.24 2026-09-01 ★ In-House Test
16 40.6% Qwen-Agent Open $0.37 2026-09-01 ★ In-House Test
17 40.2% default — 2026-10-21 ★ In-House Test
18 39.5% default — 2026-10-21 ★ In-House Test
19 39.0% default — 2026-10-21 ★ In-House Test
20 37.5% default — 2026-10-21 ★ In-House Test
21 36.8% SWE-agent v2.1 $0.67 2026-09-01 ★ In-House Test
22
Beam 10 501b A23b Moe logo
Beam 1.0 501B (A23B MoE) Beam 10 501b A23b Moe
36.0% default — 2026-10-21 ★ In-House Test
23 35.6% OpenHands $0.31 2026-09-01 ★ In-House Test
24 34.6% Qwen-Agent Open $0.82 2026-09-01 ★ In-House Test
25 34.5% default — 2026-10-21 ★ In-House Test
26
Thinking Machines logo
K2 Horizon 375B (A23B) Thinking Machines
31.8% IFM Agent Loop $1.20 2026-09-01 ★ In-House Test
27 30.2% Google Code Harness $0.67 2026-09-01 ★ In-House Test
28
Lfm25 24b A2b On Device Moe logo
LFM2.5 24B A2B (on-device MoE) Lfm25 24b A2b On Device Moe
29.0% default — 2026-10-21 ★ In-House Test
29
Inkling logo
Inkling Inkling
26.6% Thinking Machines $0.61 2026-09-01 ★ In-House Test
30 23.2% Google Code Harness $0.12 2026-09-01 ★ In-House Test
COMMAND-LINE AGENT EXECUTION • 89 Tasks • Linux Sandboxes

Terminal-Bench 2.0 Leaderboard

Test Scaffolding: Docker Ubuntu 24.04 sandbox with mock CLI • Evaluated without prompt leakage.

Top 15 Verified Models on Terminal-Bench 2.0

Horizontal performance bars with verified provider branding • Higher is better
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com
RANK MODEL & PROVIDER VERIFIED SCORE TEST HARNESS COST / TASK RUN DATE PROVENANCE & SOURCE
01 70.6% default — 2026-10-21 ★ In-House Test
02 66.4% default — 2026-10-21 ★ In-House Test
03 57.9% default — 2026-10-21 ★ In-House Test
04 57.4% default — 2026-10-21 ★ In-House Test
05
Muse Wonder logo
Muse Wonder Muse Wonder
53.0% default — 2026-10-21 ★ In-House Test
06 52.0% default — 2026-10-21 ★ In-House Test
07 51.0% default — 2026-10-21 ★ In-House Test
08 45.0% default — 2026-10-21 ★ In-House Test
09 44.5% Docker Ubuntu 24.04 sandbox with mock CLI — 2026-09-01 ★ In-House Test
10 44.0% Docker Ubuntu 24.04 sandbox with mock CLI — 2026-09-01 ★ In-House Test
11 43.8% Docker Ubuntu 24.04 sandbox with mock CLI — 2026-09-01 ★ In-House Test
12 43.5% default — 2026-10-21 ★ In-House Test
13 42.0% default — 2026-10-21 ★ In-House Test
14
Muse Spark Open 816b Moe logo
Muse Spark Open (816B MoE) Muse Spark Open 816b Moe
42.0% default — 2026-10-21 ★ In-House Test
15 40.5% Docker Ubuntu 24.04 sandbox with mock CLI — 2026-09-01 ★ In-House Test
16 40.0% Docker Ubuntu 24.04 sandbox with mock CLI — 2026-09-01 ★ In-House Test
17 39.5% default — 2026-10-21 ★ In-House Test
18 38.8% default — 2026-10-21 ★ In-House Test
19 38.5% default — 2026-10-21 ★ In-House Test
20 37.0% default — 2026-10-21 ★ In-House Test
21 35.5% default — 2026-10-21 ★ In-House Test
22
Beam 10 501b A23b Moe logo
Beam 1.0 501B (A23B MoE) Beam 10 501b A23b Moe
35.0% default — 2026-10-21 ★ In-House Test
23 34.5% default — 2026-10-21 ★ In-House Test
24 34.0% default — 2026-10-21 ★ In-House Test
25 33.5% Docker Ubuntu 24.04 sandbox with mock CLI — 2026-09-01 ★ In-House Test
26
Thinking Machines logo
K2 Horizon 375B (A23B) Thinking Machines
30.5% default — 2026-10-21 ★ In-House Test
27 30.2% default — 2026-10-21 ★ In-House Test
28
Lfm25 24b A2b On Device Moe logo
LFM2.5 24B A2B (on-device MoE) Lfm25 24b A2b On Device Moe
28.5% default — 2026-10-21 ★ In-House Test
29
Inkling logo
Inkling Inkling
25.5% default — 2026-10-21 ★ In-House Test
30 23.2% default — 2026-10-21 ★ In-House Test
CONTAMINATION-FREE ALGORITHMS • 400 Tasks • Competitive Coding

LiveCodeBench Leaderboard

Test Scaffolding: LiveCodeBench test runner with fresh problem sets • Evaluated without prompt leakage.

Top 15 Verified Models on LiveCodeBench

Horizontal performance bars with verified provider branding • Higher is better
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com
RANK MODEL & PROVIDER VERIFIED SCORE TEST HARNESS COST / TASK RUN DATE PROVENANCE & SOURCE
01 51.2% default — 2026-10-21 ★ In-House Test
02 50.5% default — 2026-10-21 ★ In-House Test
03 50.0% default — 2026-10-21 ★ In-House Test
04
Muse Wonder logo
Muse Wonder Muse Wonder
49.0% default — 2026-10-21 ★ In-House Test
05 47.5% default — 2026-10-21 ★ In-House Test
06 47.0% default — 2026-10-21 ★ In-House Test
07 45.0% default — 2026-10-21 ★ In-House Test
08 43.5% default — 2026-10-21 ★ In-House Test
09 43.0% default — 2026-10-21 ★ In-House Test
10 42.5% default — 2026-10-21 ★ In-House Test
11 42.0% default — 2026-10-21 ★ In-House Test
12 42.0% default — 2026-10-21 ★ In-House Test
13
Muse Spark Open 816b Moe logo
Muse Spark Open (816B MoE) Muse Spark Open 816b Moe
41.0% default — 2026-10-21 ★ In-House Test
14 40.5% default — 2026-10-21 ★ In-House Test
15 39.5% default — 2026-10-21 ★ In-House Test
16 38.5% default — 2026-10-21 ★ In-House Test
17 38.0% default — 2026-10-21 ★ In-House Test
18 37.5% default — 2026-10-21 ★ In-House Test
19 37.0% default — 2026-10-21 ★ In-House Test
20 35.5% default — 2026-10-21 ★ In-House Test
21 35.0% default — 2026-10-21 ★ In-House Test
22
Beam 10 501b A23b Moe logo
Beam 1.0 501B (A23B MoE) Beam 10 501b A23b Moe
34.0% default — 2026-10-21 ★ In-House Test
23 33.5% default — 2026-10-21 ★ In-House Test
24 32.5% default — 2026-10-21 ★ In-House Test
25 32.0% LiveCodeBench test runner with fresh problem sets — 2026-09-01 ★ In-House Test
26
Thinking Machines logo
K2 Horizon 375B (A23B) Thinking Machines
30.0% default — 2026-10-21 ★ In-House Test
27 29.0% default — 2026-10-21 ★ In-House Test
28
Lfm25 24b A2b On Device Moe logo
LFM2.5 24B A2B (on-device MoE) Lfm25 24b A2b On Device Moe
27.5% default — 2026-10-21 ★ In-House Test
29
Inkling logo
Inkling Inkling
25.0% default — 2026-10-21 ★ In-House Test
30 22.0% default — 2026-10-21 ★ In-House Test
MULTI-LANGUAGE CODE EDITING • 225 Tasks • 6 Languages

Aider Polyglot Benchmark Leaderboard

Test Scaffolding: Aider CLI with benchmark harness • Evaluated without prompt leakage.

Top 15 Verified Models on Aider Polyglot Benchmark

Horizontal performance bars with verified provider branding • Higher is better
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com
RANK MODEL & PROVIDER VERIFIED SCORE TEST HARNESS COST / TASK RUN DATE PROVENANCE & SOURCE
01 54.0% default — 2026-10-21 ★ In-House Test
02 54.0% default — 2026-10-21 ★ In-House Test
03 53.5% default — 2026-10-21 ★ In-House Test
04 52.0% default — 2026-10-21 ★ In-House Test
05 51.0% default — 2026-10-21 ★ In-House Test
06 49.0% default — 2026-10-21 ★ In-House Test
07
Muse Wonder logo
Muse Wonder Muse Wonder
48.0% default — 2026-10-21 ★ In-House Test
08 47.0% default — 2026-10-21 ★ In-House Test
09 46.0% default — 2026-10-21 ★ In-House Test
10 45.2% Aider CLI with benchmark harness — 2026-09-01 ★ In-House Test
11 45.0% default — 2026-10-21 ★ In-House Test
12 44.0% default — 2026-10-21 ★ In-House Test
13
Muse Spark Open 816b Moe logo
Muse Spark Open (816B MoE) Muse Spark Open 816b Moe
43.5% default — 2026-10-21 ★ In-House Test
14 43.0% default — 2026-10-21 ★ In-House Test
15 42.0% Aider CLI with benchmark harness — 2026-09-01 ★ In-House Test
16 41.0% default — 2026-10-21 ★ In-House Test
17 40.5% default — 2026-10-21 ★ In-House Test
18 39.5% default — 2026-10-21 ★ In-House Test
19 39.0% default — 2026-10-21 ★ In-House Test
20 38.0% default — 2026-10-21 ★ In-House Test
21 37.0% default — 2026-10-21 ★ In-House Test
22
Beam 10 501b A23b Moe logo
Beam 1.0 501B (A23B MoE) Beam 10 501b A23b Moe
36.5% default — 2026-10-21 ★ In-House Test
23 36.0% default — 2026-10-21 ★ In-House Test
24 35.0% Aider CLI with benchmark harness — 2026-09-01 ★ In-House Test
25 35.0% default — 2026-10-21 ★ In-House Test
26
Thinking Machines logo
K2 Horizon 375B (A23B) Thinking Machines
32.0% default — 2026-10-21 ★ In-House Test
27 31.0% default — 2026-10-21 ★ In-House Test
28
Lfm25 24b A2b On Device Moe logo
LFM2.5 24B A2B (on-device MoE) Lfm25 24b A2b On Device Moe
29.5% default — 2026-10-21 ★ In-House Test
29
Inkling logo
Inkling Inkling
27.0% default — 2026-10-21 ★ In-House Test
30 23.8% Aider CLI with benchmark harness — 2026-09-01 ★ In-House Test
EXPERT SCIENTIFIC REASONING • 198 Tasks • Graduate Level

GPQA Diamond Leaderboard

Test Scaffolding: Zero-shot and few-shot evaluation harness • Evaluated without prompt leakage.

Top 15 Verified Models on GPQA Diamond

Horizontal performance bars with verified provider branding • Higher is better
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com
RANK MODEL & PROVIDER VERIFIED SCORE TEST HARNESS COST / TASK RUN DATE PROVENANCE & SOURCE
01 96.5% default — 2026-10-21 ★ In-House Test
02 96.0% default — 2026-10-21 ★ In-House Test
03 93.0% default — 2026-10-21 ★ In-House Test
04 91.5% default — 2026-10-21 ★ In-House Test
05 90.6% default — 2026-10-21 ★ In-House Test
06
Muse Wonder logo
Muse Wonder Muse Wonder
88.5% default — 2026-10-21 ★ In-House Test
07 87.0% default — 2026-10-21 ★ In-House Test
08 86.5% default — 2026-10-21 ★ In-House Test
09 86.0% default — 2026-10-21 ★ In-House Test
10 85.0% default — 2026-10-21 ★ In-House Test
11 84.5% default — 2026-10-21 ★ In-House Test
12 84.0% default — 2026-10-21 ★ In-House Test
13
Muse Spark Open 816b Moe logo
Muse Spark Open (816B MoE) Muse Spark Open 816b Moe
83.0% default — 2026-10-21 ★ In-House Test
14 81.0% default — 2026-10-21 ★ In-House Test
15 81.0% default — 2026-10-21 ★ In-House Test
16 80.5% default — 2026-10-21 ★ In-House Test
17 80.0% default — 2026-10-21 ★ In-House Test
18 79.5% default — 2026-10-21 ★ In-House Test
19 78.0% default — 2026-10-21 ★ In-House Test
20 77.0% default — 2026-10-21 ★ In-House Test
21
Beam 10 501b A23b Moe logo
Beam 1.0 501B (A23B MoE) Beam 10 501b A23b Moe
76.0% default — 2026-10-21 ★ In-House Test
22 75.0% default — 2026-10-21 ★ In-House Test
23 74.0% default — 2026-10-21 ★ In-House Test
24 72.0% default — 2026-10-21 ★ In-House Test
25 72.0% default — 2026-10-21 ★ In-House Test
26
Thinking Machines logo
K2 Horizon 375B (A23B) Thinking Machines
70.0% default — 2026-10-21 ★ In-House Test
27 69.0% default — 2026-10-21 ★ In-House Test
28
Lfm25 24b A2b On Device Moe logo
LFM2.5 24B A2B (on-device MoE) Lfm25 24b A2b On Device Moe
67.0% default — 2026-10-21 ★ In-House Test
29
Inkling logo
Inkling Inkling
65.0% default — 2026-10-21 ★ In-House Test
30 62.0% default — 2026-10-21 ★ In-House Test
SCIENTIFIC COMPUTING & PHYSICS • 338 Sub-problems • Numerical Methods

SciCode Leaderboard

Test Scaffolding: SciCode evaluation harness with unit tests • Evaluated without prompt leakage.

Top 15 Verified Models on SciCode

Horizontal performance bars with verified provider branding • Higher is better
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com
RANK MODEL & PROVIDER VERIFIED SCORE TEST HARNESS COST / TASK RUN DATE PROVENANCE & SOURCE
01 58.0% default — 2026-10-21 ★ In-House Test
02 57.5% SciCode evaluation harness with unit tests — 2026-09-01 ★ In-House Test
03 57.0% default — 2026-10-21 ★ In-House Test
04 55.0% default — 2026-10-21 ★ In-House Test
05 54.0% default — 2026-10-21 ★ In-House Test
06
Muse Wonder logo
Muse Wonder Muse Wonder
54.0% default — 2026-10-21 ★ In-House Test
07 52.0% default — 2026-10-21 ★ In-House Test
08 51.0% default — 2026-10-21 ★ In-House Test
09 50.0% default — 2026-10-21 ★ In-House Test
10 49.0% default — 2026-10-21 ★ In-House Test
11 49.0% default — 2026-10-21 ★ In-House Test
12 48.5% default — 2026-10-21 ★ In-House Test
13
Muse Spark Open 816b Moe logo
Muse Spark Open (816B MoE) Muse Spark Open 816b Moe
47.0% default — 2026-10-21 ★ In-House Test
14 46.5% default — 2026-10-21 ★ In-House Test
15 45.0% default — 2026-10-21 ★ In-House Test
16 44.0% default — 2026-10-21 ★ In-House Test
17 43.5% default — 2026-10-21 ★ In-House Test
18 42.0% default — 2026-10-21 ★ In-House Test
19 41.5% default — 2026-10-21 ★ In-House Test
20 40.0% default — 2026-10-21 ★ In-House Test
21 39.5% default — 2026-10-21 ★ In-House Test
22
Beam 10 501b A23b Moe logo
Beam 1.0 501B (A23B MoE) Beam 10 501b A23b Moe
38.0% default — 2026-10-21 ★ In-House Test
23 37.5% default — 2026-10-21 ★ In-House Test
24 36.5% default — 2026-10-21 ★ In-House Test
25 35.5% SciCode evaluation harness with unit tests — 2026-09-01 ★ In-House Test
26
Thinking Machines logo
K2 Horizon 375B (A23B) Thinking Machines
34.0% default — 2026-10-21 ★ In-House Test
27 33.0% default — 2026-10-21 ★ In-House Test
28
Lfm25 24b A2b On Device Moe logo
LFM2.5 24B A2B (on-device MoE) Lfm25 24b A2b On Device Moe
32.0% default — 2026-10-21 ★ In-House Test
29
Inkling logo
Inkling Inkling
30.0% default — 2026-10-21 ★ In-House Test
30 28.0% default — 2026-10-21 ★ In-House Test
CANONICAL BENCHMARK SPECIFICATIONS

Detailed Benchmark Methodologies & Deep-Dives

Explore individual benchmark execution environments, task distributions, failure analysis, and Docker reproduction steps.

🛠️ 500 Tasks • Python Repositories

SWE-bench Verified

Autonomous resolution of real-world GitHub issues across popular open-source Python repositories, hand-verified by human software engineers.

Caveat: Agent scaffolding (SWE-agent vs Aider vs Devin) significantly impacts success rates; test harness timeouts must be strictly enforced.
💻 89 Tasks • Linux Sandboxes

Terminal-Bench 2.0

Autonomous command-line navigation, multi-step debugging, environment triage, and terminal state evaluation in isolated Linux sandboxes.

Caveat: Does not measure long-context codebase refactoring; highly sensitive to system prompt formatting and sub-shell execution timeouts.
⚡ 400 Tasks • Competitive Coding

LiveCodeBench

Algorithmic problem solving, complex mathematical reasoning, and edge-case handling on fresh competitive programming problems published post training cutoff.

Caveat: Focuses on isolated algorithmic challenges rather than multi-file software engineering or repository maintenance.
🌐 225 Tasks • 6 Languages

Aider Polyglot Benchmark

Multi-language code editing and refactoring across diverse programming paradigms, evaluating SEARCH/REPLACE diff format compliance and unit test verification.

Caveat: Small models frequently fail due to SEARCH/REPLACE format violations rather than fundamental algorithmic errors.
🔬 198 Tasks • Graduate Level

GPQA Diamond

Graduate-level physics, chemistry, and biology problems written and validated by domain experts to withstand search engine retrieval.

Caveat: Focuses on academic domain reasoning rather than practical software engineering or terminal agent execution.
📐 338 Sub-problems • Numerical Methods

SciCode

Complex scientific calculations, numerical simulations, and differential equations requiring multi-step mathematical programming in Python.

Caveat: Heavy domain specificity in physics and mathematics; less representative of everyday web or mobile application development.