LIVE TELEMETRY • FRONTIER AI CODING LEADERBOARD v2.4 Editorial

Vibe Coder Journal Mascot AI Coding Benchmarks & Rankings

Empirical evaluations measuring coding intelligence, agentic task resolution, execution speed, and token cost economics across 30 frontier models.

Verified Test Harness ★ Independent Telemetry Updated Weekly 30 Models Evaluated
#02 Silver
Anthropic

Claude Sonnet 5.5

55.3 Coding Intelligence★
SWE-bench 50.5%
Speed 136 t/s
Blended $4.00
Best For: Autonomous Agent Execution
👑 FRONTIER LEADER
#01 Gold
Anthropic

Claude Opus 5.5

56.6 Coding Intelligence★
SWE-bench 53.8%
Speed 95 t/s
Blended $8.00
Best For: Multi-file Code Refactoring
#03 Bronze
Google

Gemini 4 Argon (high)

53.7 Coding Intelligence★
SWE-bench 55.0%
Speed 83 t/s
Blended $4.00
Best For: Complex Mathematical Logic

Sliders create a separate weighted table from recorded GPQA, SWE-bench and Terminal-Bench scores. Missing required inputs stay unscored. Reset restores the published snapshot.

Recalculate Intelligence Index Adjust weighting distribution across canonical suites
GPQA Diamond 45%
SWE-bench Verified 40%
Terminal-Bench 15%
CREATOR:
01 ▲+15
Anthropic logo
Claude Opus 5.5
Anthropic • API
56.6%★
90.6% 53.8% 1M 95 t/s TTFT 3820ms $8.00 $0.1413
02 ▼-1
Anthropic logo
Claude Sonnet 5.5
Anthropic • API
55.3%★
93.0% 50.5% 1M 136 t/s TTFT 1160ms $4.00 $0.0723
03 ▲+1
53.7%★
96.5% 55.0% 1.048576M 83 t/s TTFT 2450ms $4.00 $0.0745
04 ▼-1
OpenAI logo
GPT-6 Astra (max)
OpenAI • API
53.2%★
71.5% 55.0% 1.1M 62 t/s TTFT 4100ms $20.00 $0.3763
05 ▼-4
Anthropic logo
Claude Fable 5.1
Anthropic • API
53.1%★
72.0% 54.6% 1M 68 t/s TTFT 2620ms $30.00 $0.5645
06 ▲+1
36.0%★
52.0% 36.8% 1M 86 t/s TTFT 1750ms $0.96 $0.0267
07 ▲+1
Alibaba logo
Qwen3.8 27B (xhigh)
Alibaba • Open weights
34.0%★
49.0% 34.6% 128K 178 t/s TTFT 850ms $0.35 $0.0103
08 ▲+1
29.9%★
44.5% 30.2% 2M 112 t/s TTFT 1400ms $1.40 $0.0468
09 ▲+1
Lfm25 24b A2b On Device Moe logo
LFM2.5 24B A2B (on-device MoE)
Lfm25 24b A2b On Device Moe • Open weights
28.8%★
67.0% 29.0% 131.072K 220 t/s TTFT 480ms $0.21 $0.0074
10 ▲+1
NVIDIA logo
Nemotron 3.5 Super VL (120B A12B)
NVIDIA • Open weights
0.0%★
74.0% 34.5% 1M 137 t/s TTFT 1200ms $0.75 —
11 ▲+1
Beam 10 501b A23b Moe logo
Beam 1.0 501B (A23B MoE)
Beam 10 501b A23b Moe • Open weights
0.0%★
76.0% 36.0% 1M 125 t/s TTFT 1250ms $0.44 —
12 ▲+1
0.0%★
78.0% 37.5% 1M 181 t/s TTFT 820ms $0.26 —
13 ▲+1
Anthropic logo
Claude Haiku 5.5
Anthropic • API
0.0%★
80.5% 39.0% 1M 210 t/s TTFT 550ms $0.20 —
14 ▲+1
OpenAI logo
GPT-6 Luna
OpenAI • API
0.0%★
81.0% 39.5% 1.1M 195 t/s TTFT 650ms $0.20 —
15 ▲+1
Zhipu / Z.ai logo
GLM-5.3 Fast (highspeed)
Zhipu / Z.ai • API
0.0%★
79.5% 40.2% 1M 490 t/s TTFT 420ms $0.80 —
16 ▲+1
Muse Spark Open 816b Moe logo
Muse Spark Open (816B MoE)
Muse Spark Open 816b Moe • API
0.0%★
83.0% 43.0% 1.048576M 115 t/s TTFT 1350ms $2.00 —
17 ▲+1
0.0%★
84.0% 43.5% 1M 186 t/s TTFT 850ms $0.26 —
18 ▲+1
xAI logo
Grok 4.7
xAI • API
0.0%★
86.5% 44.0% 500K 105 t/s TTFT 1500ms $3.50 —
19 ▲+1
0.0%★
87.0% 46.5% 1M 92 t/s TTFT 1650ms $2.07 —
20 ▲+1
OpenAI logo
GPT-6 Sol
OpenAI • API
0.0%★
91.5% 49.5% 1.1M 110 t/s TTFT 1450ms $4.00 —
21 ▲+1
Muse Wonder logo
Muse Wonder
Muse Wonder • API
0.0%★
88.5% 51.5% 1.048576M 88 t/s TTFT 2150ms $8.00 —
22 ▼-12
0.0%★
35.5% 23.2% 1M 298 t/s TTFT 380ms $0.17 —
23 ▲+1
Inkling logo
Inkling
Inkling • API
0.0%★
40.0% 26.6% 256K 132 t/s TTFT 1100ms $0.75 —
24 ▲+1
Thinking Machines logo
K2 Horizon 375B (A23B)
Thinking Machines • Open weights
0.0%★
46.0% 31.8% 1M 98 t/s TTFT 1600ms $1.20 —
25 ▲+1
0.0%★
50.5% 35.6% 1M 154 t/s TTFT 1050ms $0.47 —
26 ▲+1
Alibaba logo
Qwen3.8-Flash-Next
Alibaba • Open weights
0.0%★
56.5% 40.6% 1M 165 t/s TTFT 880ms $0.45 —
27 ▼-21
0.0%★
58.0% 41.8% 1M 145 t/s TTFT 950ms $1.31 —
28 ▲+1
0.0%★
61.5% 44.6% 1M 90 t/s TTFT 1800ms $1.20 —
29 ▼-21
Kimi logo
Kimi K3 (max)
Kimi • Open weights
0.0%★
61.8% 44.8% 2M 82 t/s TTFT 2100ms $1.80 —
30 ▼-21
Alibaba logo
Qwen3.8 Max (0902)
Alibaba • API
0.0%★
63.5% 46.0% 1M 76 t/s TTFT 1950ms $3.00 —
COMPARATIVE ANALYSIS • 5 VIEWPOINTS

Interactive Benchmark Visualizations

Explore head-to-head architectural comparisons across intelligence, inference speed, token pricing, and cost-efficiency sweet spots.

Coding Intelligence Index by Model

Horizontal rank comparison with verified provider identity • Higher is better
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com

Inference Output Speed (Tokens per Second)

Streaming throughput measured under concurrent sustained load • Higher is faster
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com

Blended Pricing per 1M Tokens (USD)

Calculated on standard 3:1 prompt to generation token ratio • Lower is cheaper
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com

Value-for-Money Matrix: Intelligence vs Blended Cost

Top-left quadrant highlights the highest performance per dollar invested
★ BEST VALUE SWEET SPOT High coding score + sub-$8 blended cost 20% 30% 40% 50% 55% CODING INTELLIGENCE (%) $0 $5 $10 $15 $20 $25+ BLENDED COST / 1M TOKENS (USD) Claude Opus 5.5 Claude Opus 5.5 Score: 56.6% Cost: $8.00/1M Claude Sonnet 5.5 Claude Sonnet 5.5 Score: 55.3% Cost: $4.00/1M Gemini 4 Argon Gemini 4 Argon (high) Score: 53.7% Cost: $4.00/1M GPT-6 Astra (max) GPT-6 Astra (max) Score: 53.2% Cost: $20.00/1M Claude Fable 5.1 Claude Fable 5.1 Score: 53.1% Cost: $30.00/1M DeepSeek V4 Pro DeepSeek V4 Pro (0813 max) Score: 36.0% Cost: $0.96/1M Qwen3.8 27B (xhigh) Qwen3.8 27B (xhigh) Score: 34.0% Cost: $0.35/1M Gemini 3.1 Pro Gemini 3.1 Pro (preview) Score: 29.9% Cost: $1.40/1M LFM2.5 24B A2B LFM2.5 24B A2B (on-device MoE) Score: 28.8% Cost: $0.21/1M Proprietary API Open Weights
Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com

Frontier Intelligence Progression (2024 – 2026)

Chronological milestones tracking state-of-the-art coding capability jumps
Q1 2024
GPT-4 Turbo Era Index ~28.0%

Initial multi-file coding and git diff generation baseline.

Q3 2024
Claude 3.5 Sonnet Release Index ~38.0%

Standardization of SWE-bench Verified and tool-use agent harnesses.

Q1 2025
DeepSeek V3 / MoE Surge Index ~42.0%

Open weights models reach parity on competitive programming benchmarks.

Q3 2026 • CURRENT
Astra & Fable Frontier Index 53.0%+

Extended test-time compute, autonomous repository debugging, and 1M+ context reasoning.

Vibe Coder Journal Benchmark Telemetry • vibecoderjournal.com
CANONICAL SUITE MATRIX

6-Benchmark Performance Heatmap

Color saturation corresponds to empirical score percentile
MODEL & PROVIDER SWE-BENCH TERMINAL 2.0 LIVECODE AIDER POLY GPQA DIAMOND SCICODE
53.8% 66.4% 51.2% 54.0% 90.6% 57.0%
50.5% 70.6% 47.0% 52.0% 93.0% 54.0%
55.0% 57.4% 47.5% 51.0% 96.5% 55.0%
55.0% 51.5% 50.5% 53.5% 71.5% 57.5%
54.6% 52.0% 50.0% 54.0% 72.0% 58.0%
36.8% 35.5% 33.5% 37.0% 52.0% 38.0%
34.6% 33.5% 32.0% 35.0% 49.0% 35.5%
30.2% 30.2% 27.5% 30.8% 44.5% 31.0%
29.0% 28.5% 27.5% 29.5% 67.0% 32.0%
34.5% 34.0% 32.5% 35.0% 74.0% 36.5%
36.0% 35.0% 34.0% 36.5% 76.0% 38.0%
37.5% 37.0% 35.5% 38.0% 78.0% 40.0%
39.0% 38.5% 37.0% 39.0% 80.5% 41.5%
39.5% 38.8% 37.5% 39.5% 81.0% 42.0%
40.2% 39.5% 38.0% 40.5% 79.5% 43.5%
43.0% 42.0% 41.0% 43.5% 83.0% 47.0%
RIGOROUS VERIFICATION PROTOCOL

Leaderboard Methodology & Evaluation Harness

Isolated Docker Execution

Every agentic benchmark (SWE-bench Verified and Terminal-Bench 2.0) is evaluated inside clean, network-isolated Docker containers with fixed system dependencies, preventing prompt leakage and external tool interference.

Zero-Data Fabrication Guarantee

We enforce strict provenance rules. Every number displayed originates from our verified in-house test harness (★) or vendor-published telemetry (†). Models awaiting re-test display "Not tested yet" rather than estimated values.

Normalized Blended Cost Formula

Blended cost per 1M tokens assumes an industry-standard 3:1 prompt to generation ratio ((3 × Input + 1 × Output) / 4). The Value Index ($/Point) measures total blended dollars required per index point scored.

Frequently Asked Questions

How is the Coding Intelligence Index calculated?
The published view preserves the composite score and rank in each stored model record. Custom Weights creates an exploratory weighted arithmetic mean of recorded GPQA Diamond, SWE-bench Verified and Terminal-Bench results, normalized by the selected weights. A missing active benchmark leaves that model unscored. This custom score is a separate comparison, not a recalculation of the published index. Reset Published View restores the original scores, ranks and value metrics; snapshot charts and podiums are hidden while custom scores are active.
Why are some SWE-bench and GPQA cells displayed with "—"?
We never estimate, extrapolate, or fabricate test scores. If a model has not completed its verified run on our physical test harness, the cell is explicitly marked as "Not tested yet" until empirical logs are published.
How often are benchmark evaluations updated?
Our telemetry runner tests new model checkpoints and frontier releases weekly. Price changes, context window upgrades, and architecture variants are refreshed within 24 hours of vendor availability.