BENCHMARK SUITE • EMPIRICAL CODE INTELLIGENCE

AI Coding Benchmarks

Standardized evaluations, empirical telemetry, and reproducible agent benchmarks. Every score is captured under isolated execution harnesses without prompt leakage.

Canonical Benchmark Suite

Breakdown across isolated execution harnesses matching Artificial Analysis methodologies.

AGENTIC CODING

SWE-bench Verified →

Resolved Rate %

Full Evaluation & Reproduction Guide →

CLI & SHELL USE

Terminal-Bench 2.0 →

Task Success %

Full Evaluation & Reproduction Guide →

POLYGLOT EDITING

Aider Polyglot →

Pass Rate %

Full Evaluation & Reproduction Guide →

ALGORITHMS & MATH

LiveCodeBench →

Pass@1 %

Full Evaluation & Reproduction Guide →

PhD REASONING

GPQA Diamond →

Accuracy %

Full Evaluation & Reproduction Guide →

SCIENTIFIC CODE

SciCode →

Problem Solving %

Full Evaluation & Reproduction Guide →

PRIMARY AGENTIC BENCHMARK

SWE-bench Verified Leaderboard

View Methodology & Deep-Dive →

Autonomous resolution of 500 real-world software engineering issues extracted from actively maintained open-source Python repositories, hand-verified by human software engineers.

Page updated: 18 Oct 2026 17:18 UTC 500 Evaluated Tasks • Metric: Resolved Rate

					SOURCE & HARNESS
#01	GPT-6 Astra (max) Benchmarks, Price & Specs (Oct 2026) OpenAI	55.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#02	GPT-6 Astra (max) Benchmarks, Price & Specs (Oct 2026) OpenAI	55.0%	$0.06	1 Sep 2026	Pc_direct_sync → Proprietary reported `OpenAI Sandbox v4`
#03	Claude Fable 5.1 (max with fallback) Anthropic	54.6%	$0.08	1 Sep 2026	Pc_direct_sync → Proprietary reported `SWE-agent v2.1`
#04	Claude Fable 5.1 (max with fallback) Anthropic	54.5%	—	19 Sep 2026	Pc_direct_sync → MIT reported
#05	Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic	52.6%	$0.11	1 Sep 2026	Pc_direct_sync → Proprietary reported `SWE-agent v2.1`
#06	Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic	52.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#07	Muse Spark 1.3 (max) Meta	49.0%	$0.03	1 Sep 2026	Pc_direct_sync → Llama-3.1-Community reported `Meta Agentic Code`
#08	Muse Spark 1.3 (max) Meta	49.0%	—	1 Sep 2026	Pc_direct_sync → Llama-3.1-Community reported
#09	GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI	48.0%	$0.04	1 Sep 2026	Pc_direct_sync → Proprietary reported `OpenAI Sandbox v4`
#10	GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI	48.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#11	Qwen3.8 Max (0902) Alibaba	46.0%	$0.05	1 Sep 2026	Pc_direct_sync → Proprietary reported `Qwen-Agent Loop`
#12	Qwen3.8 Max (0902) Alibaba	46.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#13	GLM-5.3 (max) Z AI	45.8%	$0.03	1 Sep 2026	Pc_direct_sync → Proprietary reported `Z-Agent Loop`
#14	GLM-5.3 (max) Z AI	45.8%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#15	Grok 4.6 (high) SpaceXAI	45.0%	$0.04	1 Sep 2026	Pc_direct_sync → Proprietary reported `SpaceXAI Loop`
#16	Grok 4.6 (high) SpaceXAI	45.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#17	Kimi K3 (max) Kimi	44.8%	$0.04	1 Sep 2026	Pc_direct_sync → Proprietary reported `Kimi Agent`
#18	Kimi K3 (max) Kimi	44.8%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#19	Step 5 Preview StepFun	44.6%	$0.02	1 Sep 2026	Pc_direct_sync → Proprietary reported `StepFun Code Harness`
#20	Step 5 Preview StepFun	44.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#21	GPT-5.6 Terra (max) OpenAI	43.0%	$0.03	1 Sep 2026	Pc_direct_sync → Proprietary reported `OpenAI Sandbox v4`
#22	GPT-5.6 Terra (max) OpenAI	43.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#23	GLM-5.3-Flash Z AI	42.6%	$0.01	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `Z-Agent Open Loop`
#24	GLM-5.3-Flash Z AI	42.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#25	Gemini 3.8 Flash (high) Google	41.8%	$0.03	1 Sep 2026	Pc_direct_sync → Proprietary reported `Google Code Harness`
#26	Gemini 3.8 Flash (high) Google	41.8%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#27	Qwen3.8 2.4T A95B Alibaba	40.8%	$0.05	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `Qwen-Agent Open`
#28	Qwen3.8 2.4T A95B Alibaba	40.8%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#29	Qwen3.8-Flash-Next Alibaba	40.6%	$0.01	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `Qwen-Agent Open`
#30	Qwen3.8-Flash-Next Alibaba	40.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#31	DeepSeek V4.1 Flash (max) DeepSeek	40.2%	$0.01	1 Sep 2026	Pc_direct_sync → MIT reported `OpenHands`
#32	DeepSeek V4.1 Flash (max) DeepSeek	40.2%	—	1 Sep 2026	Pc_direct_sync → MIT reported
#33	GPT-5.6 Luna (max) OpenAI	38.8%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#34	GPT-5.6 Luna (max) OpenAI	38.8%	$0.00	1 Sep 2026	Pc_direct_sync → Proprietary reported `OpenAI Sandbox v4`
#35	Claude Sonnet 5 (max) Anthropic	38.6%	$0.13	1 Sep 2026	Pc_direct_sync → Proprietary reported `SWE-agent v2.1`
#36	Claude Sonnet 5 (max) Anthropic	38.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#37	DeepSeek V4 Pro Benchmarks, Price & Specs (Oct 2026) DeepSeek	36.8%	—	1 Sep 2026	Pc_direct_sync → MIT reported
#38	DeepSeek V4 Pro Benchmarks, Price & Specs (Oct 2026) DeepSeek	36.8%	$0.02	1 Sep 2026	Pc_direct_sync → MIT reported `SWE-agent v2.1`
#39	DeepSeek V4 Flash Vision (max) DeepSeek	35.6%	$0.01	1 Sep 2026	Pc_direct_sync → MIT reported `OpenHands`
#40	DeepSeek V4 Flash Vision (max) DeepSeek	35.5%	—	1 Sep 2026	Pc_direct_sync → MIT reported
#41	Qwen3.8 27B (xhigh) Alibaba	34.6%	$0.02	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `Qwen-Agent Open`
#42	Qwen3.8 27B (xhigh) Alibaba	34.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#43	K2 Horizon 375B A23B Institute of Foundation Models	31.8%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#44	K2 Horizon 375B A23B Institute of Foundation Models	31.8%	$0.04	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `IFM Agent Loop`
#45	MiniMax-M3 MiniMax	30.6%	$0.02	1 Sep 2026	Pc_direct_sync → Proprietary reported `MiniMax Code Loop`
#46	MiniMax-M3 MiniMax	30.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#47	Gemini 3.1 Pro Benchmarks, Price & Specs (Oct 2026) Google	30.2%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#48	Gemini 3.1 Pro Benchmarks, Price & Specs (Oct 2026) Google	30.2%	$0.02	1 Sep 2026	Pc_direct_sync → Proprietary reported `Google Code Harness`
#49	Inkling Thinking Machines	26.6%	$0.02	1 Sep 2026	Pc_direct_sync → Proprietary reported `Thinking Machines`
#50	Inkling Thinking Machines	26.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported

Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

Manual / Direct Evaluation Ingest — Licence: Editorial
SWE-bench Official Leaderboard — Licence: MIT

COMMAND-LINE AGENT EXECUTION

Terminal-Bench 2.0 Leaderboard

View Methodology & Deep-Dive →

Autonomous terminal command-line navigation, multi-step debugging, environment triage, and tool execution in isolated Docker containers.

Page updated: 18 Oct 2026 17:18 UTC 89 Evaluated Tasks • Metric: Task Accuracy

					SOURCE & HARNESS
#01	Claude Fable 5.1 (max with fallback) Anthropic	52.0%	—	19 Sep 2026	Pc_direct_sync → MIT reported
#02	Claude Fable 5.1 (max with fallback) Anthropic	52.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#03	GPT-6 Astra (max) Benchmarks, Price & Specs (Oct 2026) OpenAI	51.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#04	GPT-6 Astra (max) Benchmarks, Price & Specs (Oct 2026) OpenAI	51.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#05	Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic	50.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#06	Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic	50.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#07	Muse Spark 1.3 (max) Meta	47.0%	—	1 Sep 2026	Pc_direct_sync → Llama-3.1-Community reported `standard`
#08	Muse Spark 1.3 (max) Meta	47.0%	—	1 Sep 2026	Pc_direct_sync → Llama-3.1-Community reported
#09	GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI	46.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#10	GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI	46.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#11	Qwen3.8 Max (0902) Alibaba	44.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#12	Qwen3.8 Max (0902) Alibaba	44.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#13	GLM-5.3 (max) Z AI	44.2%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#14	GLM-5.3 (max) Z AI	44.2%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#15	Step 5 Preview StepFun	44.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#16	Step 5 Preview StepFun	44.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#17	Kimi K3 (max) Kimi	43.8%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#18	Kimi K3 (max) Kimi	43.8%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#19	Grok 4.6 (high) SpaceXAI	43.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#20	Grok 4.6 (high) SpaceXAI	43.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#21	GLM-5.3-Flash Z AI	42.0%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `standard`
#22	GLM-5.3-Flash Z AI	42.0%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#23	GPT-5.6 Terra (max) OpenAI	41.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#24	GPT-5.6 Terra (max) OpenAI	41.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#25	Gemini 3.8 Flash (high) Google	40.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#26	Gemini 3.8 Flash (high) Google	40.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#27	DeepSeek V4.1 Flash (max) DeepSeek	40.2%	—	1 Sep 2026	Pc_direct_sync → MIT reported `standard`
#28	DeepSeek V4.1 Flash (max) DeepSeek	40.2%	—	1 Sep 2026	Pc_direct_sync → MIT reported
#29	Qwen3.8-Flash-Next Alibaba	40.0%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `standard`
#30	Qwen3.8-Flash-Next Alibaba	40.0%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#31	Qwen3.8 2.4T A95B Alibaba	39.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `standard`
#32	Qwen3.8 2.4T A95B Alibaba	39.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#33	Claude Sonnet 5 (max) Anthropic	38.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#34	Claude Sonnet 5 (max) Anthropic	38.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#35	GPT-5.6 Luna (max) OpenAI	37.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#36	GPT-5.6 Luna (max) OpenAI	37.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#37	DeepSeek V4 Pro Benchmarks, Price & Specs (Oct 2026) DeepSeek	35.5%	—	1 Sep 2026	Pc_direct_sync → MIT reported
#38	DeepSeek V4 Pro Benchmarks, Price & Specs (Oct 2026) DeepSeek	35.5%	—	1 Sep 2026	Pc_direct_sync → MIT reported `standard`
#39	DeepSeek V4 Flash Vision (max) DeepSeek	34.5%	—	1 Sep 2026	Pc_direct_sync → MIT reported
#40	DeepSeek V4 Flash Vision (max) DeepSeek	34.5%	—	1 Sep 2026	Pc_direct_sync → MIT reported `standard`
#41	Qwen3.8 27B (xhigh) Alibaba	33.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#42	Qwen3.8 27B (xhigh) Alibaba	33.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `standard`
#43	K2 Horizon 375B A23B Institute of Foundation Models	30.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported
#44	K2 Horizon 375B A23B Institute of Foundation Models	30.5%	—	1 Sep 2026	Pc_direct_sync → Apache-2.0 reported `standard`
#45	Gemini 3.1 Pro Benchmarks, Price & Specs (Oct 2026) Google	30.2%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#46	Gemini 3.1 Pro Benchmarks, Price & Specs (Oct 2026) Google	30.2%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#47	MiniMax-M3 MiniMax	30.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#48	MiniMax-M3 MiniMax	30.0%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`
#49	Inkling Thinking Machines	25.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported
#50	Inkling Thinking Machines	25.5%	—	1 Sep 2026	Pc_direct_sync → Proprietary reported `standard`

Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

Terminal-Bench Submissions — Licence: Apache-2.0

CANONICAL BENCHMARK DIRECTORY

Benchmark Methodologies & Specifications

Detailed specifications for each benchmark in our suite: evaluation scaffolding, test environments, resolution criteria, and known limitations.

CANONICAL BENCHMARK aider_polyglot

Aider Polyglot Leaderboard: Coding Diff Compliance (2026)

Multi-language code editing and refactoring across diverse programming paradigms, evaluating both SEARCH/REPLACE diff format compliance and unit test verification.

Caveat: Heavily reliant on Aider repo-map and diff format prompt adherence; small models frequently fail due to format violations…

View Full Methodology & Telemetry →

CANONICAL BENCHMARK

Best Free OpenRouter AI Models for Programming

Empirical benchmark of all 22 active free models on OpenRouter (:free) across N=100 availability telemetry probes and 25 deterministic coding challenges.

View Full Methodology & Telemetry →

CANONICAL BENCHMARK

GPT-4.1 vs DeepSeek V3: Hallucination Rates, Measured

DeepSeek V3 has no published hallucination score, so we ran 60 prompts against both models and graded every answer. Full results, prompts and method inside.

View Full Methodology & Telemetry →

CANONICAL BENCHMARK livecodebench

LiveCodeBench Leaderboard: Contamination-Free Coding (2026)

Algorithmic problem solving, complex mathematical reasoning, and edge-case handling on brand-new competitive programming problems published after model training cutoff dates.

Caveat: Measures competitive programming and algorithmic puzzles rather than real-world software maintenance or multi-file application development.

View Full Methodology & Telemetry →

CANONICAL BENCHMARK

LLM Benchmark News: New Benchmarks & Model Results

Every new LLM benchmark and what it reveals about models, plus the latest coding scores. Updated weekly.

View Full Methodology & Telemetry →

CANONICAL BENCHMARK

LMArena Ranking Methodology: Glicko-2 vs Bradley-Terry

Live 2026 Chatbot Arena rankings, category charts, confidence intervals, and complete methodology breakdown of Bradley-Terry, AutoEval, and Style Control.

View Full Methodology & Telemetry →

CANONICAL BENCHMARK aider_polyglot

Local quantized LLM performance running under Ollama and llama.cpp on consumer workstations, measuring empirical pass rates, edit-format compliance failure rates, tokens/sec, and peak RAM consumption.

Caveat: Hardware-bound: MacBook Air M2 16GB tier establishes a

View Full Methodology & Telemetry →

CANONICAL BENCHMARK swe_bench_verified

SWE-bench Verified Leaderboard: Top Coding Models (2026)

Autonomous resolution of real-world GitHub issues across popular open-source Python repositories, verified by human software engineers for task clarity and deterministic unit-test pass/fail.

Caveat: Unit tests can occasionally pass on partial solutions; agent performance varies significantly based on test harness scaffolding (e.g.…

View Full Methodology & Telemetry →

CANONICAL BENCHMARK terminal_bench

Terminal-Bench 2.0: CLI & Shell Agent Leaderboard (2026)

Autonomous command-line execution, shell tool use, multi-step debugging, environment triage, and terminal state evaluation in isolated Linux sandboxes.

Caveat: Does not measure long-context codebase refactoring; highly sensitive to system prompt formatting and sub-shell execution timeouts.

View Full Methodology & Telemetry →