CANONICAL BENCHMARK • EVALUATION SPECIFICATION

GPQA Diamond

What GPQA Diamond Actually Measures

High-level scientific and mathematical reasoning designed to resist web search shortcuts.

⚠ Known Boundaries & Harness Sensitivity

Multiple choice format allows guessing; Diamond subset is small (198 questions).

Intelligence Index vs. Cost per 1M Tokens

Code Intelligence Index vs. Weighted average cost (USD, Log Scale)

Most attractive quadrant Pareto frontier

GPQA Diamond tests graduate-level scientific question answering. It is a subset of GPQA, a multiple-choice benchmark covering biology, chemistry and physics. Use it to compare performance on this specific question set; a high score alone does not establish coding-agent reliability.

Methodology reference checked 27 October 2026. This page explains the benchmark and the site’s published score records; it does not announce a new evaluation run.

What GPQA Diamond measures

The original GPQA project describes expert-written science questions intended to remain difficult for non-experts even with access to web search. Diamond is the curated subset used in many model comparisons. Consult the official dataset for the exact configuration, revision and access conditions.

How to interpret a reported score

Record the numerator, denominator and scoring rule before comparing percentages. Check whether the result uses the Diamond subset, the exact model snapshot, reasoning settings, tools, prompt template and number of repeated attempts. A single-attempt accuracy figure and a multi-attempt selection result answer different questions and should not share an unlabeled column.

Preserve the evaluator’s distinction between model-only answers and tool-assisted answers. If a result uses a different subset or harness, retain a separate row and identify the difference. Missing results should remain unscored.

What this benchmark leaves open

A science multiple-choice score does not directly measure repository editing, terminal work, deployment, latency or the cost of completing your task. Combine it with SWE-bench Verified, Terminal-Bench and a small test of your own workflow when selecting a coding model.

How to check an entry on this site

Follow the source attached to the result and compare its model identity, date and evaluation settings with the table. Treat an entry without an accessible run-level source as needing verification. The model leaderboard offers a separate exploratory weighted comparison; its custom score is not the GPQA accuracy metric.

Primary sources

You made it to the end!

Get the weekly briefing

New model prices, benchmark results and Claude Code tips. One email a week. Free.

No spam. Unsubscribe anytime. Privacy policy

Keep exploring

Complete Verified Leaderboard

Fully server-side rendered table with verified primary source attributions and direct repository links.

Page updated: 198 Evaluated Tasks • Metric: Accuracy
SOURCE & HARNESS
#01 Gemini 4 Argon (high) Google 96.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#02 GPT-6 Astra (max) OpenAI 96.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#03 Claude Sonnet 5.5 Anthropic 93.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#04 GPT-6 Sol OpenAI 91.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#05 Claude Opus 5.5 Anthropic 90.6% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#06 Muse Wonder Meta 88.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#07 Mistral Large 4 (preview) Mistral 87.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#08 Grok 4.7 SpaceXAI 86.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#09 Qwen3.8 Max (0902) Alibaba 86.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#10 Kimi K3 (max) Kimi 85.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#11 Step 5 Preview (preview) StepFun 84.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#12 DeepSeek V4.1 Flash (1017 checkpoint) DeepSeek 84.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#13 Muse Spark Open (816B MoE) Meta 83.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#14 GPT-6 Luna OpenAI 81.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#15 Gemini 3.8 Flash (high) Google 81.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#16 Claude Haiku 5.5 Anthropic 80.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#17 Qwen3.8-Flash-Next Alibaba 80.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#18 GLM-5.3 Fast (highspeed) Z AI 79.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#19 MiniMax M3.1 Flash (max) MiniMax 78.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#20 DeepSeek V4 Pro (0813 max) DeepSeek 77.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#21 Beam 1.0 501B (A23B MoE) Reflection AI 76.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#22 DeepSeek V4 Flash Vision (max) DeepSeek 75.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#23 Nemotron 3.5 Super VL (120B A12B) NVIDIA 74.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#24 Qwen3.8 27B (xhigh) Alibaba 72.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#25 Claude Fable 5.1 (max with fallback) Anthropic 72.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#26 Claude Fable 5.1 (max with fallback) Anthropic 72.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#27 GPT-6 Astra (max) OpenAI 71.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#28 K2 Horizon 375B (A23B) Institute of Foundation Models 70.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#29 Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic 69.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#30 Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic 69.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#31 Gemini 3.1 Pro (preview) Google 69.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#32 LFM2.5 24B A2B (on-device MoE) Liquid AI 67.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#33 Muse Spark 1.3 (max) Meta 66.0% — 1 Sep 2026 Pc_direct_sync →
Llama-3.1-Community reported standard
#34 Muse Spark 1.3 (max) Meta 66.0% — 1 Sep 2026 Pc_direct_sync →
Llama-3.1-Community reported
#35 Inkling Thinking Machines 65.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#36 GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI 65.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#37 GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI 65.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#38 Qwen3.8 Max (0902) Alibaba 63.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#39 GLM-5.3 (max) Z AI 63.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#40 GLM-5.3 (max) Z AI 63.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#41 Gemini 3.5 Flash-Lite Google 62.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#42 Grok 4.6 (high) SpaceXAI 62.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#43 Grok 4.6 (high) SpaceXAI 62.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#44 Kimi K3 (max) Kimi 61.8% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#45 Step 5 Preview (preview) StepFun 61.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#46 GPT-5.6 Terra (max) OpenAI 59.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#47 GPT-5.6 Terra (max) OpenAI 59.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#48 GLM-5.3-Flash Z AI 59.0% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported standard
#49 GLM-5.3-Flash Z AI 59.0% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported
#50 Gemini 3.8 Flash (high) Google 58.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

What This Benchmark Does Not Measure

Does not test code execution or shell navigation.

How to Reproduce This Evaluation

All evaluations published by Vibecoder Journal follow publicly documented harness specifications. You can execute this test suite against any local or API model:

# Standardized containerized evaluation execution
git clone https://github.com/idavidrein/gpqa && cd evaluation
docker build -t vcj-harness .
docker run --rm -v $(pwd)/results:/results vcj-harness --benchmark gpqa-diamond
Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

Abdullah Zulfiqar
Abdullah Zulfiqar Founder & Technical EditorLast updated 27 October 2026

Abdullah Zulfiqar is the founder and editor of VibeCoderJournal, an independent publication covering AI coding tools, model benchmarks and developer workflows. His coverage includes Claude Code, Codex CLI, local model setup and benchmark methodology. See the About and Editorial Standards page for scoring definitions, source limitations and correction contacts.

Related AI Coding Benchmarks

benchmark 2 min read

SciCode

SciCode evaluates code generation for scientific research problems. Scientists curate problems that combine domain knowledge, reasoning and…