What GPQA Diamond Actually Measures
High-level scientific and mathematical reasoning designed to resist web search shortcuts.
⚠ Known Boundaries & Harness Sensitivity
Multiple choice format allows guessing; Diamond subset is small (198 questions).
Intelligence Index vs. Cost per 1M Tokens
Code Intelligence Index vs. Weighted average cost (USD, Log Scale)
GPQA Diamond tests graduate-level scientific question answering. It is a subset of GPQA, a multiple-choice benchmark covering biology, chemistry and physics. Use it to compare performance on this specific question set; a high score alone does not establish coding-agent reliability.
Methodology reference checked 27 October 2026. This page explains the benchmark and the site’s published score records; it does not announce a new evaluation run.
What GPQA Diamond measures
The original GPQA project describes expert-written science questions intended to remain difficult for non-experts even with access to web search. Diamond is the curated subset used in many model comparisons. Consult the official dataset for the exact configuration, revision and access conditions.
How to interpret a reported score
Record the numerator, denominator and scoring rule before comparing percentages. Check whether the result uses the Diamond subset, the exact model snapshot, reasoning settings, tools, prompt template and number of repeated attempts. A single-attempt accuracy figure and a multi-attempt selection result answer different questions and should not share an unlabeled column.
Preserve the evaluator’s distinction between model-only answers and tool-assisted answers. If a result uses a different subset or harness, retain a separate row and identify the difference. Missing results should remain unscored.
What this benchmark leaves open
A science multiple-choice score does not directly measure repository editing, terminal work, deployment, latency or the cost of completing your task. Combine it with SWE-bench Verified, Terminal-Bench and a small test of your own workflow when selecting a coding model.
How to check an entry on this site
Follow the source attached to the result and compare its model identity, date and evaluation settings with the table. Treat an entry without an accessible run-level source as needing verification. The model leaderboard offers a separate exploratory weighted comparison; its custom score is not the GPQA accuracy metric.
Primary sources
Keep exploring
Complete Verified Leaderboard
Fully server-side rendered table with verified primary source attributions and direct repository links.
| SOURCE & HARNESS | |||||
|---|---|---|---|---|---|
| #01 | Gemini 4 Argon (high) Google | 96.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #02 | GPT-6 Astra (max) OpenAI | 96.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #03 | Claude Sonnet 5.5 Anthropic | 93.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #04 | GPT-6 Sol OpenAI | 91.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #05 | Claude Opus 5.5 Anthropic | 90.6% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #06 | Muse Wonder Meta | 88.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #07 | Mistral Large 4 (preview) Mistral | 87.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #08 | Grok 4.7 SpaceXAI | 86.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #09 | Qwen3.8 Max (0902) Alibaba | 86.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #10 | Kimi K3 (max) Kimi | 85.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #11 | Step 5 Preview (preview) StepFun | 84.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #12 | DeepSeek V4.1 Flash (1017 checkpoint) DeepSeek | 84.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #13 | Muse Spark Open (816B MoE) Meta | 83.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #14 | GPT-6 Luna OpenAI | 81.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #15 | Gemini 3.8 Flash (high) Google | 81.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #16 | Claude Haiku 5.5 Anthropic | 80.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #17 | Qwen3.8-Flash-Next Alibaba | 80.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #18 | GLM-5.3 Fast (highspeed) Z AI | 79.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #19 | MiniMax M3.1 Flash (max) MiniMax | 78.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #20 | DeepSeek V4 Pro (0813 max) DeepSeek | 77.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #21 | Beam 1.0 501B (A23B MoE) Reflection AI | 76.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #22 | DeepSeek V4 Flash Vision (max) DeepSeek | 75.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #23 | Nemotron 3.5 Super VL (120B A12B) NVIDIA | 74.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #24 | Qwen3.8 27B (xhigh) Alibaba | 72.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #25 | Claude Fable 5.1 (max with fallback) Anthropic | 72.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #26 | Claude Fable 5.1 (max with fallback) Anthropic | 72.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #27 | GPT-6 Astra (max) OpenAI | 71.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #28 | K2 Horizon 375B (A23B) Institute of Foundation Models | 70.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #29 | Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic | 69.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #30 | Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic | 69.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #31 | Gemini 3.1 Pro (preview) Google | 69.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #32 | LFM2.5 24B A2B (on-device MoE) Liquid AI | 67.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #33 | Muse Spark 1.3 (max) Meta | 66.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Llama-3.1-Community
reported
standard
|
| #34 | Muse Spark 1.3 (max) Meta | 66.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Llama-3.1-Community
reported
|
| #35 | Inkling Thinking Machines | 65.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #36 | GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI | 65.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #37 | GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI | 65.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #38 | Qwen3.8 Max (0902) Alibaba | 63.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #39 | GLM-5.3 (max) Z AI | 63.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #40 | GLM-5.3 (max) Z AI | 63.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #41 | Gemini 3.5 Flash-Lite Google | 62.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #42 | Grok 4.6 (high) SpaceXAI | 62.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #43 | Grok 4.6 (high) SpaceXAI | 62.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #44 | Kimi K3 (max) Kimi | 61.8% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #45 | Step 5 Preview (preview) StepFun | 61.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #46 | GPT-5.6 Terra (max) OpenAI | 59.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #47 | GPT-5.6 Terra (max) OpenAI | 59.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #48 | GLM-5.3-Flash Z AI | 59.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Apache-2.0
reported
standard
|
| #49 | GLM-5.3-Flash Z AI | 59.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Apache-2.0
reported
|
| #50 | Gemini 3.8 Flash (high) Google | 58.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:
-
Aider Polyglot Leaderboard
—
Licence:
Apache-2.0 -
LMSYS Chatbot Arena (WebDev)
—
Licence:
CC-BY-4.0 -
Manual / Direct Evaluation Ingest
—
Licence:
Editorial -
Models.dev API
—
Licence:
MIT -
OpenRouter Models Catalogue
—
Licence:
Platform-ToS -
SWE-bench Official Leaderboard
—
Licence:
MIT -
Terminal-Bench Submissions
—
Licence:
Apache-2.0
What This Benchmark Does Not Measure
Does not test code execution or shell navigation.
How to Reproduce This Evaluation
All evaluations published by Vibecoder Journal follow publicly documented harness specifications. You can execute this test suite against any local or API model:
# Standardized containerized evaluation execution
git clone https://github.com/idavidrein/gpqa && cd evaluation
docker build -t vcj-harness .
docker run --rm -v $(pwd)/results:/results vcj-harness --benchmark gpqa-diamond
In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:
-
Aider Polyglot Leaderboard
—
Licence:
Apache-2.0 -
LMSYS Chatbot Arena (WebDev)
—
Licence:
CC-BY-4.0 -
Manual / Direct Evaluation Ingest
—
Licence:
Editorial -
Models.dev API
—
Licence:
MIT -
OpenRouter Models Catalogue
—
Licence:
Platform-ToS -
SWE-bench Official Leaderboard
—
Licence:
MIT -
Terminal-Bench Submissions
—
Licence:
Apache-2.0



