What SciCode Actually Measures
A standardized empirical evaluation suite assessing LLM code generation and editing competencies under uniform automated test constraints.
⚠ Known Boundaries & Harness Sensitivity
Results are heavily influenced by the agent harness, sampling temperature, and retry heuristics. Scores should be compared within identical scaffolding configurations.
Intelligence Index vs. Cost per 1M Tokens
Code Intelligence Index vs. Weighted average cost (USD, Log Scale)
SciCode evaluates code generation for scientific research problems. Scientists curate problems that combine domain knowledge, reasoning and implementation. The benchmark is useful for research-code workflows and should be interpreted with its evaluation configuration.
Methodology reference checked 27 October 2026. This page explains the benchmark and the site’s published score records; it does not announce a new evaluation run.
What a SciCode task looks like
The official SciCode project describes research problems decomposed into subproblems, with scientific background, reference solutions and tests. The project provides the dataset, evaluation instructions and links to the research paper. Those original materials are the starting point for reproducing an evaluation.
Identify the metric before comparing models
A subproblem accuracy result and a complete-main-problem success rate are different metrics. Keep their denominators and labels visible. Check whether background information or previous subproblem solutions were provided, which task split was used, and how the generated code was executed and tested.
Record the exact model and provider snapshot, agent or harness, dependency environment, allowed tools, reasoning settings, number of attempts and result date. A model with missing data should remain unscored. Do not convert an unrelated coding result into an estimated SciCode result.
Read results in the context of your work
Scientific research code can require specialist assumptions that an ordinary application task does not. Use a SciCode result alongside a representative task from your own field. Check correctness, numerical tolerances, reproducibility and the resources needed to obtain a working result. A single benchmark percentage does not replace that review.
Checking the site’s result table
Open the source attached to a result and verify that its metric matches the table heading. If two sources use different conditions, retain separate labeled results rather than averaging them into a supposedly universal score. The rankings hub provides model records; LiveCodeBench and SWE-bench Verified offer complementary views of coding performance.
Primary source and reproduction materials
Use the official SciCode project website for its paper, dataset, repository, evaluation instructions and stated limitations. Preserve the original dataset and code licensing terms when reusing those materials.
Keep exploring
Complete Verified Leaderboard
Fully server-side rendered table with verified primary source attributions and direct repository links.
| SOURCE & HARNESS | |||||
|---|---|---|---|---|---|
| #01 | Claude Fable 5.1 (max with fallback) Anthropic | 58.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #02 | Claude Fable 5.1 (max with fallback) Anthropic | 58.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #03 | GPT-6 Astra (max) OpenAI | 57.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #04 | Claude Opus 5.5 Anthropic | 57.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #05 | GPT-6 Astra (max) OpenAI | 56.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #06 | Gemini 4 Argon (high) Google | 55.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #07 | Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic | 55.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #08 | Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic | 55.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #09 | Claude Sonnet 5.5 Anthropic | 54.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #10 | Muse Wonder Meta | 54.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #11 | GPT-6 Sol OpenAI | 52.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #12 | Muse Spark 1.3 (max) Meta | 52.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Llama-3.1-Community
reported
standard
|
| #13 | Muse Spark 1.3 (max) Meta | 52.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Llama-3.1-Community
reported
|
| #14 | Qwen3.8 Max (0902) Alibaba | 51.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #15 | GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI | 51.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #16 | GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI | 51.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #17 | Mistral Large 4 (preview) Mistral | 50.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #18 | Grok 4.7 SpaceXAI | 49.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #19 | Kimi K3 (max) Kimi | 49.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #20 | Qwen3.8 Max (0902) Alibaba | 49.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #21 | Step 5 Preview (preview) StepFun | 48.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #22 | GLM-5.3 (max) Z AI | 48.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #23 | GLM-5.3 (max) Z AI | 48.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #24 | Grok 4.6 (high) SpaceXAI | 47.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #25 | Grok 4.6 (high) SpaceXAI | 47.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #26 | Muse Spark Open (816B MoE) Meta | 47.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #27 | Kimi K3 (max) Kimi | 47.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #28 | Step 5 Preview (preview) StepFun | 46.8% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #29 | DeepSeek V4.1 Flash (1017 checkpoint) DeepSeek | 46.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #30 | Gemini 3.8 Flash (high) Google | 45.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #31 | GPT-5.6 Terra (max) OpenAI | 45.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #32 | GPT-5.6 Terra (max) OpenAI | 45.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #33 | GLM-5.3-Flash Z AI | 44.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Apache-2.0
reported
standard
|
| #34 | GLM-5.3-Flash Z AI | 44.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Apache-2.0
reported
|
| #35 | Qwen3.8-Flash-Next Alibaba | 44.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #36 | GLM-5.3 Fast (highspeed) Z AI | 43.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #37 | Gemini 3.8 Flash (high) Google | 43.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #38 | Qwen3.8 2.4T A95B Alibaba | 42.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Apache-2.0
reported
standard
|
| #39 | Qwen3.8 2.4T A95B Alibaba | 42.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Apache-2.0
reported
|
| #40 | DeepSeek V4.1 Flash (max) DeepSeek | 42.2% | — | 1 Sep 2026 |
Pc_direct_sync →
MIT
reported
|
| #41 | DeepSeek V4.1 Flash (max) DeepSeek | 42.2% | — | 1 Sep 2026 |
Pc_direct_sync →
MIT
reported
standard
|
| #42 | GPT-6 Luna OpenAI | 42.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #43 | Qwen3.8-Flash-Next Alibaba | 42.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Apache-2.0
reported
standard
|
| #44 | Claude Haiku 5.5 Anthropic | 41.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #45 | Claude Sonnet 5 (max) Anthropic | 40.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #46 | Claude Sonnet 5 (max) Anthropic | 40.5% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #47 | MiniMax M3.1 Flash (max) MiniMax | 40.0% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
| #48 | GPT-5.6 Luna (max) OpenAI | 40.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
|
| #49 | GPT-5.6 Luna (max) OpenAI | 40.0% | — | 1 Sep 2026 |
Pc_direct_sync →
Proprietary
reported
standard
|
| #50 | DeepSeek V4 Pro (0813 max) DeepSeek | 39.5% | — | 21 Oct 2026 |
Pc_direct_sync →
MIT
reported
|
In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:
-
Aider Polyglot Leaderboard
—
Licence:
Apache-2.0 -
LMSYS Chatbot Arena (WebDev)
—
Licence:
CC-BY-4.0 -
Manual / Direct Evaluation Ingest
—
Licence:
Editorial -
Models.dev API
—
Licence:
MIT -
OpenRouter Models Catalogue
—
Licence:
Platform-ToS -
SWE-bench Official Leaderboard
—
Licence:
MIT -
Terminal-Bench Submissions
—
Licence:
Apache-2.0
What This Benchmark Does Not Measure
This benchmark does not evaluate full-repository software architecture, cross-service microservice orchestration, long-term memory maintenance, or visual graphical UI design. High accuracy on this test suite does not guarantee bug-free deployment in production enterprise codebases.
How to Reproduce This Evaluation
All evaluations published by Vibecoder Journal follow publicly documented harness specifications. You can execute this test suite against any local or API model:
# Standardized containerized evaluation execution
git clone https://scicode-bench.github.io/ && cd evaluation
docker build -t vcj-harness .
docker run --rm -v $(pwd)/results:/results vcj-harness --benchmark scicode
In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:
-
Aider Polyglot Leaderboard
—
Licence:
Apache-2.0 -
LMSYS Chatbot Arena (WebDev)
—
Licence:
CC-BY-4.0 -
Manual / Direct Evaluation Ingest
—
Licence:
Editorial -
Models.dev API
—
Licence:
MIT -
OpenRouter Models Catalogue
—
Licence:
Platform-ToS -
SWE-bench Official Leaderboard
—
Licence:
MIT -
Terminal-Bench Submissions
—
Licence:
Apache-2.0



