CANONICAL BENCHMARK • EVALUATION SPECIFICATION

SciCode

What SciCode Actually Measures

A standardized empirical evaluation suite assessing LLM code generation and editing competencies under uniform automated test constraints.

⚠ Known Boundaries & Harness Sensitivity

Results are heavily influenced by the agent harness, sampling temperature, and retry heuristics. Scores should be compared within identical scaffolding configurations.

Intelligence Index vs. Cost per 1M Tokens

Code Intelligence Index vs. Weighted average cost (USD, Log Scale)

Most attractive quadrant Pareto frontier

SciCode evaluates code generation for scientific research problems. Scientists curate problems that combine domain knowledge, reasoning and implementation. The benchmark is useful for research-code workflows and should be interpreted with its evaluation configuration.

Methodology reference checked 27 October 2026. This page explains the benchmark and the site’s published score records; it does not announce a new evaluation run.

What a SciCode task looks like

The official SciCode project describes research problems decomposed into subproblems, with scientific background, reference solutions and tests. The project provides the dataset, evaluation instructions and links to the research paper. Those original materials are the starting point for reproducing an evaluation.

Identify the metric before comparing models

A subproblem accuracy result and a complete-main-problem success rate are different metrics. Keep their denominators and labels visible. Check whether background information or previous subproblem solutions were provided, which task split was used, and how the generated code was executed and tested.

Record the exact model and provider snapshot, agent or harness, dependency environment, allowed tools, reasoning settings, number of attempts and result date. A model with missing data should remain unscored. Do not convert an unrelated coding result into an estimated SciCode result.

Read results in the context of your work

Scientific research code can require specialist assumptions that an ordinary application task does not. Use a SciCode result alongside a representative task from your own field. Check correctness, numerical tolerances, reproducibility and the resources needed to obtain a working result. A single benchmark percentage does not replace that review.

Checking the site’s result table

Open the source attached to a result and verify that its metric matches the table heading. If two sources use different conditions, retain separate labeled results rather than averaging them into a supposedly universal score. The rankings hub provides model records; LiveCodeBench and SWE-bench Verified offer complementary views of coding performance.

Primary source and reproduction materials

Use the official SciCode project website for its paper, dataset, repository, evaluation instructions and stated limitations. Preserve the original dataset and code licensing terms when reusing those materials.

You made it to the end!

Get the weekly briefing

New model prices, benchmark results and Claude Code tips. One email a week. Free.

No spam. Unsubscribe anytime. Privacy policy

Keep exploring

Complete Verified Leaderboard

Fully server-side rendered table with verified primary source attributions and direct repository links.

Page updated: 338 Evaluated Tasks • Metric: Problem Solving Rate
SOURCE & HARNESS
#01 Claude Fable 5.1 (max with fallback) Anthropic 58.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#02 Claude Fable 5.1 (max with fallback) Anthropic 58.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#03 GPT-6 Astra (max) OpenAI 57.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#04 Claude Opus 5.5 Anthropic 57.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#05 GPT-6 Astra (max) OpenAI 56.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#06 Gemini 4 Argon (high) Google 55.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#07 Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic 55.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#08 Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic 55.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#09 Claude Sonnet 5.5 Anthropic 54.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#10 Muse Wonder Meta 54.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#11 GPT-6 Sol OpenAI 52.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#12 Muse Spark 1.3 (max) Meta 52.0% — 1 Sep 2026 Pc_direct_sync →
Llama-3.1-Community reported standard
#13 Muse Spark 1.3 (max) Meta 52.0% — 1 Sep 2026 Pc_direct_sync →
Llama-3.1-Community reported
#14 Qwen3.8 Max (0902) Alibaba 51.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#15 GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI 51.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#16 GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI 51.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#17 Mistral Large 4 (preview) Mistral 50.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#18 Grok 4.7 SpaceXAI 49.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#19 Kimi K3 (max) Kimi 49.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#20 Qwen3.8 Max (0902) Alibaba 49.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#21 Step 5 Preview (preview) StepFun 48.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#22 GLM-5.3 (max) Z AI 48.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#23 GLM-5.3 (max) Z AI 48.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#24 Grok 4.6 (high) SpaceXAI 47.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#25 Grok 4.6 (high) SpaceXAI 47.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#26 Muse Spark Open (816B MoE) Meta 47.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#27 Kimi K3 (max) Kimi 47.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#28 Step 5 Preview (preview) StepFun 46.8% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#29 DeepSeek V4.1 Flash (1017 checkpoint) DeepSeek 46.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#30 Gemini 3.8 Flash (high) Google 45.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#31 GPT-5.6 Terra (max) OpenAI 45.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#32 GPT-5.6 Terra (max) OpenAI 45.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#33 GLM-5.3-Flash Z AI 44.5% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported standard
#34 GLM-5.3-Flash Z AI 44.5% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported
#35 Qwen3.8-Flash-Next Alibaba 44.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#36 GLM-5.3 Fast (highspeed) Z AI 43.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#37 Gemini 3.8 Flash (high) Google 43.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#38 Qwen3.8 2.4T A95B Alibaba 42.5% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported standard
#39 Qwen3.8 2.4T A95B Alibaba 42.5% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported
#40 DeepSeek V4.1 Flash (max) DeepSeek 42.2% — 1 Sep 2026 Pc_direct_sync →
MIT reported
#41 DeepSeek V4.1 Flash (max) DeepSeek 42.2% — 1 Sep 2026 Pc_direct_sync →
MIT reported standard
#42 GPT-6 Luna OpenAI 42.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#43 Qwen3.8-Flash-Next Alibaba 42.0% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported standard
#44 Claude Haiku 5.5 Anthropic 41.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#45 Claude Sonnet 5 (max) Anthropic 40.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#46 Claude Sonnet 5 (max) Anthropic 40.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#47 MiniMax M3.1 Flash (max) MiniMax 40.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#48 GPT-5.6 Luna (max) OpenAI 40.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#49 GPT-5.6 Luna (max) OpenAI 40.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#50 DeepSeek V4 Pro (0813 max) DeepSeek 39.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

What This Benchmark Does Not Measure

This benchmark does not evaluate full-repository software architecture, cross-service microservice orchestration, long-term memory maintenance, or visual graphical UI design. High accuracy on this test suite does not guarantee bug-free deployment in production enterprise codebases.

How to Reproduce This Evaluation

All evaluations published by Vibecoder Journal follow publicly documented harness specifications. You can execute this test suite against any local or API model:

# Standardized containerized evaluation execution
git clone https://scicode-bench.github.io/ && cd evaluation
docker build -t vcj-harness .
docker run --rm -v $(pwd)/results:/results vcj-harness --benchmark scicode
Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

Abdullah Zulfiqar
Abdullah Zulfiqar Founder & Technical EditorLast updated 27 October 2026

Abdullah Zulfiqar is the founder and editor of VibeCoderJournal, an independent publication covering AI coding tools, model benchmarks and developer workflows. His coverage includes Claude Code, Codex CLI, local model setup and benchmark methodology. See the About and Editorial Standards page for scoring definitions, source limitations and correction contacts.

Related AI Coding Benchmarks

benchmark 2 min read

GPQA Diamond

GPQA Diamond tests graduate-level scientific question answering. It is a subset of GPQA, a multiple-choice benchmark covering biology…