CANONICAL BENCHMARK • EVALUATION SPECIFICATION

Local Coding Models Benchmark

Empirical pass rates, edit-format failure compliance, generation speed, and 1Hz memory telemetry on consumer workstations.

What Local Coding Models Benchmark Actually Measures

Local quantized LLM performance running under Ollama and llama.cpp on consumer workstations, measuring empirical pass rates, edit-format compliance failure rates, tokens/sec, and peak RAM consumption.

EDITORIAL NOTE • AGENTIC DIFF HARNESS

The Aider Polyglot Benchmark is not a static code completion test. It evaluates autonomous pair-programming execution across 6 programming languages (Python, JavaScript/TypeScript, Go, Rust, C++, Java). Models must output strict git SEARCH/REPLACE blocks. A model that understands the logic but violates the delimiter syntax fails the test immediately.

⚠ Known Boundaries & Harness Sensitivity

Hardware-bound: MacBook Air M2 16GB tier establishes a <=14B 4-bit ceiling. Larger models require 32GB/64GB+ workstations.

Intelligence Index vs. Cost per 1M Tokens

Code Intelligence Index vs. Weighted average cost (USD, Log Scale)

Most attractive quadrant Pareto frontier

Highlights

Updated Daily • Empirical Telemetry

Intelligence

Updated

Artificial Analysis Intelligence Index • Higher is better

Speed

Output tokens per second • Higher is better

Cost per 1M Tokens

Weighted average blended price (USD) • Lower is better

Evaluation scope: Tests and model/version examples below are historical snapshots, including older product versions. The article edit date does not represent a new benchmark run. Remote API entries, where listed, are comparators rather than locally executed models. Verify current product limits and require the underlying test artifacts before treating numerical claims as reproduced results.

### Empirical Local LLM Coding Telemetry This benchmark evaluates quantized open-weights coding models running locally on physical consumer hardware via Ollama and llama.cpp. Rather than relying on synthetic estimates, all metrics are captured live using vcj-local-bench with continuous 1Hz memory probe sampling and isolated execution sandboxes. ### The Edit-Format Failure Rate In autonomous coding agents, small models frequently fail because they violate required diff/patch formatting, crashing the harness before unit tests can run. This leaderboard reports both pass rate and edit-format compliance failure rate as headline metrics.

No Verified Local Runs Found for This Selection

No empirical runs matching Tier "all" and Harness "mini-swe-agent" are currently published. Run vcj-local-bench on Ollama or llama.cpp and upload the exported CSV under WP Admin → Leaderboard → Benchmark Data → CSV/JSON Import.

Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

What it does not measure

Every benchmark is narrow by design. Reading this local-model comparison as a general statement about coding ability is the most common mistake people make with it. Here is what the number is silent on:

  • Frontier capability. Every model here runs on consumer hardware, which places a hard ceiling on what is being compared.
  • Cloud-hosted performance. The same weights served on datacentre hardware behave differently on throughput.
  • Anything about models that do not fit in the memory budget tested.

How to read a score honestly

  • Match the quantisation. A Q4_K_M result and a Q8_0 result are different models for comparison purposes.
  • Match the context length. Memory figures move dramatically with context, and a comparison at 4K says little about behaviour at 32K.
  • Check the runtime. Ollama, llama.cpp and LM Studio do not produce identical throughput on the same hardware.

The trap

Parameter count is the weakest predictor of local performance. Architecture — particularly how many key-value heads a model has — often decides whether it fits your card, and two 7B models can differ eightfold in cache size.

Why we publish this page

Benchmark figures circulate faster than they are checked. Numbers get copied between articles until the original source is several links away and nobody has opened the leaderboard. We maintain this page so there is a version of the figure that names its source, its date and its methodology — and so that when a number here changes, you can see what changed.

Last reviewed 7 September 2026. If a figure on this page disagrees with the official leaderboard, that is a mistake on our side and we want to know about it.

In-Depth Evaluated Local & Open-Weight Architectures

For comprehensive benchmark breakdowns, memory requirement charts, and production deployment code, read our dedicated evaluations:

You made it to the end!

Get the weekly briefing

New model prices, benchmark results and Claude Code tips. One email a week. Free.

No spam. Unsubscribe anytime. Privacy policy

Keep exploring

ComparisonKimi K2.5 vs Qwen 3.5: Which Open Model Is Better for Coding, Agents and Running Locally?ReviewWinston AI Detector Review 2026: Is It Accurate? We Ran 4 TestsGuideWhat Is Vibe Coding? Definition, Meaning and How It Works

Complete Verified Leaderboard

Fully server-side rendered table with verified primary source attributions and direct repository links.

Page updated: 225 Evaluated Tasks • Metric: Pass Rate (Pass@2)
SOURCE & HARNESS
#01 Claude Fable 5.1 (max with fallback) Anthropic 54.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#02 Claude Opus 5.5 Anthropic 54.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#03 Claude Fable 5.1 (max with fallback) Anthropic 54.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#04 GPT-6 Astra (max) OpenAI 53.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#05 GPT-6 Astra (max) OpenAI 53.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#06 Claude Sonnet 5.5 Anthropic 52.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#07 Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic 51.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#08 Claude Opus 5 (max) Benchmarks, Price & Specs (Oct 2026) Anthropic 51.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#09 Gemini 4 Argon (high) Google 51.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#10 Muse Spark 1.3 (max) Meta 49.5% — 1 Sep 2026 Pc_direct_sync →
Llama-3.1-Community reported standard
#11 Muse Spark 1.3 (max) Meta 49.5% — 1 Sep 2026 Pc_direct_sync →
Llama-3.1-Community reported
#12 GPT-6 Sol OpenAI 49.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#13 Muse Wonder Meta 48.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#14 GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI 47.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#15 GPT-5.6 Sol (max) Benchmarks, Price & Specs (Oct 2026) OpenAI 47.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#16 Qwen3.8 Max (0902) Alibaba 47.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#17 GLM-5.3 (max) Z AI 46.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#18 GLM-5.3 (max) Z AI 46.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#19 Mistral Large 4 (preview) Mistral 46.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#20 Qwen3.8 Max (0902) Alibaba 46.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#21 Step 5 Preview (preview) StepFun 45.2% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#22 Kimi K3 (max) Kimi 45.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#23 Grok 4.6 (high) SpaceXAI 45.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#24 Grok 4.6 (high) SpaceXAI 45.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#25 Step 5 Preview (preview) StepFun 44.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#26 Kimi K3 (max) Kimi 44.5% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#27 Grok 4.7 SpaceXAI 44.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#28 Muse Spark Open (816B MoE) Meta 43.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#29 DeepSeek V4.1 Flash (1017 checkpoint) DeepSeek 43.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#30 GPT-5.6 Terra (max) OpenAI 43.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#31 GPT-5.6 Terra (max) OpenAI 43.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#32 GLM-5.3-Flash Z AI 42.8% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported standard
#33 GLM-5.3-Flash Z AI 42.8% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported
#34 Gemini 3.8 Flash (high) Google 42.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#35 Gemini 3.8 Flash (high) Google 42.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#36 DeepSeek V4.1 Flash (max) DeepSeek 41.2% — 1 Sep 2026 Pc_direct_sync →
MIT reported
#37 DeepSeek V4.1 Flash (max) DeepSeek 41.2% — 1 Sep 2026 Pc_direct_sync →
MIT reported standard
#38 Qwen3.8-Flash-Next Alibaba 41.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#39 Qwen3.8 2.4T A95B Alibaba 41.0% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported standard
#40 Qwen3.8 2.4T A95B Alibaba 41.0% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported
#41 GLM-5.3 Fast (highspeed) Z AI 40.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#42 Qwen3.8-Flash-Next Alibaba 40.5% — 1 Sep 2026 Pc_direct_sync →
Apache-2.0 reported standard
#43 GPT-6 Luna OpenAI 39.5% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#44 Claude Haiku 5.5 Anthropic 39.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#45 GPT-5.6 Luna (max) OpenAI 39.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#46 GPT-5.6 Luna (max) OpenAI 39.0% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#47 Claude Sonnet 5 (max) Anthropic 38.8% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported
#48 Claude Sonnet 5 (max) Anthropic 38.8% — 1 Sep 2026 Pc_direct_sync →
Proprietary reported standard
#49 MiniMax M3.1 Flash (max) MiniMax 38.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
#50 DeepSeek V4 Pro (0813 max) DeepSeek 37.0% — 21 Oct 2026 Pc_direct_sync →
MIT reported
Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

What This Benchmark Does Not Measure

Does not test large multi-thousand line architecture planning.

How to Reproduce This Evaluation

All evaluations published by Vibecoder Journal follow publicly documented harness specifications. You can execute this test suite against any local or API model: 

# 1. Clone the official Aider repository
git clone https://github.com/Aider-AI/aider.git && cd aider
pip install -e .

# 2. Run the standardized 225-exercise Polyglot benchmark
python -m aider.benchmarks.polyglot \
    --model <model-name> \
    --edit-format diff \
    --output-dir ./eval-results
Sources & Open Data Licences CC-BY / Open Source

In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:

Abdullah Zulfiqar
Abdullah Zulfiqar Founder & Technical EditorLast updated 27 October 2026

Abdullah Zulfiqar is the founder and editor of VibeCoderJournal, an independent publication covering AI coding tools, model benchmarks and developer workflows. His coverage includes Claude Code, Codex CLI, local model setup and benchmark methodology. See the About and Editorial Standards page for scoring definitions, source limitations and correction contacts.

All Benchmarks → LinkedIn Profile →

Related AI Coding Benchmarks

benchmark 2 min read

SciCode

SciCode evaluates code generation for scientific research problems. Scientists curate problems that combine domain knowledge, reasoning and…

By Abdullah Zulfiqar • Oct 27, 2026
benchmark 2 min read

GPQA Diamond

GPQA Diamond tests graduate-level scientific question answering. It is a subset of GPQA, a multiple-choice benchmark covering biology…

By Abdullah Zulfiqar • Oct 27, 2026