What Local Coding Models Benchmark Actually Measures
Local quantized LLM performance running under Ollama and llama.cpp on consumer workstations, measuring empirical pass rates, edit-format compliance failure rates, tokens/sec, and peak RAM consumption.
The Aider Polyglot Benchmark is not a static code completion test. It evaluates autonomous pair-programming execution across 6 programming languages (Python, JavaScript/TypeScript, Go, Rust, C++, Java). Models must output strict git SEARCH/REPLACE blocks. A model that understands the logic but violates the delimiter syntax fails the test immediately.
⚠ Known Boundaries & Harness Sensitivity
Hardware-bound: MacBook Air M2 16GB tier establishes a <=14B 4-bit ceiling. Larger models require 32GB/64GB+ workstations.
Complete Verified Leaderboard
Fully server-side rendered table with verified primary source attributions and direct repository links.
| SOURCE & HARNESS | |||||
|---|---|---|---|---|---|
| #01 | Claude 3.7 Sonnet (Thinking) | 84.9% | $0.00 | 24 Feb 2025 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #02 | o3-mini (High) | 77.3% | $0.00 | 31 Jan 2025 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #03 | o3-mini (Medium) | 72.0% | $0.00 | 31 Jan 2025 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #04 | Claude 3.7 Sonnet | 70.2% | $0.00 | 24 Feb 2025 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #05 | DeepSeek R1 | 64.9% | $0.00 | 24 Jan 2025 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #06 | Claude 3.5 Sonnet (20241022) | 64.0% | $0.00 | 23 Oct 2024 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #07 | o1 (2024-12-17) | 61.8% | $0.00 | 18 Dec 2024 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #08 | DeepSeek V3 | 48.4% | $0.00 | 26 Dec 2024 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #09 | Qwen 2.5 Coder 32B Instruct | 44.0% | $0.00 | 12 Nov 2024 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
| #10 | GPT-4o (2024-11-20) | 43.1% | $0.00 | 21 Nov 2024 |
Aiderpolyglot →
Apache-2.0
reported
diff
|
In compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:
-
Aider Polyglot Leaderboard
—
Licence:
Apache-2.0 -
Manual / Direct Evaluation Ingest
—
Licence:
Editorial
What This Benchmark Does Not Measure
Does not test large multi-thousand line architecture planning.
How to Reproduce This Evaluation
All evaluations published by Vibecoder Journal follow publicly documented harness specifications. You can execute this test suite against any local or API model:
# 1. Clone the official Aider repository
git clone https://github.com/Aider-AI/aider.git && cd aider
pip install -e .
# 2. Run the standardized 225-exercise Polyglot benchmark
python -m aider.benchmarks.polyglot \
--model <model-name> \
--edit-format diff \
--output-dir ./eval-resultsIn compliance with upstream dataset licenses, all evaluated telemetry rows are credited to their primary sources:
-
Aider Polyglot Leaderboard
—
Licence:
Apache-2.0 -
LMSYS Chatbot Arena (WebDev)
—
Licence:
CC-BY-4.0 -
Manual / Direct Evaluation Ingest
—
Licence:
Editorial -
Models.dev API
—
Licence:
MIT -
OpenRouter Models Catalogue
—
Licence:
Platform-ToS -
SWE-bench Official Leaderboard
—
Licence:
MIT -
Terminal-Bench Submissions
—
Licence:
Apache-2.0
