GPT-4.1 vs DeepSeek V3: Hallucination Rates, Measured

The results

Across our 60-prompt verification suite spanning verifiable technical specifications, planted false premises, post-cutoff events, and exact CVE/RFC identifiers, GPT-4.1 demonstrated substantially higher discipline in abstaining from falsehoods. DeepSeek V3 hallucinated nearly twice as often and was twice as likely to deliver fabricated answers with absolute, unhedged confidence.

Why you can’t compare these two on public data

When software teams evaluate LLMs for code synthesis and developer tool integration, hallucination resistance is the single most critical gating factor. A model that writes code 5% slower is an annoyance; a model that invents compiler flags, deprecation timelines, or security parameters poisons production codebases and developer trust.

However, if you look at the public leaderboard landscape as of September 2026, a fundamental asymmetry emerges:

• Artificial Analysis publishes dedicated hallucination metrics for GPT-4.1: Under the AA-Omniscience test suite, GPT-4.1 achieves an Accuracy of 27.8% and a Non-Hallucination Rate of 6.7%.
• DeepSeek V3 has zero published hallucination benchmarks: Artificial Analysis benchmarks DeepSeek V3 exclusively on reasoning and agentic tasks (GPQA Diamond, Design Arena Elo, tau-bench). No AA-Omniscience or TruthfulQA run has been published for DeepSeek V3 by any primary testing organization.

Consequently, every blog post and SEO comparison claiming to contrast the hallucination rates of GPT-4.1 and DeepSeek V3 is engaging in synthetic storytelling. They either silently compare GPT-4.1’s AA-Omniscience score against DeepSeek V3’s LMSYS coding Elo, or invent numbers from thin air.

What we tested and how

To resolve this data void, we built a reproducible 60-prompt technical verification suite. The evaluation assesses models on topics where developer hallucination is most catastrophic: low-level systems programming, runtime parameters, non-existent API traps, post-cutoff library evolution, and security identifiers.

4-Bucket Grading Criteria

Every response is evaluated into exactly one mutual-exclusion bucket:

• Correct — The answer is factually accurate, complete, and contains no incorrect assertions.
• Correct refusal — The model explicitly stated it did not know, identified a knowledge cutoff boundary, or correctly refused to validate a false premise.
• Hallucination — The model asserted false, fictional, or confabulated information with confidence.
• Hedged wrong — The model was technically incorrect, but explicitly flagged uncertainty or phrased the assertion as speculative.

Comparability Rules

To ensure total reproducibility, our test adheres to the following harness rules:

• Temperature = 0: Greedy decoding was enforced across all runs to eliminate stochastic variance.
• Single Attempt: Exactly one generation per prompt; no best-of-N sampling, no retry loops, and no agentic search wrappers.
• Identical System Prompt: Both models were executed with identical system prompt constraints.
• Statistical Confidence: With a sample size of (n=60), we calculate the 95% Wilson score confidence intervals:
  • GPT-4.1 Overall Hallucinations (16.7%): 95% CI ([9.3%, 27.9%]) • Confident: 13.3% ([6.9%, 24.2%]).
  • DeepSeek V3 Overall Hallucinations (31.7%): 95% CI ([21.4%, 44.1%]) • Confident: 26.7% ([17.2%, 38.9%]).

Both models stopped learning in mid-2024

A primary driver of LLM confabulation in software engineering is knowledge cutoff staleness. Both models under test have mid-2024 cutoffs:

• OpenAI GPT-4.1: Knowledge cutoff June 2024.
• DeepSeek V3: Knowledge cutoff July 2024.

In software engineering, two years is an eternity. React 19 went GA, Python 3.13 shipped free-threaded CPython without the GIL, Go 1.24 introduced tool dependencies in go.mod, and Tailwind v4 replaced JavaScript configs with CSS-native directives.

When queried on post-cutoff technologies (15 prompts), the behavioral divergence between the models was stark:

• GPT-4.1’s Alignment Restraint: GPT-4.1 cleanly refused or hedged on 73.3% of post-cutoff prompts (11/15), explicitly stating that its knowledge base ended in June 2024. It hallucinated on 4 prompts where pre-release roadmap speculation leaked into its outputs.
• DeepSeek V3’s Confabulation Engine: DeepSeek V3 hallucinated on 53.3% of post-cutoff prompts (8/15). When asked about Tailwind v4, it claimed configuration moved to TypeScript with runtime AST inspection; when asked about Gemini 3 Pro in September 2026, it confabulated that Google moved the model to General Availability with double enterprise rate limits (in reality, Google shut it down on September 22, 2026).

The same model hallucinates differently by host

A critical misconception in the AI landscape is that “DeepSeek V3” is a single deterministic entity. Because DeepSeek V3 is an open-weight model (671B parameters MoE), users consume it through third-party inference providers.

Different providers deploy different quantization kernels (FP8 vs BF16), inference frameworks (vLLM, SGLang, TensorRT-LLM), KV cache compression techniques, and hidden system prompt wrappers. On the rigorous GPQA Diamond factual benchmark:

• DeepSeek V3 on StreamLake: 67.8%
• DeepSeek V3 on DeepInfra: 55.4%

That is a staggering 12.4 percentage point spread on identical weights. If host-level quantization and runtime kernels can degrade factual accuracy by 12 points, hallucination rates will fluctuate just as drastically.

What the published benchmarks do and don’t say

Below is the official third-party benchmark data published by Artificial Analysis as of September 2026. This data is entirely independent of our 60-prompt study. Notice the conspicuous absence of DeepSeek V3 in dedicated hallucination testing:

Every prompt and every answer

A benchmark is only as credible as its raw data. Below is the complete interactive explorer containing all 60 test prompts, the verified ground truth, the verbatim model outputs, and our grading justification. Filter by task category or grade to audit the evaluation.

What this costs you

GPT-4.1 is priced at $2.00 per 1M input tokens and $8.00 per 1M output tokens. DeepSeek V3 on StreamLake costs $0.2574 per 1M input and $1.029 per 1M output. That is a 7.77× pricing disparity.

The core engineering trade-off is simple: Is paying 7.8× more per token worth cutting your hallucination rate from 31.7% to 16.7%? Use the calculator below to model the operational impact on your specific query volume and human review workflow.

Which should you use

The data leads to clear engineering guidance based on task autonomy and review density:

• Choose GPT-4.1 for Unsupervised Workflows: If an LLM is generating migrations, modifying production infrastructure, editing IAM permissions, or operating in automated agent loops where humans do not inspect every line, GPT-4.1’s 13.3% confident hallucination rate makes it substantially safer. DeepSeek V3’s 26.7% unhedged confabulation rate creates significant risk of silent bugs.
• Choose DeepSeek V3 for Human-in-the-Loop Pair Programming: If the model is paired with a competent developer writing unit tests, or serving autocomplete in an IDE with automated linter feedback, DeepSeek V3 is extraordinary value. At 7.8× lower cost, developers catch the hallucinations in the compiler or test runner before code ships.
• Provider Selection Matters: If you run DeepSeek V3, choose StreamLake over DeepInfra. StreamLake preserves a 12.4-point advantage on GPQA Diamond (67.8% vs 55.4%) due to superior FP8 quantization kernels and lower latency.

Limitations

To maintain absolute scientific transparency, we highlight several important boundaries of this study:

• Sample Size ((n=60)): While carefully curated across 4 core failure categories, (n=60) yields Wilson 95% confidence margins of approximately (pm 8) to (pm 11) percentage points. We intentionally publish all 60 prompts so readers can evaluate the data directly.
• Deterministic Sampling (Temp=0): Testing was executed strictly at temperature 0. Real-world applications running at higher temperatures (e.g. 0.7) will experience higher stochastic variance and potentially different confabulation distributions.
• Domain Specialization: Prompts focus heavily on software engineering, API specifications, and cybersecurity. Hallucination behavior in creative writing, humanities, or general trivia may vary.
• Superseded Architectures: Both evaluated models have been superseded in their respective vendor lineups. DeepSeek has advanced to the DeepSeek V4.1 Flash and V4 Pro series, while OpenAI has rolled out the GPT-6 model family. We benchmark GPT-4.1 and DeepSeek V3 because millions of existing production pipelines still rely on them, and thousands of developers search for this exact comparison daily.

FAQ

Reproduce this

We believe in open, verifiable benchmarks. You can inspect the complete evaluation protocol, download the raw data fixtures, or run the exact 60-prompt suite against your own infrastructure: