DeepSeek Reasoning Effort Benchmark
deepseek-flash (V4.1-Flash) · 3 test suites ·
210 cells · 630 API calls · temperature 0 · deterministic graders
· $1.48 total
Bottom line
minimal
nothing above it earned its cost
Scenarios tested
35
10 easy · 10 hard · 10 repo · 5 ceiling
Tiers that beat minimal
0
on any of the 35 scenarios
Total cost
$1.48
630 graded calls
Short answer: minimal is the right default — and the higher tiers are real but
unused. Across 35 scenarios and 630 graded calls, no tier above
minimal ever produced a better score, including on a suite purpose-built to
break minimal. Where tiers differed, the difference was within-run noise, favoured
less thinking, or disappeared once a grader bug was fixed. Reasoning on vs off is
the switch that actually matters — and the ceiling suite shows it matters most exactly where
you'd expect: hard exact-computation tasks.
All suites pooled by reasoning level
| Level | Mean score | Perfect cells |
Avg latency | Avg reasoning tok | Starved |
|---|
| none |
0.923 |
30/35 |
6.3s |
0 |
0 |
| minimal recommended |
0.986 |
33/35 |
16.2s |
4140 |
0 |
| low |
0.981 |
33/35 |
14.8s |
3755 |
0 |
| medium |
0.975 |
32/35 |
22.4s |
5909 |
1 |
| high |
0.982 |
31/35 |
20.6s |
5315 |
2 |
| xhigh |
0.973 |
31/35 |
22.3s |
5773 |
1 |
The one place reasoning genuinely earns its keep. The ceiling suite (exact modular
exponent, exact rational expected value, big combinatorial sums) is the only set that broke
none: it scored 0.800 with high variance (std 0.189). Every thinking tier —
including minimal — scored a perfect 1.000 with zero variance. So the
rule is: turn reasoning on for exact arithmetic and multi-step computation; the tier you
pick above minimal does not matter.
Reasoning tokens are not monotonic in effort level.
Average reasoning tokens by level do not climb with the tier name — high and
xhigh frequently spend fewer tokens than medium. The level
name sets a ceiling, not a quota, so it is not a reliable cost knob. Latency is the
real, reliable cost of turning effort up: on the repo suite, average latency went from
2.9s at none to 24.5s at medium — an 8× slowdown — for
slightly worse scores.
Everyday business tasks
Ten ordinary tasks: reasoning, code, extraction, classification, writing, judgment. · 60 cells ·
180 calls · $0.0611
Ordinary business tasks. Ceiling was hit immediately — no headroom to separate tiers.
| Scenario | none | minimal | low | medium | high | xhigh |
Verdict |
|---|
| Multi-step math word problem |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Code generation to spec |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Find the bug in code |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Structured JSON extraction |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Support ticket classification |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Constrained customer reply |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Faithful summarization |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Multi-step logic deduction |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Strict format instruction-following |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Professional judgment / boundary |
0.92 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
gap is run-to-run noise |
| Level (this suite) | Mean | Perfect cells |
Avg latency | Avg reason tok |
|---|
| none |
0.992 |
9/10 |
1.7s |
0 |
| minimal |
1.000 |
10/10 |
3.5s |
433 |
| low |
1.000 |
10/10 |
3.1s |
406 |
| medium |
1.000 |
10/10 |
3.8s |
531 |
| high |
1.000 |
10/10 |
4.2s |
576 |
| xhigh |
1.000 |
10/10 |
3.9s |
530 |
Hard tasks (code-verified ground truth)
Adversarial set built to create headroom: real bugs, constraint counting, novel math, adversarial instruction traps, 96K long-context retrieval. · 60 cells ·
180 calls · $0.42545
Adversarial set with code-verified ground truth. Two scenarios produced real spread, but neither favoured a higher tier.
| Scenario | none | minimal | low | medium | high | xhigh |
Verdict |
|---|
| Long-context multi-needle retrieval |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Novel math, many dependent steps |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Multi-bug hunt with interaction |
0.67 | 0.56 | 0.44 | 0.56 | 0.56 | 0.44 |
gap is run-to-run noise |
| Interacting constraint satisfaction |
0.33 | 1.00 | 1.00 | 0.67 | 1.00 | 1.00 |
gap is run-to-run noise |
| Non-trivial algorithm implementation |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Precise extraction with distractors |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Instruction-following with traps |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Quantitative reasoning over a table |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Novel rule application |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Nuanced policy judgment |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Level (this suite) | Mean | Perfect cells |
Avg latency | Avg reason tok |
|---|
| none |
0.900 |
8/10 |
6.2s |
0 |
| minimal |
0.956 |
9/10 |
13.8s |
3154 |
| low |
0.944 |
9/10 |
12.3s |
2738 |
| medium |
0.922 |
8/10 |
18.4s |
4452 |
| high |
0.956 |
9/10 |
15.5s |
3635 |
| xhigh |
0.944 |
9/10 |
17.2s |
4186 |
Real repo tasks (your own `deploy` codebase)
Nine tasks derived from 4,821 lines of real Flask code in your GitHub `deploy` repo; two are executable (refactor + write tests). · 60 cells ·
180 calls · $0.41627
Tasks from your real deploy codebase. Reasoning hurt slightly: none scored 10/10 perfect.
| Scenario | none | minimal | low | medium | high | xhigh |
Verdict |
|---|
| Trace real control flow across files |
1.00 | 1.00 | 1.00 | 1.00 | 0.92 | 1.00 |
gap is run-to-run noise |
| Find a real concurrency defect |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Change-impact analysis |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.92 |
gap is run-to-run noise |
| Refactor real code safely |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Design against real constraints |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.89 |
gap is run-to-run noise |
| Derive exact runtime behaviour |
1.00 | 0.89 | 0.78 | 0.78 | 0.89 | 1.00 |
gap is run-to-run noise |
| Debug a real symptom |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Write tests for real code |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.67 |
gap is run-to-run noise |
| Security review of real code |
1.00 | 1.00 | 1.00 | 1.00 | 0.92 | 1.00 |
gap is run-to-run noise |
| Architectural summary of real code |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Level (this suite) | Mean | Perfect cells |
Avg latency | Avg reason tok |
|---|
| none |
1.000 |
10/10 |
3.0s |
0 |
| minimal |
0.989 |
9/10 |
16.6s |
3122 |
| low |
0.978 |
9/10 |
13.2s |
2324 |
| medium |
0.978 |
9/10 |
24.5s |
5165 |
| high |
0.972 |
7/10 |
21.7s |
4417 |
| xhigh |
0.947 |
7/10 |
24.1s |
4963 |
Ceiling suite (results only computable by working it out)
Five tasks whose answers are large exact integers or exact fractions, computed in code before the run — impossible to guess or pattern-match. Built specifically to break minimal and test whether any higher tier separates. · 30 cells ·
90 calls · $0.57598
Built to break minimal — answers are huge exact integers/fractions that must be worked out. none did break (0.800). Every thinking tier, including minimal, scored a perfect 5/5 with zero variance.
| Scenario | none | minimal | low | medium | high | xhigh |
Verdict |
|---|
| Exact huge modular exponent |
0.33 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
gap is run-to-run noise |
| Exact combinatorial count of strings |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Exact expected value (rational) |
0.67 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
gap is run-to-run noise |
| Exact path count with obstacles |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Exact large binomial sum |
1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
all levels tie |
| Level (this suite) | Mean | Perfect cells |
Avg latency | Avg reason tok |
|---|
| none |
0.800 |
3/5 |
14.2s |
0 |
| minimal |
1.000 |
5/5 |
31.2s |
9850 |
| low |
1.000 |
5/5 |
30.7s |
9552 |
| medium |
1.000 |
5/5 |
43.1s |
13490 |
| high |
1.000 |
5/5 |
41.1s |
12631 |
| xhigh |
1.000 |
5/5 |
43.9s |
13414 |
"Is the switch even doing anything?" — direct proof
This is the right question to ask before trusting any of the above, because a parameter that is
silently ignored would produce exactly the same "all tiers tie" result as a real-but-useless
setting. Three independent checks prove the parameter is honoured by DeepSeek's API:
- A bogus value is rejected. Sending
reasoning_effort: "bogus_level_xyz"
returns HTTP 400 naming the valid set:
none, minimal, low, medium, high, xhigh, max. The API parses the field — it is
not discarded.
- The thinking channel changes with the tier. On an identical prompt,
none returns reasoning_tokens: null and 0 characters of private
reasoning across every run, while every other tier returns a populated
reasoning_content channel.
- The volume scales, and so does the failure mode. On the modular-exponent task:
minimal ~24K reasoning tokens, medium ~33K, xhigh ~30K —
and high burned the entire 32,768-token budget on thinking and returned
finish_reason=length with zero visible output. No implementation that ignores the
parameter produces that.
Every tier also produced
different output text on the same prompt — six distinct answers,
not one answer repeated six times.
So why doesn't quality improve? Because the thinking is real but the tasks don't need it.
The tiers genuinely think harder; the extra thinking simply doesn't change the answer. On the
ceiling suite the pattern is stark: minimal spent ~24K reasoning tokens and solved
the problem, medium spent ~33K and got the same right answer. Both are correct, so
37% more thinking bought nothing measurable. This is a finding about the tasks and the
model's competence, not evidence of a broken switch.
Method & limits
How this was measured. Same prompt text at all six levels,
temperature=0,
3 passes per cell, deterministic graders (exact numeric match, JSON schema validation,
executed unit tests, brute-forced constraint counts, clause-level regex). Ground truth
was verified in code before any API call: the scheduling answer was brute-forced (13), the LRU
bugs were proved by running the code, and repo answers were derived by reading the real source.
- Any level with thinking enabled scored 1.000 on all 10 easy scenarios, all 3 passes.
Only
none ever failed there (0.992 mean, worst pass 0.917) — and it failed a
professionalism/scope judgment call.
- Harder tests created real headroom but did not favour higher tiers. The constraint
puzzle is the clearest case:
none got it right 1 of 3 times, while every thinking
level got it right 3 of 3 — including minimal. Turning thinking on fixed
it; adding more did nothing.
- On your real repo code, more thinking was mildly harmful.
none scored
10/10 perfect at 3.0s average. Every thinking level scored lower and took 4-8× longer.
The one apparent exception (writing tests) was traced to a bug in my grader — not the model —
and disappeared when the grader was fixed. See below.
- Latency is what higher effort actually costs. Quality was flat; wall-clock rose
steeply.
Correction I made mid-study — reported because it matters.
The first run showed repo_write_test climbing 0.10 → 1.00 from
none to high, which looked like a genuine "code-writing needs
reasoning" finding. It was my grader's bug: many valid test files begin with
import pytest, the grader's execution sandbox had no pytest, so
exec() raised ModuleNotFoundError and the suite scored 0.1
regardless of quality. After adding framework shims and re-grading, every level scored
1.00. Had I published the first result, it would have been confidently wrong. All numbers on
this page are post-fix.
What this does not prove. It does not prove the tiers are equivalent in general, and it
is not evidence the switch is inert — the proof section above shows it is fully honoured. What
it proves is narrower and more useful: on 35 scenarios spanning everyday business work,
adversarial reasoning traps, your own real codebase, and hard exact-computation tasks,
minimal matched or beat every higher tier. Untested here: genuinely long
agentic runs (dozens of dependent tool calls), tasks where the model must maintain state over
hours, and anything where the accumulated thinking across many turns matters more than
any single response. Those remain the plausible places a higher tier could still pay off.