Why this lesson exists
Capability maps prevent two opposite errors: using the model for work it is bad at, and failing to use it for work it is extraordinary at. This lesson gives you both edges, honestly, including the failure modes you have to engineer around.
How to read a benchmark table
DeepSeek published a substantial benchmark set with V4.1-Flash. Before looking at any number, learn to ask four questions of each row. This habit is worth more than the numbers themselves.
- What is being measured? A benchmark name is not a task. "Terminal-Bench" measures sustained autonomous work inside a shell; "GPQA Diamond" measures graduate-level scientific reasoning in a multiple-choice format. One transfers to your work; the other may not.
- With tools, or without? Models score dramatically higher when allowed to run code, search, or use a harness. V4.1-Flash reports both. The gap between them is the value of the harness, not the model.
- Which subset? A footnote can quietly change the meaning of a headline. The HLE figure is footnoted as tested on the pure-text subset only — which is a legitimate scope limitation and also a material difference.
- What is the ceiling and the spread? 90% on one benchmark and 30% on another are not contradictory. They measure different things with different headroom. A low number on a brutally hard benchmark can be a better sign than a high number on a saturated one.
The published numbers, translated
Here is the V4.1-Flash set with the translation a practitioner needs. Rows are grouped by what they actually predict about your work.
| Benchmark | Score | What it predicts for you |
|---|---|---|
| GPQA Diamond | 90.9 | Strong graduate-level scientific reasoning. It will not embarrass you on hard technical questions. |
| Codeforces (rating) | 3471 | Competitive-programming grade algorithmic ability. Far beyond what typical application code demands. |
| MathArena Apex | 65.6 | Hard competition mathematics. Solid, not perfect — expect occasional genuine errors on novel proofs. |
| HLE (text-only subset) | 36.8 / 39.1 | A deliberately near-impossible knowledge benchmark. The absolute number looks low; the difficulty is the point. |
| Terminal-Bench 2.1 | 90.6 | Sustained shell work. The most directly relevant number if you run coding agents. |
| Terminal-Bench 3.0 / 4.0 | 30.0 / 31.2 | The harder generations of the same test. This is the honest frontier: long autonomous task completion is still roughly a one-in-three proposition. |
| DeepSWE v1.1 | 74.2 | Real software-engineering issue resolution. Roughly three-quarters — good, and precisely why Lesson 25 exists. |
| NL2Repo-Bench | 65.4 | Natural language to a working repository. Two-thirds success on cold repository generation. |
| ProgramBench | 20.3 | A hard, low-saturation coding benchmark. Do not panic at a low number; note that the ceiling is low for everyone. |
| CyberGym | 88.1 | Security-relevant reasoning. Capable at finding issues — treat it as an auditor, not an authority. |
| SEC-Bench Pro / ExploitGym | 62.8 / 15.3 | Security depth drops sharply as tasks get more adversarial. Keep a human in that loop. |
| HLE with tools | 63.9 | The tools effect. Same benchmark, same model: 36.8 without tools, 63.9 with. Build the harness. |
| Automation-Bench | 54.8 | Realistic workflow automation. About half — plan for checks. |
| Agents' Last Exam | 31.8 | Long-horizon agentic tasks. Still hard. Scope your agents tightly. |
| Chartography (w/tools) | 78.9 | Visual reasoning over charts. Genuinely useful for data and reporting work. |
| BabyVision (w/tools) | 89.6 | Strong visual perception. Screenshot-driven UI work is viable. |
| ZeroBench-main (w/tools) | 49.0 | Extreme visual reasoning. Roughly a coin flip — verify anything load-bearing. |
The single most useful row in that table: HLE scoring 36.8 without tools and 63.9 with them. Same model. The difference is entirely harness design. This is why Module 7 exists and why the second half of this course is about process rather than prompts.
Benchmark-as-marketing trap
Vendor-published benchmarks are chosen by the vendor. They are not fraudulent — DeepSeek's set is unusually detailed, includes hard benchmarks where the scores are low, and footnotes its scope limitations. But they are still a curated selection. The exercise below exists because the only benchmark that governs your work is the one built from your own tasks.
Where it is genuinely excellent
- Sustained tool-using work. The Terminal-Bench 2.1 score plus a 2,500 concurrency ceiling plus 1M context is a combination built for running agents continuously. This is the model's home turf.
- Code generation and debugging. Strong algorithmic ability, and — more useful day to day — the willingness to actually read an error message and reason about it rather than pattern-match a plausible-sounding fix.
- Long-document reasoning. A million tokens of usable context is rare, and it means "read all of this and tell me" is a legitimate request rather than a trap.
- Visual inspection. Native vision with strong perception scores makes screenshot review, layout checking and data extraction from images practical.
- Cost per unit of output. With cache hits and off-peak pricing, the economics are unlike anything in the closed-model tier. Lesson 6 quantifies this.
Where it is merely good — or fails
| Failure mode | What it looks like | The guardrail |
|---|---|---|
| Confident completion of missing pieces | You ask for a function that uses an API you did not specify. It invents one that looks plausible and does not exist. | Never ask for code against an interface you have not supplied. Paste the real signature, schema, or doc. |
| Long-horizon drift | Ten steps into an autonomous run, the model is still working — on a slightly different problem than the one you asked for. | Short slices with explicit done-signals. Re-state the goal at each handoff. Lesson 22. |
| Overthinking simple work | Thinking mode burns seconds and tokens on classification or formatting, sometimes producing an over-elaborated answer. | Turn thinking off for extraction, formatting and classification. Lesson 9. |
| Self-congratulatory verification | It reports "all checks pass" for an artifact it never actually exercised, or tested with a check that cannot fail. | Verify independently of the model's summary. Acceptance tests you wrote. Lesson 25. |
| Refactoring beyond instructions | Told to change one function, it rewrites three and quietly alters behaviour you relied on. | Diff every change against the instruction. Name what must stay verbatim. Lesson 20. |
| Adversarial security depth | Good at finding issues; sharper at missing subtle ones under active adversarial conditions. | Treat security output as a first pass with a human reviewer, never as an authority. |
The ten-second triage
You will use this constantly. Given a task, answer four questions:
- Is it verifiable? Can you check the output mechanically? If yes, DeepSeek is a strong fit — you get the speed and you keep control of correctness.
- Is it tool-shaped? Does it involve reading files, running commands, calling APIs, or seeing images? If yes, this is its home turf.
- Is the interface fully specified? If the model would have to invent an API, a schema, or a library's signature, stop and supply it first.
- Would a human expert be embarrassed to be wrong here? If a confident error would be costly and hard to notice, build the verification step before you build the feature.
Checkpoint
- Why do HLE's two numbers differ so dramatically?
- What does a Terminal-Bench 3.0 score of roughly 31 actually tell you about running long autonomous tasks?
- Name the guardrail for "refactoring beyond instructions."
- In the ten-second triage, which question is a hard stop rather than a score?
Answers
- The higher figure includes tool use. The lower is the text-only subset. The gap measures harness value, not model capability.
- Long autonomous task completion is still roughly a one-in-three proposition — so scope runs tightly and verify with your own checks.
- Diff every change against the instruction and state explicitly what must stay untouched.
- Question three — an unspecified interface. If the model has to invent it, fix that before prompting.
Exercise: build your own benchmark
Select five real tasks from your own work. They should differ in shape — one extraction, one code change, one writing task, one analysis, one visual task. Write them down with a checkable success criterion for each.
Run all five against deepseek-flash. Record, per task: did it pass your criterion on the first attempt, how many tokens, how long, and what you had to correct.
Keep the results. When a new model generation ships, re-run the same five and you will know something about your own workload that no published table can tell you. Go back and re-read the paragraph you wrote in Lesson 1 — notice which parts you would now rewrite.
Next
Module 1 is complete. Module 2 gets your hands on the API: keys, a first call, and the cost model that makes everything in this course cheap to practise.