Benchmark results
Terminal-Bench 2.1
Canopy solved 79.3% of Terminal-Bench 2.1 on DeepSeek-V4.1-Flash, averaged over low and high reasoning effort. That is level with the leading harnesses on the public board for this model; the difference is inside rerun noise.
79.3%
Mean of low and high effort
Complete. Runs 10–12 Nov 2026.
Results
| Effort | Score | 95% interval | Trials passed | Tasks passed 3 of 3 | Tasks passed at least once | Run (UTC) |
|---|---|---|---|---|---|---|
| Low | 77.4% | 67.6–84.9 | 202 of 261 | 54 of 87 | 78 of 87 | 10 Nov 19:25 – 11 Nov 20:58 |
| High | 81.2% | 71.8–88.0 | 212 of 261 | 61 of 87 | 78 of 87 | 11 Nov 21:02 – 12 Nov 12:49 |
| Mean | 79.3% |
Setup
- Benchmark
- Terminal-Bench 2.1, 89 tasks, 87 scored (the board excludes qemu-startup and qemu-alpine-ssh)
- Model
- DeepSeek-V4.1-Flash, served by Azure AI Foundry
- Attempts
- 3 per task. Concurrency: 3. Runner: Harbor 0.25.0
- Memory
- Off for every trial, so no trial could recall an earlier one
- Harness version
- Not pinned; production deploys continued during both runs
Comparison with the public board
- Canopyour run79.3%
- Kimi Code77.8%
- OpenCode75.3%
- Codex75.1%
- mini-swe-agent74.3%
- Claude Code74.0%
- DeepSeek Harness73.4%
- Pi73.0%
- Qwen Code72.6%
- Hermes Agent65.9%
- OpenClaw56.5%
Where trials failed
| Reason | Trials |
|---|---|
| Wrong answer after the agent reported done | 25 |
| Ran out of time | 13 |
| Ran out of time with a server restart during the trial | 6 |
| Lost to a crash loop in our background workerCanopy fault | 4 |
| Our CLI exited at a server restartCanopy fault | 1 |
| Reason | Trials |
|---|---|
| Wrong answer after the agent reported done | 23 |
| Ran out of time | 14 |
| Ran out of time with a server restart during the trial | 19 |
| Our step limit ended the turnCanopy fault | 2 |
| Our CLI exited at a server restartCanopy fault | 1 |
The last two rows in each table are faults in Canopy, not the model. They count as failures in the scores above and have since been fixed.
Tasks never solved in three attempts (high effort)
- extract-moves-from-video
- filter-js-from-html
- gpt2-codegolf
- make-doom-for-mips
- model-extraction-relu-logits
- mteb-retrieve
- pytorch-model-cli
- query-optimize
- train-fasttext
Read this before comparing
Limits of this comparison
- Our route to the model (Azure) differs from the board’s providers.
- On our route, low effort was not shown to reason less than high effort.
- The board reports that re-running the same harness flips about 21% of Terminal-Bench tasks, so differences of a few points are noise.
- Per-trial transcripts are not published yet.
- Cost: $33.59 recorded for the low run over the 221 trials that reported one; not recorded for the high run.
