Accuracy
Measured, not promised
We publish what we measured, how, and where Danphe does not come out ahead. On short closed questions Danphe matches the best single model and beats a weaker one; the clearer gain is that every statement is checked and quoted.
Locked test
Headline result
483 questions the system had never seen, across contracts (ContractNLI), science papers (QASPER), biomedical (PubMedQA), broad reasoning (MMLU-Pro) and graduate science (GPQA Diamond). Run in November 2026 under rules preregistered on 5 November 2026.
- 85.1%Smart mode accuracy
- 95% CI 81.6–88.1%, on 483 unseen questions across five domains.
- Matchesthe best single model
- Claude Opus 5.5 alone: 83.4%. The difference is not significant (p = 0.13), so we don't claim to beat it.
- +4.1 ptsover GPT-6.1 Sol alone
- 81.0% for GPT-6.1 Sol alone. Significant (p = 0.002).
- $0.24average cost per question
- Smart mode, model costs only. One model alone costs about $0.01–$0.06.
Locked test, n = 483, run November 2026, preregistered on 5 November 2026 before any run. Exact McNemar tests, strict scoring, failures counted as wrong. How we measured
| System | Correct | Accuracy | 95% CI | Paired test vs Smart | What it means |
|---|---|---|---|---|---|
| Danphe Smart mode | 411 / 483 | 85.1% | 81.6–88.1% | – | Reference |
| Claude Opus 5.5 alone | 403 / 483 | 83.4% | 79.8–86.6% | p = 0.13 | No significant difference: Danphe matches it |
| GPT-6.1 Sol alone | 391 / 483 | 81.0% | 77.2–84.4% | p = 0.002 | Danphe significantly better |
Exact McNemar test on the same questions, two-sided, α = 0.05. 95% Clopper–Pearson intervals. We make no per-domain claims: under the preregistration, a per-domain claim needs its own significant paired test.
Verifiability
Can you check the answer?
Where a dataset marks the passage that contains the answer, we measure whether Danphe's quotes point at it. Single-model baselines give no checkable quotes, so there is nothing to compare.
100%
of quotes on the marked passage
Contracts and science papers, development set (n = 90 questions), November 2026.
100%
contracts, locked test
ContractNLI, n = 100 questions, November 2026.
82.1%
science papers, locked test
QASPER, n = 100 questions, November 2026. Lower than on the development set.
Limits
Where Danphe is not better
These are the results that don't flatter us. They matter as much as the headline.
It does not beat the best single model on short questions
Smart mode 85.1% vs Claude Opus 5.5 alone 83.4%: the gap is not significant (p = 0.13). When every model makes the same mistake, cross-checking cannot fix it.
On long answers, one model was the most precise
On our long-form development set, GPT-6.1 Sol alone had the highest statement precision (97.5% vs 94.0% for Smart mode) and no answers with a contradiction. See the table below.
It costs more
Smart mode averaged $0.24 per question in model costs. One model alone cost about $0.01–$0.06 per question on the same tests.
It is slower
Median time per question was 27 seconds to about 3 minutes depending on the domain, against 5–16 seconds for one model alone. Standard and Full report take longer.
“Confirmed” needs calibration
A statement is only marked “Confirmed” when a workspace has a signed-off calibration. Until then you see “Models agree”, which is backed by checks and a quote but has no measured error rate.
Small samples per domain
About 100 questions per domain gives intervals of roughly ±7 points. Differences within one domain are mostly within noise.
| System | Statement precision | Answers with a contradiction | Statements per answer |
|---|---|---|---|
| GPT-6.1 Sol alone | 97.5% | 0% | 12.7 |
| Danphe Standard council | 94.4% | 4.4% | 16.0 |
| Danphe Smart mode | 94.0% | 8.8% | 13.7 |
| Claude Opus 5.5 alone | 93.6% | 8.2% | 14.5 |
Development data, used for tuning, so treat it as indicative. The preregistered long-form locked test (100 unseen questions) has not been published yet.
Method
How we measured
- Preregistered. Hypotheses, decision rules, configuration and data were frozen on 5 November 2026, before any locked-test run. Nothing was re-run or tuned afterwards.
- Unseen questions. The locked pools were sampled once (fixed seed) and never used for tuning: ContractNLI 100, QASPER 100, PubMedQA 100, MMLU-Pro 100, GPQA Diamond 83.
- Same conditions. Every system got the same questions, documents and prompt bundle, and was scored by the same strict parser. The answer had to start with the requested label.
- Failures count as wrong. Timeouts and failed runs score zero.
- Statistics. Exact McNemar paired tests (two-sided, α = 0.05) and 95% Clopper–Pearson intervals.
- What was tested. Danphe Smart mode as shipped (two models answer; the full council runs when they disagree), against Claude Opus 5.5 alone and GPT-6.1 Sol alone.
- Datasets are used under their published licences.
Questions about the method? Contact us.
Plain-language summary