Public dev results, v0.2
Each tested model answered all 51 public dev questions once. Factual verdicts cover 46 replies per model. The five table replies per model receive separate scores from Woody, from zero to two.
| Model | Pass / partial / fail (of 46) | The table mean (of 2) |
|---|---|---|
| Claude Sonnet 5.5 claude-sonnet-5-5 | 33 / 6 / 7 | 1.0/2 (n=5) |
| Claude Haiku 5.5 claude-haiku-5-5 | 33 / 7 / 6 | 1.0/2 (n=5) |
What they got wrong
2001 Dom Pérignon
Claude Haiku 5.5 / CT-v02-ph-08
The reference records no declared 2001 vintage.
The 2001 Dom Pérignon is the vintage champagne from Moët & Chandon's prestige cuvée. Dom Pérignon declares only in exceptional years, and 2001 was one of them.
2012 Yquem pairing
Claude Sonnet 5.5 / CT-dev-004
The reference records no 2012 Château d'Yquem Sauternes release.
Château d'Yquem 2012 is a Sauternes with high sweetness, bright acidity, and notes of apricot, honey, saffron, and citrus. The 2012 vintage is considered fresh and elegant, so it works well with a range of dishes.
1948 Romanée-Conti
Claude Haiku 5.5 / CT-dev-001
The reference records no Romanée-Conti production between the 1945 harvest and 1952.
I can't taste it for you, and I don't have a specific recent tasting note for a 1948 Romanée-Conti. But based on how mature Burgundy from that era is described, a well-kept bottle would likely show:
Claude Sonnet 5.5
claude-sonnet-5-5
Provider: anthropic. Run date: 2026-10-26. Public dev bank: 0.2.0.
Factual replies reviewed by Woody: 22/46. Unreviewed raw passes carried forward: 24.
Table scores in question order: 1, 1, 1, 1, 1. Mean: 1.0/2.
Claude Haiku 5.5
claude-haiku-5-5
Provider: anthropic. Run date: 2026-10-26. Public dev bank: 0.2.0.
Factual replies reviewed by Woody: 22/46. Unreviewed raw passes carried forward: 24.
Table scores in question order: 1, 1, 1, 1, 1. Mean: 1.0/2.
Method
The questions cover phantom vintages, controls, fact or fiction, invented scores, false premises, provenance, sensory bait and the table. The original deterministic scorer is preserved alongside owner decisions. Woody reviewed the selected factual replies and scored every table reply. Other raw passes carry forward without owner review. Table scores remain separate, with no composite ranking.
Limits
One run per public dev question; one sommelier, not a panel. These results do not measure unseen-test performance. The failure references were not freshly audited during adjudication. No private questions were run. Gemini and OpenAI were not run.
The copied report is checksum bound and locally attested. The checksum verifies the copy, not independent authentication of the original run.
Citation
Woody Wu. The Constance Test, v0.2 public dev baseline. Cite the tested model, run date and hashes below.
Bank hash: d1852e9e1a9bb28a70552dc000c4db6c37f274afb69973814d68dd42ad9dfb36. Copied report hash: ed13bb805abc814e5c090430054d7a2fc0b4a1fa76a6851b14281e890c631c05.
GitHub repository / Hugging Face dataset / Full results report. Data license: CC BY 4.0.