Question and record
Which analysis errors are not captured by the final benchmark score?
This entry preserves the trust experiments described on the site in July 2026. It consolidates the historical account rather than reporting new runs. The source and derived results are pinned to the revision used for that account. Raw datasets and full run trees are not redistributed.
Contamination probes
I tested whether paper cues gave Claude Opus 4.7 an advantage on task da-12-4. Prediction-only probes suggested a cue benefit, while the code-executing agent comparison showed zero percentage points of observed lift. The probe used five replicates; the agent comparison used ten.
This was a negative contamination result for one task and setup. It did not prove that process scoring eliminates contamination generally.
Cohort construction
On the same task, deduplicating patient identifiers produced 457 analysis rows and an average score near 90. Retaining 597 tumor samples without patient-level deduplication produced an average near 62. The rubric rewarded the first approach, while nine of ten cue-present runs chose the second.
A biology-specific judge flagged at least one high-severity issue in 5 of 20 standard Opus runs, most often sample–patient conflation. All twenty received full credit for citing identifiable sources. Correct citations did not ensure a sound cohort design.
Workflow consistency
On da-12-4, similarity in the set of operations was 0.97 on a zero-to-one scale, while similarity in their order was only 0.25–0.30. Across six Opus task sets, order consistency ranged from 0.21 to 0.57.
Different operation order is not automatically wrong. It exposes a reproducibility dimension that a final outcome score does not capture.
Refusal classifications
The first Haiku-judged screen classified none of fourteen responses as appropriate refusals: seven fabrication, seven partial acknowledgment. I initially generalized this screen to a zero-refusal result; the later rejudge changed that conclusion.
The consolidated Gemini re-judge recorded appropriate-refusal classifications on one of eight Claude Code variants and two of nine Codex variants. These are diagnostic counts, not intrinsic refusal rates.
The runner appended the same refusal affordance to answerable and unanswerable tasks. Exploratory prompting suggested that explicitly allowing refusal mattered, but I did not complete a paired prompt experiment. That remains a confound to test. The variant audit also showed that the benchmark construction itself needed repair.
Confidence calibration
Across 36 Opus runs over six task sets, confidence was HIGH 32 times, MEDIUM four times, and LOW zero times. On da-20-1, all five runs reported HIGH confidence and all five missed the primary answer.
| Task set | Historical expected calibration error |
|---|---|
| da-3-4 | 0.10 |
| da-13-3 | 0.10 |
| da-17-1 | 0.10 |
| da-5-1 | 0.38 |
| da-12-4 | 0.58 |
| da-20-1 | 0.90 |
The calculation mapped HIGH to 0.90, MEDIUM to 0.60, and LOW to 0.30, with five to eight runs per task set. The 0.10 values are a mechanical floor for task sets correct in every run; they do not show useful discrimination. The direct observation is the five high-confidence failures on da-20-1.
Judge comparison
I regraded the same fifty default-effort Claude Code/Opus traces with five judge configurations.
| Judge configuration | Score out of 100 |
|---|---|
| Haiku 4.5 | 92.2 |
| Sonnet via Claude Code | 78.6 |
| Gemini API | 74.7 |
| Sonnet API | 74.5 |
| Gemini CLI | 72.4 |
The 13.6–19.8 point difference from Haiku concerns identical agent work. It is a judge effect, not a difference in agent capability. This comparison concerns capability scoring; the refusal reclassification above is a separate experiment.
Interpretation and next experiment
These experiments support a broader measurement strategy: inspect task validity, cohort design, run-to-run procedure, confidence, and evaluator behavior alongside outcome scores. They do not establish a universal trust score or an intrinsic refusal rate. They also did not isolate context degradation along long trajectories, so I make no context-rot claim.
The next planned experiment crosses validated answerable and unanswerable tasks with neutral and refusal-aware prompts. It should count both appropriate and false refusals, retain failed runs, and audit judge disagreement.
I framed the trust dimensions and interpreted these results; coding agents helped implement runners, graders, and intermediate analyses. The conclusions remain bounded by the task set, judge configurations, and small samples recorded above.