Question
Does the tournament’s internal ranking predict correctness well enough to justify its additional computation?
This entry reviews the historical Elo–accuracy analysis. It does not report a new benchmark run.
Setup and provenance
The historical GPQA-bio run reported 41 graded responses across six questions. The concordance analysis grouped responses into five Elo buckets, then correlated bucket midpoints with accuracy within each bucket.
The historical account used source revision d7ee1a6. The local analysis code inspected for this entry is in bench/concordance.py at committed revision 66dc159. Source is now private, and the complete historical response-level bundle is absent from the committed evidence. I cannot present a fresh statistical interpretation as a reproduced result.
Observations
| Observation | Interpretation boundary |
|---|---|
| Spearman ρ = −0.56, p = 0.32 | Computed on five aggregated buckets, not 41 independent observations |
| No improvement over difficulty-corrected base sampling | No demonstrated benefit in this small evaluation |
| Six distinct questions | Too little independent question coverage for a broad capability claim |
The negative correlation does not establish that tournaments generally fail. This result also does not validate the current tournament.
Statistical limitations
The code also computes a response-level logistic regression of correctness on Elo. That preserves information discarded by bucket averages, but responses to the same question share a problem and a search history. Forty-one responses are not forty-one independent scientific questions.
The existing baseline-spread calculation resamples whole questions. A revised primary analysis should also account for dependence within a question and include enough independent questions to estimate uncertainty usefully.
Implementation updates
The local implementation now retrieves PubMed evidence during generation and checks factual claims during reflection. It records consulted sources and distinguishes unavailable retrieval from a completed search with no matching record. It can also identify citations never supplied to the generating model.
These changes make errors inspectable. They do not establish verifier calibration. A mistaken contradiction can remove a useful hypothesis from the active pool, so quarantined hypotheses remain recoverable. Evaluation reports need to include quarantine counts; reporting only active hypotheses would omit excluded candidates.
Next experiment
- Run a small operational pilot to check each selected model’s output contracts, evidence retrieval, and logging. A hosted smaller model can enter through the compatible-endpoint adapter. This pilot tests readiness, not relative scientific capability.
- Freeze the evaluation questions, retrieval conditions, candidate-generation budget, and analysis plan. Choose the question count through power analysis before the main run.
- Compare tournament Elo, direct rubric scoring by a distinct judge, and base-model sampling. For a focused selection test, score a shared, fixed candidate set. Study the full adaptive tournament separately, because rankings also affect which hypotheses are developed.
- Preserve inputs, candidates, judgments, model identities, resource use, failures, quarantines, and analysis outputs. Report uncertainty at the question level.
There is implementation work before this comparison is executable. Absolute scoring currently belongs to the rediscovery benchmark; GPQA concordance requires Elo. Ranking mode participates in the run-cache identity. The current harness therefore does not provide all three procedures over one frozen candidate set merely by changing a flag.
Sources and responsibility
This account uses the historical project summary, the local bench/concordance.py, bench/cli.py, bench/runner.py, tools/verify.py, and docs/ROADMAP.md at the revision above. These are author-held sources, not a released reproducibility package.
I define the evaluation and review its interpretation. Codex assisted with this source audit and draft.