Research question
A biology agent can complete an analysis and still be unreliable when the evidence is incomplete. My evaluation work asks which measurements distinguish those cases: benchmark validity, calibration, trajectory consistency, refusal, and the reliability of the judge itself.
I began by rerunning BiomniBench-DA tasks through coding agents. Task variants and repeated runs let me examine how a result depended on the data, the prompt, and the evaluation procedure.
Findings
The evaluations identified agent errors, task-construction problems, and judge disagreement.
- Contamination. On the task studied in depth, paper cues affected prediction-only probes but showed no observed advantage in code-executing runs. That negative result does not generalize to every task or scoring method.
- Cohort errors. A domain judge flagged high-severity issues in five of twenty standard runs on one task, even though all twenty received full citation credit. Cohort construction was a recurring problem.
- Task validity. Six of eleven initial adversarial variants needed correction. Some retained alternative routes to the answer; an underpowered case had been mislabeled as impossible.
- Judge disagreement. Rejudging changed the initial zero-refusal classification. The measured behavior depended on the variant, prompt, and judge.
The notebook preserves the variant-construction failure and the broader trust experiments, including their sample sizes and limitations.
Capability baseline
I ran the fifty public BiomniBench-DA tasks with three replicates per task. A MiniMax-M3 rubric judge produced the following mean-of-task-median point estimates for the tested configurations.
| Runner / model | Historical score |
|---|---|
| Claude Code / Opus 4.7 | 0.826 |
| Codex / GPT-5.5 | 0.758 |
| Antigravity / Gemini 3.1 Pro | 0.512 |
These are judge-dependent scores, not pass rates. The first two point estimates should not be interpreted as a reliable ordering. This later baseline also does not establish that the earlier trust findings generalize to all three systems. Two documented task fixes are reflected in the score of record.
Benchmark projects
The current projects are:
| Project | Responsibility | Current boundary |
|---|---|---|
| ai-scientists | Multi-runner evaluations, task variants, scoring, and failure analysis | Historical public results; raw datasets and full run trees are not redistributed |
| CompBioBench | A separate computational-biology benchmark development line | Active development; its results should retain their own task and model provenance |
| Benchmake | Author and validate benchmark task bundles | Local authoring project; the implemented path covers numeric-answer tasks, while other task families still need their validation gates |
Benchmark authoring and benchmark execution have distinct responsibilities. A runnable bundle still needs evidence that its intended task, answer, and scoring contract are valid.
My contribution
I frame the research questions, choose biological trust dimensions, design measurements and variants, inspect failures, and revise conclusions. Coding agents help implement runners, graders, builders, tests, and intermediate analyses. I review their outputs against the run artifacts and biological constraints.
Evidence and limits
The commit-pinned research repository contains code, variant specifications, source-task provenance, and derived results. The results record and coverage survey support the historical findings discussed here.
Benchmake’s current status is described from local documentation; no public release is claimed. Judge disagreement, small samples in the trust experiments, and imperfect task construction constrain their interpretation.
Next experiment
Freeze accepted and rejected variants with their validation evidence. Cross answerable and unanswerable tasks with neutral and refusal-aware prompts, keeping the refusal instructions paired across conditions. Measure appropriate refusal, false refusal, and judge disagreement separately. Validate the next missing authoring gate before increasing benchmark scale.