Research question
CoScientist is an experimental multi-agent hypothesis-generation system. I want to understand whether collaboration and critique improve scientific reasoning enough to justify their additional computation.
I began with an implementation inspired by Google’s AI co-scientist paper: generate hypotheses, critique assumptions, compare explanations, and revise selected hypotheses. Drug repurposing, including Duchenne muscular dystrophy, motivates the work.
Implementation
The current local implementation has six specialist roles: generation, reflection, ranking, proximity, evolution, and meta-review. A supervisor adapts the task mix; an Elo tournament prioritizes candidates. SQLite records preserve hypotheses, reviews, comparisons, and the evidence consulted.
An experimental company layer runs research teams through disease-dossier, hypothesis, mechanism, and translational stages, with human gate decisions and shared resource budgets. The company structure is an orchestration experiment; it does not establish that its outputs are sound scientific programs.
Recent local work connects generation and claim review to PubMed. Checked claims retain their sources and rationale. An unavailable search remains distinguishable from a completed search with no matching record. Contradicted claims can quarantine a hypothesis for review. These checks are model judgments whose error rates still need measurement.
GPQA-bio evaluation
The early evaluation did not establish that Elo rankings predict correctness.
| Historical GPQA-bio summary | Reported value |
|---|---|
| Graded responses | 41 |
| Questions | 6 |
| Elo buckets used in the correlation | 5 |
| Spearman correlation | −0.56 |
| p-value | 0.32 |
The correlation uses five bucket-level observations. The reported correlation was negative, and the system did not outperform the difficulty-corrected base-sampling comparison. The small, inconclusive result neither validates this tournament nor establishes that tournaments generally fail.
The full historical response-level bundle is absent from the committed evidence. This is a reported summary, not a result readers can currently recompute from a released artifact. Read the analysis and next experiment.
Models and smaller-model experiments
Provider adapters cover Claude, Codex, and OpenAI-compatible endpoints, including hosted smaller models. Each run currently shares one provider with fast and strong model tiers. Separate runs can compare provider/model configurations; arbitrary mixtures of providers across individual agents remain future work.
The next smaller-model step is an operational pilot against a remote endpoint: check output contracts, retrieval, logging, and resource accounting before attempting a capability comparison. No new hosted-SLM result is claimed here.
Evidence and limits
Current source is private. The capability description reflects local committed revision 66dc159 and subsequent restart work reviewed in September 2026. Some local changes are ahead of remote main. The earlier website described source revision d7ee1a6; its public-release wording no longer reflects repository visibility.
Deterministic offline checks exercise orchestration, persistence, budgets, and failure handling. Their synthetic answers provide no measurement of scientific performance. Evidence verification still needs calibration, and no validated DMD candidate is claimed.
My contribution
I set the research direction, architecture, and evaluation question. Coding agents assisted with implementation, tests, and documentation. I retain responsibility for reviewing the work and its scientific interpretation.
Next experiment
Compare tournament ranking, independent rubric scoring, and base-model sampling under matched conditions. First preserve a fixed candidate set and complete the harness work needed to score it across selection procedures. Freeze the questions, retrieval conditions, budgets, and analysis plan before the main comparison; report question-level uncertainty and failures alongside scores.