Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies
2608.16645

Authors

Qingqing Mao,Ritankar Das,Shaolong Chen,Xinmiao Yu,Lei Li

Abstract

Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea.

Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%).

We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx.

23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline.

This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.

Resources

Ray graphicRay graphicRay graphicRay graphic

Stay in the loop

Every AI paper that matters, free in your inbox daily.

Details

  • takara.ai
  • Custom AI and machine learning from the Frontier Research Team.
  • © 2026 takara.ai Ltd
  • Content is sourced from third-party publications.
Ray graphicRay graphicRay graphic