Executive summary
In ScholarCatalyst, a GPT-4.1 search agent scored 43% Recall@20 on 687 field-specific questions; Qwen3-Emb-8B scored 51%. On 207 broader questions, both scored 37%. Recall@20 measures recovery of author-labelled useful papers, not correct answers.
Context: Preprint; uneven computer-science coverage and retrospective author labels. GPT-4.1 was chosen for its older training cutoff. A newer Fable 5.1 comparison did better but may have seen the completed papers. This tests finding useful references, not doing science.
Source published 2026-10-01 · Selected claims checked 2026-10-03
October 1 preprint; benchmark of 2025–26 research projects, not a new security incident
Published 2026-10-03T15:02:38.801885Z
What happened
The October 1 preprint built its labels from 184 researchers describing 207 computer-science projects. Each question asks for prior papers that could help a project, with retrieval restricted to literature preceding that project.
Table 3 reports the two question types separately. The tool-calling agent uses GPT-4.1; the direct embedding baseline uses Qwen3-Emb-8B. A lower field-specific score and a tied broad-question score are the receipt.
These results concern literature retrieval in a curated corpus, not autonomous discovery, completed research or a universal comparison of current agents. Better candidate coverage remains an important part of the authors’ proposed explanation.
The 2050 museum label
The research assistant needed a research assistant.
What this does—and doesn’t—show
Preprint; uneven computer-science coverage and retrospective author labels. GPT-4.1 was chosen for its older training cutoff. A newer Fable 5.1 comparison did better but may have seen the completed papers. This tests finding useful references, not doing science.
