ReproBAIT: Can Coding Agents Rebuild Bio AI Tools?

Background and Context

Large language models can already draw on scientific knowledge and existing software to run complex workflows. A narrower question has received almost no data: can an agent rebuild a tool when it gets only a limited description of it? The ReproBAIT benchmark, short for Reproducing Biological AI Tools, was built to answer that. It measures how well coding agents can design, implement and train biological AI tools taken from published research.

This is a different task from ordinary code-generation tests. Writing a program that runs is not enough. The agent must also train the model to a usable level inside a fixed time and compute budget. The paper makes three contributions. It offers a framework for comparing agents under different levels of access to information. It selects real tools of varying complexity. It also adds a memorization check, because open tools may already sit in a model's pretraining corpus. Without that check, a high score could simply be recitation.

Deep Analysis

The setup is strict. Each agent works in a sandbox with no internet access, so it cannot download the original code or weights. It receives training data, a fixed budget of time and compute, and limited information about the target tool. The amount of information can be adjusted, which lets the authors see how performance changes as access changes. The agent must choose the network design, write the data-processing and training code, and finish training within budget. Scoring ignores how closely the code resembles the original. Instead, the rebuilt tool is evaluated on held-out data using the metrics that belong to that specific tool. Three tools were tested. ProteinMPNN designs sequences for a given protein structure, a task known as inverse folding. RFdiffusion3 is an all-atom generative diffusion tool, and its structure and training are more complex. The third is a closed-source molecular design tool. It is very unlikely to appear in any training corpus, so it works as a control that helps separate real reproduction skill from memory.

The results come in layers. Several leading closed and open agents reproduced some of the tools to a notable degree. That shows they can complete the loop from design to trained model on limited information. For the highly complex tool, reproduction was limited. Larger systems still seem to defeat current agents on design, implementation or the training budget. On memorization, the authors found moderate evidence that is consistent with it, and the amount varies a lot between agents. The wording matters. The evidence is consistent with memorization. It does not prove it. Part of the success on open tools may come from material seen in pretraining, but not to the same degree for every agent. The abstract gives no specific scores, so this article makes no numeric comparisons.

Industry Impact

The first effect is on intellectual property and safety governance. The authors say tool developers should be more careful about what they publish. If an agent can rebuild a tool from public descriptions, the tool's commercial value may shrink. A developer who spent years on a model could see it reproduced by an agent working from a paper and a training set.

The second effect touches dual-use risk. Some tools carry that risk, and their developers often manage it through controlled access. If others can reproduce a tool from public information, that policy may stop working. Restricting the weights would then protect little if the paper itself gives enough detail to rebuild them.

For the open-source community, ReproBAIT offers a reusable way to track what agents can do when they develop biological AI tools on their own. For companies, the paper suggests treating the level of detail in papers, technical reports and documentation as a risk decision that needs review. The closed-source control tool also gives a template: testing against a tool the model cannot have seen is a practical way to check a reproduction claim.

Outlook

Two open problems follow directly from the paper. The first is how to tell reproduction from memorization more reliably. The current check yields moderate evidence that differs by agent, which leaves room for a sharper method. The second is coverage. The benchmark tests a small set of tools, and the one highly complex tool already exceeded what the agents could do. Adding more and harder tools would show where the limit sits.

Because the information given to the agent can be adjusted, the framework can be rerun as new agents appear. A rising score on the complex tool would be an early signal that disclosure policies need to change. Until then, the practical message for developers is plain: decide how much to publish before release, not after an agent has rebuilt the tool.