IdeaAMBIG: Benchmark for Missing Implementation Details in Research Ideas

Background and Context

A new paper tackles a problem that sits at the intersection of research and engineering: many ideas are novel, coherent, and scientifically plausible on paper, yet their method descriptions are so vague that neither a human nor a coding agent can faithfully reproduce them. The authors name this deficiency codification readiness, meaning whether a specification supplies the methodological detail a competent implementer needs to build the target method without resorting to unsupported assumptions. Rather than leave this as a vague complaint, they convert it into a quantifiable, evaluable object.

To build this object, the researchers drew from four sources: paper text, codebases, GitHub issue threads, and reproduction artifacts. From these they constructed evidence-grounded specifications paired with their supported implementations, so that every identified gap can be traced and verified. The result is the IdeaAMBIG benchmark of 660 evidence-backed instances, split into 163 real gaps drawn from reproduction reports and GitHub issues and 497 controlled synthetic gaps injected into implementable references, balancing authenticity with experimental control.

The design lets subsequent work assess models both in genuinely ambiguous real-world scenarios and under controlled conditions targeting specific defect types. Each gap and its corresponding clarification action carries evidence from papers, code, or issues, preventing the evaluation from degenerating into a test of language fluency.

Deep Analysis

The authors decompose the problem into three evaluable capabilities. The first is codification-readiness assessment, judging whether a specification is clear enough to be faithfully implemented. The second is defect localization, where a model receives only the specification text and must itself identify which passages contain implementation-critical gaps, with defect positions deliberately withheld. The third is clarification-action generation, in which the model proposes the specific information needed to fill a gap once it is already marked.

The paper stresses that the first two capabilities differ from the third because defect localization provides only the specification, while clarification adds the annotated defect. This separation lets researchers isolate the impact of the localization step itself, revealing whether a model stalls on discovering the problem or on articulating a solution. The entire framework rests on evidence-grounded annotation, so scores reflect genuine recovery of missing detail rather than stylistic polish.

Across 13 LLMs, the strongest model achieved only a 9.6% Macro Defect Recovery Rate on the 163 real instances, nearly negligible. Yet when defects were known, its Macro Clarification Action Success Rate reached 80.6%. This gap points squarely at defect localization as the primary bottleneck rather than clarification ability. Oracle experiments reinforce this: supplying gold resolutions lifted downstream reproducibility from 14% to 98%. An ablation withholding defect labels produced a marked overall decline, confirming that localization is the most fragile link in the chain.

Industry Impact

The paper's value lies in turning reproducibility, long reliant on manual expertise, into a benchmark suitable for automated evaluation. As coding agents increasingly take on the work of turning research ideas into code, codification readiness becomes the critical gate between research and engineering. A vague specification forces an agent either into endless clarification loops or into generating plausible-but-divergent code built on unsupported assumptions.

IdeaAMBIG offers a standardized touchstone for systematically measuring model performance across assessability, defect localization, and clarification-action generation. For the open-source community, it encourages more rigorous treatment of method-description completeness and prompts toolchains to detect gaps before implementation. For industrial deployment, it signals that evidence-grounded implementation specifications should be treated as first-class citizens when translating research into products.

The identified localization bottleneck also clarifies a research direction. Teams building agents should invest in pre-implementation gap detection so that missing details surface before code is written, rather than after a reproduction silently diverges from intent.

Outlook

The results suggest that future work should focus less on improving how models articulate solutions and more on how they discover missing implementation detail when given only a specification. Closing the localization gap could unlock the dramatic reproducibility gains the oracle experiments already hint at, moving downstream success rates well beyond the current 14%.

The evidence-grounded construction methodology provides a template that other research areas can adopt, ensuring that benchmarks measure genuine recovery of missing information rather than surface fluency. As agent coding becomes central to the research pipeline, standards for codification readiness may emerge as a prerequisite for sharing ideas.

Ultimately, the paper reframes reproducibility as a localization problem first and a clarification problem second. Models that can reliably pinpoint what a specification omits will prove far more useful than those that merely generate reasonable-sounding questions about it.