ReproBridge preprint reports lower API costs for reusable ML artifacts

ReproBridge proposes preserving an ML reproduction attempt as four reusable components: refined code, environment setup, experiment launchers and a reproduction guide. The [November 5 preprint](https://arxiv.org/abs/2611.08452) argues that this handoff should retain the decisions and evidence recovered while getting an experiment to run, instead of leaving them in a conversation or shell history. For a researcher considering reuse, the important distinction is between that executable handoff and the cost of producing it. The paper reports lower spending in later reproductions, but its main API comparison excludes construction, auditing and GPU costs. Below, we turn the paper’s [artifact specification](https://arxiv.org/html/2611.08452v1#S3.SS2) and [cost accounting](https://arxiv.org/html/2611.08452v1#A5) into an inspection record for a prospective reproducer; we have not run the workflow or independently verified its results. Code and data are under internal review; the [project page](https://wyyyb.github.io/reprobridge_website/) makes open‑sourcing contingent on approval.

The authors’ [selection protocol](https://arxiv.org/html/2611.08452v1#S2.SS2) begins with 742 ICML 2026 Spotlight papers, identifies 241 with code and non‑purely‑theoretical content, and randomly samples 50. Availability, compute and closed‑model access exclusions leave 36 test papers. From 1,463 curated main‑table measurements, 1,029 have human‑verified reference results compatible with the published scores and enter the downstream comparison. These are selected measurements from retained papers, not a conference‑wide reproduction rate. Each task represents a specified measurement, including its dataset, metric and reporting statistic. The [reference procedure](https://arxiv.org/html/2611.08452v1#S2.SS3) uses five human‑verified executions and a fixed numerical compatibility interval, with a 1% tolerance floor. A multi‑seed mean is one observation of that statistic; it is not interchangeable with a single‑seed score. The study’s four coding agents—Sonnet 4.6, Sonnet 5, Opus 5 and Gemini 3.8 Flash—attempt the same selected tasks with raw code or refined artifacts. The [task scope](https://arxiv.org/html/2611.08452v1#S2.SS4) includes 631 full‑procedure and 398 evaluation‑only tasks; 730 are GPU tasks and 299 are CPU‑only. The [evaluation setup](https://arxiv.org/html/2611.08452v1#S4.SS1) gives both conditions a 24‑hour limit per paper, and GPU experiments use eight H100‑80GB GPUs. These are the study’s conditions, not minimum hardware or time requirements for every reproduction. [Table 3](https://arxiv.org/html/2611.08452v1#S4.T3) reports means and standard deviations across three evaluations. The equally weighted cross‑model mean success rate rises from 76.4% to 92.6%; the mean of each model’s relative downstream API reduction is 19.7%. That percentage is not a ratio of pooled expenditure or a measured reduction in total research cost. The paper also cautions that vendor harnesses differ, so cross‑model comparisons are descriptive. A successful task must be numerically compatible and survive the primary execution‑integrity audit. The [audit protocol](https://arxiv.org/html/2611.08452v1#S4.SS1) traces a reported value to its output, execution record and generating script. A matching number alone does not establish faithful execution, and unresolved suspicion alone is not an invalidation. The authors’ [human cross‑check](https://arxiv.org/html/2611.08452v1#S4.SS6) covers 818 outcomes from eight selected papers and finds missed violations by both automated Auditors. This is evidence with a defined review scope, not a guarantee of scientific validity.

The main cost table sums downstream Reproducer sessions, including failed attempts, using the paper’s frozen accounting rates. Construction, audit and GPU costs are outside that comparison. The accounting appendix additionally distinguishes human reference collection, maintenance and unrecorded cache‑storage fees. Readers should keep these categories separate even when the reported downstream saving is positive.

The paper’s Sonnet 5 example makes the distinction concrete. Mean aggregate API spending for the 36‑paper workload is $1,726 with raw code and $1,317 with refined artifacts: a $409 difference. Constructing all 36 artifacts costs $5,026 in API usage. Dividing $5,026 by $409 gives about 12.29, so the stated scenario reaches an API crossover after 13 complete reuse rounds of the 36‑paper workload. It does not mean 13 individual paper reproductions or 13 uses of one artifact.

That calculation holds prices, caching, artifact contents and agent behavior fixed. It excludes independent auditing, GPU execution and maintenance, and it is not a forecast of a reader’s full‑cost return. The limitations explicitly say that human time savings and comprehension need a separate study.

The paper’s four‑component handoff, execution audit and accounting appendix can be combined into the following practical record. This is our unexecuted inspection aid, not a released ReproBridge tool or an additional benchmark result.

1. Measurement identity: record the paper/table, task, dataset or split, variant, metric and reporting statistic. State whether the job requires training or permits an official checkpoint. Retain the eligible‑task denominator and exclusions alongside any success percentage.
2. Executable handoff: identify the pinned source, documented repairs, environment and data/checkpoint acquisition, and the launcher that connects the task to outputs and score extraction. Preserve the guide’s unresolved choices and reasons for adopted settings. A setup smoke test or a shipped score file is not the requested measurement.
3. Evidence chain: retain the actual configuration, command, execution record and output location for a reported value, with available script and output hashes. Separate diagnostic reruns from scored repetitions and record why a run was replaced. Keep audit findings and unresolved deviations visible.
4. Two cost ledgers: record construction and verification separately from later reproduction sessions. Include failures in the latter; list GPU, human review, maintenance and access constraints separately. Preserve pricing dates, cache assumptions and missing amounts rather than filling unknown costs with zero.
5. Reuse assumptions: compare the same workload and reporting unit in the raw and refined conditions. State who bears construction cost and which repeated‑use conditions must remain fixed. Keep any API‑only crossover separate from a full‑budget decision.

This record joins two questions the headline figures cannot answer by themselves: whether another researcher can trace and execute the intended measurement, and which expenses a reuse decision still leaves unpaid. Neither an archived artifact nor a numerically close result answers both.

The study limitations say refined artifacts can preserve incorrect assumptions. The intervention also includes investigation, repair, review and packaging; it does not isolate the effect of rearranging identical outputs. Its five‑reference protocol is a choice, not a guarantee of sufficient uncertainty estimation. Those limits matter when deciding what the inspection record must expose.

The first‑evaluation case analysis records 813 raw‑code failures becoming refined‑artifact successes, alongside 136 reversals. These are paired task–model outcomes across the four agents; the 136 are not unique tasks, and the analysis does not assign every reversal to packaging. These counts belong to the first evaluation, separate from the three‑evaluation means above.

The AuxiliaryMCMC example shows how the inspection record changes a decision. Its prescribed Gaussian sweep calls for 100 rounds at each of three acceptance rates. The Opus 5 refined‑artifact attempt completes only 55 rounds per rate after the workload exceeds the available clock at its chosen concurrency. The primary audit invalidates numerical matches affected by that changed measurement procedure. The raw‑code attempt also falls short of 100 rounds in part of its sweep, so this case does not establish that refinement alone caused the deviation.

Applied to that case, our unexecuted inspection record would retain the 100‑round target, the 55‑round execution, the affected output and audit finding, and the decision that those numerical matches do not meet the specified procedure. It would leave the case’s API, GPU, human‑review and maintenance amounts unknown where unreported. A close score would not repair a changed measurement, and a completed setup would not establish that the intended experiment finished.

The useful application is to ask for an inspectable procedure and a complete budget, then assess them against the intended research task. The paper’s selected‑task results support examining that approach under its tested conditions. They do not establish universal model rankings, total‑cost savings, production reliability or successful reproduction of all ML research.

Illustration: four artifact‑output icons adapted from Figure 2 in ReproBridge: Building Reproducibility into ML Research Artifacts, by Yubo Wang, Rui Meng, Jaehyun Nam, Tomas Pfister and Jinsung Yoon, CC BY 4.0. Icons rearranged in a new folio illustration; the original figure also depicts construction and independent review roles.