Reproducing published economics (economics)

Can an AI agent reproduce a published economics paper from its replication package, and then improve on it? An LLM workflow reruns the analyses behind published articles and looks for numerical improvements and extensions.

For many years several economics journals have required each published article to come with a replication packageThe data and code that an article’s authors deposit with the journal so that others can rerun the calculations behind the published numbers., the data and code behind its numbers. In an open-source workflow (NBER Working Paper 35782, by Matthew D. Schwartz, Isaiah Andrews, and Jesse M. Shapiro), a large language model (LLM) reads such a package, reruns the article’s calculations, and compares the results with the printed numbers. It then looks for faster ways to do the same calculations and proposes further results under the article’s own assumptions. Applied to 4,452 packages from five journals, the workflow reports a discrepancy somewhere in the article or its appendix in 3,460, speeds a calculation up more than tenfold in 496, and develops an extension in 923. The authors present these as findings to date from work that is still under way.

Replication packages

Computationally, an empirical economics article is a chain of steps: data are cleaned and merged, estimators and simulations run on the result, and the output becomes the numbers printed in the tables. The National Academies’ 2019 report Reproducibility and Replicability in Science distinguishes two ideas. Reproducibility means that the same data and code return the published numbers; replicability means that a new study with its own data reaches a consistent conclusion. The workflow tests reproducibility.

Depositing data and code with a journal goes back in economics at least to the Journal of Money, Credit and Banking replication project that Dewald, Thursby, and Anderson reported in 1986, an early study of the practice. The five journals studied here are the American Economic Review, Econometrica, the Journal of Political Economy, the Quarterly Journal of Economics, and the Review of Economic Studies. They are the five highest-ranked peer-reviewed general-interest economics journals in the RePEc (Research Papers in Economics) rankings. The first full year in which the American Economic Review and Econometrica required data and code from authors of accepted articles was 2005; for the Journal of Political Economy and the Review of Economic Studies it was 2006, and for the Quarterly Journal of Economics, 2017. The American Economic Association’s Data and Code Availability Policy shows what such a rule looks like today.

To check a deposited package, someone has to install the software the authors used, which may need a license, and obtain the data, some of which may be confidential and left out of the deposit. Then every script has to be run and its output compared with the printed tables. Earlier large-scale studies of reproducibility and replicability, from McCullough, McGeary, and Harrison in 2008 to Brodeur and coauthors in Nature in 2026, covered dozens or hundreds of articles; this one covers 4,452. Three recent efforts have had LLMs assess or carry out reproductions of social-science research. A fourth, by Muravyev, uses an LLM to reconstruct the main results of 1,328 finance articles from source databases rather than from packages.

The LLM workflow

The starting hypothesis is that a modern LLM combines the skills of a software engineer with broad knowledge of mathematics, computer science, and other sciences, a combination that few if any people have. It might therefore see ways to improve a piece of research that a human reader, even an expert in the subject, could miss. The test bed is every available replication package for articles published from 2000 through 2026 in the five journals. Each package comes with its published article and any online appendix, treated together as “the article.”

Before any article is checked, a multi-agent LLM workflow builds an open-source library that can reproduce the kinds of computation found across the corpus. Writing a package’s calculations in open-source code puts every step of each calculation in a form the LLM can read, which helps it understand and later improve them. The workflow built on that library has four tasks: check reproducibility, analyze sensitivity, improve computations, and extend results.

Reproduction begins with the package’s documentation, which may declare that some calculations cannot be rerun because their data, confidential records for example, are left out of the package. If the documentation says this of every calculation, the workflow does not attempt a reproduction; instead it compares the package’s code and any log files with the text of the article and records each apparent discrepancy, which it cannot verify. Otherwise it attempts every calculation that the documentation does not exclude, working from the open-source library, the package’s own code, and the article’s description of its methods. Two things count as a discrepancy: a calculation blocked by a data omission the documentation did not declare, and a calculation that runs but returns a value different from the published one.

Every recorded discrepancy then has to pass two checks. First, another LLM agent, acting as an adversary, tries to overturn it and uphold the reproducibility of the original finding. Second, wherever licensed access to the original software could be arranged, the calculation is rerun there. For a discrepancy that survives both, the workflow also records the change to the article’s text that would align it with the recomputed value.

Reproduction and sensitivity

The diagram below follows the 4,452 packages through the reproduction procedure. A reproduction is attempted in 87.6% of packages, and among those the workflow finds and verifies a discrepancy in 87.3%. As shares of all packages, 11.2% reproduce fully at the precision the article reports and 76.4% do not. The other 12.4% declared every calculation dependent on omitted data, so only an apparent discrepancy in code or logs can be recorded, which happens in 1.3% of all packages. In 6.3% of the packages with a discrepancy, the earliest one is a data omission rather than a mismatched value. Counting verified and apparent discrepancies together, the workflow reports a discrepancy in 3,460 articles or their appendices.

Flow diagram reading left to right. A dark gray bar labeled Replication packages audited, 4,452, splits into two bands: a wide light-gray band labeled Reproduction attempted, 87.6%, and a narrow lower band labeled All calculations declared data-gated; not attempted, 12.4%. The attempted band splits into a wide blue band labeled Not fully reproduced, 76.4%, and a narrower green band labeled Fully reproduced, 11.2%. The blue band fans out into four blue endpoints on the right: Earliest discrepancy in abstract, 3.1%; Earliest discrepancy in introduction, 12.9%; Earliest discrepancy in body of paper, 54.0%; Earliest discrepancy in appendix, 6.4%. The lower gray band splits into two light-gray endpoints: Apparent discrepancy in code or logs, 1.3%, and No apparent discrepancy, 11.1%. Every percentage is a share of all 4,452 audited packages.

Where the 4,452 replication packages end up. Read left to right; each band’s width is proportional to the number of packages, with very small groups drawn at a minimum width so they stay visible. Lower gray bands: packages whose documentation declares every calculation dependent on omitted data (“data-gated” in the figure’s label), so nothing is rerun (12.4%); in 1.3% the code or logs still disagree with the text. Blue: packages with at least one recomputed value or undeclared data omission, sorted by where in the article the earliest one falls, from the abstract (3.1%) to the appendix (6.4%). Green: packages in which every attempted calculation matches the article at its printed precision (11.2%). All percentages are shares of the 4,452. Most packages that could be rerun have at least one discrepancy, and it most often first appears in the body of the paper. (Figure 2 of the paper.)

Many of the discrepancies are small. The 3,166 articles whose earliest discrepancy is a recomputed number are each classified by the significant digit of the published value that the discrepancy affects. The mildest bin holds articles whose recomputed value $\hat{v}$ and published value $v$ satisfy

$$\lvert \hat{v} - v \rvert \;\le\; 1.0\,u(v), \qquad u(v) = \text{one unit in the last printed digit of } v ,$$

in this page’s notation; the paper states the rule in words. If an article prints 0.047, then $u(v)$ is 0.001, and a recomputed value of 0.046 or 0.048 lands in this bin, a difference consistent with rounding or truncation. This bin holds 35.6% of the 3,166 articles. The rest are placed by the earliest significant digitThe meaningful digits of a printed number, counted from its first nonzero digit; a difference at the first significant digit changes the leading figure itself. of the published value at which the recomputed value, rounded to the published precision, differs: beyond the third digit for 3.9% of the articles, the third digit for 10.5%, the second for 25.1%, and the leading digit for 24.9%.

Dot chart, one dot per article, with a boxed legend reading One dot = one article (N = 3,166). Five columns of stacked dots whose blue shade darkens from left to right. Column labels and percentages, left to right: Last digit 35.6%, Beyond 3rd 3.9%, 3rd digit 10.5%, 2nd digit 25.1%, 1st digit 24.9%. The Last digit column is the tallest, reaching above 1,100 articles; Beyond 3rd is the shortest; 2nd digit and 1st digit are nearly equal at just under 800. Vertical axis labeled Articles with gridlines at 300, 600, 900. Horizontal axis label: First significant digit of the published value at which the earliest calculation discrepancy occurs.

How deep the earliest discrepancy reaches. Each dot is one of the 3,166 articles whose earliest discrepancy is a number recomputed at a value different from the published one. “Last digit”: the two values agree to within one unit of the last printed digit. The other columns give the first significant digit of the published value at which they differ, from beyond the third digit to the leading digit; darker blue marks a more significant digit. About a third of the earliest discrepancies are of rounding size, and about a quarter change the leading digit. (Figure 3 of the paper.)

A reproduced estimate can also be tested for how much it depends on a few data points. Giordano, Meager, and Broderick proposed checking published findings for sensitivity to removing a small amount of data and demonstrated the idea on nine articles; here it is applied to 1,908. The method rests on a first-order approximation. For a point estimateA single reported number estimating some quantity, such as a regression coefficient or an average effect, as opposed to a range or an interval. $\hat{\theta}$ computed from $n$ units that the article’s own statistical method treats as independent, rows of a data table or clusters of rows, deleting unit $i$ changes the estimate to approximately

$$\hat{\theta}_{-i} \;\approx\; \hat{\theta} \;-\; \frac{1}{n}\,\psi_i ,$$

where $\hat{\theta}_{-i}$ is the estimate with unit $i$ removed and $\psi_i$ is the empirical influenceA number measuring how strongly one observation pulls the estimate; the units with the largest influence are the ones whose removal should move it most. of unit $i$. (The display is the textbook form; the paper describes the step in words.) The workflow uses this approximation only to rank the units: it picks the 5 of greatest approximate influence, removes them one at a time, and recomputes the estimate in full each time. Each change is reported as a percentage of the workflow’s own reproduction of the published value,

$$\Delta \;=\; 100 \times \frac{\lvert \tilde{\theta} - \hat{\theta} \rvert}{\lvert \hat{\theta} \rvert} ,$$

with $\hat{\theta}$ the reproduced value and $\tilde{\theta}$ the value after one unit is removed, and each article is classified by its largest $\Delta$. A $\Delta$ above 100 means the estimate moved by more than its own size: it more than doubled or changed sign. Removing a single unit does that to some reported point estimate in 963 of the 1,908 articles, and in 512 of those the published value is printed to at least two significant digits.

Improving the computations

The improvement step looks for calculations that involve numerical optimization, integration, simulation, or another approximate numerical method. The workflow could reproduce at least one such calculation in 1,781 articles, and in each of those articles it tries two kinds of improvement. A reimplementationThe term used here for speeding up a calculation without changing its algorithm, for example by vectorizing or removing overhead; the output must match the original to floating-point rounding. keeps the algorithm and speeds up the code, by profiling it and applying standard practice such as cutting overhead or vectorizingRewriting a loop that handles one number at a time as a single operation on whole arrays, which numerical libraries execute far faster.. Its output must match the original to within floating-point roundingThe tiny error, around the sixteenth significant digit, that computers make because they store numbers with finite precision., and it is timed against the workflow’s open-source rendering of the original code on the same machine. A reformulation changes the algorithm itself, for instance by using an exact characterization of the solution known from other research or another discipline but absent from the package. Because a reformulation may genuinely improve on the original, an exact match is not always possible or even wanted. Reformulations are instead tested for fidelity to the mathematical intent of the calculation, for example against a high-precision computation or a smaller instance of the same problem.

In the chart below, each article is binned by the largest speedup any of its calculations received, running time before divided by running time after. Reimplementations speed something up in 1,535 articles, by more than a factor of 10 in 252. Reformulations help in 1,324 and exceed a factor of 10 in 308, often with better accuracy as well. A reformulation’s speedup is measured against the already reimplemented code, so the two kinds of gain are never counted twice. Counting both, computation time falls by more than a factor of 10 in 496 of the 1,781 articles.

Grouped bar chart. Horizontal axis: largest speedup of any calculation in the article, run time before divided by after, in six bins: no gain (1 times), 1 to 10 times, 10 to 100 times, 100 to 1,000 times, 1,000 to 10,000 times, over 10,000 times. Vertical axis: share of the 1,781 articles, 0 to 80 percent. Dark slate bars, reimplementation (same algorithm, faster code): 13.8, 72.0, 11.4, 2.5, 0.3 and 0.0 percent. Orange bars, reformulation (a different algorithm): 25.7, 57.0, 12.5, 3.2, 1.1 and 0.4 percent. A violet bracket over the four right-hand bins reads: more than tenfold, by either route: 496 of the 1,781 articles.

Largest speedup per article, for the 1,781 articles with at least one reproducible calculation that uses an approximate numerical method. Slate bars: reimplementation, the same algorithm in faster code, timed against an open-source translation of the authors’ public replication code. Orange bars: reformulation, a different algorithm, timed against the already reimplemented code so that no gain is counted twice. Bins are factors of ten, and the percentages, of the 1,781 articles in each case, are those printed in the paper. Most gains are under tenfold, but a tail of articles runs hundreds to tens of thousands of times faster, more often through reformulation. (Data from Figure 5 of the paper.)

Two examples, one of each kind, both come from the Review of Economic Studies. Lee (2026) has a package the workflow reproduces exactly, at every printed digit, for every calculation it can run. Its bootstrap confidence intervals come from 1,000 replications, each built from its own random draws and processed one at a time in a loop. The workflow notices that each replication enters the intervals only through means of quantities that do not vary across replications, so all of them can be computed at once as one matrix product. Replaying the original seeded random draws, the new code returns the published intervals exactly, roughly 440 times faster. Akcigit, Pearce, and Prato (2025) calibrate seven model parameters to seven data momentsSummary statistics of the data, such as averages, variances, or shares, that a model’s parameters are chosen to match. by minimizing a numerical objective with a stochastic search followed by a local solver. The workflow observes that the minimizing parameters can be written in closed form, with no numerical optimization at all, and uses interval arithmetic to certify that this solution is the only root in its neighborhood. The closed form matches every printed digit of the published calibration and evaluates roughly 24,000 times faster than the workflow’s open-source rendering of the authors’ original calibration code.

Extending the results

The last step looks for results the article could have added under its own assumptions. For each quantitative conclusion, the workflow asks whether existing mathematical or scientific knowledge yields a further result without different or additional substantive assumptions, statistical or economic. Two adversarial LLM agents then check each proposed extension. One probes for unstated assumptions that the extension needs and the original analysis did not; only extensions that need none are kept. The other argues that the extension is of no use to the article’s readers, and here the workflow makes a subjective judgment about which extensions to keep.

Of 4,418 articles screened, the workflow proposes an extension for 923 and none for the other 3,495. Counting each article once, by the extension relevant to its earliest section, 143 add a new counterfactualA model’s prediction for a situation that did not occur, such as a policy change of a different size from the one actually studied. or other substantive calculation, 367 quantify uncertainty for a number the article reported without any, and the rest add sensitivity analysis.

Two examples follow. Kaas, Lalé, and Siassi (2026), in Econometrica, use an overlapping-generations macroeconomic model with a job ladderA model feature in which employed workers keep searching and move up to better-paying jobs over time, falling back down when they lose a job. to simulate a 10 percent cut in unemployment-insurance benefits in Germany. The workflow solves the same general-equilibriumSolved so that all markets clear at once: prices and wages adjust until the choices of households and firms are mutually consistent. system, to the article’s own convergence tolerances, for a 25 percent cut, and finds that most reported outcomes scale less than proportionately with the size of the cut. Carrera and coauthors (2022), in the Review of Economic Studies, estimate from a structural modelAn economic model whose equations describe people’s preferences and decisions, estimated from data so that it can put a value on welfare or on untried policies. that participants who take up a commitment contract for gym attendance incur a welfare loss equivalent to $18.69 per person. The article gives no measure of that number’s uncertainty. The workflow applies a bootstrap analogous to one used elsewhere in the same article and obtains a 95% confidence interval from $12.75 to $23.99.

Evaluation and caveats

The authors state that the workflow does not aim to evaluate research. A separate experiment does look at evaluation by LLMs, with the emphasis on how stable such judgments are. Fresh agents of two models, Claude Opus 5 and Claude Fable 5.1, read 100 articles from the corpus under three conditions: the article alone, the article with its replication package, or both together with the workflow’s reproduction. Ten agents per model and condition rated each article’s conclusions and implementation and, when given the reproduction, whether the findings reproduce. Access to the reproduction led both models to rate more articles as correct. With the reproduction in hand, the two models rated the findings as reproduced with minor differences in 83.1% and 83.6% of gradings, respectively. Three sessions of the same model, drawn at random from its ten, would be expected to agree on the reproduction rating for 88.6% of articles with Claude Opus and 89.1% with Claude Fable.

The paper

Supplementary material

References

National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science, National Academies Press, Washington, DC (2019)cited on reproducibility as part of the scientific process; its usage separates reproducibility from replicability
W. G. Dewald, J. G. Thursby and R. G. Anderson, Replication in empirical economics: The Journal of Money, Credit and Banking project, Am. Econ. Rev. 76 (1986) 587the early journal replication project; the paper’s first reference on depositing data and code
B. S. Bernanke, Editorial statement, Am. Econ. Rev. 94 (2004) 404announced the American Economic Review requirement to submit data and code
B. D. McCullough, K. A. McGeary and T. D. Harrison, Do economics journal archives promote replicable research?, Can. J. Econ. 41 (2008) 1406the earliest of the eight large-scale reproducibility and replicability studies cited
A. Brodeur, D. Mikola, N. Cook et al., Reproducibility and robustness of economics and political science research, Nature 652 (2026) 151the most recent of those eight studies
A. Brodeur, D. Valenta, A. Marcoci et al., Comparing human-only, AI-assisted, and AI-led teams on assessing research reproducibility in quantitative social science, IZA Discussion Paper 17645, IZA Institute of Labor Economics (2025)the first of three cited studies of LLMs assessing or carrying out social-science reproductions
C. Hu, L. Zhang, Y. Lim, A. Wadhwani, A. Peters and D. Kang, REPRO-bench: Can agentic AI systems assess the reproducibility of social science research?, in Findings of the Association for Computational Linguistics: ACL 2025 (2025) 23616the second of those three studies
B. Kohler, D. Zollikofer, J. Einsiedler, A. Hoyle and E. Ash, Read the paper, write the code: Agentic reproduction of social-science results, arXiv:2604.21965 (2026)the third of those three studies
D. Muravyev, Does empirical finance replicate?, working paper, University of Illinois Urbana-Champaign (2026)an LLM reconstruction of 1,328 finance results from source databases rather than packages
R. Giordano, R. Meager and T. Broderick, An automatic finite-sample robustness metric: When can dropping a little data change conclusions? Part I: definitions and experiments and Part II: theory and intuition, Phil. Trans. R. Soc. A 384 (2026) 20250001 and 20240614; earlier preprint version T. Broderick, R. Giordano and R. Meager, arXiv:2011.14999 (2020)the dropping-data sensitivity check, demonstrated on nine articles and applied here to 1,908
W. Lee, Identification and estimation of dynamic random coefficient models, Rev. Econ. Stud. (2026)the reimplementation example: a bootstrap loop as one matrix product, about 440 times faster
U. Akcigit, J. Pearce and M. Prato, Tapping into talent: Coupling education and innovation policies for economic growth, Rev. Econ. Stud. 92 (2025) 696the reformulation example: a closed-form calibration, about 24,000 times faster
L. Kaas, E. Lalé and N. Siassi, Job ladder and wealth dynamics in general equilibrium, Econometrica 94 (2026) 1449the extension example with a new 25 percent benefit-cut counterfactual
M. Carrera, H. Royer, M. Stehr, J. Sydnor and D. Taubinsky, Who chooses commitment? Evidence and welfare implications, Rev. Econ. Stud. 89 (2022) 1205the extension example that adds a confidence interval to a welfare estimate

← back to the web summaries