Evaluating the Reproducibility of Articles in Management Science

Abstract

When researchers do publish, especially in peer-reviewed journals such as Management Science, there is an unspoken presumption that others can reproduce those findings with the same code and data. But in reality, that doesn't typically happen — or get attempted at all. By actively testing whether replication packages actually work—and how difficult or costly it is to get them running—the purpose of this experiment is to expose the gap between the ideal of reproducible science and the reality researchers face. Articles with available replication packages or supplementary materials were identified and processed through TDX to assess the extent to which reported results could be independently verified. The experiment systematically tracked performance metrics like run time(Avg: 2.087 hours), computational power utilized, expense(Avg: $1.149), and success rates. For results and accessibility verification, each replication package was run on Intel TDX and Google Cloud. Particular attention was paid to monitor replication challenges such as big file sizes, broken code, excessive runtimes, or exorbitantly high computational costs. Faced with a 75% reproduction success rate so far, using TDX and Cloud, the results show that while most Management Science studies with replication material available can be reproduced, a large proportion of it—25%—cannot be reproduced, showing persistent challenges with research transparency and reliability.