University of Illinois Urbana-Champaign

Trust, but verify: An investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility

Willis, Craig Alan

Content Files
WILLIS-DISSERTATION-2020.pdf

Permalink

https://hdl.handle.net/2142/108468

Description

Title
Trust, but verify: An investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility
Author(s)
Willis, Craig Alan
Issue Date
2020-07-13
Director of Research (if dissertation) or Advisor (if thesis)
Stodden, Victoria
Doctoral Committee Chair(s)
Stodden, Victoria
Committee Member(s)
  • Darch, Peter
  • Ludaescher, Bertram
  • Taufer, Michela
Department of Study
Information Sciences
Discipline
Library & Information Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Date of Ingest
2020-10-07T20:59:43Z
Keyword(s)
  • Computational reproducibility
  • Peer review
  • Reproducibility
Abstract
In this study, I investigate how research communities are addressing concerns about the quality and rigor of computational research. I focus specifically on initiatives to expand the peer review and publication process to include new requirements for the assessment and dissemination of computational research artifacts. I report the results of a multiple-case analysis of two primary (American Journal of Political Science, ACM/IEEE Supercomputing) and five supplemental cases in political science, computer science, economics, mathematics, and statistics. Cases were developed through qualitative analysis based on interviews with key stakeholders including editors, reviewers, and verifiers; a sample of verified artifacts; and documentary evidence including policies, guidelines, and workflows. The central argument of this dissertation is that these reproducibility initiatives represent a set of experiments across the sciences exploring how changes to the incentives and information requirements of authors impact the quality, rigor, reproducibility, and trustworthiness of published research. These initiatives are part of a broader effort to change community norms with respect to the dissemination of the results of research that involves computation, elevating the importance of computational artifacts and clearly signaling that authors cannot be trusted to provide this information voluntarily. Expanding peer review and increasing the information required of authors through publication reproducibility audits is just one approach -- and a costly one -- to improving research quality and trustworthiness. The effect of these changes on research quality has yet to be demonstrated or studied. Based on the cases, I identify key factors that influence the operationalization of policies and workflows; the elements that each community considers important to the assessment of computational transparency and reproducibility; as well as the tools and infrastructure that they leverage to aid in the creation, assessment and dissemination of reproducible research artifacts. I develop a framework to analyze the reproducibility initiatives and a conceptual model of reproducible research artifacts. I relate my findings to recommendations from the recent National Academies of Science, Engineering and Medicine (NASEM) report on \emph{Reproducibility and Replicability in Science} and provide a set of normative guidelines for communities interested in pursuing similar initiatives with implications for journal and conference leadership; tool and infrastructure developers; and funding bodies. I conclude that, while promising, further efforts should be made to increase our understanding of the effect of initiative policies and technological advancements on research quality.
Graduation Semester
2020-08
Type of Resource
Thesis
Permalink
http://hdl.handle.net/2142/108468
Copyright and License Information
Copyright 2020 Craig Willis

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois
Dissertations and Theses - Information Sciences
Dissertations and theses from the School of Information Sciences