Evaluating an automatic data extraction tool for evidence synthesis through real-life case studies
Author(s)
Hoang, Linh K.
Cao, Linh T.
Guan, Yingjun
Cheng, Yi-Yun
Schneider, Jodi
Issue Date
2017
Keyword(s)
Systematic Review
Automation
Data Extraction
Evidence Synthesis
Abstract
Computer support tools are increasingly designed to help reduce the time required for evidence synthesis tasks such as manually extracting information from scientific papers. We present two case studies evaluating RobotReviewer, an automatic data extraction tool used to help synthesize evidence from primary research papers for use in review papers. We use primary research papers related to oral health and dental medicine for both our case studies. The first case study uses the same published review we presented at last year's Research Showcase, with a new evaluation metric and 3 novice annotators. Through manual annotation and a content analysis of the six studies synthesized in the review paper, we compare how well (1) novices and (2) the RobotReviewer data extraction compare to the Cochrane Review paper (seen here as a expert gold standard). (Feedback at last year's Research Showcase was instrumental, and we have improved the methodology we presented last year in several ways; for instance those preliminary results had just one novice annotator.) The second case study is based on a real systematic review project being conducted in part at the School of Dental Medicine at the University of Buffalo. Real reviewers collaborate by comparing RobotReviewer's results with their own manual extraction results and describing how well it meets their expectations. By evaluating existing tools such as RobotReviewer, we would be able to identify gaps between what computer support tools are available and how well these tools work. This could help propose new directions for automated support systems that would help to reduce the time and human labor required for evidence synthesis tasks such as data extraction.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
