A fundamental concern associated with automatically or semi-automatically eliminating candidate records is that important studies may be missed, compromising the comprehensiveness of the review and potentially the validity of its conclusions. Mounting interest in the use of ML tools to expedite title and abstract screening has been accompanied by skepticism and distrust by review teams and end users of reviews, and adoption has been slow. The relevance predictions produced by ML tools can also be leveraged by review teams to semi-automate title and abstract screening by eliminating records predicted to be irrelevant. By prioritizing relevant records, such tools provide reviewers with the opportunity to identify relevant studies earlier and move forward with subsequent review tasks (e.g., data extraction, risk of bias appraisal) sooner. Freely-available, off-the-shelf tools like Abstrackr, RobotAnalyst, and Rayyan allow review teams without ML expertise and/or limited resources to create efficiencies during title and abstract screening. Machine learning (ML) tools provide the potential to expedite title and abstract screening by predicting and prioritizing the relevance of candidate records.Īt the time of writing the SR Tool Box, an online repository of software tools that support and/or expedite evidence synthesis processes, referenced 37 tools aimed at supporting title and abstract screening. The process requires substantial effort and time to return a relatively small body of relevant studies. Often, two reviewers screen through the records retrieved, first by title and abstract and then by full text, to identify those that are relevant. To avoid missing relevant studies, rigorously conducted evidence syntheses typically include comprehensive searches of multiple sources. Systematic evidence syntheses provide the foundation of informed decision-making however, the large and growing body of primary studies makes it difficult to complete them efficiently and keep them up-to-date. Many missed records would be identified via other means. Using Abstrackr to assist one of two reviewers in systematic reviews saves time with little risk of missing relevant records. The median (range) proportion missed records for both approaches was 6 (0–22)%. For the rapid reviews, the fully and semi-automated approaches saved time (median (range) 9 (2–18) hours and 3 (1–10) hours, respectively), but less so than for the systematic reviews. The cited references search identified 59% ( n = 10/17) of the records missed. Resultsįor systematic reviews, the semi-automated, dual independent screening approach provided the best balance of time savings (median (range) 20 (3–82) hours) and reliability (median (range) proportion missed records, 1 (0–14)%). We performed cited reference searches to determine if missed studies would be identified via reference list scanning. We calculated the proportion missed, workload savings, and time savings compared to single-reviewer and dual independent screening by human reviewers. We subjected 11 systematic reviews and six rapid reviews to four retrospective screening simulations (automated and semi-automated approaches to single-reviewer and dual independent screening) in Abstrackr, a freely-available machine learning software. We investigated the feasibility of using a machine learning tool’s relevance predictions to expedite title and abstract screening.
0 Comments
Leave a Reply. |