Background: Systematic reviews employ comprehensive search strategies to identify all relevant studies on a topic. Methods projects assessing novel search approaches often benchmark the sensitivity of searches by checking whether they find the same studies as comprehensive, systematic searches. However, results of studies are frequently reported in multiple publications. Defining the threshold that classifies a study as 'identified' by a search partially determines the validity of the search strategy under assessment: The least time-intensive approach is to check if the major publication of that study is found. A second approach is to check if any publication of a study was identified. The strictest threshold requires every publication belonging to a study to be found.
Objectives: Our aim is to test the validity of three methods of classifying a study as found by a search compared to a gold standard.
Methods: As part of a larger methods project (1) we randomly chose 60 Cochrane Reviews on various clinical topics. We reproduced their MEDLINE, CENTRAL, and EMBASE searches, and used a 'content-based' approach to determine if a study was found: We read all identified publications belonging to this study and checked if they reported the outcomes relevant for the main meta-analyses. This is our gold standard method: It allows us to define a study as identified when all relevant data are found. For this study we will compare this exhaustive process with the three pragmatic approaches: a study is considered as 'identified' if a) the main study publication is found, b) one publication is found or c) all publications belonging to one study are found. We will assess to what degree these three methods agree with the results of the content-based approach.
Results will be available at the Summit.
Conclusions: To our knowledge, this is the first study to inform methodologists what is the most valid approach to define a study as found by a search strategy.
(1) Nussbaumer-Streit B, et.al. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Sys rev. 2016 Nov 22;5(1):197.