Including non-randomised studies or letting them go unchallenged? - Experiences from a Cochrane review on screening for malignant melanoma




Poster session 2 Thursday: Evidence synthesis - methods / improving conduct and reporting


Thursday 14 September 2017 - 12:30 to 14:00


All authors in correct order:

Johansson M1, Brodersen J2, Gøtzsche P3, Juhl Jørgensen K4
1 Department of Public Health and Community Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden, Sweden
2 Research Unit for General Practice and Section of General Practice, Department of Public Health, University of Copenhagen, Copenhagen, Denmark., Denmark
3 Nordic Cochrane Centre, Rigshospitalet, Copenhagen, Denmark., Denmark
4 Nordic Cochrane Centre, Rigshospitalet, Copenhagen, Denmark., Denamrk
Presenting author and contact person

Presenting author:

Minna Johansson

Contact person:

Abstract text
Background: High-quality evidence from randomised trials is a general requirement for the introduction of population-based screening programmes. However, for some types of screening no relevant data from randomised trials exist, while the intervention is nonetheless offered in nationwide programmes based on apparent beneficial effects seen in non-randomised studies. When performing a Cochrane review on screening for malignant melanoma, we ended up in a dilemma; should we include non-randomised studies in our review or should we ignore them?

Objectives: To find out how to deal with problems arising when high-quality evidence from randomised trials is lacking, while low-quality evidence from non-randomised studies has been used to justify a screening programme.

Method: Reflections based on experience from encountering this dilemma in a Cochrane review on screening for malignant melanoma.

Results: We had preplanned not to include non-randomised studies in our review. However, while working on the review we encountered a number of non-randomised studies frequently used to justify this type of screening. In the discussion part of the review, we chose post hoc to evaluate those non-randomised studies that had influenced national policy, using the ROBINS-I tool (Risk Of Bias In Non randomised Studies of Interventions).

Conclusion: Including non-randomised studies leads to a massive extra workload compared to restricting inclusion to randomised trials only, which will inevitably increase time to publication. Additionally, including data from non-randomised studies in a Cochrane review on population screening might legitimise the use of such interventions based on low-quality evidence for other screening programmes. However, if we do not assess the merits of studies that have been pivotal for far-reaching public health decisions, it would reduce the relevance of the corresponding Cochrane reviews. A compromise might be to evaluate the most relevant non-randomised studies in the discussion section of the review. However, to evaluate some non-randomised trials without performing a systematic search might introduce selection bias.