Background: Replication is a cornerstone of the scientific method. However, unnecessary duplication rather than replication is unethical and a cause of research waste. Moreover, what appear to be duplicate systematic reviews (SRs) often come to different conclusions. Multiple overlapping SRs with not infrequent discordance lead to confusion among users (e.g. patients, healthcare workers, social workers). A better understanding of the reasons for discord among overlapping SRs may contribute to the development of guidance on when to replicate a SR, and when not to.
Objectives: To develop a checklist to identify reasons for discordance among overlapping SRs.
Methods: Based on a review of the literature and consultation with experts, we developed a checklist of items to understand reasons for discord among overlapping SRs. We tested the feasibility and usefulness of the checklist on several overlapping SRs with discordant results or conclusions.
Results: The checklist itemises components of the objectives, methods for study inclusion, selection of outcomes, data synthesis, reporting and interpretation of findings, which may contribute to discordant findings in overlapping SRs. Information on author discipline and affiliation, conflict of interest, and SR quality was also recorded. The checklist was tested on a diverse selection of discordant reviews in controversial areas including deworming, glucosamine, vitamin D supplementation, payment for environmental services, and pre-school programmes. The most frequent reasons for discord included differences in study eligibility criteria and definition of outcomes, leading to differences in the primary studies being reviewed. We noted several examples where review conclusions supported possible bias related to reviewer conflict of interest.
Conclusions: The checklist for discordant SRs is a useful tool for explaining discordant findings among overlapping SRs. Development of this tool is part of a larger project to establish guidance on when replication of SRs may be useful, and when it would be wasteful. This work aims to support reliance on high-quality SRs rather than low-quality duplication.