Unnecessary pooling of heterogeneous data: Analysis of a sample of systematic reviews




Poster session 2 Thursday: Evidence synthesis - methods / improving conduct and reporting


Thursday 14 September 2017 - 12:30 to 14:00


All authors in correct order:

Deshpande S1, Westwood M1, Misso K1, Stirk L1, de Kock S1, Kleijnen J1, Clayton D1, Kleijnen J1
1 Kleijnen Systematic Reviews Ltd., United Kingdom
Presenting author and contact person

Presenting author:

Sohan Deshpande

Contact person:

Abstract text
Background: A common criticism of systematic reviews (SRs) is that they carry out statistical pooling without adequately considering clinical heterogeneity. Findings based on such estimates could be misleading as readers often consider the pooled summary estimates and ignore any underlying heterogeneity.

Objectives: To analyse the systematic reviews in KSR Evidence (a database of systematic reviews) which were assessed to be at high risk of bias (RoB) because pooled analysis was not appropriate due to presence of clinical heterogeneity.

Methods: All SRs in the database are critically appraised using the validated Risk of Bias in Systematic Reviews (ROBIS) tool, specifically designed to assess RoB in SRs and are checked for quality by a second independent reviewer. We analysed a sample of appraisals independently checked by 10 experienced reviewers. We identified reviews judged at high RoB for the synthesis as the authors pooled the studies despite of the clinical heterogeneity across the studies.

Results: The database has >30,000 SRs, as identified by our searches to date. This analysis includes a sample of 651 SR critical appraisals (checked independently) of which 49% (321) were judged at high RoB on ROBIS domain 4 (Synthesis and Findings). Pooling was judged to be inappropriate, due to the presence of clinical heterogeneity, in 17.5% (56/321) of the SRs judged to be at high RoB. Sources of heterogeneity across the 56 SRs included pooling of different study designs, time points, scales used for measuring outcomes and interventions/comparators with different dose regimens, etc. The majority of these SRs (45/56) also showed statistical evidence of heterogeneity.

Conclusions: Consideration of clinical heterogeneity before pooling studies is crucial. A large number of studies ignore clinical heterogeneity and report summary estimates which may mask clinically important variations in effectiveness. SRs should not conduct pooled analysis where inappropriate and further improve reporting of synthesis.