Analysing rank statistics: The Cochrane Kit Kat Trial




Poster session 2 Thursday: Evidence synthesis - methods / improving conduct and reporting


Thursday 14 September 2017 - 12:30 to 14:00


All authors in correct order:

Le J1, Fusco N1, Lindsley K2, Chaimani A3, Cao M4, Chen Y5, Li T6
1 Johns Hopkins University - Bloomberg School of Public Health, USA
2 Cochrane Eyes and Vision, US Satellite, USA
3 Paris Descartes, France
4 University of Texas School of Public Health, USA
5 Department of Biostatistics, Epidemiology and Informatics; Perelman School of Medicine, USA
6 Cochrane United States, USA
Presenting author and contact person

Presenting author:

Jimmy Le

Contact person:

Abstract text
Background: We illustrated an approach of teaching randomised-controlled trials through a 'Kit Kat' trial with participants at the 24th Cochrane Colloquium in Seoul, Korea. The trial ran in four cohorts of tasters where three out of four Kit Kats from the United States (US), United Kingdom (UK), France, and South Africa were compared in each cohort.

Objective: To illustrate the challenges in analysing and interpreting rank statistics in the context of a network of trials

Methods: We used a crossover design. Each taster tried a piece of three different Kit Kats, wrapped in foil, in a random order. We concealed the random sequence using opaque stickers that were peeled off at time of randomisation. Participants indicated which Kit Kat they liked the 'most', 'second most', and 'least'. This design created a network of four 'nodes' for comparison where the outcome of interest is rank statistics.

We summarised the ranks by cohort. We examined whether there is evidence for carry-over effect and cohort effect. We pooled the differences of proportion of each rank in a fixed effect meta-analysis. We also fit Bradley-Terry models to estimate the latent 'likeness' of each Kit Kat by pooling ranks across cohorts while acknowledging the cohort effects. Such latent likeness can be used to rank the preference of Kit Kats, or predict the outcome of a comparison, e.g. probability of one Kit Kat is preferred over another.

Results: 126 conference attendees from 33 different countries participated from 24 - 26 October 2016. We observed a cohort effect, which may explain inconsistent rankings between cohorts (Figure). The pairwise meta-analyses suggest that Kit Kats from France were never 'liked the most' and Kit Kats from the UK were never 'liked the least'.

Bradley-Terry model suggested that Kit Kats from South Africa (likeness value= 0.35; the higher the more likable) are liked more than Kit Kats from the UK (0.22), US (0.14), or France (reference= 0) (Table).

Conclusions: Meta-analysis may be an appropriate method to combine rank statistics. The statisticians we consulted also suggested different approaches, which we will present at the Summit.