P Cuijpers1, E Weitz1, I A Cristea2, J Twisk3. 1. Department of Clinical, Neuro and Developmental Psychology,Vrije Universiteit Amsterdam,The Netherlands. 2. Department of Clinical Psychology and Psychotherapy,Babes-Bolyai University,Cluj-Napoca,Romania. 3. EMGO Institute for Health and Care Research,The Netherlands.
Abstract
Aims The standardised mean difference (SMD) is one of the most used effect sizes to indicate the effects of treatments. It indicates the difference between a treatment and comparison group after treatment has ended, in terms of standard deviations. Some meta-analyses, including several highly cited and influential ones, use the pre-post SMD, indicating the difference between baseline and post-test within one (treatment group). METHODS: In this paper, we argue that these pre-post SMDs should be avoided in meta-analyses and we describe the arguments why pre-post SMDs can result in biased outcomes. RESULTS: One important reason why pre-post SMDs should be avoided is that the scores on baseline and post-test are not independent of each other. The value for the correlation should be used in the calculation of the SMD, while this value is typically not known. We used data from an 'individual patient data' meta-analysis of trials comparing cognitive behaviour therapy and anti-depressive medication, to show that this problem can lead to considerable errors in the estimation of the SMDs. Another even more important reason why pre-post SMDs should be avoided in meta-analyses is that they are influenced by natural processes and characteristics of the patients and settings, and these cannot be discerned from the effects of the intervention. Between-group SMDs are much better because they control for such variables and these variables only affect the between group SMD when they are related to the effects of the intervention. CONCLUSIONS: We conclude that pre-post SMDs should be avoided in meta-analyses as using them probably results in biased outcomes.
Aims The standardised mean difference (SMD) is one of the most used effect sizes to indicate the effects of treatments. It indicates the difference between a treatment and comparison group after treatment has ended, in terms of standard deviations. Some meta-analyses, including several highly cited and influential ones, use the pre-post SMD, indicating the difference between baseline and post-test within one (treatment group). METHODS: In this paper, we argue that these pre-post SMDs should be avoided in meta-analyses and we describe the arguments why pre-post SMDs can result in biased outcomes. RESULTS: One important reason why pre-post SMDs should be avoided is that the scores on baseline and post-test are not independent of each other. The value for the correlation should be used in the calculation of the SMD, while this value is typically not known. We used data from an 'individual patient data' meta-analysis of trials comparing cognitive behaviour therapy and anti-depressive medication, to show that this problem can lead to considerable errors in the estimation of the SMDs. Another even more important reason why pre-post SMDs should be avoided in meta-analyses is that they are influenced by natural processes and characteristics of the patients and settings, and these cannot be discerned from the effects of the intervention. Between-group SMDs are much better because they control for such variables and these variables only affect the between group SMD when they are related to the effects of the intervention. CONCLUSIONS: We conclude that pre-post SMDs should be avoided in meta-analyses as using them probably results in biased outcomes.
Authors: Erica S Weitz; Steven D Hollon; Jos Twisk; Annemieke van Straten; Marcus J H Huibers; Daniel David; Robert J DeRubeis; Sona Dimidjian; Boadie W Dunlop; Ioana A Cristea; Mahbobeh Faramarzi; Ulrich Hegerl; Robin B Jarrett; Farzan Kheirkhah; Sidney H Kennedy; Roland Mergl; Jeanne Miranda; David C Mohr; A John Rush; Zindel V Segal; Juned Siddique; Anne D Simons; Jeffrey R Vittengl; Pim Cuijpers Journal: JAMA Psychiatry Date: 2015-11 Impact factor: 21.596
Authors: Pim Cuijpers; Erica Weitz; Jos Twisk; Christine Kuehner; Ioana Cristea; Daniel David; Robert J DeRubeis; Sona Dimidjian; Boadie W Dunlop; Mahbobeh Faramarzi; Ulrich Hegerl; Robin B Jarrett; Sidney H Kennedy; Farzan Kheirkhah; Roland Mergl; Jeanne Miranda; David C Mohr; Zindel V Segal; Juned Siddique; Anne D Simons; Jeffrey R Vittengl; Steven D Hollon Journal: Depress Anxiety Date: 2014-11 Impact factor: 6.505
Authors: Jeffrey R Vittengl; Robin B Jarrett; Erica Weitz; Steven D Hollon; Jos Twisk; Ioana Cristea; Daniel David; Robert J DeRubeis; Sona Dimidjian; Boadie W Dunlop; Mahbobeh Faramarzi; Ulrich Hegerl; Sidney H Kennedy; Farzan Kheirkhah; Roland Mergl; Jeanne Miranda; David C Mohr; A John Rush; Zindel V Segal; Juned Siddique; Anne D Simons; Pim Cuijpers Journal: Am J Psychiatry Date: 2016-02-12 Impact factor: 18.112
Authors: Rosalind van der Lem; Nic J A van der Wee; Tineke van Veen; Frans G Zitman Journal: Psychother Psychosom Date: 2012-05-11 Impact factor: 17.659
Authors: Rosalind van der Lem; Wouter Wh de Wever; Nic Ja van der Wee; Tineke van Veen; Pim Cuijpers; Frans G Zitman Journal: BMC Psychiatry Date: 2012-11-08 Impact factor: 3.630
Authors: Irving Kirsch; Brett J Deacon; Tania B Huedo-Medina; Alan Scoboria; Thomas J Moore; Blair T Johnson Journal: PLoS Med Date: 2008-02 Impact factor: 11.069
Authors: David W Pantalone; Kimberly M Nelson; Abigail W Batchelder; Christopher Chiu; Hamish A Gunn; Keith J Horvath Journal: J Sex Res Date: 2020-02-20
Authors: Wenhao Dai; Ryan Palmer; Aashna Sunderrajan; Marta Durantini; Flor Sánchez; Laura R Glasman; Fan Xuan Chen; Dolores Albarracín Journal: Psychol Addict Behav Date: 2020-04-20
Authors: Kelly Allott; Kristi van-der-El; Shayden Bryce; Emma M Parrish; Susan R McGurk; Sarah Hetrick; Christopher R Bowie; Sean Kidd; Matthew Hamilton; Eoin Killackey; Dawn Velligan Journal: Schizophr Bull Date: 2020-07-08 Impact factor: 9.306
Authors: Stephen P H Whiteside; Leslie A Sim; Allison S Morrow; Wigdan H Farah; Daniel R Hilliker; M Hassan Murad; Zhen Wang Journal: Clin Child Fam Psychol Rev Date: 2020-03