S H Downs1, N Black. 1. Department of Public Health and Policy, London School of Hygiene and Tropical Medicine.
Abstract
OBJECTIVE: To test the feasibility of creating a valid and reliable checklist with the following features: appropriate for assessing both randomised and non-randomised studies; provision of both an overall score for study quality and a profile of scores not only for the quality of reporting, internal validity (bias and confounding) and power, but also for external validity. DESIGN: A pilot version was first developed, based on epidemiological principles, reviews, and existing checklists for randomised studies. Face and content validity were assessed by three experienced reviewers and reliability was determined using two raters assessing 10 randomised and 10 non-randomised studies. Using different raters, the checklist was revised and tested for internal consistency (Kuder-Richardson 20), test-retest and inter-rater reliability (Spearman correlation coefficient and sign rank test; kappa statistics), criterion validity, and respondent burden. MAIN RESULTS: The performance of the checklist improved considerably after revision of a pilot version. The Quality Index had high internal consistency (KR-20: 0.89) as did the subscales apart from external validity (KR-20: 0.54). Test-retest (r 0.88) and inter-rater (r 0.75) reliability of the Quality Index were good. Reliability of the subscales varied from good (bias) to poor (external validity). The Quality Index correlated highly with an existing, established instrument for assessing randomised studies (r 0.90). There was little difference between its performance with non-randomised and with randomised studies. Raters took about 20 minutes to assess each paper (range 10 to 45 minutes). CONCLUSIONS: This study has shown that it is feasible to develop a checklist that can be used to assess the methodological quality not only of randomised controlled trials but also non-randomised studies. It has also shown that it is possible to produce a checklist that provides a profile of the paper, alerting reviewers to its particular methodological strengths and weaknesses. Further work is required to improve the checklist and the training of raters in the assessment of external validity.
OBJECTIVE: To test the feasibility of creating a valid and reliable checklist with the following features: appropriate for assessing both randomised and non-randomised studies; provision of both an overall score for study quality and a profile of scores not only for the quality of reporting, internal validity (bias and confounding) and power, but also for external validity. DESIGN: A pilot version was first developed, based on epidemiological principles, reviews, and existing checklists for randomised studies. Face and content validity were assessed by three experienced reviewers and reliability was determined using two raters assessing 10 randomised and 10 non-randomised studies. Using different raters, the checklist was revised and tested for internal consistency (Kuder-Richardson 20), test-retest and inter-rater reliability (Spearman correlation coefficient and sign rank test; kappa statistics), criterion validity, and respondent burden. MAIN RESULTS: The performance of the checklist improved considerably after revision of a pilot version. The Quality Index had high internal consistency (KR-20: 0.89) as did the subscales apart from external validity (KR-20: 0.54). Test-retest (r 0.88) and inter-rater (r 0.75) reliability of the Quality Index were good. Reliability of the subscales varied from good (bias) to poor (external validity). The Quality Index correlated highly with an existing, established instrument for assessing randomised studies (r 0.90). There was little difference between its performance with non-randomised and with randomised studies. Raters took about 20 minutes to assess each paper (range 10 to 45 minutes). CONCLUSIONS: This study has shown that it is feasible to develop a checklist that can be used to assess the methodological quality not only of randomised controlled trials but also non-randomised studies. It has also shown that it is possible to produce a checklist that provides a profile of the paper, alerting reviewers to its particular methodological strengths and weaknesses. Further work is required to improve the checklist and the training of raters in the assessment of external validity.
Authors: William A Ghali; Jacques Cornuz; Finlay A McAlister; Jean-Blaise Wasserfallen; P J Devereaux; C David Naylor Journal: CMAJ Date: 2002-04-30 Impact factor: 8.262
Authors: Geisa Cristina da Silva Alves; Samuel Dutra da Silva; Virginia Paula Frade; Danielle Rodrigues; André de Oliveira Baldoni; Whocely Victor de Castro; Cristina Sanches Journal: Eur J Clin Pharmacol Date: 2017-08-04 Impact factor: 2.953
Authors: Darren Er Warburton; Sarah Charlesworth; Adam Ivey; Lindsay Nettlefold; Shannon Sd Bredin Journal: Int J Behav Nutr Phys Act Date: 2010-05-11 Impact factor: 6.457
Authors: Nancy K Grote; Jeffrey A Bridge; Amelia R Gavin; Jennifer L Melville; Satish Iyengar; Wayne J Katon Journal: Arch Gen Psychiatry Date: 2010-10
Authors: Glynnis A McDonnell; Christina G Salley; Marie Barnett; Antonio P DeRosa; Rachel S Werk; Allison Hourani; Alyssa B Hoekstra; Jennifer S Ford Journal: J Adolesc Health Date: 2017-07-17 Impact factor: 5.012
Authors: Kora DeBeck; Tessa Cheng; Julio S Montaner; Chris Beyrer; Richard Elliott; Susan Sherman; Evan Wood; Stefan Baral Journal: Lancet HIV Date: 2017-05-14 Impact factor: 12.767
Authors: Dean T Eurich; Finlay A McAlister; David F Blackburn; Sumit R Majumdar; Ross T Tsuyuki; Janice Varney; Jeffrey A Johnson Journal: BMJ Date: 2007-08-30