Ruth Lebet1, Jennifer Hayakawa, Tracy B Chamblee, Joana A Tala, Nakul Singh, David Wypij, Martha A Q Curley. 1. Ruth Lebet, MSN, CCNS-P, is Lecturer, School of Nursing, University of Pennsylvania, Philadelphia. Jennifer Hayakawa, DNP, PCNS-BC, CNRN, CCRN, is Clinical Nurse Specialist, Pediatric Intensive Care Unit, CHOC Children's Hospital, Orange, California, and Clinical Faculty, Western University of Health Sciences, Pomona, California. Tracy B. Chamblee, PhD, APRN, PCNS-BC, is Clinical Nurse Specialist, Pediatric Intensive Care Unit, Children's Medical Center Dallas, Texas. Joana A. Tala, MD, is Research Coordinator, Pediatric Intensive Care Unit, Yale New Haven Hospital/Yale University, Connecticut. Nakul Singh, ScM, is Biostatistician, Department of Cardiology, Boston Children's Hospital, Massachusetts. David Wypij, PhD, is Senior Biostatistician, Department of Cardiology, Boston Children's Hospital; Associate Professor, Department of Pediatrics, Harvard Medical School; and Senior Lecturer, Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, Massachusetts. Martha A. Q. Curley, RN, PhD, FAAN, is Ellen and Robert Kapito Professor in Nursing Science, School of Nursing, University of Pennsylvania, Philadelphia, and Nurse Scientist, Boston Children's Hospital, Massachusetts.
Abstract
BACKGROUND: RESTORE (Randomized Evaluation of Sedation Titration for Respiratory Failure) was a cluster randomized clinical trial evaluating a sedation strategy in children 2 weeks to <18 years of age with acute respiratory failure supported on mechanical ventilation. A total of 31 U.S. pediatric intensive care units (PICUs) participated in the trial. Staff nurse rater agreement on measures used to assess a critical component of treatment fidelity was essential throughout the 4-year data collection period. OBJECTIVE: The purpose of the study is to describe the method of establishing and maintaining interrater agreement (IRA) of two core clinical assessment instruments over the course of the clinical trial. METHODS: IRA cycles were carried out at all control and intervention sites and included a minimum of five measurements of the State Behavioral Scale (SBS) and Withdrawal Assessment Tool-Version 1 (WAT-1). Glasgow Coma Scale scores were also obtained. PICUs demonstrating <80% agreement repeated their IRA cycle. Fleiss's kappa coefficient was used to assess IRA. RESULTS: Repeated IRA cycles were required for 8% of 226 SBS cycles and 2% of 222 WAT-1 cycles. Fleiss's kappa coefficients from more than 1,350 paired assessments were .86 for SBS and .92 for WAT-1, demonstrating strong agreement and similar to .91 for the Glasgow Coma Scale. There was no difference in Fleiss's kappa for any of the instruments based on unit size or timing of assessment (earlier or later in the study). For SBS scores, Fleiss's kappa was significantly different in larger and smaller PICUs (.82 vs. .92, p = .003); however, Fleiss's kappa for both groups indicated excellent agreement. CONCLUSION: Monitoring measurement reliability is an essential step in ensuring treatment fidelity and, thus, the validity of study results. Standardization on the use of these core assessment instruments among participating sites was achieved and maintained throughout the trial.
RCT Entities:
BACKGROUND: RESTORE (Randomized Evaluation of Sedation Titration for Respiratory Failure) was a cluster randomized clinical trial evaluating a sedation strategy in children 2 weeks to <18 years of age with acute respiratory failure supported on mechanical ventilation. A total of 31 U.S. pediatric intensive care units (PICUs) participated in the trial. Staff nurse rater agreement on measures used to assess a critical component of treatment fidelity was essential throughout the 4-year data collection period. OBJECTIVE: The purpose of the study is to describe the method of establishing and maintaining interrater agreement (IRA) of two core clinical assessment instruments over the course of the clinical trial. METHODS: IRA cycles were carried out at all control and intervention sites and included a minimum of five measurements of the State Behavioral Scale (SBS) and Withdrawal Assessment Tool-Version 1 (WAT-1). Glasgow Coma Scale scores were also obtained. PICUs demonstrating <80% agreement repeated their IRA cycle. Fleiss's kappa coefficient was used to assess IRA. RESULTS: Repeated IRA cycles were required for 8% of 226 SBS cycles and 2% of 222 WAT-1 cycles. Fleiss's kappa coefficients from more than 1,350 paired assessments were .86 for SBS and .92 for WAT-1, demonstrating strong agreement and similar to .91 for the Glasgow Coma Scale. There was no difference in Fleiss's kappa for any of the instruments based on unit size or timing of assessment (earlier or later in the study). For SBS scores, Fleiss's kappa was significantly different in larger and smaller PICUs (.82 vs. .92, p = .003); however, Fleiss's kappa for both groups indicated excellent agreement. CONCLUSION: Monitoring measurement reliability is an essential step in ensuring treatment fidelity and, thus, the validity of study results. Standardization on the use of these core assessment instruments among participating sites was achieved and maintained throughout the trial.
Authors: D G Altman; K F Schulz; D Moher; M Egger; F Davidoff; D Elbourne; P C Gøtzsche; T Lang Journal: Ann Intern Med Date: 2001-04-17 Impact factor: 25.391
Authors: Martha A Q Curley; David Wypij; R Scott Watson; Mary Jo C Grant; Lisa A Asaro; Ira M Cheifetz; Brenda L Dodson; Linda S Franck; Rainer G Gedeit; Derek C Angus; Michael A Matthay Journal: JAMA Date: 2015-01-27 Impact factor: 56.272
Authors: Nienke J Vet; Erwin Ista; Saskia N de Wildt; Monique van Dijk; Dick Tibboel; Matthijs de Hoog Journal: Intensive Care Med Date: 2013-06-19 Impact factor: 17.440
Authors: Linda S Franck; Sion Kim Harris; Deborah J Soetenga; June K Amling; Martha A Q Curley Journal: Pediatr Crit Care Med Date: 2008-11 Impact factor: 3.624
Authors: Carmen Flores-Pérez; Luis Alfonso Moreno-Rocha; Juan Luis Chávez-Pacheco; Norma Angélica Noguez-Méndez; Janett Flores-Pérez; María Fernanda Alcántara-Morales; Luz Cortés-Vásquez; Lina Sarmiento-Argüello Journal: Saudi Pharm J Date: 2022-05-23 Impact factor: 4.562
Authors: JoAnne E Natale; Lisa A Asaro; Jill G Joseph; Christine Ulysse; Judith Ascenzi; Cindy Bowens; David Wypij; Martha A Q Curley Journal: Ann Am Thorac Soc Date: 2021-01