Julia Vishnevetsky1, Chasity Burrows Walters2, Kay See Tan3. 1. Patient and Caregiver Engagement, Memorial Sloan Kettering Cancer Center, New York, USA. Electronic address: vishnevj@mskcc.org. 2. Patient and Caregiver Engagement, Memorial Sloan Kettering Cancer Center, New York, USA. 3. Epidemiology-Biostatistics, Memorial Sloan Kettering Cancer Center, New York, USA.
Abstract
OBJECTIVE: To assess the interrater reliability (IRR) and usability of the Patient Education Materials Assessment Tool (PEMAT) and the relationship between PEMAT scores and readability levels. METHODS: One hundred ten materials (80 print, 30 audiovisual) were evaluated, each by two raters, using the PEMAT. IRR was calculated using Gwet's AC1 and summarized across items in each PEMAT domain (understandability and actionability) and by material type. A survey was conducted to solicit raters' experience using the PEMAT. Readability of each material was assessed using the SMOG Index. RESULTS: The median IRR was 0.92 for understandability and 0.93 for actionability across all relevant items, indicating good IRR. Eight PEMAT items had Gwet's AC1 values less than 0.81. PEMAT and SMOG Index scores were inversely correlated, with a Spearman's rho of -0.20 (p=0.081) for understandability and -0.15 (p=0.194) for actionability. While 92% of raters agreed the PEMAT was easy to use, survey results suggested specific items for clarification. CONCLUSION: While the PEMAT demonstrates moderate to excellent IRR overall, amendments to items with lower IRR may increase the usefulness of the tool. PRACTICE IMPLICATIONS: The PEMAT is a useful supplement to reading level alone in the assessment of educational materials.
OBJECTIVE: To assess the interrater reliability (IRR) and usability of the Patient Education Materials Assessment Tool (PEMAT) and the relationship between PEMAT scores and readability levels. METHODS: One hundred ten materials (80 print, 30 audiovisual) were evaluated, each by two raters, using the PEMAT. IRR was calculated using Gwet's AC1 and summarized across items in each PEMAT domain (understandability and actionability) and by material type. A survey was conducted to solicit raters' experience using the PEMAT. Readability of each material was assessed using the SMOG Index. RESULTS: The median IRR was 0.92 for understandability and 0.93 for actionability across all relevant items, indicating good IRR. Eight PEMAT items had Gwet's AC1 values less than 0.81. PEMAT and SMOG Index scores were inversely correlated, with a Spearman's rho of -0.20 (p=0.081) for understandability and -0.15 (p=0.194) for actionability. While 92% of raters agreed the PEMAT was easy to use, survey results suggested specific items for clarification. CONCLUSION: While the PEMAT demonstrates moderate to excellent IRR overall, amendments to items with lower IRR may increase the usefulness of the tool. PRACTICE IMPLICATIONS: The PEMAT is a useful supplement to reading level alone in the assessment of educational materials.
Authors: Angela G Brega; Megan A G Freedman; William G LeBlanc; Juliana Barnard; Natabhona M Mabachi; Maribel Cifuentes; Karen Albright; Barry D Weiss; Cindy Brach; David R West Journal: J Health Commun Date: 2015
Authors: Erin Sarzynski; Hamza Hashmi; Jeevarathna Subramanian; Laurie Fitzpatrick; Molly Polverento; Michael Simmons; Kevin Brooks; Charles Given Journal: BMJ Qual Saf Date: 2016-05-06 Impact factor: 7.035
Authors: Ndidi I Unaka; Angela Statile; Julianne Haney; Andrew F Beck; Patrick W Brady; Karen E Jerardi Journal: J Hosp Med Date: 2017-02 Impact factor: 2.960
Authors: Amr Salama; Janet Panoch; Elhaam Bandali; Aaron Carroll; Sarah Wiehe; Stephen Downs; Mark P Cain; Richard Frankel; Katherine H Chan Journal: J Pediatr Urol Date: 2019-12-04 Impact factor: 1.830
Authors: Mary Kate Skalitzky; Trevor R Gulbrandsen; William Lorentzen; Burke Gao; Alan G Shamrock; Stuart L Weinstein; Jose A Morcuende Journal: Iowa Orthop J Date: 2021
Authors: Stephanie A Hooker; Michelle D Sherman; Katie A Loth; Marc James A Uy; Andrew H Slattengren Journal: J Clin Psychol Med Settings Date: 2022-01-20
Authors: Burke Gao; Alan G Shamrock; Trevor R Gulbrandsen; Olivia C O'Reilly; Kyle R Duchman; Robert W Westermann; Brian R Wolf Journal: Orthop J Sports Med Date: 2022-07-28
Authors: Carissa Bonner; Michael Anthony Fajardo; Samuel Hui; Renee Stubbs; Lyndal Trevena Journal: J Med Internet Res Date: 2018-02-01 Impact factor: 5.428
Authors: Trevor Robert Gulbrandsen; Mary Kate Skalitzky; Alan Gregory Shamrock; Burke Gao; Obada Hasan; Benjamin James Miller Journal: JMIR Cancer Date: 2022-03-24