Richard S Rosenberg1, Steven Van Hout2. 1. California State University, Long Beach, CA. 2. American Academy of Sleep Medicine, Darien, IL.
Abstract
STUDY OBJECTIVES: The American Academy of Sleep Medicine (AASM) Inter-scorer Reliability program provides a unique opportunity to compare a large number of scorers with varied levels of experience to determine agreement in the scoring of respiratory events. The objective of this paper is to examine areas of disagreement to inform future revisions of the AASM Manual for the Scoring of Sleep and Associated Events. METHODS: The sample included 15 monthly records, 200 epochs each. The number of scorers increased steadily during the period of data collection, reaching more than 3,600 scorers by the final record. Scorers were asked to identify whether an obstructive, mixed, or central apnea; a hypopnea; or no event was seen in each of the 200 epochs. The "correct" respiratory event score was defined as the score endorsed by the most scorers. Percentage agreement with the majority score was determined for each epoch and the mean agreement determined. RESULTS: The overall agreement for scoring of respiratory events was 93.9% (κ = 0.92). There was very high agreement on epochs without respiratory events (97.4%), and the majority score for most of the epochs (87.8%) was no event. For the 364 epochs scored as having a respiratory event, overall agreement that some type of respiratory event occurred was 88.4% (κ = 0.77). The agreement for epochs scored as obstructive apnea by the majority was 77.1% (κ = 0.71), and the most common disagreement was hypopnea rather than obstructive apnea (14.4%). The agreement for hypopnea was 65.4% (κ = 0.57), with 16.4% scoring no event and 14.8% scoring obstructive apnea. The agreement for central apnea was 52.4% (κ = 0.41). A single epoch was scored as a mixed apnea by a plurality of scorers. CONCLUSIONS: The study demonstrated excellent agreement among a large sample of scorers for epochs with no respiratory events. Agreement for some type of event was good, but disagreements in scoring of apnea vs. hypopnea and type of apnea were common. A limitation of the analysis is that most of the records had normal breathing. A review of controversial events yielded no consistent bias that might be resolved by a change of scoring rules.
STUDY OBJECTIVES: The American Academy of Sleep Medicine (AASM) Inter-scorer Reliability program provides a unique opportunity to compare a large number of scorers with varied levels of experience to determine agreement in the scoring of respiratory events. The objective of this paper is to examine areas of disagreement to inform future revisions of the AASM Manual for the Scoring of Sleep and Associated Events. METHODS: The sample included 15 monthly records, 200 epochs each. The number of scorers increased steadily during the period of data collection, reaching more than 3,600 scorers by the final record. Scorers were asked to identify whether an obstructive, mixed, or central apnea; a hypopnea; or no event was seen in each of the 200 epochs. The "correct" respiratory event score was defined as the score endorsed by the most scorers. Percentage agreement with the majority score was determined for each epoch and the mean agreement determined. RESULTS: The overall agreement for scoring of respiratory events was 93.9% (κ = 0.92). There was very high agreement on epochs without respiratory events (97.4%), and the majority score for most of the epochs (87.8%) was no event. For the 364 epochs scored as having a respiratory event, overall agreement that some type of respiratory event occurred was 88.4% (κ = 0.77). The agreement for epochs scored as obstructive apnea by the majority was 77.1% (κ = 0.71), and the most common disagreement was hypopnea rather than obstructive apnea (14.4%). The agreement for hypopnea was 65.4% (κ = 0.57), with 16.4% scoring no event and 14.8% scoring obstructive apnea. The agreement for central apnea was 52.4% (κ = 0.41). A single epoch was scored as a mixed apnea by a plurality of scorers. CONCLUSIONS: The study demonstrated excellent agreement among a large sample of scorers for epochs with no respiratory events. Agreement for some type of event was good, but disagreements in scoring of apnea vs. hypopnea and type of apnea were common. A limitation of the analysis is that most of the records had normal breathing. A review of controversial events yielded no consistent bias that might be resolved by a change of scoring rules.
Authors: C W Whitney; D J Gottlieb; S Redline; R G Norman; R R Dodge; E Shahar; S Surovec; F J Nieto Journal: Sleep Date: 1998-11-01 Impact factor: 5.849
Authors: Stephen D Pittman; Mary M MacDonald; Robert B Fogel; Atul Malhotra; Koby Todros; Baruch Levy; Amir B Geva; David P White Journal: Sleep Date: 2004-11-01 Impact factor: 5.849
Authors: Subooha Zafar; Indu Ayappa; Robert G Norman; Ana C Krieger; Joyce A Walsleben; David M Rapoport Journal: Chest Date: 2005-01 Impact factor: 9.410
Authors: Richard B Berry; Rohit Budhiraja; Daniel J Gottlieb; David Gozal; Conrad Iber; Vishesh K Kapur; Carole L Marcus; Reena Mehra; Sairam Parthasarathy; Stuart F Quan; Susan Redline; Kingman P Strohl; Sally L Davidson Ward; Michelle M Tangredi Journal: J Clin Sleep Med Date: 2012-10-15 Impact factor: 4.062
Authors: Susan Redline; Rohit Budhiraja; Vishesh Kapur; Carole L Marcus; Jason H Mateika; Reena Mehra; Sariam Parthasarthy; Virend K Somers; Kingman P Strohl; Loreto G Sulit; David Gozal; Merrill S Wise; Stuart F Quan Journal: J Clin Sleep Med Date: 2007-03-15 Impact factor: 4.062
Authors: Ulysses J Magalang; Ning-Hung Chen; Peter A Cistulli; Annette C Fedson; Thorarinn Gíslason; David Hillman; Thomas Penzel; Renaud Tamisier; Sergio Tufik; Gary Phillips; Allan I Pack Journal: Sleep Date: 2013-04-01 Impact factor: 5.849
Authors: Thijs E Nassi; Wolfgang Ganglberger; Haoqi Sun; Abigail A Bucklin; Siddharth Biswal; Michel J A M van Putten; Robert J Thomas; M Brandon Westover Journal: IEEE Trans Biomed Eng Date: 2022-05-19 Impact factor: 4.756
Authors: Martin Glos; AbdelKebir Sabil; Katharina Sophie Jelavic; Christoph Schöbel; Ingo Fietze; Thomas Penzel Journal: J Clin Sleep Med Date: 2018-03-15 Impact factor: 4.062
Authors: Huy Phan; Fernando Andreotti; Navin Cooray; Oliver Y Chen; Maarten De Vos Journal: IEEE Trans Biomed Eng Date: 2018-10-22 Impact factor: 4.538
Authors: Shelley R Berson; Jaclyn Klimczak; Elizabeth A Prezio; Shirley Hu; Manoj Abraham Journal: Int Forum Allergy Rhinol Date: 2018-02-20 Impact factor: 3.858
Authors: Renan C Castello-Branco; Thiago Cerqueira-Silva; Alisson L Andrade; Beatriz M M Gonçalves; Camila B Pereira; Iuri F Felix; Leila S B Santos; Louise M Porto; Maria E L Marques; Marilia B Catto; Murilo A Oliveira; Paulo R S P de Sousa; Pedro J R Muiños; Renata M Maia; Saul Schnitman; Jamary Oliveira-Filho Journal: J Am Heart Assoc Date: 2020-03-13 Impact factor: 5.501