Raymond W Tse1,2, Eugene Oh1,2, Joseph S Gruss1,2, Richard A Hopper1,2, Craig B Birgfeld1,2. 1. Seattle, Wash. 2. From the Division of Plastic Surgery, Department of Surgery, University of Washington; and the Division of Craniofacial and Plastic Surgery, Seattle Children's Hospital.
Abstract
BACKGROUND: Lack of convenient and reliable methods to grade aesthetic outcomes limits the ability to study results and optimize treatment of unilateral cleft lip. Crowdsourcing methods solicit contributions from a large group to achieve a greater task. The authors hypothesized that crowdsourcing could be used to reliably grade aesthetic outcomes of unilateral cleft lip. METHODS: Fifty deidentified photographs of 8- to 10-year-old subjects (46 with unilateral cleft lip and four controls) were assembled. Outcomes were assessed using multiple pairwise comparisons that produced a rank order (Elo rank) of nasal appearance and, on a separate survey, by Asher-McDade ratings. Both surveys were repeated to assess reliability. A group of expert surgeons repeated the same tasks on a smaller subset of photographs. RESULTS: The authors obtained 2500 and 1900 anonymous, layperson evaluations by means of crowdsourcing on each Elo rank and Asher-McDade survey, respectively. Elo rank and Asher-McDade scores were highly reproducible (correlation coefficients, 0.87 and 0.98), and crowd evaluations agreed with those by expert surgeons (0.980 and 0.96 for Elo rank and Asher-McDade score, respectively). Crowdsourcing surveys were completed within 9 hours, whereas the expert surgeons required 3 months. On further analysis of their cleft subject sample set, the authors found that greater initial cleft severity was associated with worse aesthetic outcome. CONCLUSIONS: Outcomes assessed by crowds were reliable and correlated well with expert assessments. Crowdsourcing allows acquisition of massive numbers of layperson assessments on an unprecedented scale, and is a convenient, rapid, and reliable means of assessing aesthetic outcome of treatment for unilateral cleft lip. CLINICAL QUESTION/LEVEL OF EVIDENCE: Diagnostic, IV.
BACKGROUND: Lack of convenient and reliable methods to grade aesthetic outcomes limits the ability to study results and optimize treatment of unilateral cleft lip. Crowdsourcing methods solicit contributions from a large group to achieve a greater task. The authors hypothesized that crowdsourcing could be used to reliably grade aesthetic outcomes of unilateral cleft lip. METHODS: Fifty deidentified photographs of 8- to 10-year-old subjects (46 with unilateral cleft lip and four controls) were assembled. Outcomes were assessed using multiple pairwise comparisons that produced a rank order (Elo rank) of nasal appearance and, on a separate survey, by Asher-McDade ratings. Both surveys were repeated to assess reliability. A group of expert surgeons repeated the same tasks on a smaller subset of photographs. RESULTS: The authors obtained 2500 and 1900 anonymous, layperson evaluations by means of crowdsourcing on each Elo rank and Asher-McDade survey, respectively. Elo rank and Asher-McDade scores were highly reproducible (correlation coefficients, 0.87 and 0.98), and crowd evaluations agreed with those by expert surgeons (0.980 and 0.96 for Elo rank and Asher-McDade score, respectively). Crowdsourcing surveys were completed within 9 hours, whereas the expert surgeons required 3 months. On further analysis of their cleft subject sample set, the authors found that greater initial cleft severity was associated with worse aesthetic outcome. CONCLUSIONS: Outcomes assessed by crowds were reliable and correlated well with expert assessments. Crowdsourcing allows acquisition of massive numbers of layperson assessments on an unprecedented scale, and is a convenient, rapid, and reliable means of assessing aesthetic outcome of treatment for unilateral cleft lip. CLINICAL QUESTION/LEVEL OF EVIDENCE: Diagnostic, IV.
Authors: Giap H Vu; Anthony Azzolini; Laura S Humphries; Daniel M Mazzaferro; Christopher L Kalmar; Carrie E Zimmerman; Jordan W Swanson; Jesse A Taylor; Scott P Bartlett Journal: Plast Reconstr Surg Glob Open Date: 2020-07-15
Authors: Nirvana S S Kornmann; Robin A Tan; Frans J Mulder; Joseph T Hardwicke; Bruce M Richard; Brian B Pigott; Ronald W Pigott Journal: Cleft Palate Craniofac J Date: 2018-11-21
Authors: Abdulrahman Takiddin; Mohammad Shaqfeh; Osman Boyaci; Erchin Serpedin; Mitchell A Stotland Journal: Plast Reconstr Surg Glob Open Date: 2022-01-18
Authors: Alexandra Bucknor; Joani Christensen; Parisa Kamali; Sabine Egeler; Charlotte van Veldhuisen; Hinne Rakhorst; Irene Mathijssen; Samuel J Lin; Heather Furnas Journal: Plast Reconstr Surg Glob Open Date: 2018-04-16