Optimal Outcomes Reporting (OOR): A New Value-Based Metric for Outcome Reporting Following Cleft Palate Repair.

OBJECTIVE: Palatoplasty outcome measurements vary widely among institutions. A standardized outcome metric would help provide quality benchmarks.
DESIGN: Retrospective review of primary palatoplasty patients from 2007 to 2013.
SETTING: Tertiary care children's hospital. MAIN OUTCOME MEASURES: We created a novel conceptual quality metric called "OOR" (Optimal Outcome Reporting). Optimal Outcome Reporting is designed to reflect the percentage of patients with cleft palate who experience the best outcomes: one operation, velar competence by age 5 years, and no unintended palatal fistula.
RESULTS: Optimal Outcome Reporting was 72.3% (68/94). Eight patients had "suboptimal" outcomes for having undergone more than one operation. Eighteen patients failed for velar incompetence. No additional patients fell out of the algorithm for fistula. A significantly higher proportion of nonsyndromic patients demonstrated an "optimal" result compared to syndromic patients (61/80, 76.3% vs 7/14, 50.0%; P = .04). Patients who required more than one procedure had significantly more clinic visits (32.6 vs 14.9; P < .01) and accrued higher costs compared to "optimal" patients (US$34 019.88 vs US$15 357.25; P < .01).
CONCLUSIONS: Optimal Outcome Reporting represents a novel quality metric that can provide meaningful information for patients with cleft palate. Optimal Outcome Reporting utilization can help cleft centers adopt changes that matter to patients and their families. By allowing for cross-institutional comparisons in a clear and objective manner, OOR can promote competition, innovation, and value in cleft palate care.

Introduction

The primary goal of cleft palate repair is to establish adequate palatal and velar structure and function for the subsequent production of normal speech (Kummer et al., 2019). Institutional and national database reviews over the last few decades have demonstrated surgical success based largely on (1) postoperative speech evaluations and (2) the absence of clinically relevant fistulae (Furlow, 1986; Marrinan et al., 1998; Kirschner et al.,1999; LaRossa et al., 2004; Salyer et al., 2006; Andrades et al., 2008; Sullivan et al., 2009; Jackson et al., 2013; Ahl and Harding-Bell, 2018; Jodeh et al., 2019; Kummer et al., 2019). However, current outcome studies vary significantly in inclusion criteria (ie, age range, types and severity of clefts, fistulas, syndromes, etc) and methods (ie, surgical techniques, speech assessment protocols, and use of patient-reported outcome measure). Most concerning is that there is substantial variation in what is considered a successful speech outcome. Although most clinicians agree that normal speech is the ultimate goal, a subjective judgment of “acceptable,” intelligible,” and even “improved” speech is considered a successful outcome in many centers (Kummer et al., 2012; Kummer et al., 2019). Lastly, burden of care related to the number of surgical procedures needed to achieve the ultimate speech outcome is not uniformly reported. The diversity in cleft palate outcomes reporting makes comparisons difficult and quality benchmarking impossible.

A shared concept of “success” that remains constant, despite variations in clinical care, would provide useful information to patients with clefts and their providers. Common quality metrics can facilitate a shared language for value-based health care discussions and can ultimately lead to improved processes that benefit our patients. Michael Porter, from the Harvard Business School, has been a proponent of measuring outcomes that matter to the patient by creating simple, understandable metrics that demonstrate care quality and drive competition and improvement (Porter and Lee, 2016; Porter et al., 2016; Patient Protection and Affordable Care Act, 2020). From the patient and family perspective, the singular goal of the cleft palate repair—the optimal outcome—is to have one palate procedure, resulting in normal velar function and no fistula. Because this is what all families and patients want, this should be the goal of every cleft center.

This study introduces that concept in the form of a new metric: What percentage of patients with cleft palate obtain velar competence after only one cleft palate procedure and have no fistula? We have called this the “Optimal Outcome Reporting” (OOR) metric and have used this with our patients in this study. We believe that this metric is a simple way to report results of surgical care for cleft palate, communicate with patients and families, and finally, benchmark among cleft centers in order to improve the way we care for patients. By adding a detailed clinical and cost analysis, this metric can create a common language to effectively and accurately compare outcomes that are most important to the patient and family.

Methods

We conducted a retrospective review of all patients who had undergone a primary palatoplasty for cleft palate by a single surgeon (J.G.M.) from 2007 through 2013. This review included patients with a syndromic codiagnosis. All patients had at least 5 years of postoperative follow-up. A speech assessment performed by our speech-language pathologists was conducted by an average of 5 years of age. Demographic and clinical data included age at operation, ethnicity, sex, Veau classification, and assessments of hypernasality and audible nasal escape. Patients with a submucous clefts were not included in the study.

We then applied the OOR metric using a sequential branching tree algorithm to all patients presenting with a cleft palate. With the OOR system, patients were assigned to the “optimal outcome” category only if they met all 3 of the following requirements:

One palate operation: Patients who required more than one procedure on their palate for any reason related to cleft care (eg, velopharyngeal insufficiency (VPI) operation or fistula repair) did not meet this requirement.

Competent speech: Velopharyngeal function was assessed at the 5-year visit by our speech and language pathologists using a formal resonance assessment. We elected to use the most stringent criteria to determine palatal function: patients who exhibited any hypernasality OR any audible nasal escape were not considered “optimal.”

No fistula: Patients with evidence of an unintended oronasal fistula did not meet this requirement. Anterior palatal and alveolar fistulae (Pittsburgh VI and VII) were not considered in our analysis. We routinely delay repair of fistulae in these locations until the time of alveolar bone grafting.

Each patient captured in our review also underwent a detailed clinical and cost analysis from birth to age 7 years. Each clinic visit was counted under the appropriate specialty. Cost for the professional components of the visit were based on standardized Current Procedural Terminology codes that the provider used to report the medical, surgical, and diagnostic procedures and services rendered. Data for direct and indirect costs were broken out to single item categories (ie, room and board, laboratory, other diagnostics, pharmaceuticals, therapies, etc) and were taken from a chargemaster for the facility component based on actual utilization at the point of care. Relative value units (RVUs) are assigned as the primary basis for allocating costs. In addition, actual direct costs, time-based values, and industry RVUs are integrated where possible. For example, a time-based value includes the calculation of operating room activity based on a fixed per minute variable. Costs were updated monthly to reflect market changes using the hospital accounting software package (Strata Decision Technology).

Differences between the “optimal” group and the “suboptimal” group were analyzed using t tests for continuous variables and χ2 tests for categorical variables. All statistics were performed using SPSS v24 (IBM Corp).

Results

Ninety-four patients were identified who underwent primary cleft palate repair by one surgeon (J.G.M.) between 2007 and 2013 and had adequate follow-up for a speech evaluation at 5 years of age. All palates were repaired using a single-stage, 2-flap palatoplasty with intravelar veloplasty in a modified Sommerlad style. Vomerine flaps were used for nasal lining when necessary. Table 1 depicts the demographics of this study population. There was an even distribution of cleft severity by Veau type, with a slight predominance of Veau I (31.9%). Slightly more than half of all patients were female (55.3%) and Caucasian (54.3%).

Table 1.

Patient Demographics.a

Median age at operation, mean (SD)	10.0 (11.4)
Female sex, n (%)	52 (55.3)
Cleft type, n (%)
Veau I	30 (31.9)
Veau II	19 (20.2)
Veau III	26 (27.7)
Veau IV	19 (20.2)
Race/ethnicity, n (%)
Caucasian	51 (54.3)
African American	4 (4.3)
Asian	16 (17.1)
Hispanic	3 (3.2)
Middle Eastern	3 (3.2)
Other	3 (3.2)

Abbreviation: SD, standard deviation.

a n = 94.

All patients underwent categorization as “optimal” or “suboptimal” based on our predefined criteria. The flowchart algorithm is presented in Figure 1. Our total OOR was 72.3% (68/94). Eight patients had “suboptimal” outcomes for having undergone more than one operation associated with their cleft palate. Seven (87.5%) of those were secondary operations for VPI and 1 (12.5%) was for palatal fistula and dehiscence. At the second stage of the OOR evaluation, 18 of 86 patients failed for suboptimal speech by demonstrating either hypernasality or audible nasal escape on resonance evaluation. No additional patients fell out of the algorithm at the third stage because no additional patients demonstrated postoperative fistulae. One patient who demonstrated a fistula required a reoperation and was thus already excluded in the first stage of this iterative process and considered “non optimal.”

Figure 1.

Flowchart for patient categorization. Optimal Outcome Reporting (OOR) describes the “best case scenario” in cleft palate treatment, for example, patients who present with a cleft palate and have one operation, velar competence, and no postoperative fistulae.

Within cleft types, patients with Veau I cleft palates had the highest OOR percentage at 80.0% (24/30). Patients with Veau II palates had a 57.9% OOR (11/19), those with Veau III had 88.5% (23/26), and Veau IV patients demonstrated an OOR of 52.6% (10/19). The OOR rate was significantly different across all cleft types (P = .02).

Table 2 depicts differential characteristics between “optimal” and “suboptimal” patient groups. There were no significant differences between the populations with respect to sex or age at palatoplasty. A significantly higher proportion of nonsyndromic patients demonstrated an “optimal” result compared to those patients with a documented syndromic codiagnosis (61/80, 76.3% vs 7/14, 50.0%; P = .04), and a significantly higher proportion of “suboptimal” patients had a syndromic diagnosis (7/26, 26.9% vs 7/68, 10.3%; P = .04).

Table 2.

Differences Between “Optimal” and “Suboptimal” Populations.

Patient Variables	Optimal	Suboptimal	P value
N	68	26	NA
Mean age at repair, mean (SD)	13.1 (11.0)	16.9 (12.2)	.15
Female sex, n (%)	38 (55.9)	14 (53.8)	.86
Syndromic diagnosis, n (%)	7 (10.3)	7 (26.9)	.04
Cleft type, n (%)
Veau I	24 (35.3)	6 (23.1)	.26
Veau II	11 (16.2)	8 (30.8)	.12
Veau III	23 (33.8)	3 (11.5)	.03
Veau IV	10 (14.7)	9 (34.6)	.03

Abbreviation: NA, not applicable; SD, standard deviation.

We calculated the number of clinic visits and costing data for each patient in the study through age 7 years. Patients who required more than one procedure for their palate had a significantly higher number of clinic visits compared to “optimal” patients (32.6 vs 15.6, respectively; P < .01). There was no significant difference in the number of clinic visits between patients who incompetent velar mechanisms following palate repair, but did not receive an additional operation, and “optimal” outcome patients (17.9 vs 14.7, respectively; P = .19).

Average total treatment costs associated with those patients who required more than one procedure for their cleft palate were significantly higher than “optimal” patients (US$34 019.88 vs US$15 455.55, respectively; P < .01). There was no statistically significant difference in the mean aggregate costs between patients with incompetent postoperative velar mechanisms that did not need another operation and “optimal” patients (US$15 828.61 vs US$15 327.92, respectively; P = .19). Differences in cost and clinic visits between the groups are depicted in Figure 2.

Figure 2.

Number of clinic visits, total cost, and speech outcomes for the study population (N = 91). Each bubble represents an individual patient, and the size of the bubbles is relative to the number of clinic visits each patient had. For those patients with suboptimal outcomes who underwent a secondary speech operation, the arrows indicate the magnitude to which their subsequent speech score improved (secondary and tertiary speech scores).

Discussion

This study describes the use of a novel quality metric called OOR, which can be used for the assessment and reporting of velopharyngeal outcomes following cleft palate surgery. With this metric, an “optimal outcome” is defined as the achievement of a competent velopharyngeal mechanism, as determined by the absence of both hypernasality AND audible nasal escape, following one cleft palate operation in addition to the absence of an unintended fistula.

The OOR metric was used in a retrospective review of all patients who had undergone a primary palatoplasty from 2007 through 2013 with a single surgeon. Results of this review revealed achievement of an optimal outcome in 76.3% of nonsyndromic patients and 50% of syndromic patients. The Veau classification was significant. Differences in OOR between age at palate repair were not statistically significant in this study, but the trend suggests that palatoplasty prior to 12 months may be associated with a higher OOR in all patients.

Although we can report our results in a consistent way using the OOR metric, we cannot compare our results with other centers or surgeons who use different methods for assessing and reporting outcomes. This is because, as Kummer and colleagues reported (Kummer et al., 2019), there are significant inconsistencies in how outcomes are measured and reported among cleft palate centers. There are even fundamental differences in the definition of success; some centers consider “normal’ speech as a successful outcome, whereas others include “improved,” “acceptable,” and intelligible” speech as a successful outcome (Kummer et al., 2012).

The Pittsburgh Weighted Score for Speech (PWS) has traditionally been used at our institution to provide a reference benchmark during the post-palatoplasty speech assessment. However, the PWS has not been universally accepted as a gold standard, and it can be influenced by articulation errors that do not adequately reflect velopharyngeal function. In an attempt to address these inherent limitations, we selected the most restrictive measures of success with respect to velopharyngeal function. If patients exhibited any degree of hypernasality or audible nasal escape during their resonance evaluation at age 5 years, they were considered a “suboptimal” result. Some patients with mild and even moderate hypernasality can eventually improve without a subsequent speech operation, but we decided to define our outcomes to reflect what most practitioners would consider to be an ideal result. Until centers compare outcomes in the same manner, preferably by comparing the percentage of patients with the optimal outcome, benchmarking for success will remain impossible.

In this study, we also compared the average number of visits and the total treatment costs for those patients who achieved the optimal outcome with one surgery versus those who required more than one procedure. As expected, the number of visits and overall costs were significantly higher for those patients who required more than one procedure as compared to those who only needed a single surgery. Achieving competent velopharyngeal function for speech in one surgical procedure rather than 2 or more operations certainly improves patient/family satisfaction and the patient’s quality of life, but it also significantly decreases the overall costs and burden of care.

With a recent focus on health care reform in the United States, beginning with the Affordable Care Act (2010), health care providers are challenged to achieve exceptional outcomes and a high degree of patient/family satisfaction while at the same time, decrease the overall cost of care (Affordable Care Act, 2019). Therefore, consistent outcome measurement and reporting is essential when evaluating surgery designed to achieve adequate velopharyngeal function for speech. In addition, collection of cost data is an important part of the measurement of value. Collecting these data will allow for quality benchmarking and value-based competition, which will provide incentives for all providers to improve the quality of their care while decreasing the cost (Porter, 2010; Porter et al., 2016; Porter and Lee, 2016).

The powerful aspect to the OOR metric is that it includes a measurement of positive surgical outcomes for velopharyngeal function and also a measurement of the cost to achieve those outcomes (based on the number of procedures and clinic visits). Those centers that have the best surgical outcomes with the fewest surgical procedures and therefore, lowest costs would serve as models for others and promote competition and learning based on overall value. This will ultimately result in better care by all providers and would help to meet the overall aim of health care reform by improving outcomes while decreasing costs.

Limitations

The limitations of this study include the fact that there was a relatively small cohort of patients and most of the patient cohort is English speaking. Further studies will be required to define the OOR metric in other languages. Another limitation is that the PWSS is not the universally accepted standard for speech evaluation and requires a subjective analysis. In an effort to combat this, we elected to focus on the most stringent criteria for success that would also most accurately reflect velopharyngeal function. Several different speech and language pathologists were involved in the resonance assessments, and our data may be affected by variable interrater reliability between providers. In future research, the use of objective measures, such as nasometry, will help to further define the optimal speech outcome. In addition, reassessment at a later age may be helpful to determine whether the outcomes are stable. Additionally, because the sample size is small, a multivariate analysis could not be performed to tease out the relative contributions of syndromic diagnosis versus Veau classification to the OOR metric. Regarding costing, accuracy of cost estimates could be limited by an inability to capture out of network charges such as speech pathology. Finally, it is well known that cost calculations using RVU-based algorithms are not as accurate as more granular “micro costing” methods such as time-driven activity-based costing (Abbott and Meara, 2011; Abbott et al., 2014; Kaplan et al., 2014).

Summary

We have proposed the use of a simple, understandable metric for assessing surgical outcomes after cleft palate repair. Using this metric, the percentage of patients who obtain the optimal outcome can be determined for individual surgeons, for various cleft palate centers, and even for various surgical procedures. This percentage can then be compared with others for the purpose of benchmarking and performance improvement. This metric can also be used for improved communication with families and patients. Benchmarking will increase our value to patients and families by driving improvements in speech outcomes, while at the same time, reducing the number of procedures to achieve the optimal outcome and, therefore, the cost. Payers (government, private sector, or self-pay) must be able to make informed health care purchasing decisions based on an overall value framework that includes outcomes that matter to the patient and the cost to deliver those outcomes.