Implementation of an Outcome Measure in Pediatric Behavioral Health: A Process Improvement Initiative.

INTRODUCTION: Efforts to monitor outcomes in pediatric behavioral health are becoming a quality, financial, and regulatory imperative. The implementation of a broad-based measure to assess patient functioning at the start of pediatric psychology services, as well as at subsequent visits, has not been demonstrated. This article describes the systematic implementation of a measure of health-related quality of life (HRQOL) to assess functional impairment across an entire clinic population using quality improvement science and methodologies.
METHODS: The Pediatric Quality of Life Inventory Generic Core 4.0 (PedsQL) was administered at initial and subsequent visits for all patients seeking treatment at a large, tertiary care pediatric psychology clinic in an academic pediatric medical center (Nationwide Children's Hospital, Columbus, Ohio). The goal of this project was to design a process change to support a 90% completion rate of this measurement tool by all clinicians.
RESULTS: Within 16 months, the completion rate of the PedsQL increased from a baseline of 39% to the identified goal of 90%. This process change was within control limits (over 80%) for over 12 months.
CONCLUSION: This study demonstrates the implementation of a systematic process for collection of outcome measures in a pediatric behavioral health care setting. Successful administration of an outcome measure at multiple time points during the care of children and adolescents in a large psychology clinic can allow for quantitative assessment of treatment progress and identify a pathway for administration of additional measures.

INTRODUCTION

Problem Description

High quality care is the goal of any clinician in service delivery and is an increasingly important focus in pediatric behavioral health care. Clinical standards that use evidence-based practices are the model for clinicians in the field of pediatric psychology, but the use of established measurement tools to assess the effectiveness of psychosocial interventions is lacking. In the context of the quality imperatives that exist in large pediatric academic medical settings that emphasize continuous improvement, monitoring toward best outcomes, and efficiency in clinical care, collecting data on treatment effectiveness is imperative. This study will demonstrate the capacity of a high volume pediatric psychology outpatient clinic to reliably collect a quality of care measure. The demonstration of process and practice change that supports collection of an outcome measure is a necessary step toward evaluating progress over time on a patient population presenting with a wide array of pediatric and psychosocial conditions.

Available Knowledge

The importance of developing procedures that will systematically monitor treatment progress will soon be an accreditation mandate. Beginning in 2018, the Joint Commission will require the use of a “standardized tool or instrument to monitor the individual’s progress in achieving his or her care, treatment, or service goals.” Health care providers will be expected to analyze data generated through standardized monitoring and use the results on both the individual and aggregate levels to inform individual’s treatment goals and to evaluate outcomes of the populations served.[1] It is critically important to demonstrate that such measures can be reliably completed and collected in pediatric psychology clinics.

Although choosing a measure to assess clinical improvement can be challenging, such measures are essential to assess best outcomes. Quality of life measures hold promise as tools to assess patient perception of treatment effectiveness, to help inform treatment decisions, and to provide guidance for treatment modifications.[2] Recent research suggests a link between health care improvement and health-related quality of life (HRQOL). Several studies have demonstrated the potential for HRQOL measures to assist in analysis of patient and parent experiences of care, as well as facilitate patient and physician communication.[3-5]

The Pediatric Quality of Life Inventory Generic Core 4.0 (PedsQL) is one example of a well-established measure that has been endorsed in pediatric behavioral health care and has been used more broadly in pediatric psychology clinics.[2] By accurately measuring a patient’s own perception of functioning, the PedsQL helps to determine comorbid emotional, social, and physical problems that can be addressed during treatment. This measure has also been cross validated with other measurement tools assessing physical and emotional functioning in patients with a medical overlay.[6] The PedsQL has been used widely among different medical populations, including chronic conditions such as renal disease, oncology, obesity, and numerous other medically complicated pediatric populations.[7-9] In recent years, disease-specific measures of quality of life have been developed and normed. Tailoring this measure to specific populations not only illustrates the versatility of the measure but helps to guide the treatment process. For example, one study has suggested links between parental overprotection and perceived child vulnerability among patients with pediatric cancer diagnoses.[10] Other studies have demonstrated disease-specific PedsQL to be associated with better treatment adherence among male patients with type 1 and 2 diabetes.[11]

Although disease-specific measures are promising, the applicability to general behavioral health care settings, where medical conditions may not be prominent, is limited. One study utilized the PedsQL and identified a correlation to other measures of anxiety and depression.[12] However, the focus of other quality improvement (QI) initiatives among mental health populations has been limited to symptom reduction and medication management,[13] not in assessing aspects of quality of life. Only one goal of effective behavioral health care is symptom reduction; an additional and important overarching goal is to reduce functional impairments and promote overall adjustment and adaptability. Therefore, a more comprehensive utilization of a quality of life measure used in treatment across a pediatric population with diverse medical and mental health diagnoses still needs to be demonstrated. The current project describes a process improvement initiative to implement the PedsQL in a pediatric psychology outpatient clinic treating a large range of medical and mental health conditions.

Before this QI initiative, there was no common metric to assess patient functioning upon entrance into treatment in our clinic. The aim of this quality improvement project was to implement a measure to assess patient functioning at the first visit and maintain utilization at specified visits during treatment across a complex medical population. By implementing at the initial visit, and subsequent sessions, the measure could provide a step toward optimizing a means to inform interventions and assess patient progress in treatment.

In summary, the study aims include:

Effective application of an HRQOL measure at multiple time periods for every new patient seeking treatment, regardless of diagnosis in a large pediatric psychology clinic;

Create and monitor a process for ongoing implementation; and,

Achieve consistency across practitioners in the utilization and input of this measure to a completion rate of 90%, thereby helping to identify a potential measure of treatment effectiveness.

METHODS

A quality improvement project was launched in the Division of Pediatric Psychology and Neuropsychology at Nationwide Children’s Hospital. The study followed procedures consistent with the Institutional Review Board guidelines at Nationwide Children’s Hospital and was ultimately considered exempt from the review process. This hospital is a large pediatric primary and tertiary hospital serving 37 of 88 counties in Ohio, with a population of more than 2 million people including contiguous counties, and a secondary service area of rural Appalachian southeastern Ohio. The Section of Psychology is integrated into 25 pediatric medical subspecialties and provides more than 23,000 visits per year. At the time of the project, over 800 new patients were seen yearly in the Psychology Department at Nationwide Children’s hospital. The top visit diagnoses in 2016 were mood and anxiety related to comorbid medical conditions, feeding difficulties, and pain symptoms across several populations, predominantly headache and abdominal pain. Other mental health conditions (tics, trichotillomania) are referred to this clinic. Columbus has a majority white population (62%) with the more populous racial and ethnic minority groups composed of African American (28%), Hispanic/Latino (5.6%), and Asian (4.1%). At the time this project was initiated, this clinic was staffed by 17 providers and 6 trainees, with 470 average monthly visits.

Measure

The PedsQL assesses several domains of functioning, including physical, social, emotional, and academic functioning. The PedsQL is widely used as it demonstrates good reliability for both the parent and child self-report measures[14] across a wide age range[15]; for example, children as young as 5 years of age who are able to accurately self-report HRQOL can complete a self-report.[15] Studies suggest good reliability between child and parent formats as well.[15-17] Thus, this measure appears to adequately assess a variety of areas of functioning among a wide array of patients. This measure has also shown effectiveness in assessment of functioning for patients with primary presentation of mental health needs, such as depressive symptomatology.[9,17,18] The PedsQL has validity in numerous populations to assess the significance of an acute or chronic condition on a person’s functioning.[12] Guidelines for clinical cutoffs of the PedsQL have been suggested to determine need for services, as these scores can be compared with healthy controls and other disease-specific populations.[19,20]

Procedures

All patients were provided with the PedsQL at registration for their first appointment and at designated times during treatment. Non–English-speaking families had the assistance of an interpreter who was present at all visits.

Interventions

A multidisciplinary team consisting of psychologists, operations staff, electronic health record (EHR) report writers and a Quality Improvement coordinator was convened on several occasions to identify the steps needed for establishing consistent assessment. Because of the large size of the department, 2 plan-do-study-act (PDSA) cycles were conducted to determine the best approach at initial implementation across the department. The decision was made to roll out this process change in phases. Identification of a small subset of clinicians who would also serve as consultants for the data collection process was the first step. Subsequently, the process was translated to the entire department after the second PDSA cycle was completed.

Registration staff fulfilled a primary role for this project. Considering this team of 4 individuals already provided other measures to targeted patient populations, they were instrumental in standardizing a process for administration of the HRQOL measure across the department. In addition to having direct contact with all patients, registration staff also had direct contact with clinicians. In this way, they were able to ensure administration and completion of measures that clinicians may overlook within the context of a treatment relationship (Fig. 1).

Fig. 1.

Process map developed by quality improvement team to determine key drivers and identify interventions.

During the first PDSA cycle, the team developed a process map from which a key driver diagram was created (Figs. 1, 2). Several interventions were identified to: (1) target patients needing assessment measures at specified visits; (2) determine a standardized process of administration; (3) create a mechanism whereby family members completed the form during identified session, and (4) guarantee entry of data into medical records.

Early on, the importance of automating this process by use of EHR was deemed necessary, given the large number of patients registered at any time. Using computer-generated data, families identified as needing the measure were noted on the registration page, and adequate time was given for completion. This also allowed for forms to be linked correctly with each patient, and time given for the family to complete this measure to minimize burden. By providing this questionnaire at registration, clinicians could review answers during their visit.

A third PDSA cycle was conducted midway through process change implementation to further improve completion rate. An automated report was generated to determine existence of PedsQL entry from the previous week. By providing timely information, study staff and registration were able to work with clinicians on efficient means to enter data. As some clinicians did not routinely enter data, operation staff began to make duplicate copies of this measure for a subset of clinicians, to further alleviate missing data. This helped to ensure consistent entry of the measure, which may be impacted by clinician mishandling or resistance.

The fourth visit was identified as the targeted visit to gather follow-up data. The team arrived at this session number by looking at a subset of patients who participated in a standardized treatment for pain and concussion symptoms. An average of 4 visits for this particular population was found to match the overall average of outpatients visits seen in the department as a whole. Once this time point was established, registration staff was educated on the process flow to assure administration occurred at these visits (Fig. 2).

Fig. 2.

Aim and key drivers diagram for instituting QI project to increase the rate of completion of PedsQL questionnaire in the section of pediatric psychology and neuropsychology.

To monitor progress of administration, the percentage of PedsQLs entered at every fourth visit was compared with the number of total of patients who should have received and entered this measure for the targeted visit. This value only included PedsQL scores entered into the EHR. Statistics were calculated for the outcome measure, frequency of data collection, and the data source. Automated reports ran data completion rates on a weekly basis, whereas the control chart reflected monthly numbers recorded on a p-chart.[21] Baseline data for initial 8 months was entered into the p-chart and a centerline (average) and control limits (+/- 3 SD, representing inherent process variation) were calculated. As the QI project progressed, monthly data was entered into the p-chart but holding the baseline constant. Statistical process control (SPC) rules were applied to detect special cause variation versus common cause variation and to determine when to shift the centerline.

RESULTS

From idea to inception, this project took 2 years to establish consistency. As represented in the p-chart in Figure 3, completion of PedsQL by families increased from 39% during initiation of this study among the subset of clinicians to 75% in 6 months, after training all the participating psychologists and psychology trainees (P < 0.000). Compliance of 90% completion rate (P < 0.001) was established within 16 months by enlisting registration staff to the process of identifying missing PedsQL forms (Fig. 2). On average, this measure captured ratings for 77 families per month (Fig. 3).

Fig. 3.

Annotated control chart (p-chart) with percentage of PedsQL scores documented in EHR at every fourth visit from February 2013 to February 2016. Two centerline shifts were achieved: first shift from 39% to 75% when PedsQL was expanded to all the psychologists and trainees and a second shift to 90% completion rates when the registration staff was enlisted to identify missing PedsQL forms.

The control chart reveals 2 process shifts, highlighting key improvements to this project. Engagement efforts to educate clinicians on procedures was necessary on several occasions to increase awareness about using the measure consistently. Change behaviors were further supported by an automated report highlighting missing data, which led to subsequent interventions to improve measurement collection procedures. Use of the process map suggested other areas of delegation by which supportive staff could aid clinicians in obtaining the data from families. In addition, short cuts for data entry were created as well as tables reflecting results of the data collected. This allowed for clinicians to access clinical data, which may have also improved motivation to enter data in a timely manner.

A special cause was identified, with a noticeable dip in data collection following the implementation of the process shift. This change in data collection coincided with shift in leadership, such that focus of this project also had a momentary shift, resulting in lost data. This revealed the need to incorporate a system of checks needed to ensure standardization within a changing environment.

DISCUSSION

The goal of this QI project was to create a comprehensive and consistent process change in a hospital-based psychology clinic for administration and collection of a measure monitoring functional impairment and quality of life assessment. Establishing process accountability was paramount when implementing new interventions across a large department of professionals. This project demonstrated the ability to introduce and maintain a process that administered the measure, sustaining that process for over a year, at or above 80% compliance. Providing measures consistently over the course of treatment, and in a streamlined fashion among numerous clinicians, could have a significant impact in the enhancement of evidence-based care across a wide variety of patients.

Several steps were necessary to improve this process. Weekly feedback regarding data entry, periodic checks on process implementation, and education of staff to promote flexibility collectively led to process improvement. In this project, identifying key staff members to be involved in process change allowed for a more thorough analysis of whether changes were, in fact, improving methods of measurement collection. In turn, this same team helped establish reliable methods to maintain change. Introducing automated data collection further decreased human error and omissions, thus creating standards by which all patients could be assessed. The automated report also identified potential barriers for data entry and allowed for time to add data or recover missing data. The QI team was instrumental in improving these gaps in collection.

Clear communication of procedural changes was essential to maintain compliance, despite significant shifts in personnel. As indicated on the control chart, disruptions in data collection occurred when project leadership changed. However, if a process is to be sustainable long-term, it should be able to recover quickly despite environmental shifts. This study demonstrated robustness in this process change and revealed possibility for the introduction of other outcome measures to further assess treatment outcome.

Challenges and Limitations

The implementation of the PedsQL was met with several challenges and limitations, most notably during the maintenance phase. The importance of the project was not always apparent, or felt as relevant, to clinicians or clinic staff, especially as leadership shifted. Changes in personnel who distributed the measure complicated the sustainability of process improvement. In addition, as expected in any quality improvement project, strategies of implementation need to be adjusted periodically to achieve greater consistency. Future efforts exploring use of tablet-based questionnaires that can be uploaded directly to the patient’s EHR, thus eliminating paper formats and clinician-based errors, will allow for greater flexibility and sustainability.

Variability of usage exhibited by the treating psychologists was also a significant challenge. Incomplete and inconsistent responsiveness from clinicians to be accountable for collection of the PedsQL and data entry required multiple changes during the implementation of this project. Re-education of the process to clarify the importance and steps in the collection and data entry procedures occurred verbally in one-on-one contexts or in staff meetings, which was a likely limitation to this study. Other educational formats or visual reminders may have resulted in more engagement and less error. Moreover, clinicians may have benefitted from ongoing updates of study results to demonstrate measurement utility, thus impacting collective ownership toward the common goal of implementation. In other words, sharing the results of the process, as improvements and inconsistencies happened, may have helped draw awareness to the overarching goal and reason behind the project. This idea of collective engagement has been successful in another study looking at global measure implementation across a treatment setting, by having clinicians provide ongoing self-reflection and input about implementation.[22] In addition, by revealing the next steps of longer range departmental goals to look at the results of these treatment progress measures over time would have strengthened the rationale of this current QI effort and helped with accountability around a central, shared initiative.

Another limitation to the current study is the absence of a balance measure. Although this project successfully created a process for PedsQL collection, there was no metric to assess the burden that practitioners may have experienced when required to enter this data. Sampling feedback from providers may have forged conversations with project leaders on ways to create more streamlined entry of the data. In addition, sampling potential burden on patients and families would have provided useful information for possible modifications needed in the future.

Future Goals

With the success of this process change for measurement collection in our clinic population, QI projects identifying the clinical benefits of the PedsQL is the next step to establishing a measurement of treatment progress. Recent efforts consistent with Joint Commission standards to monitor treatment effectiveness have begun in 2 small subset of patients seeking treatment for headache and concussion. Collecting and examining the longitudinal assessment of functional disability among particular patient groups as well as comprehensively has the potential to inform treatment planning and improve therapeutic gains. Further, our plan is to use this measure in conjunction with ratings of our patients’ experience of care will help to validate the value of this measure as an indicator of treatment success.

In addition, efforts have begun across our behavioral health setting to gain access to electronic version of this and other measures, thus allowing for direct entry into the medical chart once EHR technology is available. This will minimize error as well as ensure thoroughness of data entry. For instance, one study did demonstrate that use of electronic screeners lessened burden on families and was more efficient to administer, thus allowing for the potential of added measures in the future.[23]

There are additional benefits that come along with developing an effective strategy for monitoring patient progress in pediatric behavioral health care. As financial pressures grow in health care, including limits on reimbursement, measuring effectiveness of care to ensure improved outcomes will likely become essential for any practice. For instance, it has been argued that health care providers should be driven by performance incentives that emphasize more patient-centered outcomes such as quality of care, clinical outcomes, and patient experiences of care.[24] Tools to identify risks in our patients can facilitate focused interventions within a cost-effective context and promote faster access to care, a timely application in today’s health care system.[1] Also, with quality measures in place, clinicians and payors can better compare the effectiveness of 2 or more treatment approaches and payment plans can also be improved over time.[24]

CONCLUSIONS

This project describes a systematic approach to achieving implementation of a clinic-wide behavioral health care measure, which will be useful to many organizations needing to comply with new Joint Commission standards on collecting and using outcome measures. This current study suggests feasibility in systematic implementation of the PedsQL over a large pediatric psychology clinic caring for a medically complicated patient population. With a reliable measurement process in place, our organization is better prepared to identify and actualize effective treatment approaches that can achieve beneficial outcomes, perhaps in fewer sessions, by better informing clinicians of functional changes during treatment. Treatment goals can then be aligned with emerging health care revenue models such as pay for performance contracts and accreditation standards on behavioral health care outcome measurement.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.