Scott C Moore1, Katerine Osatuke1, Steven R Howe1. 1. Scott C Moore, Katerine Osatuke, United States Veterans Health Administration, National Center for Organization Development, Cincinnati, OH 45249, United States.
Abstract
AIM: To present a conceptual and measurement strategy that allows to objectively, sensitively evaluate intervention progress based on data of participants' perceptions of presenting problems. METHODS: We used as an example an organization development intervention at a United States Veterans Affairs medical center. Within a year, the intervention addressed the hospital's initially serious problems and multiple stakeholders (employees, management, union representatives) reported satisfaction with progress made. Traditional quantitative outcome measures, however, failed to capture the strong positive impact consistently reported by several types of stakeholders in qualitative interviews. To address the paradox, full interview data describing the medical center pre- and post- intervention were examined applying a validated theoretical framework from another discipline: Psychotherapy research. The Assimilation model is a clinical-developmental theory that describes empirically grounded change levels in problematic experiences, e.g., problems reported by participants. The model, measure Assimilation of Problematic Experiences Scale (APES), and rating procedure have been previously applied across various populations and problem types, mainly in clinical but also in non-clinical settings. We applied the APES to the transcribed qualitative data of intervention participants' interviews, using the method closely replicating prior assimilation research (the process whereby trained clinicians familiar with the Assimilation model work with full, transcribed interview data to assign the APES ratings). The APES ratings summarized levels of progress which was defined as participants' assimilation level of problematic experiences, and compared from pre- to post-intervention. RESULTS: The results were consistent with participants' own reported perceptions of the intervention impact. Increase in APES levels from pre- to post-intervention suggested improvement, missed in the previous quantitative measures (the Maslach Burnout Inventory and the Work Environment Scale). The progress specifically consisted of participants' moving from the APES stages where the problematic experience was avoided, to the APES stages where awareness and attention to the problems were steadily sustained, although the problems were not yet fully processed or resolved. These results explain why the conventional outcome measures failed to reflect the intervention progress; they narrowly defined progress as resolution of the presenting problems and alleviation of symptomatic distress. In the Assimilation model, this definition only applies to a sub-segment of the change continuum, specifically the latest APES stages. The model defines progress as change in psychological processes used in response to the problem, i.e., a growing ability to deal with problematic issues non-defensively, manifested differently depending on APES stages. At early stages, progress is an increased ability to face the problem rather than turning away. At later APES stages, progress involves naming, understanding and successfully addressing the problem. The assimilation approach provides a broader developmental context compared to exclusively symptom, problem-, or behavior- focused approaches that typically inform outcome measurement in interpersonally based interventions. In our data, this made the difference between reflecting (APES) vs missing (Maslach Burnout Inventory, Work Environment Scale) the pre-post change that was strongly perceived by the intervention recipients. CONCLUSION: The results illustrated a working solution to the challenge of objectively evaluating progress in subjectively experienced problems. This approach informs measuring change in psychologically based interventions.
AIM: To present a conceptual and measurement strategy that allows to objectively, sensitively evaluate intervention progress based on data of participants' perceptions of presenting problems. METHODS: We used as an example an organization development intervention at a United States Veterans Affairs medical center. Within a year, the intervention addressed the hospital's initially serious problems and multiple stakeholders (employees, management, union representatives) reported satisfaction with progress made. Traditional quantitative outcome measures, however, failed to capture the strong positive impact consistently reported by several types of stakeholders in qualitative interviews. To address the paradox, full interview data describing the medical center pre- and post- intervention were examined applying a validated theoretical framework from another discipline: Psychotherapy research. The Assimilation model is a clinical-developmental theory that describes empirically grounded change levels in problematic experiences, e.g., problems reported by participants. The model, measure Assimilation of Problematic Experiences Scale (APES), and rating procedure have been previously applied across various populations and problem types, mainly in clinical but also in non-clinical settings. We applied the APES to the transcribed qualitative data of intervention participants' interviews, using the method closely replicating prior assimilation research (the process whereby trained clinicians familiar with the Assimilation model work with full, transcribed interview data to assign the APES ratings). The APES ratings summarized levels of progress which was defined as participants' assimilation level of problematic experiences, and compared from pre- to post-intervention. RESULTS: The results were consistent with participants' own reported perceptions of the intervention impact. Increase in APES levels from pre- to post-intervention suggested improvement, missed in the previous quantitative measures (the Maslach Burnout Inventory and the Work Environment Scale). The progress specifically consisted of participants' moving from the APES stages where the problematic experience was avoided, to the APES stages where awareness and attention to the problems were steadily sustained, although the problems were not yet fully processed or resolved. These results explain why the conventional outcome measures failed to reflect the intervention progress; they narrowly defined progress as resolution of the presenting problems and alleviation of symptomatic distress. In the Assimilation model, this definition only applies to a sub-segment of the change continuum, specifically the latest APES stages. The model defines progress as change in psychological processes used in response to the problem, i.e., a growing ability to deal with problematic issues non-defensively, manifested differently depending on APES stages. At early stages, progress is an increased ability to face the problem rather than turning away. At later APES stages, progress involves naming, understanding and successfully addressing the problem. The assimilation approach provides a broader developmental context compared to exclusively symptom, problem-, or behavior- focused approaches that typically inform outcome measurement in interpersonally based interventions. In our data, this made the difference between reflecting (APES) vs missing (Maslach Burnout Inventory, Work Environment Scale) the pre-post change that was strongly perceived by the intervention recipients. CONCLUSION: The results illustrated a working solution to the challenge of objectively evaluating progress in subjectively experienced problems. This approach informs measuring change in psychologically based interventions.