Use of Computerized Adaptive Testing to Develop More Concise Patient-Reported Outcome Measures.

BACKGROUND: Patient-reported outcome measures (PROMs) are essential tools that are used to assess health status and treatment outcomes in orthopaedic care. Use of PROMs can burden patients with lengthy and cumbersome questionnaires. Predictive models using machine learning known as computerized adaptive testing (CAT) offer a potential solution. The purpose of this study was to evaluate the ability of CAT to improve efficiency of the Veterans RAND 12 Item Health Survey (VR-12) by decreasing the question burden while maintaining the accuracy of the outcome score.
METHODS: A previously developed CAT model was applied to the responses of 19,523 patients who had completed a full VR-12 survey while presenting to 1 of 5 subspecialty orthopaedic clinics. This resulted in the calculation of both a full-survey and CAT-model physical component summary score (PCS) and mental component summary score (MCS). Several analyses compared the accuracy of the CAT model scores with that of the full scores by comparing the means and standard deviations, calculating a Pearson correlation coefficient and intraclass correlation coefficient, plotting the frequency distributions of the 2 score sets and the score differences, and performing a Bland-Altman assessment of scoring patterns.
RESULTS: The CAT model required 4 fewer questions to be answered by each subject (33% decrease in question burden). The mean PCS was 1.3 points lower in the CAT model than with the full VR-12 (41.5 ± 11.0 versus 42.8 ± 10.4), and the mean MCS was 0.3 point higher (57.3 ± 9.4 versus 57.0 ± 9.6). The Pearson correlation coefficients were 0.97 for PCS and 0.98 for MCS, and the intraclass correlation coefficients were 0.96 and 0.97, respectively. The frequency distribution of the CAT and full scores showed significant overlap for both the PCS and the MCS. The difference between the CAT and full scores was less than the minimum clinically important difference (MCID) in >95% of cases for the PCS and MCS.
CONCLUSIONS: The application of CAT to the VR-12 survey demonstrated an ability to lessen the response burden for patients with a negligible effect on score integrity.

In modern orthopaedic medicine, patients are burdened by the administration of several questionnaires that are designed as data collection tools to obtain patient-reported outcome measures (PROMs). PROMs have several functions, including to aid research by assigning an overall function score, to evaluate the value of care through objective outcomes, and to direct treatment and reimbursement rates[1-3]. While the benefits of PROMs to the medical and scientific community are clear, there is demand from both the patient and the physician standpoint to develop more efficient PROMs that improve compliance while maintaining the integrity of the outcome score.

The Veterans RAND 12 Item Health Survey (VR-12) is an example of a widely used PROM that calculates physical and mental health outcome scores for patients receiving orthopaedic care (Table I). The VR-12 was developed from the Veterans RAND 36 Item Health Survey (VR-36) with use of extensive research to identify the 12 most important questions with the greatest influence on scoring variability[4]. Currently, it is one of the most popularly used general outcome measures, with applications across several orthopaedic subspecialties, and has been used to characterize subjects in numerous population-based studies[5-8]. As a result, reducing the question burden of this particular PROM could have a widespread impact on streamlining patient care.

Advances in data science have shown that the score of an outcome measure can be accurately predicted from fewer questions if the correct questions are asked. Predictive models, developed through a process known as computerized adaptive testing (CAT), offer a potential solution. The goal of CAT is to identify the correct subset of questions selected from the full questionnaire to ask each patient on the basis of on his/her previous responses. CAT is trained through so-called machine learning programs, also described as artificial intelligence, that analyze how response patterns affect overall outcome scores. The CAT model then uses its own recognition of these patterns to self-improve its efficiency and minimize question burden in an accurate manner. Technology for this purpose has been successfully developed and applied in other fields, demonstrating potential to effectively improve the patient experience[9,10].

Prior to real-time use, CAT models specific to each PROM must be validated by comparing the accuracy of scores generated using fewer questions with that of scores generated using the full questionnaires. A CAT version of the VR-12 was recently developed within the OBERD software system (Universal Research Solutions, www.oberd.com), a general tool used for outcome data collection. A CAT model developed for a specialty-specific PROM using the same software system was recently validated[11], but its application for other PROMs remains undetermined. The purpose of this study was to evaluate the success of this CAT model in improving VR-12 efficiency by (1) decreasing the question burden of responders and (2) maintaining accuracy of outcome scores.

Materials and Methods

This was a retrospective, single-institution analysis evaluating 19,523 patients presenting to 5 different orthopaedic clinics across 4 different subspecialties (shoulder and elbow, hand, sports medicine, and spine). Subjects with a variety of ages and diagnoses were included in the analysis (Fig. 1, Table II). Data were collected with OBERD, which we have used for several years to collect outcome data from patients. Through this process, the database has provided extensive resources for training and evaluating predictive models. During the initial visit to our orthopaedic surgery clinics, each patient completed a full VR-12 survey using OBERD by means of a tablet device (iPad; Apple), resulting in the baseline calculation of both a physical component summary score (PCS) and a mental component summary score (MCS). This CAT model developed for the VR-12 was trained using a random sample of 27,800 de-identified administrations of the VR-12 collected routinely by users of the software. The algorithms constructed by the CAT system through this training were then retrospectively applied to each set of patient responses stored on the instrument (i.e., not live while the patients were answering the survey). The 19,523 responses were not part of the original training set that helped develop the algorithms. Beginning with the first VR-12 item, the CAT takes each set of responses through a range of VR-12 items guided by previous answers. This resulted in the calculation of a CAT-specific score of both the PCS and the MCS for each individual subject.

Fig. 1

Distribution of ages of patients included in the CAT model analysis of the VR-12.

The decrease in question burden was measured by assessing the percentage difference in the number of questions between the CAT model and the full survey. Several statistical methods were used to compare the accuracy of the CAT scores with the full scores, derived by analyses recommended by Bland and Altman[12]. These methods included (1) comparing the means and standard deviations (SDs) of both sets of scores, (2) calculating a Pearson correlation coefficient to measure the strength of the linear correlation between scores, (3) calculating an intraclass correlation coefficient to determine the extent to which score differences were explained by inherent variability of the VR-12, (4) plotting the frequency distributions of scores for the CAT and full model against one another, (5) plotting the distribution of the score differences (full score minus CAT score) for analysis, and finally (6) generating a Bland-Altman plot to assess the patterns in score differences. Analyses were performed with the R software suite (version 3.4.2; R Foundation for Statistical Computing), with the Python programming language (version 3.4.5; Python Software Foundation), or using Microsoft Excel spreadsheets.

The accuracy of CAT was viewed in the context of the minimal clinically important difference (MCID) for the VR-12, which is the minimum deviation in score that must occur for a noticeable change in health to be present. Previous literature has supported use of a 6-point change in the MCS and PCS as the MCID[13,14].

Results

While the full VR-12 form comprises 12 questions, the CAT model required 8 questions to be answered in its application for all subjects, representing a 33% decrease in question burden. The model found the most useful initial question to be Question 6c (Table I), so the algorithm started with this question for each application. Questions 2b, 3a, 4b, and 6a were eliminated by CAT.

The mean CAT score was 1.3 points lower than the mean full score for the PCS (41.5 ± 11.0 versus 42.8 ± 10.4) and 0.3 point higher for the MCS (57.3 ± 9.4 versus 57.0 ± 9.6), with very similar SDs. The Pearson correlation coefficients were 0.97 for the PCS and 0.98 for the MCS, representing strong linear relationships between scores, and the intraclass correlation coefficients were 0.96 and 0.97, respectively, indicating strong agreement between scores as well. For each individual practice cohort, these values varied no more than 0.02. The mean scores for the individual sites are provided in Tables III and IV. The distribution of the CAT and full scores showed significant overlap for both the PCS (Fig. 2) and the MCS (Fig. 3). For the PCS, the difference between the CAT score and the full score was less than the MCID in >95% of cases (Fig. 4), and demonstrated a slight skew of the CAT to underestimate the score. For the MCS, the difference between the CAT score and the full score was also less than the MCID in >95% of cases (Fig. 5), with the differences evenly clustered around zero.

Fig. 2

Distribution of PCS scores on the full (long) VR-12 (orange) overlaid with the distribution of the PCS scores on the CAT model (blue). Green shows where the full and CAT scores are the same.

Fig. 3

Distribution of MCS scores on the full (long) VR-12 (orange) overlaid with the distribution of the MCS scores on the CAT model (blue). Green shows where the full and CAT scores are the same.

Fig. 4

Distribution of the differences between the full VR-12 and CAT PCS scores. Less than 5% of the absolute values of score differences were greater than the MCID. The differences are clustered around zero with very slight bias for lower CAT scores relative to full scores.

Fig. 5

Distribution of the differences between the full VR-12 and CAT MCS scores. Less than 5% of the absolute values of score differences were greater than the MCID. Most of the differences are clustered around zero.

The Bland-Altman plot demonstrated that the differences between the CAT and the full VR-12 scores were largely independent of the overall score for both the PCS (Fig. 6) and the MCS (Fig. 7), although a slight decrease in the score difference was demonstrated at the overall score extremes (highest and lowest scores). This pattern is more identifiable in the PCS Bland-Altman plot than in the MCS plot. However, a greater score error was unbiased toward either single extreme. In other words, greater score errors were not seen with higher scores compared with lower scores, or vice versa, for either the PCS or the MCS.

Fig. 6

Bland-Altman plot of the difference between the CAT and full VR-12 PCS scores versus the mean of the 2 scores for each case. Most of the score differences are less than the MCID of 6 points, indicating that the CAT would not affect clinical interpretation of the outcomes. The differences in scores are shown to be slightly decreased at the overall score extremes, but bias is not seen toward larger versus smaller scores or vice versa.

Fig. 7

Bland-Altman plot of the difference between the CAT and full VR-12 MCS scores versus the mean of the 2 scores for each case. Most of the score differences are less than the MCID of 6 points, indicating that the CAT would not affect clinical interpretation of the outcomes. The differences in scores are shown to be independent of the overall score (i.e., no bias of greater differences at higher or lower scores).

Discussion

PROMs are an essential part of orthopaedic care. The VR-12 is particularly useful as a well-validated, non-proprietary, and relatively short outcome measure compared with other PROMs. Recent literature has shown that it has become an increasingly popular measure for characterizing outcomes of hip and knee arthroplasty[15-17] as well as various arthroscopic procedures including rotator cuff repair and SLAP (superior labral tear from anterior to posterior) repair[18-20]. It has additional practical value due to its integration in the Medicare Health Outcomes Survey and its use by the Centers for Medicare & Medicaid Services as a quality-of-life measure and source of performance assessment[21,22]. In this role, it is used to assess the quality of programs including the Veterans Affairs, Medicare Advantage, and other health-care plans, and thereby contributes to the direction of reimbursements and other financial incentives for various plans and providers. Given this utility and widespread application, countless patients are asked to complete the VR-12 survey in its full form every day as a fixture of participation in the health-care system. For this reason, the CAT model was developed in an effort to help reduce the question burden placed on patients, thereby improving the patient experience by shifting focus away from data collection instruments and toward patient-driven goals and patients’ relationship with their physician.

In order to determine the value of the CAT model for this purpose, we first evaluated its ability to decrease question burden for potential responders. Our analysis showed that the model required a fixed number of questions for each response set with a uniform decrease in question burden (33%) across the board compared with the full VR-12. It remains unclear, however, to what degree the actual gross reduction in question burden would improve the patient experience. On the basis of public reporting of survey completion time, we can estimate that removal of 4 questions saves an average of about 140 seconds in response time. While these time savings may seem trivial when the survey is viewed as a singular event, this perception fails to take into account the demands placed on a patient during a typical health-care visit. In this setting, patients are frequently asked to provide various categories of data, from personal biographical information to medical history to problem-specific questionnaire responses. Every engagement to streamline these individual burdens may benefit the process as a whole, and combined efforts could certainly create more substantial improvements in health-care efficiency.

The second part of our evaluation of CAT concerned the accuracy of its score outputs relative to the scores generated from the full VR-12 survey. The accuracy of these scores was viewed in the context of the MCID to ensure that the analysis was anchored in the reality of subjective patient experience. The MCID for the VR-12 has been shown to vary somewhat based on the method of calculation (i.e., distribution-based versus anchor-based approach) and the patient cohort, but we estimated the MCID to be 6 points for both the PCS and the MCS on the basis of a review of the available literature[13,14,23]. In this context, our results demonstrate that the scores strongly resemble each other not only in terms of summary statistics, including mean, SD, and Pearson correlation coefficient, but more importantly in terms of individual score breakdowns. The essential recreation of the score distributions by the CAT model was an important finding especially given that the score frequencies were not distributed normally. Additionally, for both the PCS and the MCS, the difference between the CAT score and the full VR-12 was less than the MCID in >95% of cases. This indicates that the CAT model outcome scores are faithful to the full version not just at the population level but also at the individual patient level. Lastly, in terms of test-retest reliability, the intraclass coefficients demonstrated a stronger agreement between the CAT and full VR-12 scores than between scores of the same measure administered to the same individual twice[24]. This suggests that there is likely more variability within the full VR-12 itself than between the full VR-12 and the CAT model. Taken together, these findings demonstrate support for the implementation of CAT in a live setting (while patients are responding to the survey) to elicit VR-12 outcome scores.

The CAT used by OBERD is distinct from alternative CAT systems, including those developed in PROMIS (Patient-Reported Outcomes Measurement Information System), which use methodology based on item response theory that requires a separate set of questions (i.e., “item bank”)[24,25]. The OBERD CAT, however, relies on machine learning methods rather than item response theory. It utilizes the questions of the historical forms rather than separate item banks to collect outcomes, making them interchangeable with the existing PROMs such as the VR-12. In its practical application from a patient perspective, the administration of the CAT strongly resembles that of the full questionnaire. Both questionnaires are completed using a tablet device in a private clinic room, and questions are delivered to the patient’s screen 1 at a time. The patient is unaware whether the questions are a fixed set or generated by a system of algorithms such as the CAT model, and thus the patient experience remains unchanged. This consistency is important as it has been demonstrated that the method of administration of general health surveys can affect the outcome score[26].

Because CAT was able to eliminate 4 questions from the VR-12, it would seem that the remaining 8 questions could generate a satisfactory “VR-8 short form,” but rigorous assessment of such a PROM awaits further study. An important element to consider for CAT is that, unlike standard PROMs, the order of questions may change from patient to patient on the basis of the earlier responses that they provided. As a result, theoretically the CAT model is valid only for PROMs in which the individual questions are independent from one another; in other words, when outcome scores would be the same regardless of the question order, which is not always the case. For example, it has been demonstrated that there may be a difference in how respondents answer a certain “energy” item depending on whether it is integrated in a 12-item (VR-12) or 36-item (Short Form [SF]-36) questionnaire[27]. Importantly, however, it has not been shown that a change in item order would affect responses within the VR-12. In fact, multiple methods have been validated for converting item-based responses of the VR-12 to relevant counterparts through scoring algorithms, which have supported the basis of item independence for this survey[21,25,27].

A few more limitations must be considered to place these results in proper context. By nature of being a retrospective comparative analysis, the model could be used only on previously stored responses and not in a live setting (while the patients were actually responding to the survey). Only comparative prospective studies will be able to confirm the assumptions of certain measure requirements, including item independence, and truly validate the CAT model’s fidelity to the full survey. Additionally, although our cohort included patients with a range of demographic and diagnostic characteristics to maximize generalizability of the CAT model, our population had a higher mean PCS (42.8) and MCS (57.0) compared with a recent report of the contemporary U.S. population (40.1 and 50.2, respectively)[7]. Therefore, our patient population likely represents a slightly healthier cohort than the general population. Additionally, although this study did not compare outcomes on the basis of demographic or diagnostic factors, there was a consistent level of agreement for both the MCS and the PCS at multiple clinic sites, each of which treated patients with a distinct demographic and diagnostic make-up (Table II). Determining whether these factors impact the accuracy of the CAT model likely requires a prospective comparative approach.

In conclusion, the CAT system was designed to incorporate machine learning algorithms into PROM collection in order to improve PROM efficiency and improve patient experience. In this study, the application of the CAT model to the VR-12 survey demonstrated an ability to lessen the response burden for patients and had very little impact on score integrity. Additional studies that validate CAT-generated scores for specialty-specific questionnaires will help determine the potential scope of the model’s application.

TABLE I

Questions and Response Options for Veterans RAND 12 Item Health Survey (VR-12)*

Question	Response Options
1. In general, would say your health is:	Excellent
	Very good
	Good
	Fair
	Poor
2. The following questions are about activities you might do during a typical day. Does your health now limit you in these activities? If so, how much?
a. Moderate activities, such as moving a table, pushing a vacuum cleaner, bowling, or playing golf.	Yes, limited a lot
	Yes, limited a little
	No, not limited at all
b. Climbing several flights of stairs.	Yes, limited a lot
	Yes, limited a little
	No, not limited at all
3. During the past 4 weeks, have you had any of the following problems with your work or other regular daily activities as a result of your physical health?
a. Accomplished less than you would like.	No, none of the time
	Yes, a little of the time
	Yes, some of the time
	Yes, most of the time
	Yes, all of the time
b. Were limited in the kind of work or other activities.	No, none of the time
	Yes, a little of the time
	Yes, some of the time
	Yes, most of the time
	Yes, all of the time
4. During the past 4 weeks, have you had any of the following problems with your work or other regular daily activities as a result of any emotional problems (such as feeling depressed or anxious)?
a. Accomplished less than you would like.	No, none of the time
	Yes, a little of the time
	Yes, some of the time
	Yes, most of the time
	Yes, all of the time
b. Didn’t do work or other activities as carefully as usual.	No, none of the time
	Yes, a little of the time
	Yes, some of the time
	Yes, most of the time
	Yes, all of the time
5. During the past 4 weeks, how much did pain interfere with your normal work (including both work outside the home and housework)?	Not at all
	A little bit
	Moderately
	Quite a bit
	Extremely
These questions are about how you feel and how things have been with you during the past 4 weeks. For each question, please give the one answer that comes closest to the way you have been feeling.
6. How much of the time during the past 4 weeks:
a. Have you felt calm and peaceful?	All of the time
	Most of the time
	A good bit of the time
	Some of the time
	A little of the time
	None of the time
b. Did you have a lot of energy?	All of the time
	Most of the time
	A good bit of the time
	Some of the time
	A little of the time
	None of the time
c. Have you felt downhearted and blue?	All of the time
	Most of the time
	A good bit of the time
	Some of the time
	A little of the time
	None of the time
7. During the past 4 weeks, how much of the time has your physical health or emotional problems interfered with your social activities (like visiting with friends, relatives, etc.)?	All of the time
	Most of the time
	Some of the time
	A little of the time
	None of the time

The Veterans RAND 12 Item Health Survey was developed from the Veterans RAND 36 Item Health Survey, which was developed and modified from the original RAND version of the 36-Item Health Survey version 1.0 (also known as the “MOS SF-36”). “VR-12: How to create VR-12 scales and PCS/MCS summaries” © 2014 by Trustees of Boston University. All rights reserved. (All questions should be directed to Professor Lewis Kazis, Boston University School of Public Health. E-mail: lek@bu.edu.)

TABLE II

Diagnostic Information of Patients Whose Stored Responses to the Full VR-12 Were Applied to the CAT Model

Site	No. of Patients	Common Diagnoses
1	3,163	Degenerative disc disease, intervertebral disc disorder, spinal stenosis
2	2,860	Rotator cuff sprain, osteoarthritis of shoulder, rupture of rotator cuff
3	7,593	Carpal tunnel syndrome, trigger finger, tenosynovitis
4	2,478	Tear of meniscus, transient synovitis of knee, disruption of knee ligament
5	3,429	Rotator cuff sprain, rupture of rotator cuff, osteoarthritis of shoulder

TABLE III

Summary of Statistical Data Comparing Accuracy of PCS of Full VR-12 with CAT Model at Different Sites

Site	No. of Patients	Mean ± SD		R	ICC*
		Full VR-12	CAT VR-12
1	3,163	37.1 ± 10.7	36.0 ± 10.8	0.97	0.97
2	2,860	43.0 ± 9.7	41.0 ± 10.5	0.97	0.95
3	7,593	45.8 ± 9.7	44.7 ± 10.5	0.97	0.96
4	2,478	41.8 ± 10.1	41.6 ± 10.1	0.97	0.97
5	3,429	41.7 ± 10.0	39.7 ± 10.7	0.97	0.95
Overall	19,523	42.8 ± 10.4	41.5 ± 11.0	0.97	0.96

ICC = intraclass correlation coefficient.

TABLE IV

Summary of Statistical Data Comparing Accuracy of MCS of Full VR-12 with CAT Model at Different Sites

Site	No. of Patients	Mean ± SD		R	ICC*
		Full VR-12	CAT VR-12
1	3,163	52.1 ± 11.2	52.5 ± 11.0	0.98	0.98
2	2,860	57.4 ± 8.7	58.0 ± 8.5	0.97	0.97
3	7,593	58.0 ± 8.6	59.1 ± 8.3	0.98	0.97
4	2,478	57.5 ± 9.1	57.8 ± 8.8	0.97	0.97
5	3,429	56.2 ± 9.6	56.8 ± 9.5	0.97	0.96
Overall	19,523	57.0 ± 9.6	57.3 ± 9.4	0.98	0.97

ICC = intraclass correlation coefficient.