Measurement Equivalence of "Touch-Screen" versus "Paper-Based" Assessments of OHRQoL: A Randomized Crossover Trial.

PURPOSE: To determine the measurement equivalence of computer touch screen assessment (CTSA) and paper based assessment (PBA) of the oral health impact profile (OHIP-14). PATIENTS AND METHODS: A randomized crossover trial was conducted. Sixty participants were randomized to either i) Arm A: completed CTSA then PBA of OHIP-14, or ii) Arm B: PBA and then CTSA of OHIP-14 within the same day. User preference and time taken to complete the assessments were recorded. Agreement between CTSA and PBA was determined using directional difference (DD), absolute difference (AD), and intraclass correlation coefficient (ICC).
RESULTS: There was no significant difference in CTSA and PBA OHIP-14 scores (P>0.05). The magnitude of the DD in scores between assessment methods was small for overall scores and all domains (<0.3). The AD in OHIP-14 scores was small (~6% for overall score, between 8-16% for domains). Agreement between CTSA and PBA was high (ICC=0.9; 95% CI=0.8-0.9) for overall OHIP-14 scores, but ICC values varied across domains. Most (78%) preferred CTSA. There was no significant difference in time taken to complete assessments (P=0.09). Regression analyses did not identify any significant socio-demographic factor associated with absolute difference between CTSA and PBA scores.
CONCLUSION: There is equivalence of measurements in OHRQoL assessments from CTSA and PBA, and the time taken to complete assessment by either means is similar. There is a greater preference for CTSA. This has implications to support the use of CTSA in OHRQoL assessments.

Introduction

There is an acceptance of the importance of patient-reported outcome measures (PROMS) in assessing oral health needs and in determining outcomes from care.1 To this end, assessment of oral health-related quality-of-life (OHRQoL) is increasingly employed in population health surveys, in research and in clinical practice.2,3 Traditionally, OHRQoL assessments have been conducted through “paper-based assessments” (PBA). However, PBA have many disadvantages and limitations, particularly when collecting large amounts of data. In PBA, respondents frequently either miss items or mark an item ambiguously, resulting in frequently “missing data”.4 Furthermore, post-data collection PBA have to be scanned or responses manually inputted into statistical packages for analyses, and this may further increase the likelihood of error in the data.5 Moreover, PBA are labor intensive to produce, distribute, and collect; thus not surprisingly more costly to use in research and clinical practice.6

In the new millennium with advances in computer technologies, it has become feasible to collect data by alternative means; namely computerized touch-screen assessments (CTSA). Arguably, CTSA offers several advantages over PBA to ensure high quality data sets (less errors in data entry), less missing data and greater efficiency.7,8 What remains unclear is whether CTSA and PBA questionnaires provide similar assessment outcomes, especially in terms of OHRQoL assessments.

We aimed to determine the measurement equivalence of CTSA and PBA of OHIP-14 (one of the most commonly used OHRQoL measures) specifically to determine agreement at “the group level” and “individual level”. Furthermore, to determine participants’ preference of the modes of OHRQoL assessment and to determine differences (if any) in time taken to complete OHRQoL assessments. Finally, to identify if any socio-demographic factors were associated with absolute differences in OHRQoL assessments (OHIP-14 scores) from CTSA and PBA.

Patients and Methods

Research Design and Sample

This study was a randomized crossover trial (Trial registration: ClinicalTrials.gov NCT02108470; National Medical Research Register NMRR-12-1289-13605). The study was conducted in accordance with the Declaration of Helsinki. Participants (60) were recruited from residential colleges at a university campus in April 2014. The only inclusion criteria were that participants could read and speak Malay. Participants were randomly assigned through block randomization in groups of four (ABBA) into two arms of the trial: i) Arm A, where participants self-completed the CTSA followed by the PBA; or ii) Arm B, where participants self-completed PBA followed by CTSA. The first assessment was conducted in the morning (AM) session and the second assessment was conducted in the afternoon (PM) session. The period of “washout” was a “lunch period” between morning and afternoon sessions. Assignment to trial arm was concealed in envelopes that were opened when written informed consent was obtained from participants. Participation was entirely voluntary but participants did receive a small token in-lieu of their time. A sample size of 60 participants was determined based on the size necessary to assess agreement using the intraclass correlation coefficient (ICC). The null hypothesis for the ICC was set at 0.2 (poor agreement) and the ICC was set at 0.6 as moderate agreement (indicating a significant level of agreement). Consequently, with α at 0.05 and β at 0.2, the minimum number of participants needed was 30 per group; a total of 60 subjects.9 The study was approved by the local institutional review board (Medical Ethics Committee, University of Malaya DF CO 1205/0074(L).

Data Collection

OHRQoL assessments were conducted employing the Malaysian version of OHIP-14 (S-OHIP [M]) comprising 14 items across seven domains (functional limitation, physical pain, psychological discomfort, physical disability, psychological disability, social disability, and handicap).10,11 The response to each item was recorded on a five-point Likert scale representing an event “not having occurred” to “occurring all of time/most of the time” within the past month.

PBA involved “pen and paper” for participants to self-complete questions. Data from PBA were entered manually into the computer package of SPSS for the analyses. For the CTSA a Samsung tablet with Android operating system was used. The respondents entered their responses by touching the relevant buttons on the screen. The navigation buttons at the bottom of the screen allowed the screen to be moved backwards and forwards through the questions. The responses were compiled automatically into an excel file that could be accessed in SPSS. The formats of the questions were similar in both CTSA and PBA. Burden on participants was assessed in two ways; i) time and ii) preference. Time taken for participants to complete PBA and CTSA was recorded by stopwatch. On completion of the trial participants were asked to rate their preference for PBA or CTSA (touch screen, paper, or no preference). Details of their age, gender, and ethnicity was recorded (Figure 1).

Figure 1

Flow chart of data collection.

Data Analysis

Statistical analysis was carried out using statistical packages for social sciences software (IBM SPSS, version 20, USA). Data cleaning was performed prior to data analysis. Imputation was carried out for missing data or “do not know“ responses using the mean score of overall respondents for each of the 14 questions. Descriptive statistics of summary total and domain scores (mean (SD) values) were produced and the profile of the participants determined. Agreements with OHIP-14 between PBA and CTSA were determined in a number of ways i) to identify if there was a significant difference in scores between the assessment methods; ii) to calculate the mean directional difference (MDD) in scores (mean of values obtained from PBA scores minus CTSA scores) and to determine the magnitude of this by calculation effect sizes (ES = MDD/SD of the MDD); iii) by calculating the mean absolute difference (the difference between PBA and CTSA in a positive integer); and iv) by determining the Intraclass Correlation Coefficient (ICC) of PBA and CTSA OHIP-14 scores (mixed models) – this assesses agreement at the individual level. Difference in time taken to complete PBA and CTSA was determined. Preference for mode of assessment was determined through descriptive statistics. Following on, regression analyses were conducted where the absolute difference in OHIP-14 scores was the dependent variable and the independent variables were participants age, gender, ethnicity, and sequence of completion of assessments. The datasets used and/or analyzed during this study are available from the corresponding author on reasonable request from April 2020 until April 2022.

Results

The assessment of one participant was excluded because of failure to complete the PBA (5 of the 14 questions were incomplete). The mean age of the participants was 21.4 years (SD=1.3); half were female (51%, 30) and most were of Malay ethnicity (70%, 41) [20% (12) were Chinese, 3% (2) were Indian, and 7% (4) were of other ethnicities]. The internal consistency of OHIP (Cronbach alpha values) for CTSA and PBA were 0.86 and 0.85, respectively. There was no significant difference in summary OHIP-14 scores (or across any of its seven domains) obtained from PBA and CTSA (P>0.05) (Table 1). The MDD was 0.2 (SD=4.7) for summary OHIP-14 scores, and the magnitude of the difference (effect size) was 0.0. Across the domains the ES of directional differences ranged from 0 (physical pain) to 0.3 (physical disability and psychological disability). The mean absolute difference for summary OHIP-14 scores was 3.6 (SD=2.9), constituting approximately a 6% difference in scores (3.6/56). Across domains, the largest difference was in “psychological discomfort” assessments; more than 16% difference (1.3/8). ICC value obtained of PBA and CTSA was 0.90 (95% CI=0.8–0.9). Across domains for two of the seven domains, ICC values were <0.70 (Functional limitation and Psychological discomfort).

Table 1

Agreement Between CTSA and PBA Score

Scale	Mean CTSA (SD)	Mean PBA (SD)	Mean directional differencesa (SD)	Standardized difference (Dd)	Mean absolute differencesb (SD)	Correlationsc (95% CI)
Total OHIP Score	13.0 (7.8)	13.2 (7.5)	0.2 (4.7)	0.0	3.6 (2.9)	0.9f (0.8–0.9)
Sub-scales
Functional limitation	2.4 (1.5)	2.2 (1.8)	−0.2 (1.7)	−0.1	1.2 (71.1)	0.6f (0.4–0.8)
Physical pain	2.6 (1.5)	2.6 (1.5)	0.0 (1.1)	0.0	0.8 (0.8)	0.9f (0.8–0.9)
Psychological discomfort	2.9 (1.8)	2.6 (1.2)	−0.3 (1.6)	−0.2	1.3 (1.1)	0.6f (0.3–0.8)
Physical disability	1.4 (1.5)	1.8 (1.6)	0.4e (1.3)	0.3	0.9 (1.0)	0.8f (0.6 −0.9)
Psychological disability	1.5 (1.8)	1.7 (1.8)	0.3 (1.1)	0.3	0.7 (0.9)	0.9f (0.8–0.9)
Social	0.9 (1.4)	1.0 (1.2)	0.1 (0.9)	0.1	0.6 (0.7)	0.9f (0.8–0.9)
Handicap	1.3 (1.4)	1.4 (1.5)	0.1 (1.1)	0.1	0.7 (0.9)	0.8f (0.7–0.9)

Notes: aDifference between CTSA and PBA score (PBA score minus CTSA scores) accounting for directional difference (indicator of bias). bDifference between CTSA and PBA scores irrespective of the direction of differences (indicator of agreement). cObtained using Intraclass Correlation Coefficient. dStandardized difference D=mean directional difference/standard deviation of directional differences (D≤0.2, small; 0.2

Abbreviations: CTSA, computer touch screen assessment; PBA, paper based assessment; SD, standard deviation; CI, confidence interval; OHIP, oral health impact profile.

Multiple linear regression analyses did not identify any significant association between socio-demographics (age, gender, ethnicity) or sequence of completing OHRQoL assessments, and (R2=0.091) absolute difference in Table 2.

Table 2

Multiple Regressions for Directional Difference and Absolute Difference of Total OHIP Scores

	Age	Gender	Ethnicity	Test Group
Directional difference
B (β)	0.3 (0.1)	−0.6 (−0.1)	−0.2 (−0.0)	4.6 (0.5)
SE	0.5	1.3	0.7	1.2
P-valuea	0.57	0.67	0.79	< 0. 001b
CI (95%)	−0.7–1.3	−3.2–2.1	−1.6–1.2	2.2−6.9
Absolute difference
B (β)	−0.6 (−0.3)	0.4 (0.1)	0.1 (0.0)	−1.3 (−0.2)
SE	0.3	0.9	0.5	0.8
P-valuea	0.1	0.7	0.8	0.1
CI (95%)	−1.2–0.1	−1.4–2.2	−0.8–1.1	−2.9–0.3

Notes: aObtained from t-test. bSignificant at 5%. The dependent variable is generated directional differences and absolute differences of total OHIP scores.

Abbreviations: OHIP, oral health impact profile; B, beta; SE, standard error; CI, confidence interval.

The mean (SD) time spent for PBA was 163.5 seconds (SD=60.8) compared to 172.2 seconds (SD=38.9). There was no significant difference in time taken to completed assessment: PBA versus CTSA (P=0.09). Among the respondents, 46 respondents (78%) had a preference for the touch screen version, 10 (17%) respondents preferred the written version, and 3 (5%) respondents had no preference.

Discussion

The value and use of OHRQoL measures has long been accepted, but the challenge has been how to employ them in research and clinical practice in effective and efficiency means.12,13 There have been reports on the equivalence of assessments made by interview and questionnaires that have informed the debate on how to collect OHRQoL.14,15 Our trial sought to inform the debate further by comparing CTSA and PBA. The study benefits from being a randomized cross-over trial with a priori hypothesis of an appropriate sample size and response rate. Some studies have reported that the lack of randomization reduce the quality of the evidence provided by the study, thus introduces an important bias.16

There was no significant difference between scores obtained from CTSA and PBA in terms of overall scores or any of its seven domain scores. Furthermore, the magnitude of the directional difference could be interpreted as “insignificant or small” ES <0.20.17 In the domain of “psychological disability” there was a greater lack of concordance which may reflect that psychological attributes rather than physical attributes may vary.18 In terms of absolute differences, again the proportion of dissimilarity relative to absolute scores was low, at ~6%, but again larger in the aspects of “psychological disability” (~16%) for potential research as aforementioned. Thus, the findings suggest at “the group” level there was good agreement and in practical terms either method could be used to obtain OHRQoL assessments.19,20 At the individual level, here was also high levels of agreement with ICC valued >0.70. Of note again it was lowest in terms of “psychological discomfort”. Thus, at “individual level”, in a case of patient care, either method is appropriate for OHRQoL assessments.21 When comparing respondents by age or gender, there were no significant differences highlighted, as was also recently stated by Casola et al.22

There was no significant difference in time taken to complete assessments. However, given that CTSA are likely to be more time saving because data is automatically entered and amenable to statistical analyses this would imply that overall CTSA offers greater efficiency and most likely greater cost-efficiency.7

Of note most respondents reported to prefer CTSA method over PBA. Perhaps in the era of widespread (and continued use of) hand help devices CTSA have become more “second nature” than PBA. It is worth considering through mix methods and qualitative research whether this is in fact the case.23,24 There are some potential limitations of this trail. First, the study was conducted among university level students who are likely to have greater familiarity with computer-based technologies than perhaps the general population of similar age or older people and, thus, this may influence their ability to complete and preference for CTSA. Second, the study population reported to experience relatively few oral health impacts (ie, had low OHIP scores) which is to be expected given their age and educational attainment. It is plausible that among study populations with poorer oral health who experience more oral health impacts that differences between the CTSA method over PBA may exist. Lastly, both assessments were conducted on the same day so as to ensure oral health state was stable but, given the limited washout period (“lunch time”), it is feasible that participants could recall previous assessments made.

Conclusion

In conclusion, OHRQoL assessments obtained from computer touch screen assessment (CTSA) or paper-based assessments (PBA) are equivalent. There is concordance of measurement at both “the group”, which is refers to the mean absolute difference and “the individual”, which refers to ICC levels. Furthermore, they take an equivalent amount of time to perform assessments and arguably CTSA are likely to be more efficient and cost-effective. Moreover, CTSA appears to be a more preferred means of OHRQoL assessment that PBA. This has implications to inform the practice of OHRQoL assessments in research, epidemiological surveys, and in clinical practice.