Data entry quality of double data entry vs automated form processing technologies: A cohort study validation of optical mark recognition and intelligent character recognition in a clinical setting.

BACKGROUND AND AIMS: Patient-reported outcome measures (PROMs) are increasingly used in health services. Paper forms are still often used to register such data. Manual double data entry (DDE) has been defined as the gold standard for transferring data to an electronic format but is laborious and costly. Automated form processing (AFP) is an alternative, but validation in a clinical context is warranted. The study objective was to examine and validate a local hospital AFP setup.
METHODS: Patients over 18 years of age who were scheduled for knee or hip replacement at Stavanger University Hospital from 2014 to 2017 who answered PROMs were included in the study and contributed PROM data. All paper PROMs were scanned using the AFP techniques of optical mark recognition (OMR) and intelligent character recognition (ICR) and were processed by DDE by health secretaries using a data entry program. OMR and ICR were used to capture different types of data. The main outcome was the proportion of correctly entered numbers, defined as the same response recorded in AFP and DDE or by consulting the original paper questionnaire at the data field, item, and PROM level.
RESULTS: A total of 448 questionnaires from 255 patients were analyzed. There was no statistically significant difference in error proportions per 10 000 data fields between OMR and DDE for data from check boxes (3.52 95% confidence interval (CI) 2.17 to 5.72 and 4.18 (95% CI 2.68-6.53), respectively P = .61). The error proportion for ICR (nine errors) was statistically significantly higher than that for DDE (two errors), that is, 3.53 (95% CI 1.87-6.57) vs 0.78 (95% CI 0.22-2.81) per 100 data fields/items/questionnaires; P = .033. OMR (0.04% errors) outperformed ICR (3.51% errors; P < .001), Fisher's exact test.
CONCLUSIONS: OMR can produce an error rate that is comparable to that of DDE. In our setup, ICR is still problematic and is highly dependent on manual validation. When AFP is used, data quality should be tested and documented.

INTRODUCTION

Patient‐reported outcome measures (PROMs) have been increasingly used in both orthopedic surgery and health services in general over the last decades. , , Paper forms are still often used to document such data. Traditionally, data have been manually entered into a database when the conversion of data from paper forms to an electronic format has been required. Single manual data entry (SDE), in which data are manually entered once, has been shown to be inferior to double manual data entry (DDE; also called duplicate data entry, DE). , Manual data entry should ideally be performed through a dedicated data entry program with program control. A dedicated data entry program should restrict invalid response options (out‐of‐range values), check and flag missing data, and perform automatic comparisons and direct flagging of conflicting double‐entered data. Manual two‐person DDE has been defined as the gold standard for transferring data to an electronic format, but the process is laborious and costly. Automated form processing (AFP) is an alternative, but there are few validation studies in a clinical context.

AFP (also called electronic data capture, EDC) is a method for storing information entered into data fields by scanning and converting it to an electronic format. Different types of AFP exist; for example, optical mark recognition (OMR) recognizes checked boxes on a paper form, and intelligent character recognition (ICR; also called optical character recognition, OCR) recognizes machine‐printed and handwritten characters. Due to the more complex automatic interpretation and higher error rate , , of data obtained through ICR, these data should always be reviewed for accuracy.

In many parts of health services, there is a need to convert data from paper forms to an electronic format. Different AFP packages are typically used for this purpose. Trusting AFP software to be accurate and failing to test the error rates in the specific clinical setting, including errors from incorrect manual validation by staff, increase the risk of erroneous results, which may have grave consequences in a clinical setting. Different AFP setups may give different error rates, warranting the validation of local AFP setups. This report may be used as a guide for data quality validation in local hospital setups.

METHODS

Ethical considerations

The study was submitted for registration to the Regional Committee for Ethics in Medical Research for Western Norway (2017/538) and was approved by the local data protection officer (journal number 2017/26). This project examined the technical and methodological development of data capture methods and used only anonymized data. Due to the use of only anonymous data from patient journals, there was no need for informed written consent from patients. The study was conducted in accordance with the World Medical Association's Declaration of Helsinki.

Design

Our prospective cohort consisted of patients over 18 years of age who were scheduled for knee or hip replacement at Stavanger University Hospital, a medium‐sized university hospital with approximately 800 beds that serves a population of 366 500, from 2014 to 2017. All patients received two different PROMs, one generic and one condition‐targeted PROM, before the operation, 6 to 8 weeks postoperatively, and 1 year postoperatively. The generic EuroQoL‐5D‐5L (EQ‐5D‐5L) , , and either the Knee Injury and Osteoarthritis Outcome Score (KOOS) , for patients scheduled for knee replacement or the Hip Dysfunction and Osteoarthritis Outcome Score (HOOS) for patients scheduled for hip replacement. Both EQ‐5D‐5L (https://euroqol.org/) and HOOS/KOOS (http://www.koos.nu/) are available in Norwegian‐language versions.

Items and fields

We defined an item as a single question in the PROM and a data field as a possible answer category for an item. The EQ‐5D‐5L contains six items in total, five index items plus the EQ‐VAS. Each of the five index items has five answer categories/data fields (checkboxes), and the EQ‐VAS has one data field in which the respondent writes an integer from 0 to 100. The KOOS and HOOS contain 42 and 40 items, respectively, each with five answer categories/data fields (checkboxes).

All items from the KOOS, the HOOS, and the five index items on EQ‐5D were processed by OMR, whereas the EQ‐VAS had to be read by ICR.

Scanning setup

The scanning setup consisted of a Kodak i5200 scanner (Kodak Canada Inc, Toronto, Ontario). OCR for AnyDoc H version 15.0.0.97 (Hyland Software Inc, Westlake, Ohio) was used for questionnaire setup and processing. Kodak Capture Pro version 5.0.4 was used to import data from the scanner. A Dell OptiPlex 7040 computer (Dell Inc, Round Rock, Texas) with Microsoft Windows version 7 and the Microsoft Office 2016 packages (Microsoft Corporation, Redmond, Washington) were used. The data were stored in ORPlan, a local EPJ data program for operation planning, coding, and PROM data collection (ORPlan, version 7.3, Stavanger, Norway). Secretaries at the Department of Orthopaedic Surgery, Stavanger University Hospital performed both DDE and AFP (DDE was performed manually by secretaries who entered the data twice in close succession in dedicated data entry software).

Manual validation

When the AFP system cannot convert an item response due to poor or ambiguous questionnaire completion, manual validation is necessary. The scanner/data importer stops, and the secretaries must manually validate the correct code for the questionnaire answer before the data import can continue. As such, all missing values, invalid values, and out‐of‐range values were manually validated by reference to the original completed questionnaires. All ICR data were manually validated in the data import process.

Manual data entry

The ORPlan data entry software module (ORPlan, Stavanger, Norway) was used for DDE and the program control of the data entry. This software program contains data validation and branching logic and restricts invalid response options, flags missing data, and performs an automatic comparison and direct flagging of conflicting DDE data.

Data processing

We compared AFP and DDE data at the item level and defined correct data entry when both AFP and DDE recorded the same response. In case of differences, we manually consulted the original paper questionnaire response and found the correct answer in accordance with the manuals for handling the questionnaires as well as the coding guidance books for each PROM (how to interpret correct/incorrectly completed questionnaires). , , , , In case of differences, when the original paper questionnaire response was missing, we assumed error in the AFP. Four secretaries with no prior AFP knowledge performed the AFP after 10 hours of training and supervision.

Statistical methods

IBM SPSS Statistics version 24 was used for data processing and descriptive statistics, and Stata version 16 was used for confidence intervals and tests. Descriptive statistics are presented as counts and percentages and as the means or medians and full ranges. Error proportions were calculated as the number of units with errors per 10 000 data fields, per 1000 questionnaire items, and per 100 questionnaires. Proportions are presented with 95% confidence intervals (CI) estimated by the Wilson method using the immediate function cii. Differences between processing methods or between groups of observations were tested with Chi‐square tests or Fisher's exact tests as appropriate, using immediate function tabi with options chi2 and exact. The sample consisted of 448 consecutive questionnaires processed by both double manual entry and optical scanning that were collected during the hospital's transition from using double manual entry to using optical scanning (in the time period 2014‐2017).

RESULTS

A total of 448 PROM questionnaires (255 EQ‐5D, 143 HOOS, and 50 KOOS) from 255 patients were included. An overview of patient characteristics for the PROM responses included in the study is given in Table 1, and the observed numbers of PROMs, items, and data fields are listed in Table 2. In the study, three questionnaires initially marked with AFP/DDE discrepancies were excluded because the scanning was technically correct (no errors were found when the original paper questionnaire response was manually consulted). In two HOOS questionnaires, patients had included additional information on the questionnaire, and the secretaries took this information into account during the data entry. In one KOOS questionnaire, page three of the questionnaire had mistakenly been shredded before scanning and was, therefore, missing in the scanning data and in the original paper version at control.

TABLE 1

Patient characteristics for the PROM responses included in the study

	EQ‐5D‐5L (n = 255)	HOOS (n = 143)	KOOS (n = 50)
Age, median (full range)	69 (18‐95)	69 (18‐89)	69 (45‐84)
Female sex, n (%)	152 (60%)	91 (64%)	25 (50%)

Abbreviations: EQ‐5D‐5L, EuroQoL‐5D‐5L; HOOS, Hip Dysfunction and Osteoarthritis Outcome Score; KOOS, Knee Injury and Osteoarthritis Outcome Score; PROM, patient‐reported outcome measure.

TABLE 2

Observed number of questionnaires, items, and data fields in relation to processing methods and PROMs

Scanning method/questionnaire	Observed number of PROMs	Observed number of items	Observed number of data fields
Processed with OMR
EQ‐5D‐5L *	255	1275	6375
HOOS	143	5720	28 600
KOOS	50	2100	10 500
Processed with ICR
EQ‐VAS	255	255	255

Abbreviations: EQ‐5D‐5L, EuroQoL‐5D‐5L; HOOS, Hip Dysfunction and Osteoarthritis Outcome Score; KOOS, Knee Injury and Osteoarthritis Outcome Score; OMR, optic mark recognition; PROM, patient‐reported outcome measure.

a Excluding EQ‐VAS.

For OMR, there was no statistically significant difference (P = .61) between AFP and DDE, with error proportions per 10 000 data fields of 3.52 (95% CI, 2.17‐5.72) and 4.18 (2.68‐6.53), respectively (see Table 3). Error proportions per questionnaire item and per questionnaire were also similar between the processing methods, that is, 0.88 (95% CI 0.45, 1.73) and 1.10 (95% CI 0.60, 2.02) per 1000 items and 1.34 (95% CI 0.62, 2.89) and 1.79 (95% CI 0.91, 3.48) per 100 questionnaires for AFP and DDE, respectively.

TABLE 3

Observed errors for optic mark recognition and double‐key data entry processing of PROMs

	OMR		DDE
Unit of observation	With error(s)	Error proportion (95% CI)	With error(s)	Error proportion (95% CI)	P value
Data fields (n = 45 475)	16	3.52 (2.17, 5.72)	19	4.18 (2.68, 6.53)	.61
Items (n = 9095)	8	0.88 (0.45, 1.73)	10	1.10 (0.60, 2.02)	.64
Questionnaires (n = 448)	6	1.34 (0.62, 2.89)	8	1.79 (0.91, 3.48)	.59

Note: Error proportions given as errors per 10 000 data fields, per 1000 items, and per 100 questionnaires. P‐values from Chi‐square tests.

Abbreviations: DDE, double‐key data entering; OMR, optic mark recognition; PROM, patient‐reported outcome measure.

For ICR processing, the number of observed data fields equaled the number of observed items, which equaled the number of observed questionnaire responses; ie, the error proportions were the same for all units of observation. The error proportion for ICR (nine errors) was statistically significantly higher than that for DDE (two errors), that is, 3.53 (95% CI, 1.87‐6.57) vs 0.78 (0.22‐2.81) per 100 data fields/items/questionnaires; P = .033. It was also higher than for OMR (0.04% errors; P < .001, Fisher's exact test).

Compared with the error proportion per data field observed for the questionnaires included in the OMR part of the study (0.04%), DDE had worse performance on the EQ‐VAS (0.78%, P = .006, Fisher's exact test). The time required for DDE of the different PROMs at the questionnaire and item levels is listed in Table 4.

TABLE 4

Seconds used for double‐key entering data (DDE)

PROM	Seconds used per questionnaire, mean (range)	Seconds per Item (mean)
EQ‐5D‐5L*	73 (26‐883)	12.2
HOOS	271 (63‐2585)	6.8
KOOS	291 (88‐942)	6.9

Abbreviations: EQ‐5D‐5L, EuroQoL‐5D‐5L; HOOS, Hip dysfunction and Osteoarthritis Outcome Score; KOOS, Knee injury and Osteoarthritis Outcome Score; PROM, patient‐reported outcome measure.

a Including EQ‐VAS.

DISCUSSION

Summary

We found a low level of error with AFP using OMR and a high level of error with AFP using ICR. Only one item (the EQ‐VAS) had to be processed using ICR. Our error level using OMR of 3.52 per 10 000 data fields read is a good and acceptable result.

Jenkins et al reported an error rate of 6.7 errors per 10 000 data fields in their AFP system. Jørgensen and Karlsmose reported an error rate of 7.2 errors per 10 000 data fields overall in their system, which is still far below the acceptable quality level of 50 errors per 10 000 data fields overall (0‐10 errors per 10 000 data fields for critical variables, 20‐100 errors per 10 000 data fields for noncritical variables) established by the Society for Clinical Data Management. SDE of data may be problematic for critical data, as the error rate of 36 errors per 10 000 data fields shows in the study of Wahi et al. Error detection rates of 88.3% for the two‐person double‐entry approach, compared to 69.0% for the single‐person double‐entry approach, have been published. DDE can reduce the error rate from 0.17 to 0.08% (P = .001), or by 15 per 10 000 data fields.

An earlier study with a similar scanning setup found that AFP can be a valid alternative to DDE and can be superior to SDE (even if a data entry program is used), with an error rate of 0.46 errors per 10 000 data fields. The previous study had access to an entire AFP department that was highly experienced in handling AFP, whereas, in the present study, four secretaries with no prior AFP knowledge performed the entire AFP after only 10 hours of training and supervision. Five of the eight original paper questionnaires with AFP/DDE discrepancies in OMR (Table 3) had mistakenly been shredded, and error in AFP was assumed (worst case scenario).

The time required for DDE of the PROMs/items (Table 4) was comparable to an earlier study by Paulsen et al with regard to seconds per item for HOOS but not for the entire questionnaire. This can be explained by the use of a 19‐item short‐form version in the previous study instead of the 40‐item original version used in the present study. In the present study, our secretaries required less than half the time for DDE of EQ‐5D, even though we used the 5 L version with more answer options compared to the 3 L version used in the earlier study. This may explain some of the increase in the error rate for DDE of EQ‐VAS (78.1/10000 data fields vs 33.7/10000 data fields) in the present study.

Optimization of AFP is important to reduce errors. To reduce the error rate and enhance both performance and data quality, we used highly structured paper PROMs containing only adequately sized check boxes and numbers (0‐9) but no complex codes or other writing. , The importance of validation of the data collection process when using manual data entry or AFP systems to achieve acceptable data quality, especially in a clinical setting, is emphasized. The low level of error with AFP using OMR and the high level of error with AFP using ICR found in this study may be indicative of structural problems in the use of ICR compared to OMR in a clinical setting. In our study, OMR was less dependent on manual validation by staff and seemed to be a more robust technology. We advocate the optimization of AFP by focusing on OMR where possible to reduce the risk of data errors.

Benefits

AFP benefits include more efficient data capturing, lower cost, , and acceptable data quality. , Manual data entry may produce erroneous results, which can be problematic and potentially dangerous in a clinical setting. The cost of AFP can be challenging to calculate, and direct comparison with DDE is difficult because commercial AFP services often include setting up and adjusting PROMs for AFP, controlling the status of patients (living or dead) before sending out the PROMs, communication with the printing company, printing expenses, sorting questionnaires and patient information, stapling questionnaires, mail merging, enveloping, receiving and opening envelopes from the patients, sorting the PROMs, removing the staples, scanning the PROMs, manually validating the PROMs, sending the data in an electronic format, manually checking out‐of‐range values, controlling the status of patients (living or dead) before sending out the reminders, and managing first and second reminder letters. In addition, there are the costs of training the scanning operators, hardware, and scanning software. Fifolt et al compared the cost of DDE and OMR and found that the $3.03 savings per survey processed via OMR paid for the capital and noncapital investment fixed cost at the 1400 survey threshold and that the potential benefit of DDE, in terms of data accuracy, did not outweigh the operational efficiency and financial savings of OMR.

Limitations and strengths

Several methodological limitations should be taken into consideration when interpreting the results of the present study. The results of this study cannot be generalized to all AFP systems or forms, but similar setups (setups using OMR/ICR on highly structured paper questionnaires containing only check boxes and numbers (0‐9) but no complex codes or other writing) may give similar results. The ORPlan data entry software module has not previously been validated. We used well‐validated PROMs for the patient group with validated feasibility (HOOS, EQ‐5D) and a previously validated AFP system (Kodak scanner/OCR for AnyDoc).

CONCLUSION

We found a low level of error with AFP using OMR and a high level of error with AFP using ICR. Our error level using OMR of 3.52 per 10 000 data fields read is a good and acceptable result. When using OMR, AFP of PROMs can produce an error rate comparable with manual double entry of data. In our setup, ICR was still problematic and highly dependent on manual validation by staff. When AFP of PROMs is performed, data quality should be tested and documented.

AUTHOR CONTRIBUTIONS

Conceptualization: Aksel Paulsen, Knut Harboe, Ingvild Dalen

Data Curation: Aksel Paulsen, Ingvild Dalen

Formal Analysis: Aksel Paulsen, Ingvild Dalen

Methodology: Aksel Paulsen, Knut Harboe, Ingvild Dalen

Software: Knut Harboe

Writing‐Original Draft Preparation: Aksel Paulsen

Writing‐Review and Editing: Aksel Paulsen, Knut Harboe, Ingvild Dalen

All authors have read and approved the final version of the manuscript.

Aksel Paulsen had full access to all of the data in this study and takes complete responsibility for the integrity of the data and the accuracy of the data analysis.

FUNDING

No financial support was received for this study/work. No funder had any involvement regarding study design, data collection, analysis, interpretation of data, writing of the report, or the decision to submit the report for publication.

CONFLICT OF INTEREST

The authors have no conflicts of interest to declare.

TRANSPARENCY STATEMENT

Aksel Paulsen affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Appendix S1: Supporting Information

Click here for additional data file.

Appendix S2: Supporting Information

Click here for additional data file.