Smartphone Data Capture Efficiently Augments Dictation for Knee Arthroscopic Surgery.

INTRODUCTION: The objectives of this study are (1) to develop a provider-friendly, evidence-based data capture system for lower-limb orthopaedic surgeries and (2) to assess the performance of the data capture system on the dimensions of agreement with operative note and implant log, consistency of data, and speed of provider input.
METHODS: A multidisciplinary team developed a database and user interfaces for Android and iOS operating systems. Branching logic and discrete fields were created to streamline provider data input. One hundred patients were randomly selected from the first four months of data collection (February to June 2015). Patients were limited to those undergoing anterior cruciate ligament reconstruction, meniscal repair, partial meniscectomy, or a combination of these procedures. Duplicate data on these 100 patients were collected through chart review. These two data sets were compared. Cohen's kappa statistic was used to assess agreement.
RESULTS: The database and smartphone data capture tool show almost perfect agreement (kappa > 0.81) for all data tested. In addition, data are more comprehensive with near-perfect provider completion (100% for all data tested). Furthermore, provider data entry is extremely efficient (median 151-second completion time).
CONCLUSION: A well-designed database and user-friendly interface have greater potential for research utility, clinical efficiency, and, thus, cost-effectiveness when compared with standard voice-dictated operative notes. Widespread utilization of such tools can accelerate the pace and improve the quality of orthopaedic clinical research. LEVEL OF EVIDENCE: Level IV.

As health care becomes increasingly value driven, the ability to justify treatment is enhanced by prospective, high-quality, standardized databases.[1] Examples of large anterior cruciate ligament reconstruction (ACLR) databases include the Swedish National Anterior Cruciate Ligament (ACL) Register (SNKRL), Danish Cruciate Ligament Registry (DKKR), Kaiser Permanente Anterior Cruciate Ligament Reconstruction Registry (KPACLRR), and the Multicenter Orthopaedic Outcomes Network and Multicenter ACL Revision Study cohort.[23456] The uses of high-quality surgical data include, but are not limited to, internal quality improvement initiatives, large-scale comparative effectiveness research, outcomes research, cost-effectiveness research, and clinical trials.[7-9] Such data have greatly contributed to our understanding of surgical practice.[10] The availability of prospective, standardized, high-quality data is a foundational component for the advancement of the field of orthopaedic surgery.

Several practical challenges exist in constructing large high-quality data sets. Much of the important data that are captured in the electronic medical record (EMR) cannot be accessed easily for statistical analysis.[11] Error rates in the manual extraction of data from EMRs have been reported from 8% to 23%, with data varying by site, clinical area, and surgical specialty.[12] Increasing the accuracy of such data through quality assurance and data review increases cost.[12] Operative reports, a key data source for surgical research, tend to infrequently report quantitative data, markedly limiting the precision of research and quality measurement.[13] Even best-in-class registry data are collected through a secured web-based or paper form, which have obvious limitations in workflow (both), data accessibility (paper), and an exhaustivity of 85% to 90% for surgeon-reported data.[10]

To further advance orthopaedic outcomes measurement nationally, investigators developed the OrthoMiDaS (Orthopaedic Minimal Data Set) Episode of Care (OME) database and data collection methodology. OME's goal is to accurately and consistently collect patient-reported outcome measures (PROMs) immediately before and at the time of peak function after high-volume elective orthopaedic surgeries, as well as to accurately and consistently collect information about the actual surgical intervention in a manner that is both faster and more detailed than previously designed outcomes database systems.

This study describes the development of the OME database and provider-friendly, evidence-based smartphone data collection methodology and assesses the performance of the OME data capture system on the dimensions of agreement with operative note and implant log, consistency of data, and speed of provider input. We hypothesized that the use of the unique, provider-friendly, smartphone-based OME data capture system would increase the quality of orthopaedic procedure data in the context of ACLR and meniscal repair arthroscopic procedures.

Methods

OrthoMiDaS Episode of Care Database Design

A multidisciplinary team including administrators, orthopaedic surgeons, and software developers was assembled for design purposes. The OME database collects the following two distinct classes of data: PROMs and procedural data about the orthopaedic surgery itself. PROMs collected for arthroscopic knee surgeries include the Knee injury and Osteoarthritis Score, The Hospital for Special Surgery Pediatric Functional Activity Brief Scale, and the Veterans Rand 12-Item Health Survey.[14151617] Upon check-in at the surgery center, patients receive an iPad with their PROM form and complete it before going back into the preoperative holding area. This process is designed to be built into the standard clinical workflow so as not to slow down the operating rooms or require any additional staffing to execute.

The second class of data, the procedural details about surgery, are entered into OME by the surgeons themselves immediately after surgery using their hospital-issued smartphones or desktop/laptop computers (Figure 1). Demographics such as height, weight examination under anesthesia findings, commonly cited operative report parameters, and key predictors of surgical outcomes for ACLR, meniscal repair, and cartilage repair as identified in the literature are collected in the form of discrete data entry fields.[6,18-20] In total, OME currently provides 449 fields in the knee arthroscopy surgery data set. Branching logic is used to streamline provider data input. For primary meniscal repair and ACLR procedures, the branching logic system will require the surgeon to input approximately 13 fields for each meniscal repair and approximately 16 fields for the ACLR. However, the number of fields will expand as complexity increases. The basic set of meniscal pathology and repair data are shown in Appendix 1. Within the user interface, the branching logic displays fields only relevant to the individual procedure and pathology, greatly expediting the data collection process and decreasing the cognitive load on the surgeon. For example, selecting meniscal repair presents the surgeon with repair types, followed by manufacturers of implants/devices, and then specific device/implant offerings by that company, guiding the surgeon to procedure-specific details rather than requiring unnecessary data input as is often the case in EMR template usage. On the morning of surgery, surgeons receive reminder e-mails for each surgery they are about to perform that day with links to the surgeries' corresponding OME surgeon forms; like the PROMs-collection workflow, surgical data collection is built into surgeons' workflows to minimize delays in the daily routine.

Figure 1

OrthoMiDaS Episode of Care iPhone interface demonstrating detailed data capture and speed dial provider–specific templates. ACL = anterior cruciate ligament, ACLR = anterior cruciate ligament reconstruction

Architecturally, OME exists as a collection of research electronic data capture (REDCap) databases managed by the custom-built software that manages the multiplatform data entry and distribution.[21,22] All software is hosted locally and was approved by the local institutional review board (IRB) and information security. This study was also approved by the local IRB (IRB# 06-196).

Patient Selection

The OME database was launched on February 18, 2015. One hundred patients undergoing ACLR, meniscal repair, partial meniscectomy, or a combination of these procedures were included in the data set from the first four months of data collection (February through June 2015). All 12 surgeons at the sports health center who performed these procedures were included in the data set.

Data Collection and Validation

OME data used in this study were prospectively captured by the surgeons directly following their surgical procedures and were exported into a study database in an automated fashion. To evaluate the agreement with surgeons' operative dictation and/or implant logs, an independent REDCap database was established. Independent chart review data for comparison were collected from the operative report and the implant log in the Epic EMR system (Epic Systems). Reviewers of the operative report and the implant log were blinded to the OME REDCap results. Before analysis, the two data sets were reviewed for discrepancies, and all unmatching data were rechecked.

Assumptions in Chart Review Data Collection

The following assumptions were made in obtaining chart-reviewed operative dictation data: (1) If only one tunnel or reamer size was specified, both tibial and femoral tunnels were assumed to be the same size. (2) The operative report was used to determine the implant number and type; when a discrepancy or lack of clarity arose, the reviewer deferred to the implant log. (3) If the bone reamer or tunnel sizes were not specified, the graft size was used to approximate the tunnel size. (4) Regarding ACL status, normal status and no status were considered equivalent.

Statistical Analyses

All analyses were performed using R software (R version 3.2.3 [2015-12-10]). Agreement on nominal variables was measured by Cohen's (unweighted) kappa. 0.81 to 1.00 was considered almost perfect agreement; 0.61 to 0.80 was considered substantial agreement; and 0.41 to 0.60 was considered moderate agreement.[23,24] Agreement on numeric variables was measured by the concordance correlation coefficient (CCC). 0.99 to 1.00 was considered almost perfect agreement; 0.95 to 0.99 was considered substantial agreement; 0.90 to 0.95 was considered moderate agreement; and <0.90 was considered poor agreement.[25] Operative report and OME completion rates are presented as percent completion and are compared using the McNemar test (with continuity correction).

Results

Basic Demographics

The sample included 100 patients undergoing arthroscopic ACLR, arthroscopic meniscal repair, or both. The median age of the patients was 23 years (95% confidence interval [CI] [24.2, 28.7]), and 48% (n = 48) were women. This sample included 94 ACLRs, 58 partial meniscectomies, and 26 meniscal repairs as described in Table 1.

Table 1

Knee Arthroscopy Patient Characteristics

Factor	N (%)	N
Characteristic
N		100
Female	48 (48.0%)
Median age (95% CI)	23 (24.2, 28.7)
Operative limb		100
Both	1 (1.00%)
Left	45 (45.0%)
Right	54 (54.0%)
Medial meniscus
Meniscus status		100
Complete tear	28 (28.0%)
Normal	55 (55.0%)
Partial tear	17 (17.0%)
Main tear type		45
Bucket-handle	9 (20.0%)
Complex	7 (15.6%)
Horizontal	2 (4.44%)
Longitudinal	27 (60.0%)
Location to horns		45
Both	8 (17.8%)
Posterior only	37 (82.2%)
Location to blood supply		36
Red-red	14 (38.9%)
Red-white	16 (44.4%)
White-white	6 (16.7%)
Main tear length		45
<6 mm	5 (11.1%)
6–10 mm	4 (8.89%)
11–15 mm	11 (24.4%)
16–20 mm	18 (40.0%)
>20 mm	7 (15.6%)
Treatment		45
Abrade + trephine	1 (2.22%)
No treatment for tear	9 (20.0%)
Partial excision	15 (33.3%)
Repair	20 (44.4%)
Repair technique		20
All inside	19 (95.0%)
Both inside-out and all-in	1 (5.00%)
Implant manufacturer		20
Arthrex	3 (15.0%)
Smith & Nephew	17 (85.0%)
Implant system		20
DartStick	3 (15.0%)
FAST-FIX 360	17 (85.0%)
Number of implants Med [Q1, Q3]	2.00 [2.00; 3.00]	20
Lateral meniscus
Meniscus status		100
Complete tear	48 (48.0%)
Normal	43 (43.0%)
Partial tear	9 (9.00%)
Main tear type		57
Bucket-handle	5 (8.77%)
Complex	12 (21.1%)
Horizontal	3 (5.26%)
Longitudinal	12 (21.1%)
Oblique/flap	9 (15.8%)
Radial	14 (24.6%)
Root	2 (3.51%)
Location to horns		57
Anterior only	4 (7.02%)
Both	13 (22.8%)
Posterior only	40 (70.2%)
Location to blood supply		17
Red-red	3 (17.6%)
Red-white	7 (41.2%)
White-white	7 (41.2%)
Main tear extent		16
Complete to periphery	4 (25.0%)
Partial, periphery intact	12 (75.0%)
Main tear length		57
<6 mm	10 (17.5%)
6–10 mm	14 (24.6%)
>20 mm	11 (19.3%)
11–15 mm	19 (33.3%)
16–20 mm	3 (5.26%)
>20 mm	11 (19.3%)
Treatment		57
No treatment for tear	8 (14.0%)
Partial excision	43 (75.4%)
Repair	6 (10.5%)
Repair technique		6
All inside	5 (83.3%)
Inside-out	1 (16.7%)
Implant system		5
DartStick	1 (20.0%)
FAST-FIX 360	4 (80.0%)
Number of implants, Med [Q1, Q3]	3.00 [2.00; 4.00]	5
Anterior cruciate ligament
ACL status		100
Complete tear	91 (91.0%)
Normal	6 (6.00%)
Partial tear	3 (3.00%)
ACLR performed		94
Primary	93 (98.9%)
Revision	1 (1.06%)
Graft		94
ALLO Achilles	5 (5.32%)
ALLO other (HG, TA, PT, and ITB)	1 (1.06%)
Auto BTB	32 (34.0%)
Auto HG	56 (59.6%)
Number of strands, Med [Q1, Q3]	4.00 [1.00; 5.00]	94
Femur tunnel (mm), Med [Q1, Q3]	9.00 [8.50; 10.0]	94
Tibia tunnel (mm), Med [Q1, Q3]	9.00 [8.62; 10.0]	94
Primary femoral fixation		94
Cross-pin	8 (8.51%)
Interference screw	32 (34.0%)
Suspensory	54 (57.4%)
Primary femoral screw type		32
Bioabsorbable	14 (43.8%)
Metal	18 (56.2%)
Primary tibial fixation		94
Interference screw	77 (81.9%)
Suspensory	17 (18.1%)
Primary tibial screw type		77
Bioabsorbable	41 (53.2%)
Metal	26 (33.8%)
PEEK	10 (13.0%)

ACL = anterior cruciate ligament, ACLR = anterior cruciate ligament reconstruction, ALLO = allograft, BTB = bone-patellar tendon-bone, CI 2= confidence interval, HG = hamstring graft, ITB = iliotibial band, PEEK = polyetheretherketone, PT = patellar tendon, TA = tibialis anterior

Agreement

OME and chart-reviewed data comparison showed “near perfect” (kappa ≥ 0.81) or “substantial” to “almost perfect” agreement (CCC > 0.95) for all data tested (Table 2). The highest agreement level among nominal variables occurred in the reporting of graft type (kappa = 1.000, 95% CI [1.000, 1.000]), medial meniscus implant system (kappa = 1.000, 95% CI [1.000, 1.000]), lateral meniscus implant system (kappa = 1.000, 95% CI [1.000, 1.000]), and lateral meniscus treatment (kappa = 1.000, 95% CI [1.000, 1.000]). The lowest degree of agreement occurred in reporting of lateral meniscus status (kappa = 0.859, 95% CI [0.759, 0.960]). For the three numeric variables, the CCC was lowest for femur tunnel size (CCC = 0.977, 95% CI [0.964, 0.985]) and highest for graft strand number (CCC = 0.998, 95% CI [0.997, 0.999]).

Table 2

Agreement Between Chart-Reviewed and OrthoMiDaS Episode of Care Data

Measure	n	Agreement Statistic	95% Confidence Interval
Operative limb	100	0.940	0.873, 1.000
ACL
Status	100	0.904	0.716, 1.000
Graft type	93	1.000	1.000, 1.000
Graft strand number	75	0.998[a]	0.997, 0.999
Femur tunnel size	79	0.977[a]	0.964, 0.985
Primary femoral fixation type	94	0.961	0.908, 1.000
Femoral screw type	31	0.871	0.699, 1.000
Tibial tunnel size	79	0.990[a]	0.984, 0.993
Primary tibial fixation type	93	0.925	0.821, 1.000
Tibial screw type	76	0.934	0.860, 1.000
Medial meniscus
Status	100	0.898	0.812, 0.985
Treatment	37	0.952	0.860, 1.000
Implant system	18	1.000	1.000, 1.000
Lateral meniscus
Status	100	0.859	0.759, 0.960
Treatment	46	1.000	1.000, 1.000
Implant system	5	1.000	1.000, 1.000

ACL = anterior cruciate ligament

Agreement between numeric variables is measured through the concordance correlation coefficient.

Provider Completion Rate

Graft strand number was reported in OME in 100% of the ACLR patients, but only in 80% of the ACLR patients in the operative reports. Femur tunnel size was reported in OME in 100% of the ACLR patients, but only in 85% of the ACLR patients in the operative reports. Tibial tunnel size was reported in OME in 100% of the ACLR patients, but only in 85% of the ACLR patients in the operative reports (Table 3).

Table 3

Provider Data Completion Rates

Measure	Operative Report		OrthoMiDaS Episode of Care Database		P Value
	n	% Completion	n	% Completion
ACL
Graft strand number	75	80	94	100	P < 0.001
Femur tunnel size	80	85	94	100	P < 0.001
Tibial tunnel size	80	85	94	100	P < 0.001

ACL = anterior cruciate ligament

Time

The median provider time to complete the data entry for a single patient was approximately 2 minutes.

Discussion

Although all data tested showed near-perfect agreement, ACL status (kappa = 0.904), lateral meniscus status (kappa = 0.898), femoral screw type (kappa = 0.871), and medial meniscus status (kappa = 0.859) displayed the least agreement of all data tested. Despite the obvious need for accuracy in determining these parameters, a high degree of variability exists in surgeons' operative dictation description of these lesions and implants. Some surgeons provided detailed, quantitative descriptions (eg, 5-mm, full-thickness, radial tear in the posterior horn), whereas other surgeons gave very brief descriptions (eg, small tear in the lateral meniscus), leading to the inability to extract accurate data from the operative report.

Improved Completion Rate

Historically, the operative report is used as a key data source for most orthopaedic retrospective research. Despite widespread use, the quality of these data is limited.[26] Operative reports tend to infrequently report quantitative data, markedly limiting the precision of research and quality measurement.[13] Moreover, late dictation of reports, or dictation by residents, may also increase error rates.[27,28] This error is compounded by errors in data extraction from the medical record itself. This limited quality of data contributes to the large gap, both in validity and reliability, between prospective and retrospective research.[29] Together, these limitations of operative report data delay the progress of the field and cause increased cost burden.

Some improvements in the quality of capturing surgical data have been previously made. Customized, computerized, templated operative reporting systems are available and can improve the consistency of reporting of key operative parameters.[30] The branching logic of OME not only improves the completeness of reporting operative parameter details but also maximizes efficiency in data input. Such systems can dramatically decrease completion delays and reduce the cost of surgical documentation.[31]

Before this study, templates and computer assistance are effective in increasing the completeness of operative report data.[30] This finding was consistent with our findings that graft strand number, femur tunnel size, and tibial tunnel size were all more frequently collected in the OME data capture system. The OME user interface incorporates dropdown menus that likely function as memory aids for key surgical parameters.[30] The OME computer defaults and branching logic will not accept missing key risk factors identified in the most recent ACLR and meniscal outcomes research.[6,18-20] Branching logic and the smartphone interface speed the data entry process. Finally, an automated e-mail reminder system is used to ensure the high data capture rate.

Limitations

Currently, although the operative report is widely used, no true benchmark exists for operative reporting. Thus, the absolute accuracy of the OME database (ie, OME data compared directly with true occurrences in the operating room) is difficult to assess. However, the present study indicates OME data are consistent with the current most pervasive methodology and demonstrate less information loss.

Conclusion

The OME data capture system demonstrated “almost perfect” agreement (kappa ≥ 0.81) on all 13 nominal variables and “substantial” agreement (CCC > 0.95) to “almost perfect” agreement (CCC > 0.099) on all three numeric variables tested. In addition, the OME data capture system improved the reporting of key operative parameters necessary for outcomes research and internal quality improvement (100% for all data tested). Furthermore, the developers of this system continue to develop branching logic and data capture tools for additional high-volume orthopaedic surgical procedures. Moreover, widespread use of this system can bring the highest level of prospective quality data to everyday surgical practice. Finally, this technology could potentially replace standard narrative/dictation-based operative reporting, and transform observational and retrospective orthopaedic clinical research, to a prospective cohort model.

Data Field	Dropdown Selections
Meniscus status	Normal
	Complete tear
	Partial tear
	Previous partial excision
	Previous repair
New tear	None
	Partial
	Complete
Main tear type	Oblique/flap
	Longitudinal
	Bucket-handle
	Radial
	Root
	Horizontal
	Complex
Location to horns	Anterior only
	Posterior only
	Both
Location to blood supply	Red-red
	Red-white
	White-white
Main tear extent	Partial, periphery intact
	Complete to periphery
Main tear length (mm)	<6 mm
	6–10 mm
	11–15 mm
	16–20 mm
	>20 mm
Treatment	No treatment for tear
	Partial excision
	Repair
	Abrade + trephine
	Meniscal transplant
Repair technique	All inside
	Inside-out
	Both inside-out and all in
	Outside-in
Number of inside-out and/or outside-in sutures	Arthrex
	Biomet
	Covidien
	CONMED
	Cayenne
	DePuy Mitek
	Smith & Nephew
	Others
Implant system	Select from preprogrammed implant system library specific to manufacturer selection
Number of implants	Select integer value between 0 and 10
Discoid meniscus	No
	Yes, partial
	Yes, complete