Level of agreement on health information technology adoption and use in survey data: a mixed-methods analysis of ambulatory clinics in 1 US state.

OBJECTIVE: Adoption of health information technology (HIT) is often assessed in surveys of organizations. The validity of data from such surveys for ambulatory clinics has not been evaluated. We compared level of agreement between 1 ambulatory statewide survey and 2 other data sources: a second survey and interviews with survey respondents.
MATERIALS AND METHODS: We used 2016 data from 2 surveys of ambulatory providers in Minnesota-the Healthcare Information and Management Systems Society (HIMSS) survey and the Minnesota HIT Ambulatory Clinic Survey-and primary data collected through qualitative interviews with survey respondents. We conducted a concurrent triangulation mixed-methods assessment of the Minnesota HIT survey by assessing level of agreement between it and HIMSS, and a thematic analysis of interview data to assess the respondent's understanding of what was being asked and their approach to responding.
RESULTS: We find high agreement between the 2 surveys on questions related to common HIT functionalities-such as computerized provider order entry, medication-based decision support, and e-prescribing-which were widely adopted by respondents' organizations. Qualitative data suggest respondents found wording of items about these functionalities clear but encountered multiple challenges including interpreting items for less commonly adopted functionalities, estimating degree of HIT usage, and indicating relevant barriers. Respondents identified multiple errors in responses and likely reported greater within-group homogeneity than actually existed.
CONCLUSIONS: Survey items related to the presence or absence of widely adopted HIT functionalities may be more valid than items about less common functionalities, degree of usage, and barriers.

BACKGROUND AND SIGNIFICANCE

As use of electronic health records (EHR) and other forms of health information technology (HIT) increases, there has been continued interest in measuring the adoption and use of specific HIT capabilities, and examining the relationships between HIT and outcomes such as reductions in safety events, reductions in costs, and improvements in quality of care. Data to support these measurement efforts are typically collected through surveys of health systems and providers of care. Called “enterprise surveys,” they ask a representative of an organization about the organization’s adoption and use of HIT, which may require input from multiple respondents within the organization.

Previous studies have questioned the validity of HIT utilization surveys for hospitals, finding poor levels of agreement between different surveys on similar items related to EHR status and adoption of computerized provider order entry (CPOE)., A more recent study of a range of HIT functionalities in hospitals comparing survey responses to Medicare Meaningful Use reports as a gold standard, found a more favorable assessment—the survey items investigated predicted meaningful use attestation with a sensitivity of 0.82 and specificity of 0.72. Similar validation assessments have not been conducted on enterprise surveys used to acquire data from ambulatory providers. It is important when using data from these surveys to have confidence that reported responses accurately reflect the intended meaning. To this end, we assessed 1 ambulatory HIT survey’s level of agreement with a second survey encompassing similar items that cover a range of functionalities, and with survey respondents’ stated perspectives of the survey items when interviewed.

MATERIALS AND METHODS

Data

We targeted the state of Minnesota (MN) for data collection in this study. To our knowledge, MN is one of the few if not the only states that has systematically measured HIT capabilities of clinics statewide. We utilized 3 sources of data. First, the 2016 MN HIT Ambulatory Clinic Survey, conducted by MN Community Measurement (MNCM) on behalf of the MN Department of Health (MDH), is administered on a yearly basis to approximately 234 medical groups (representing 1300 ambulatory clinics) and covers most clinics in MN, with a medical group response rate above 90%. This survey is a reporting requirement of the MDH and results are used by MDH, MN e-Health Initiative, MNCM, and others to report the status and use of EHR, health information exchange, and other HIT across MN. MNCM and MDH developed the survey items collaboratively. Staff from these organizations conducted beta testing on the survey items. (Wellbrock D. Manager, Accounts, Communications & Programs. Personal communication; 2018).

Second, the Healthcare Information and Management Systems Society (HIMSS) Analytics Ambulatory Survey, which is available from HIMSS Analytics LOGIC™ Market Intelligence Platform, targets ambulatory “clinics” defined as facilities providing preventative, diagnostic, therapeutic, surgical, and/or rehabilitative outpatient care in which the treatment duration is less than 24 h. The purpose of the survey is to inform HIT-related decision-making. We used data from the 2016 survey, which contains information on more than 75% of US health system-associated ambulatory care practices. HIMSS defines a health system as an organization composed of at least 1 hospital and its associated nonacute facilities. “Associated” is defined as having a governance relationship (ie, they are owned, leased, or managed by a health system).

Third, we conducted semistructured interviews with representatives of a purposive sample of 8 medical groups that responded to the MN HIT survey. As the goal of this qualitative analysis was exploratory and not inferential, we used the MN HIT survey responses to select a sample of medical groups that all have an EHR but that vary in terms of size (large/small number of clinics) and location (urban/rural groups). We spoke by telephone with 1 individual from each medical group, requesting to speak with the person most involved in responding to the survey. We developed an interview guide that covered the following topics: the survey respondent’s approach to gathering data to answer the survey items, their interpretation of a subset of the survey items, challenges they faced in answering survey items, and whether the recorded answers were, in retrospect, consistent with their understanding of what was being asked.

Mixed-method analysis and survey items

We conducted a concurrent triangulation mixed-methods assessment of the MN HIT survey. Specifically, for the quantitative analysis, we selected MN HIT survey items that included common HIT functionalities and were worded in both surveys in a manner that was similar enough to allow comparison of MN HIT survey items with HIMSS survey items. These items covered topics such as the presence of certified EHR technology, CPOE, e-prescribing, clinical decision support (CDS) functionalities, health information exchange, and patient portal capabilities. We conducted an analysis of the level of agreement between the MN HIT and HIMSS responses by calculating the percentage of completed responses for the 4 possible combinations for dichotomous variables (yes to both surveys, yes to MN HIT/no HIMSS, no to MN HIT/yes to HIMSS, and no to both surveys). Concurrently, we interviewed MN HIT survey respondents about their interpretation and understanding of the same survey items that we used in the quantitative analysis, as well as some additional survey items related to those same functionalities (eg, barriers to adopting the HIT functionalities, degree of HIT use) and selected additional functionalities to capture a wide range (eg, telemedicine, registries, use of reports). (See Supplementary Appendix for more details on this survey item content.) Finally, we compared quantitative and qualitative findings.

Quantitative analysis

In total, the MN HIT survey included 1417 clinics in MN, and the HIMSS survey included 971. Using names and addresses, we matched clinics that were part of a medical group and that that responded to both the MN HIT and HIMSS surveys. Our final sample included 577 clinics whose representatives responded to both surveys in 2016. (These 577 clinics represent 76 medical groups.) Even where content was similar, the wording of the survey items was often slightly different across the 2 surveys and several of the MN HIT items were designed with Likert scales or percentage intervals which we dichotomized for this analysis so that they could be compared with the HIMSS items which were binary. (See Supplementary Appendix for details.) For each item in which there was a response completed in both surveys, we computed the percentage of clinics responding in agreement and disagreement to both surveys.

Qualitative analysis

We contacted a total of 28 medical groups that responded to the 2016 MN HIT survey and were able to interview representatives of 8 medical groups (29% response rate). These 8 medical groups represent a total of 228 clinics. We interviewed the individual at the medical group who was responsible for completing the survey for each clinic that was part of the group. We audio recorded and transcribed all interviews. Using the constant comparative method, 2 researchers (S.H.F., R.S.R.) reviewed the transcripts for emerging themes and developed a hierarchical code book which we then used to code the transcripts. Disagreements between reviewers were resolved by discussion leading to consensus.

RESULTS

Clinic characteristics

Characteristics of MN clinics represented in the MN HIT and HIMSS surveys overall, matched clinics used in the quantitative analysis, and for clinics within medical groups whose representatives participated in our interviews are shown in Table 1. Overall, the characteristics of the clinics were largely similar, with those in the matched sample more similar to the HIMSS sample than the MN HIT. For the qualitative analysis, we interviewed respondents from 8 medical groups representing clinics that had similar representation of larger and small, and of urban and rural, compared with the survey samples.

Table 1.

Characteristics of ambulatory clinics that make up the MN HIT, HIMSS, and the matched samples in 2016

	MN HIT (n = 1, 417), n (%)	HIMSS (n = 971), n (%)	Matched for level of agreement analysisa (n = 577), n (%)	Interviewed clinicsbn = 228 (group n = 8), n (%)
Geographical location
Rural	256 (18)	248 (26)	154 (27)	52 (23)
Urban	1150 (82)	721 (74)	420 (73)	171 (77)
Size of clinic (no. of physicians)
1	171 (14)	112 (15)	50 (10)	23 (11)
2–5	425 (33)	285 (37)	183 (35)	58 (28)
6–9	228 (19)	139 (18)	97 (19)	41 (20)
10–19	208 (17)	104 (13)	101 (20)	34 (17)
20–49	148 (12)	103 (13)	69 (13)	44 (21)
50+	39 (3)	22 (3)	19 (4)	6 (3)
Size of medical group (no. of clinics)
1	100 (7)	12 (1)	16 (2.8)	0 (0)
2–10	417 (29)	149 (15)	110 (19)	25 (11)
11–20	106 (8)	58 (6)	39 (7)	13 (6)
21+	797 (56)	752 (78)	412 (71)	190 (83)

Reported characteristics did not always agree between the 2 surveys. For these numbers, we use the MN HIT’s data.

Reported only in MN HIT.

Quantitative analysis of level of agreement across surveys

We found high levels of agreement between the MN HIT and HIMSS survey responses for the presence or absence of common HIT functionalities (Table 2). For 5 of the 10 survey items we evaluated, the levels of agreement exceeded 90%; only 1 item had agreement below 80%. The 3 HIT functionality items with the highest level of agreement were: e-prescribing capabilities (100%); CPOE (96.5%); and CDS (94.3%). The 3 items with the lowest level of agreement were related to health information exchange with government agencies (71.7%), health information exchange with ambulatory clinics (82.9%), and CDS for preventive medicine (83.6%). Most of the lack of agreement was due to the MN HIT survey items indicating greater adoption compared with HIMSS. The survey items in the sample with the highest levels of agreement also reported the highest levels of adoption or use: survey items of the 5 functionalities with greater than 90% agreement also reported greater than 90% adoption on both surveys.

Table 2.

Agreement between reports of HIT functionalities in the HIMSS and MN HIT survey data in the matched sample of clinic sites (n = 577)

	% Agreement	“Yes” to both surveys, % (n)	“No” to both surveys, % (n)	“Yes” to MN HIT, “No” to HIMSS, % (n)	“Yes” to HIMSS, “No” to MN HIT, % (n)
Clinic with certified EHR technology	92.9	92.9 (521/561)	0 (0)	7.0 (39/561)	0.1 (1/561)
EHR system with CPOE	96.5	96.5 (380/394)	0 (0)	3.2 (13/394)	0.3 (1/394)
EHR with CDS for
Medication guides	93.0	93.0 (436/469)	0 (0)	6.6 (314/469)	0.4 (2/469)
Clinical guidelines	94.3	93.8 (440/469)	0.4 (2/469)	2.4 (11/469)	3.4 (16/469)
Preventive medicine	83.6	82.5 (387/469)	1.1 (5/469)	8.7 (41/469)	7.7 (36/469)
Health information exchange with
Governmental agencies	71.7	66.6 (355/533)	5.1 (27/533)	23.8 (127/533)	4.5 (24/533)
Hospitals	85.6	78.8 (420/533)	6.8 (36/533)	13.5 (72/533)	0.9 (5/533)
Ambulatory clinics	82.9	78.0 (416/533)	4.9 (26/533)	14.3 (76/533)	2.8 (15/533)
e-Prescribing capabilities	100	100 (468/468)	0 (0)	0 (0)	0 (0)
Patient portal	85.4	85.4 (374/438)	0 (0)	13.5 (59/438)	1.1 (5/438)

Qualitative analysis of respondent understanding of survey items

In the interviews, medical group respondents provided insight into their understanding of the MN HIT survey items and how they arrived at their answers, and identified challenges answering certain items, particularly the items about magnitude and frequency of adoption and barriers. We identified 4 key thematic domains in their responses: clarity, relevance, process for completing the survey, and potential errors in responses. We elaborate on each below and show examples of the challenges they reported in Table 3.

Table 3.

Respondent reported challenges answering MN HIT survey items

Challenge	Survey item(s)	Examples
Clarity
Determining how often functions are used	CPOE and CDS use (Q12, Q14) with response options “routinely,” “occasionally,” “not available,” or “function turned off/not in use.”	“Routinely” could be interpreted to mean 95% of the time or daily or weekly; similarly, “occasionally” could mean 10–90%, monthly. The reason for infrequent use could also be because of lower need or because the system was transitioning to new EHR and hadn’t turned on many alerts yet; the question did not incorporate these possibilities. Also, variability in usage by individual clinician is not captured.
Parsing 2 concepts that were combined	Clinic uses its EHR for quality improvement efforts (Q34), routinely identifies and reminds patients who are due for preventive care (Q35)	The answer options do not distinguish the EHR’s ability to be used for these functions from the clinic’s ability to export data and then use the data for quality improvement. In some cases, the clinic issues reminders separate from the EHR, such as through practice management system, and it is unclear how to respond in that case.
	Option to select: “cannot or do not” electronically exchange information (Q41)	This question is difficult to answer for clinics that are actively working on implementation, because the question asked about 2 different concepts: ability to do something and whether it is actually used in that way.
	HIE-related items (Q38–45)	These questions do not distinguish different types of data exchange, such as query-based versus sending a DIRECT message. It also combines “inadequate setup” and “subscription fees” as a barrier for HIE within 1 response option (Q45), and the meaning of “needing” to share information with an organization (Q40) could be interpreted related to timely clinical needs or for regulatory requirements.
	Telemedicine (Q53)	Definition of an “originating site” is unclear to respondents if both sites are internal to the health system. “Lack of demand” as a potential barrier could be interpreted as lack of demand from providers or from patients.
	Portals (Q51)	This question’s definition of “patient portal” doesn’t clearly distinguish those that allow patient access to their provider-generated EHR data from those that allow patient access to data they entered themselves (Q51). For example, “Access to allergies list” may mean only allowing patients to add allergies, not integrated with the allergy list in the EHR.
Relevance
Applying to type of clinic	Preventive care reminders (Q35)	The question of preventive care is not as relevant to specialty clinics; thus, a lower rate of usage may be expected and appropriate.
Expressing key barriers	HIE barriers (Q41)	The question doesn’t capture these barriers: limited ability to incorporate external data and use it for care, lack of return on investment, lack of infrastructure on the part of other potential data sharing partners, faxes still required for notes, lack of patient permission to exchange, or limits to which EHR vendors the clinics can share data with.
	Quality measure and reporting barriers (Q34, Q27/28),	These questions do not include these barriers: lack of standardized quality measures across clinics, lack of use of reports for anything other than required reporting.
	Patient portal (Q50/51)	Answer options do not include usability of the patient portal as a barrier.
	CDS (Q14)	The question does not allow to indicate that CDS was designed for adults and mostly not relevant for a pediatric population.
	CPOE challenges (Q13)	The question does not include barriers to configuration that may constrain time during a medical visit.
Process for answering survey
Limited time to answer survey	Multiple (especially Q36 preventative care reminders)	Respondents used rough estimates or selected “unsure” rather than looking at data, used previous year’s survey responses as a starting point, answered the same answers for all clinics in group even if there was some variation.
Lack of 1 person knowing everything	N/A	Respondents asked multiple other staff for input and collected the responses. In 1 case, the medical director had final review of the responses.
Errors
N/A	Multiple	Respondents identified examples of typos, cases in which they had not read the questions carefully when responding to the survey, and responses which they could not explain.

Analysis of interview data with 8 respondents who answered the survey for their medical groups and associated ambulatory clinic sites.

Clarity of survey items

Respondents found some survey items and the concepts they asked about clear and straight-forward to answer. The item that respondents indicated having the least challenges understanding and answering was the one that asked if their medical group used an ONC-certified EHR (Q8). (All respondents we spoke to answered “yes” to that survey item.) Respondents indicated other items were clear as written because the medical group was either not using a particular functionality (eg, some medical groups were not electronically exchanging any data, or not using telemedicine), was using the functionality routinely without problems (eg, 2 respondents reported no ongoing barriers to CPOE use because it was part of routine care), or was familiar with items because they had to supply the same data for Meaningful Use requirements (eg, items about CPOE use). In most cases, respondents said they understood the meaning of items related to CPOE, CDS for medications, and patient portals, and were able to indicate if their medical group’s HIT system supported these functionalities (all did).

Respondents also identified examples of survey items and/or the concepts underlying them that were unclear to them. The most frequently cited example of an unclear item was related to an item about CDS use (Q14) in which the survey item included response choices: “routinely,” “occasionally,” “not available,” or “function turned off/not in use.” Respondents varied in their interpretation of these response choices, with one calling the response categories “very subjective.” In several cases, 2 or more distinct questions were combined into one, making interpretation challenging. For example, HIE-related items (Q38-45) did not distinguish different types of data exchange, such as query-based versus sending a DIRECT message—even though the answer to 1 question could be different than the answer to the other. Other challenges include, for example, understanding the meaning of an “originating site” in telemedicine if both the transmitting and the receiving site are internal to a single health system and defining a patient portal as one that allows patients access to their provider-generated EHR data rather than one that allows patients access only to data they themselves have entered (Q51).

Relevance of items

Respondents identified multiple instances in which they were unsure how to respond because survey items were not relevant to their organization or did not reflect the most relevant aspects of their experience implementing and using HIT. For example, respondents representing specialty medical groups reported that some items were clearly oriented toward primary care clinicians and not relevant to them, such as items related to preventive care reminders (Q35). An item about e-prescribing of controlled substances seemed irrelevant to several respondents given the regulatory rules in their state. One asked, “How can I e-prescribe controlled substances, when that’s not something the Board of Pharmacy allows here?” Key challenges that medical groups in MN face in their use of EHR functionalities were not captured in the survey: In 1 striking example, 1 medical group shortened their appointment time from 15 to 10 min after a new scheduling software installation because they did not have the authority to change that setting. As a result, they indicated barriers related to CPOE as having “time too limited during patient encounter to use,” but no survey item captured the underlying reason for this challenge.

Process for completing the survey

The process respondents used to develop the answers to the survey items varied in terms of the individuals involved, the kinds of data used, and how differences across clinics within the medical group were documented. Most respondents asked other staff—such as analysts, application directors, compliance officers, IT staff, or operations staff—for help with responding to the items, and those other staff sometimes were unable to answer the items which resulted in responses that may not have been accurate. Some respondents were directly familiar with the HIT functionalities and how they were implemented and were confident that they were able to answer the items accurately. However, some respondents were not familiar with the survey times, made rough estimates, or used their previous year’s survey responses as a starting point and modified only those items that they believed had changed. One said, “I kind of make the assumption that we would not lose any functionality.” All respondents answered the items for the medical group as a whole, but most acknowledged that their responses did not capture extant variation across clinics and most medical groups supplied the same item responses for all clinics, whether or not there was variation (including answers that asked about percentages). One respondent said, “We are not staffed to be able to sit down with every clinic and complete this in a good way.”

Potential errors in survey responses

In 4 of our 8 interviews, we identified survey responses that respondents themselves said were incorrect. These errors impacted items about adoption and use of some functionalities, frequency of use, and variation across clinics within a medical group. For example, when asked about an item related to EHR data for quality measurement, 1 respondent said, “I probably would change some of those responses, at this point even looking back, because I’m seeing reality more clearly.” Example of errors included: claiming to utilize telemedicine when the medical group did not; claiming use of CDS for high-tech imaging when the medical group did not; claiming use of preventive service reminders when the medical group’s EHR did not support them; claiming that the medical group did not use preventive service reminders when the medical group’s EHR actually did support them; counting an administrative office as a clinic; and including facsimile (“fax”) in their definition of health information exchange despite survey instructions to the contrary. Respondents said that the reasons for these errors included typographical errors, not reading the items carefully, and lack of time to devote to making sure their survey responses were correct. In some cases, the respondents could not explain the reason for the incorrect answer.

DISCUSSION

The principal finding of our study is that quantitative and qualitative data provide some evidence supporting the validity of some of the survey items included in the MN HIT and HIMSS enterprise surveys related to the presence or absence of HIT functions in ambulatory clinics operated by medical groups. Since there is no external gold standard, this validity is convergent: separate surveys report high agreement on the presence or absence of common functions, and interviews with knowledgeable medical group staff confirm those answers. MN HIT survey items related to functionalities that were highly adopted and required under federal incentive programs—such as CPOE, e-prescribing, and patient portals—had high levels of agreement with comparable HIMSS survey items and our interview respondents also indicated that these survey items were clearly interpreted. However, the findings from our qualitative interviews suggest that survey items that ask respondents for estimates of level of adoption, involve less well-established HIT functionalities, or ask about barriers to adoption may be more challenging for respondents to answer, and because of that respondents may omit critical aspects, and the resulting survey findings may contain invalid responses. Respondents at the medical group who are representing multiple clinics may also indicate greater homogeneity than actually exists across clinics within their medical group. The large number of challenges we identified in our interviews should give pause to analyses that depend on certain survey items without further evidence of validity. If inaccurate data are used in analyses to inform policy, there is a risk that important decisions will be misguided.

While this study was focused on ambulatory care, the findings may help to explain previous assessments of hospital survey data. Previous studies that assessed level of agreement for the adoption of EHR and CPOE during 2005–2008 found limited agreement among 2 different surveys at a time when adoption was low—1.5% of US hospitals for a comprehensive EHR and 17% for CPOE according to 1 survey during that time period—but a 2012 assessment (after the federal incentive program was established) using Medicare Meaningful Use attestation as a gold standard found “satisfactory” agreement with 1 survey. It is possible that when HIT functionalities are not widely adopted or not required as part of a national financial incentive program, survey respondents are less familiar with the functionalities and therefore have a more challenging time answering items related to those functionalities accurately. In fact, respondents in our interview study reported that they were able to answer some of the MNCM items accurately because they had already produced similar data for Meaningful Use attestation. This interpretation suggests that validation of some survey items for well-established functionalities does not necessarily support that other items in the same survey about less utilized HIT functionalities are also as valid.,

If true, a major challenge to using surveys for analyses of HIT is that the emerging functionalities that may be of greatest interest to policymakers are the most difficult to measure using surveys. If surveys are the only option (eg, to assess perceived barriers), additional effort is needed in their design and implementation to adequately ensure the respondents interpret the items correctly and answer them accurately. Without additional evidence of validity, survey results about HIT other than those related to widely adopted functionalities should be interpreted with caution. More automated methods for assessing HIT usage would likely be more reliable and should be pursued. Although surveys by nature cannot capture all of the nuances of real-world phenomena, additional qualitative investigation that informs survey design and implementation would help to capture more of the full range of health-care provider activity and experience using HIT, and provide greater assurance that the responses are accurate. Survey designers will likely need to test survey items using modes that are similar to how the ultimate survey will be fielded in addition to performing cognitive testing.

This work has several limitations. First, as already mentioned, there is no gold standard for HIT implementation, such as on-site inspection by a third party for the presence or absence of HIT functions. Second, the items that we compared were phrased somewhat differently in the 2 surveys and could be interpreted in different ways. This would likely result in lower levels of agreement than actually exist. For example, the HIMSS’ HIE items specifically refer to data exchange with entities outside of the health-care organization; MN HIT’s’ HIE items do not. However, our study was not an assessment of test–retest reliability, which would be the case if the items on the 2 surveys had been identically worded. Third, our sample was limited to 1 US state that had high levels of adoption of HIT and our sample of respondents’ may be biased.

CONCLUSION

Analyses of HIT survey findings that inform public policy should take into consideration the validity of the data used to develop estimates of prevalence and use. Responses to survey items that inquire about more widely adopted and well-established functionalities are more likely to be valid. Adequately capturing data on less established functionalities, degree of usage, variation across clinics within a health system, and barriers to implementation will likely require a more in-depth understanding of provider experience with HIT and a better understanding of how respondents to surveys arrive at their answers.

FUNDING

This work was supported through the RAND Center of Excellence on Health System Performance, which is funded through a cooperative agreement (1U19HS024067-01) between the RAND Corporation and the Agency for Healthcare Research and Quality. The content and opinions expressed in this publication are solely the responsibility of the authors and do not reflect the official position of the Agency or the US Department of Health and Human Services.

AUTHOR CONTRIBUTORS

R.S.R. conceived of the study, led the qualitative data collection, led the analysis, and drafted and revised the manuscript. Y.S. conceived of the study, conducted the quantitative analyses, and revised the manuscript. S.H.F. analyzed the qualitative data and revised the manuscript. P.S. conceived of the study, analyzed the data, and revised the manuscript. A.A. analyzed the quantitative data, analyzed the qualitative data, and revised the manuscript. B.S. analyzed the qualitative data, and revised the manuscript. M.S.R. provided critical comments on the protocol, and revised the manuscript. C.L.G. oversaw the study, and revised the manuscript. All authors approved of the manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Conflict of interest statement. None declared.

Click here for additional data file.