Factors impacting physician use of information charted by others.

OBJECTIVES: To identify factors impacting physician use of information charted by others.
MATERIALS AND METHODS: A 4-round Delphi process was conducted with physicians and non-physicians publishing in the healthcare data quality literature to identify and characterize factors impacting physician use of information charted by others (other people or devices), either within or external to their organization. Factors with high average importance and reliability were categorized according to similarity of topic.
RESULTS: Thirty-nine factors were ultimately identified as impacting physician use of information charted by others. Five categories of factors included aspects of: the information source, the information itself, the information user, the information system, and aspects of healthcare as an institution. In addition, 4 themes were identified: (1) value of narrative text in providing context, (2) importance of mental models and personal heuristics in deciding whether, and how to use information, (3) loss of confidence in, and decreased use of information due to errors encountered, and (4) existence of a trust hierarchy potentially influencing information use. DISCUSSION: Five similarly focused studies have recently probed clinician willingness to use information in decision-making. Our results mostly confirmed factors identified by prior studies, and uniquely identified aspects of the information user as important.
CONCLUSION: According to the participants in this study, information quality is prominent among factors impacting physician use of information charted by others. Based on this and similar studies, it appears that despite concerns about information quality, physicians use information charted by others.

INTRODUCTION

Increased efficiency, safety and quality of care through health IT depends on clinicians documenting and using data describing patient characteristics, care, and outcomes. In healthcare, there have been reports of reluctance to use information of questionable quality or information systems containing such. The National Institute of Standards and Technology (NIST) explicitly calls out inaccurate and incomplete data as root causes of Electronic Health Record (EHR) use errors. However, information of questionable quality or containing known deficiencies is used by clinicians,,, healthcare executives,, and international health authorities. Relationships between information quality and information use are predicted by information system theory and have been tested in other sectors. While some have probed the relationship between health IT and aspects of information quality—for example in computerized physician information or order entry, computerized physician documentation, and overall level of health IT implementation—the relationship between information quality and information use by clinicians has not been directly tested in healthcare. In the last decade through Federal legislation in the United States alone, over 500 million dollars were allocated toward the realization of health information exchange. Significant resources have and continue to be focused on bringing data from other sources into an aggregated patient-centric EHRs in pursuit of better care. Yet, clinician willingness and ability to use existing clinical information in EHRs has not been widely explored. Calls for such studies continue, and emphasize the need for broad assessments and including a wide range of factors.

BACKGROUND AND SIGNIFICANCE

There have been multiple attempts at enumerating dimensions of data quality important to or assessed in healthcare data., Most are targeted toward supporting quality assessment of EHR data for use in research. With few notable exceptions, there has been little work to date identifying and defining the information quality dimensions important to clinicians or their willingness to use information provided by others in clinical decision-making.

The quality of data in medical records has been questioned for decades with multiple reports of data quality problems and their impact appearing in the literature. The importance of information quality in healthcare has been noted in consensus reports and national efforts., These reports emphasize that our “knowledge of data accuracy in electronic medical records is not commensurate with it is importance” and emphasize the need for more studies.,,, In other industries as evidenced by inclusion in the Malcom Baldrige National Quality Award criteria, the importance of data quality to organizational performance is widely recognized. In healthcare, as our reliance on information grows, likely so will the impact of data quality.

Assuming importance of data quality in health care presumes: (1) that clinicians use data provided by others, (2) that data quality impacts clinical decision-making, and (3) that impact on decision-making translates to outcomes. Four recent studies focusing on clinician perception of the quality of data used in decision-making or clinical decision support have been reported. A fifth study reviewed literature and summarized reported data quality issues in EHRs likely to impact information use. All 5 studies focused on clinician perceptions of information quality or information quality problems impactful to clinicians.

Courtney et al surveyed clinicians attending a medical conference to examine clinicians’ perceptions of the trustworthiness and usefulness of information based on the source of the information. The survey addressed limited information sources for example, specialty providers and types of data such as vital signs. The study reported a moderate positive correlation between perceived trustworthiness and perceived usefulness of information in clinical decision-making, noted differences based on information source, and noted that technology data sources were rated at least as favorable as traditional sources.

McCormack and Ash analyzed existing ethnographic data gathered though iterative interviews and observations from 10 institutions using electronic data. The study included observation or interviews of 26 physicians, 21 nurses, 11 pharmacists, and an imaging technician. McCormack and Ash reported concern among study participants about clinical information used as a foundation for clinical decision support. Five data quality dimensions were reported as being important to clinical decision-making: completeness, accessibility, context specificity, accuracy, and reliability with the most common being completeness.

Skyttberg et al explored factors affecting vital sign data quality with respect to use in clinical decision support. Similar to McCormack and Ash, they used iterative interviews and observations to evaluate the clinicians’ perception of information quality. The study was conducted in 9 emergency departments. The research participants included 2 medical doctors and 14 registered nurses with at least 5 years of experience. The study concluded that workflow and process issues caused lack of currency, completeness, and interoperability resulting in a predictable lack of confidence in clinical information by healthcare providers.

Galster abstracted, categorized and reported information quality issues from the literature. Four primary reasons to refrain from using health information were identified and encompass intrinsic data quality dimensions as well as broader organizational and environmental factors such as access and availability.

West et al probed clinician perception of patient-generated data (PGD) as evidence for clinical decisions through a literature review and synthesis of 23 empirical studies of self-tracking tools. They report accuracy and reliability, completeness, context, patient motivation, and representation of information as concerns as well as differences in reliance on PGD as evidence in different situations, for example, seeking more trustworthy information for diagnosis versus increased willingness to consider PGD for new hypothesis generation or to stimulate patient recall.

The research methods from the aforementioned studies have many similarities. Two of the five studies, used interviews and selected participants from institutions using EHRs. In 3 studies, a variety of clinicians were included, with the Skyttberg et al study having multiple interactions with participants. The McCormack and Ash and Skyttberg et al, studies included an observational component for support of the interview data. Four studies, applied an inductive coding scheme allowing factors and categories to arise from the data rather than imposing a preconceived set of factors or structure. Two studies, synthesized information from published articles. All 5 studies report information quality problems impacting clinician use of information. However, the studies were each limited in scope which in turn limited generalizability. For example, 3 of the studies,, only addressed specific information sources and types of data. McCormack and Ash study was possibly limited by data having been collected as part of a larger program of research into “the perceptions and attitudes of clinicians who use clinical decision support (CDS).

To date, clinician use of data charted by others has rarely been studied. None of the available studies provides a comprehensive list of factors impacting use of information charted by others. Progress toward such a list is necessary so that the factors can be studied to determine which have an actual impact, under what conditions and to what extent. This information is needed prior to expending effort on data quality improvement efforts. This study was conducted to begin filling this knowledge gap.

METHODS

A 4-round Delphi process was conducted with those publishing in the healthcare data quality literature identified through the published literature. A total of 3400 citations were returned from PubMed using the search terms “data quality” or “data error” or “information quality” or “clinical documentation.” The 2000 articles within the prior decade were screened according to the criteria in Table 1 producing 89 citations for the full text review. The International Conference on Information Quality and Journal of Data and Information Quality (likely to contain articles about information quality in healthcare although not indexed in PubMed) were also searched returning 37 additional articles for the full text review. A total of 126 articles were reviewed resulting in 70 articles meeting the criteria in Table 1.

Table 1.

Inclusion and exclusion criteria

Inclusion criteria: Articles about data or information quality in patient care clinical settings, ie, primary rather than secondary data use. Articles must be about data or information quality, eg, assessment or intervention where the data or information quality is the topic of research or is a major finding rather than only mention of data or information quality. Articles must be about data currently captured in healthcare rather than evaluation or pilot of new data capture.

Exclusion criteria: Articles no longer available in PubMed. Articles about device data quality. Articles in languages other than English. Articles by the research team.

Approval was received from the Duke University institutional review board under Pro00030327 for the research. Attempts were made to contact by email the first or corresponding authors of the 70 identified articles for recruitment. Publically available contact information from each article was used. Contact information on thirteen articles was no longer active. Eight articles were written by an author already identified through another paper (5 of these were also invalid emails) Three contact attempts were made with the remaining 54 distinct authors identified through the literature. Authors 10 (18.5%) responded with interest in participating. The 10 respondents were consented to participate in the study to ensure a minimum of 7 participants remaining at the end of the last Delphi round. A total of 8 participants, including 3 health informatics researchers, and 5 medical doctors, remained actively engaged through the last round.

In Round 1 of the Delphi process, participants responded to 2 open-ended prompts: (1) “please list up to 10 things that, in your experience or opinion, impact a Physician or Physician Extender’s use of health information charted by others,” and (2) “please list the 5 most important aspects of EHR data quality with respect to data use in clinical decision-making, that is, to you, what dimensions of health information quality are important when the data are used for clinical decision-making.” Question 1 was asked first and did not mention information quality. The purpose was to solicit any factors impacting physician use of information in health records. In contrast, Question 2 specifically probed information quality factors. Following Round 1, all responses were reviewed and coded to obtain a list of distinct factors. Two clinical informatics researchers independently reviewed the coding.

For Round 2, each of the coded factors was presented as a statement, for example, “The extent to which information is concisely stated impacts a clinician's choice of whether to use information charted by others.” Participants were asked to rate their level of agreement with these statements on a 5-point Likert scale [(1) strongly disagree, (2) mildly disagree, (3) neither agree nor disagree, (4) mildly agree, or (5) strongly agree]. In Round 2, participants were also prompted but not required to list comments for each factor.

In Round 3, the participants were each provided an individualized report of their previous responses versus the Round 2 aggregate. In Round 3, participants were asked for more information about randomly selected factors where their response was within 1 point of the aggregate and all factors where their responses differed by more than 1 point from the aggregate. Participants were allowed to change their responses. One point was chosen because a difference of 1 point is the difference between the categories on the Likert scale. Requesting participants to tell the interviewer more about their responses enabled researchers to ensure consistent understanding of the factors and provided the researcher with more information and possible reasons where there were disagreements.

All factors identified during Round 1 were carried forward and rated in Rounds 2 and 3. Following Round 3, factor ratings were analyzed by average importance and reliability. Average importance corresponded directly to the average Likert rating. Reliability was operationalized as the standard deviation of the ratings. In Round 4 the participants were each provided the draft report of the analyzed data and given the opportunity to provide feedback for inclusion in the report.

Rounds 1 and 2 were conducted using the REDCap web-based survey system. Round 3 was conducted via structured phone interview, and Round 4 was conducted via email. The quantitative ratings were analyzed in Microsoft Excel (Microsoft corporation, Redmond, WA, USA). Qualitative data, that is, Round 2 reports and the Round 3 interview transcripts, were imported and coded by the first author in Nvivo qualitative analysis software (QSR International Pty Ltd, Victoria, Australia). Thematic analysis was performed in 6 phases including familiarization with data, generating initial codes, searching for themes among codes, reviewing themes, defining and naming themes, and producing the report.

Lastly, the factors obtained from the Delphi process were compared with 5 studies identified from the literature probing clinician perception of or experience with the quality of data used in decision-making or clinical decision support., These studies were identified from the original search and from the same search applied to the literature since the start of the study.

RESULTS

Round 1 of the Delphi identified 53 factors impacting clinician use of information charted by others. Importance and reliability ratings are provided in the appendix for each factor. Seven factors had importance ratings less than 3.5 and were dropped due to low importance. Thirteen factors, 6 of them overlapping with those also rated as having low importance, were dropped due to low reliability (standard deviation greater than 1.0). Any factor with either a low importance or a low reliability rating (factors with shaded mean or standard deviation in Supplementary Appendix S1) was discarded leaving 39 factors after Round 3. There were no changes to factors in Round 4. Together, these 39 factors represent the opinion of the Delphi process participants of factors impacting physician use of information charted by others.

The 39 factors were categorized according to similarity of topic forming 5 categories. The resulting 5 high-level categories (Table 2) include aspects of: the information source, the information itself, the information user, the information system, and aspects of healthcare as an institution. Twenty-five (64.1%) of the factors are either likely used as indicators of information quality or are directly considered dimensions of information quality (Supplementary Appendix S1). Thus, according to the participants in the Delphi process, information quality is prominent among factors impacting physician use of information charted by others.

Table 2.

Categorization of factors

Factor category	Initial factors	Remaining factors
Aspects of the information source	8	7
Aspects of the information	22	14
Aspects of the information user	5	4
Aspects of information systems	14	10
Aspects of health care as an institution	4	4

Initial factors are those identified in Delphi Round 1. Remaining factors are those remaining after dropping low importance and low reliability factors.

Coded interview results were analyzed to identify patterns across participant responses important to the description of clinician use of information charted by others. Topics noted by more than 1 participant in the qualitative data were noted as themes. Four themes were identified (Table 3).

Table 3.

Summary of themes identified from qualitative analysis of Round 3 interviews

Theme I	Value of subjective and narrative text: narrative text was valued by participants because it often contains information not found elsewhere in the record and because the construction and content can indicate the competence of the recorder.
Theme II	Use of mental models and heuristics to gauge information quality: different mental models or heuristics were mentioned by multiple participants as being used to identify questionable information or to weight questionable information lower in decision-making.
Theme III	Loss of confidence in information and decreased future use of information after encountering poor quality data: Multiple participants commented that past encounters with errant information caused them to discount future information from the same source or decreased their willingness to use the system in which they previously encountered the errant information.
Theme IV	Potential existence of a hierarchy of trust in information sources: multiple participants described hierarchies with respect to their confidence in information from others, eg, information from individuals personally known to be competent was trusted more than information from individuals known only by reputation or from individuals unknown to the user.

Theme I: value of subjective and narrative text for the information content and as an indication of competence of the recorder

Multiple participants noted that subjective information in the chart can be valuable to clinicians. One participant stated that, “some specialties are very narrative driven” and that, the “rationale for decisions [often found in the narrative text] invariably describes a clinical context that strikes a chord with clinicians who use the EHR.” One participant provided the example of seeing “is a pain drug seeker” in the chart and that while subjective and judgmental, it was valuable information that would not have been communicated through the structured and more objective data. Another participant pointed out that subjective information from an “inexperienced colleague carries little currency,” whereas “subjective from an experienced senior clinician has significant value.” Three participants noted that evidence of clear or rational thinking in the narrative record increases their confidence in data while lack thereof decreases their confidence. This finding underscores the value of narrative text in providing clinical context not usually available in structured data and as an indicator of the competence of documenter used by the physician reader in gauging the confidence to place in the information. Another participant noted that reliance on subjective, and often narrative, information varies by specialty. The participant gave the example of surgical thinking being more propositional and relying less on subjective information while documentation from a pediatrician with a complex case in family context will tend to be more subjective and narrative. The participant further added that a pediatrician will usually be more welcoming of narrative text as will professionals in areas such as psychology and social care. These findings add depth to the “structured data versus narrative text” debate. Based on these comments, it seems that both are needed and that the extent of each needed and used depends on the medical specialty.

Theme II: existence of mental models and heuristics used to gauge information quality

A second theme that arose from the qualitative data was a clinician’s ability to look at data values and through personal heuristics, come to a decision about whether or not to use the information. Clinician participants gave examples such as, “consistent with other information,” “precisely stated with objective facts,” “a relevant and reliable test,” “does the information make sense in terms of scientific and clinical validity,” and is the information “relevant to the problem at hand” in explanation of how they assess whether to use data in decision-making and if so, to what extent they will rely upon it. From the Delphi participant responses, it seems likely that clinicians develop personal mental models or heuristics upon which they rely, possibly subconsciously, to determine whether or to what extent to use data found in the health record in their decision-making. Two participants stated that their training emphasized the physician’s responsibility for decisions and the importance of evaluating data upon which decisions are based, for example, “you were held accountable for decisions that you made, and the data on which they were based.” The same participant metaphorically described depending on information charted by others as “betting your [medical] license.” Areas for future investigation include understanding these mental models or personal heuristics, their development and the extent of use to identify what if any information could be provided with information in the chart to better support clinicians in decisions of whether or how to use data in clinical decision-making, and once those have been determined and tested, how best to incorporate them into graduate medical education.

The mention of relevance of the information to the problem at hand is an important finding with respect to the value of information in decision-making and the timing of the physician becoming aware of the information. For example, if a decision has already been made the physician will need to adjust the initial assessment based on the new information. Further, in a hypothetico-deductive context new data coming after an initial decision has been made may represent missing an earlier opportunity to order tests or otherwise collect additional information. Thus, information late to the decision-making process may have diminished value.

Theme III: loss of confidence in information and decreased future use of information after encountering poor quality data

A third theme that arose was the relationship between information quality and information use. Multiple participants noted that recollections of errors in information from a person, information system, or institution decreased their willingness to use other information from that source. For example, a participant speaking about relying on further data from another clinician stated, “If results from an individual have been previously wrong, physicians will not use them [further data from the same individual] for patient care.” Another participant stated, “If I had experience of a system and more than once came across errors, I would lose confidence.” This supports the findings reported by others, that a positive correlation exists between information quality and willingness to use information. Half of the participants in this study made direct statements of this nature.

The impact of information quality on information use is possibly mediated by relevance of the information to a present clinical problem and by the level of effort required to access the information. Based on participant responses, clinical relevance has a strong impact. Regarding the level of effort required to obtain the information versus the need for the information, a participant stated that extensive effort required to obtain information “would affect whether to use [information] only if it was too much hard work, but if it is relevant and they can find it, then yes, they would use it,” and followed the statement with, “When it comes to lower level of priority, you put it off.” Another participant stated that, “If you think it is useful information, you don’t mind about ease of access,” and they mentioned research having shown that relevance is more important than ease of use. Finally, another participant noted that the impact of information quality on information use is less often a choice of whether to use data than it is a decision of how to use it.

Theme IV: Potential existence of a hierarchy of trust in information sources

The last theme identified from the Delphi process describes a hierarchy of trust in information sources. Participants in Round 1 of the Delphi process enumerated multiple aspects of trust or lack thereof in information sources including, individuals, institutions, and information systems. As these were further described in Rounds 2 and 3, a potential hierarchy emerged. The trust hierarchy of individuals charting information, ordered by highest trust first, included (1) individuals personally known to the information user, (2) individuals known through the information-user’s familiarity with their documentation, and (3) individuals known to the information-user by reputation only. Where there was no knowledge of the individual charting the information, information users factored in the seniority, and medical specialty/sub-specialty of the individual charting the information. Similarly, a hierarchy was suggested for institutions including (1) personal knowledge of data or documentation practices from an institution and (2) an institution’s reputation.

DISCUSSION

In this study, 39 factors and 4 themes impacting use of information charted by others have been identified (Supplementary Appendix S1 and Table 3). The factors have been further sorted into 5 higher-level categories (Table 2). The small Delphi process reported here focused directly on factors impacting information use and was not constrained to any specific clinical setting, information source or type of data. In this regard, our study was more comprehensive than other studies. However, by focusing on perceptions rather than actual behavior and in terms of the number of participants, our study was limited. With 1 exception, our results confirmed the factors identified by prior studies. Our study uniquely identified aspects of the information user including (1) pre-existence of or propensity for trust in information or information sources, (2) familiarity with the information system and local data flow or workflow, and (3) familiarity with local clinical documentation practices as important.

The Courtney et al study surveyed clinicians’ perceptions of information trustworthiness and use based on the source of the information. Thus, it is expected that the factors identified by the study are by design limited to aspects of the information source. McCormack and Ash analyzed existing ethnographic data gathered though iterative interviews and observations from 10 institutions using electronic data. The detailed analysis reported in McCormack and Ash was of one theme, “data as a foundation for CDS,” arising from an earlier study. Because the theme is quite similar to ours in scope, a high degree of overlap in the factors identified was expected. However, the McCormack and Ash study did not identify aspects of the information source, aspects of the information user or aspects of healthcare and health information exchange as an institution. We attribute this to the McCormack and Ash data having been collected as part of a larger program of research into “the perceptions and attitudes of clinicians who use CDS.” Topics covered in the initial data gathering broadly included backgrounds, roles, culture, history, barriers and facilitators, knowledge management, governance, and clinician views with respect to CDS. Thus the questions during the initial data collection were broad possibly limiting the detail achieved in any one area. An alternative explanation was that because participants in the Delphi process had published articles about healthcare data quality, they were predisposed to focus on data quality even in the presence of the initial broad question about factors impacting information use. All 6 factors identified in the McCormack and Ash study were also identified by the Delphi Process.

The focus of the Skyttberg et al study, vital signs data, was the most narrow of the prior studies. However, the study reported factors in 4 of the 5 areas identified by the Delphi process. There was no opportunity for contamination since the Delphi process had concluded before publication of the Skyttberg results. With high overlap at the category level and having been conducted by very different methodology (iterative interviews and observations) the concordance between the high-level categories is encouraging. However, at the detail level, the Skyttberg et al study identified only 11 (28%) of the 39 factors identified by the Delphi process. The lower number of factors may be attributable to the Skyttberg et al study focusing on vital signs data. This suggests that the individual factors identified by the Delphi process may not all be applicable to every type or source of data and remains an area for further research. Further, the Skyttberg study collected data through direct observation, limiting the results to observable factors. The Delphi process, on the other hand collected perceptions. There is often a difference between perceptions and what individuals actually do. Thus, the methodological differences are another possible reason for differences between our results and the Skyttberg et al results. The Skyttberg et al conclusion that workflow and process issues caused lack of currency, completeness and interoperability resulting in a lack of confidence in clinical information by healthcare providers is a cause for concern and reason for further inquiry.

Galster abstracted, categorized, and reported information quality issues from the literature. Four primary reasons to refrain from using health information were identified and include intrinsic data quality dimensions as well as broader organizational and environmental factors such as access and availability. Findings from the study matched the Delphi Process results in 4 out of the 5 high-level factor categories.

It is notable that the Delphi Process was the only study to identify aspects of the information user as factors in physician use of information charted by others. All of the factors in this category pertained to lack of trust in or knowledge of a data source held by the information user. Though, it makes sense from first principles that uncertainty or altogether lack of knowledge about a data source would impact use of data from the source, this category of factors requires further inquiry regarding their importance. This is the sole category of factors identified by the Delphi Process that was not corroborated by another study.

The West et al study probed clinician perception of PGD as evidence for clinical decisions through a literature review and synthesis of 23 empirical studies of patient self-tracking tools. They report factors in 4 of the 5 categories identified by the Delphi Process as well as an additional factor not identified by the Delphi Process—data representation. As a literature review, their findings are limited by the factors reported in the source articles. Focusing on clinician use of patient provided self-tracking data the scope of the study was restricted to one source of data. The importance of representation to information use by and performance of humans has been repeatedly demonstrated in the cognitive science and engineering literature. Lack of report of this factor elsewhere, including our Delphi Process signals again that factors applicable to one data source may not be as impactful to data from other sources. It also signals that the list of factors is likely not yet complete and beckons further study in the area.

All but one of the studies used a bottom-up approach with factors arising from the data, increasing the likelihood of completeness over the domain of inquiry. The questions posed in Round 1 of the Delphi process in our study were the most broad of the aforementioned studies (with the exception being the McCormack and Ash study as previously noted). Thus, we anticipated that our resulting factors would encompass those identified by the other studies. With the exception of information representation identified by the West et al review, this was the case.

Significant resources have been expended toward regional health information exchange and institutional data aggregation. Both with the goal of bringing data from other sources into an aggregated patient-centric EHR and assuming that better care would result. Few would argue that the desired improvement is dependent on use of the aggregated data in clinical decision-making. To the extent that data quality precludes, inhibits or enhances productive use of data in clinical decision-making, the importance of data quality in healthcare increases.

To date studies, including the Delphi Process reported here, have probed clinician use of information charted by others, and only assessed physician perceptions of the impact of data quality on clinical decision-making. Ultimately, we need to understand the impact of information quality on clinical decisions, clinical processes and clinical outcomes. Only then will institutions be able to gauge (1) the effort that should be allocated to achieving and maintaining information quality, (2) the dimensions of information quality important to clinical decision-making, and (3) acceptable quality levels. While the impact of information quality on information use is important to clinical decision-making, this is an intermediate relationship. Establishing whether causal relationships exist between information quality, clinical processes and their outcomes, and if so, characterizing the conditions under which these relationships hold is a necessary precursor to information quality management in healthcare. Though the Delphi Process reported here and the similarly focused studies included in the discussion are a step towards such knowledge, additional studies are necessary for understanding the existence and extent of impact of information quality on clinical processes and outcomes.

Limitations

The results here are limited by the small size of the study. This limitation is somewhat mitigated by interpreting the results in the context of the other similarly focused studies. As previously noted, methodological differences in the relevant studies are a possible reason for differences in results. For example, some studies—ours included—collected perceptions while others collected data through direct observation. The latter are limited to directly observable factors and the former are limited by the difference between what people say and what they actually do in practice. Our knowledge on the topic is richer through this methodological diversity. Interpretation of our results is constrained by the limited amount of demographic information collected by the participants. Length of time practicing medicine and the specialty would have further contextualized the results. However, divulging these details would risk inadvertently identifying participants. The 4 themes identified from the qualitative analysis arose from responses to targeted questions about the factors in the context of this study. Thus, they are only representative of the Delphi participants.

CONCLUSIONS

Based on the Delphi process participants, information quality is prominent among factors impacting physician use of information charted by others. Consistent with recent and similarly focused studies, the Delphi results supported the notion that despite quality concerns, data charted by others is routinely used in clinical decision-making. With one exception, our results confirmed the factors identified by prior studies. Our study uniquely identified aspects of the information user including (1) pre-existence of or propensity for trust in information or information sources, (2) familiarity with the information system and local data flow or workflow, and (3) familiarity with local clinical documentation practices as important. Though, we do not yet understand the impact of data quality on clinical decision-making or ultimately outcomes, significant effort continues toward availability of data for use in routine care. More studies are needed.

FUNDING

The Delphi study and literature synthesis reported here was funded by grants K99-LM011128 and R00-LM011128 from the National Library of Medicine, a component of the National Institutes of Health (NIH) and by Duke University’s institutional commitment to the grant. Further, this research utilized resources supported by the Duke Clinical and Translational Science Award grant UL1TR001117. The CTSA initiative is led by the National Center for Advancing Translational Sciences (NCATS) at the National Institutes of Health. The opinions expressed here belong solely to the authors and are not necessarily representative of the NIH.

CONTRIBUTORS

M.N.Z. drafted initial versions of the manuscript, data collection forms, Delphi report, administered the Delphi process and analyzed the responses. M.P. participated in preparation of the manuscript and conducted the synthesis of the similar studies. W.E.H. served as the mentor for the K99 award supporting this research. Dr Hammond participated in the design of the study, the data collection forms and the analysis of responses from the Delphi rounds. He reviewed and commented on multiple versions of the final Delphi report and the manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Click here for additional data file.