Analysis of Clinical Variations in Asthma Care Documented in Electronic Health Records Between Staff and Resident Physicians.

Clinical documentation using free text to describe a patient's medical status is an essential component of electronic health records (EHRs), and the quality of information in documents plays a critical role in clinical practice and translational research. Physicians are the primary creators of EHRs, but their clinical practices vary substantially, resulting in variations in clinical documentation. These variations can represent a source for potential bias in clinical outcomes and downstream applications using EHRs. Asthma is one example, presenting an inconsistent ascertainment process and criteria. A recent study revealed that resident physicians' knowledge of asthma diagnosis and management is relatively limited. In this study, we examined clinical documentation variations in asthma care between staff and resident physicians using individual words, topics, and asthma-related concepts in EHR clinical narratives. Additionally, we discuss potential biases in building an informatics model and further compare asthma diagnosis and outcomes between two physician groups.

Introduction

Asthma is one of the five most burdensome diseases [1] and the most common chronic childhood illness, affecting approximately 4%–17% of children in the United States [2; 3]. Structured data (e.g., ICD-9 codes) in electronic health records (EHRs) reveal a low sensitivity level to the accurate identification and management of children with asthma. The recent growth of unstructured clinical documents in EHRs and clinical text analytics using machine learning and natural language processing (NLP) have demonstrated great potential in the areas of clinical practice and translational research. These techniques have been successfully applied in asthma [4] and other various clinical applications [5-12].

The clinical documentation of a patient’s medical status using free text is an essential component in EHRs and the quality of information accumulated in clinical documents plays a critical role in patient care and clinical research. Physicians are the primary creators of EHRs, but their clinical practices vary substantially [13], which may significantly affect the quality of both structured (e.g., billing codes) and unstructured data (e.g., clinical notes) and cause potential bias in the secondary use of EHR data. Lack of a consistent approach to patient care may also result in higher costs and inevitably makes it necessary to reduce clinical variations at the point of care [14; 15]. For example, asthma has an inconsistent ascertainment process and criteria that cause delays in diagnosis, despite the availability of effective asthma therapies. Further, it has been reported that junior healthcare providers’ (e.g., residents) knowledge of asthma diagnosis and management is in development and needs to be improved through an interactive course [16]. Resident physician documentation often contains inaccurate information in progress notes [17] and as a result, the effect of a point-of-care personal digital assistant has been examined as a means of reducing resident documentation discrepancies [18]. Another study investigates the effect of an EHR template on family medicine resident documentation to improve asthma severity classification and treatment [19].

The quality of clinical documentation is crucial for downstream applications of EHRs to ensure the strong performance of informatics systems developed based on them. However, there are gaps between experienced (i.e., staff) and inexperienced (i.e., residents) healthcare providers in clinical documentation, and this can result in unintended consequences. To build accurate informatics systems, clinical variations between two provider groups need to be identified and addressed. Inappropriate or missing representations of a patient’s medical status as a result of inexperienced healthcare providers may diminish preventive opportunities and lead to inefficient healthcare delivery. This challenges physicians in their efforts to improve the value of healthcare delivery.

To address this problem, it is important to identify major variations in clinical documentation and understand the nature of those variations. Previously, researchers have investigated differences in laboratory tests in a physician’s practice to identify overlooked information regarding physician behavior [13]. The association between residency training and clinical practice [20; 21] was studied to identify variations in terms of clinical education. Regional variances among healthcare providers were also explored in diagnostic practices measured in Medicare claims [22]. Although it has been observed that the recent trend of novel efforts to promote rational use may cause variations in physician practice [23], the nature of such variations are not well understood. These previous studies utilized structured data and educational backgrounds to analyze physician behavior rather than the rich source of unstructured data in EHRs (i.e., clinical narratives).

In this study, we assessed variations in clinical documentation, especially clinical narratives, between staff and resident physicians in asthma care using individual words, topics, and asthma-related concepts. Further, we compared actual asthma diagnoses and outcomes between two physician groups and discussed the potential impacts of clinical document variations on quality of care and an informatics model.

Methods

We examined basic statistics and similarities in clinical documents between staff and resident physicians to identify clinical documentation variations. Asthma diagnoses and outcomes statistics were then compared to analyze gaps in clinical practice. This study was approved by the Mayo Clinic Institutional Review Board. Detailed methods are described in the following subsections.

Patient Cohort

Our study subjects are children who received medical care from Mayo Clinic pediatric practice, Rochester, Minnesota. As of 2015 September, 19,064 children were cared for by Mayo Clinic pediatric practice. Of these, 9,277 children were included for this study who participated in an ongoing asthma intervention study.

To examine document statistics and similarities we used the all patient cohort (n=9,277) and associated clinical notes. For the comparison of asthma diagnoses and outcomes, we selected patients from 2013 to 2015 who had clinical visits with either staff or resident physicians in Community Pediatric & Adolescent Medicine (CPAM) based on the 2015 physician list (n=4,102).

Clinical Document Statistics and Similarities

Basic statistics on clinical documents (i.e., total number of clinical documents/tokens, frequency of asthma-related concepts, note types, and sections in clinical notes) were compared between staff and residents. Staff and resident clinical document similarities were also compared in terms of word-level, topic-level, and asthma-related concepts. We preprocessed the documents (i.e., tokenization, removing stop words, and stemming) and created a vector space model for each case to compare similarity. The similarity was measured by cosine similarity since it is useful for comparing how two documents or corpora are likely to be in terms of their subject matter [24].

The clinical documents were represented by tf-idf (term frequency-inverse document frequency) and latent topic-based vector for computing word-level and topic-level similarity, respectively. The tf-idf for the term t is defined by summation of tf (t) idf (t) for all documents in the corpus divided by total number of documents in the corpus, i.e., Σ (t) idf (t)/N.

In order to compare clinical documents in the latent topic space, we employed a topic modeling method, Latent Dirichlet Allocation (LDA) [25; 26], to generate the document distributions in the latent topic space, i.e., p(zk|di). In the latent topic-based vector representation, the topic zk for the corpus C is defined as: To compute similarity of asthma-related concepts we extracted concepts used in the predetermined asthma criteria (PAC) (see Table 1) and represented them in a vector space model using the same notion of tf-idf representation.

Table 1

Asthma-related concepts used in predetermined asthma criteria.

Cough, Asthma, Infantile Eczema, Wheeze, Night-time disturbance (nocturnal cough/wheezing), Hay fever, Dyspnea, Bronchiolitis, Pulmonary Test, Bronchodilator, Bronchospasm, Methacholine Test, Positive Skin Test, COPD, Nasal Polyps

Asthma-related Concept Extraction

We used the MedTaggerIE module [27] in MedTagger to extract asthma-related events and episodes. MedTagger is the open-source pipeline developed by Mayo Clinic that contains a suite of programs including three major components: indexing based on dictionaries, information extraction based on patterns, and machine learning-based named entity recognition [27; 28]. Asthma-related concepts (Table 1) were compiled from PAC [29] that were originally developed by Yunginger et al.[29] and have been used extensively in research for asthma epidemiology. Each concept consists of a set of keywords.

Asthma Diagnoses and Outcomes

We compared asthma diagnosis statistics based on both physician-diagnosis and PAC between staff and residents. Physician-diagnosed asthma was determined by examining the diagnosis section in clinical notes, and PAC-based asthma diagnosis was determined by applying our NLP-based PAC system [4]. Further we compared outcomes of asthma care, such as ER visits for asthma, hospitalization for asthma, and asthma exacerbation (i.e., oral steroid use for asthma).

Results

Clinical Document Statistics

The basic descriptive clinical document statistics of staff vs. resident physicians are shown in Table 2. The list of providers (i.e., staff or residents) was based on the year 2015. Staff have a higher number of patients and documents because they tend to stay longer than residents at the clinic. Interestingly, residents have a much higher median number of tokens (words) per document than staff. Both staff and residents have a similar proportion of documents that contain asthma-related concepts and median number of asthma-related concepts per document.

Table 2

Basic statistics of clinical notes written by staff and residents for pediatric patients

	Staff (n=21)	Resident Physicians (n=56)
Date range	1998 – 2015	2011 – 2015
No. patients seen by staff or residents	8,434	4,252
Total no. documents	97,443	16,268
No. documents/year, median (IQR)	4,888 (5,491)	2,120 (5,238)
No. tokens/document, median (IQR)	274.0 (392)	530.5 (655)
No. documents with asthma-related concepts (%)	22,504 (23.1)	4,301 (26.4)
No. asthma-related concepts/document, median (IQR)	2 (4)	2 (4)

Figure 1 shows the distribution of asthma-related concepts between staff and residents. The concept cough appears most dominant in both providers’ notes, but staff documented it more frequently than residents. The next most frequent concepts for staff were asthma, eczema, and wheezing, and for residents, asthma, dyspnea, eczema, and wheezing (asthma and dyspnea appear in very similar patterns). Residents’ documents have a much higher frequency for dyspnea than staff. It should especially be noted that asthma appeared less frequently in resident documentation, even though the other concepts appeared in similar proportion, including wheezing, which is the most important concept in determining a patient’s asthma status.

Figure 1

Asthma-related concept distribution (proportion>=1%).

Figure 2 shows the distribution of note types that contain asthma-related concepts. Both staff and residents recorded asthma-related concepts dominantly in limited exam (LE) and multi-system evaluation (ME) notes. Residents also recorded these concepts in progress (PRG) notes and also used admission (ADM), observation (AOB), and consultant (CON) notes, while staff rarely used these note types.

Figure 2

Distribution of note types that contains asthma-related concepts (proportion>=1%). LE: limited exam, ME: multi-system evaluation, SV: subsequent visit, MIS: miscellaneous, SUP: supervisory, TOM: test-oriented miscellaneous, PRG: progress, ADM: admission, AOB: observation, CON: consultation

The distribution of sections that contain asthma-related concepts between staff and residents is shown in Figure 3. Staff used history of present illness (HPI) most frequently, followed by impression/report/plan (IRP), but the order is reversed for residents. Residents also used the subjective section, which was rarely used by staff.

Figure 3

Distribution of sections that contains asthma-related concepts (proportion>=1%). HPI: history of present illness, IRP: impression/report/plan

Clinical Document Similarities

Similarities between staff and resident clinical documents were examined in terms of word-level, topic-level, and asthma-related concepts (Table 3). Word-level similarity was not high, but topics and asthma-related concepts between two provider groups were very similar.

Table 3

Similarities of clinical documents between staff and resident physicians

	Cosine similarity
Word	0.718
Topic	0.991
Asthma-related concept	0.964

Asthma Diagnosis and Outcome

Table 4 contains statistics pertaining to asthma diagnoses and outcomes for patients whose CPAM physicians have been either staff or residents for the past three years (2013 to 2015). Staff diagnosed asthma more than twice as often as residents in both physician-diagnosed (10.4% and 4.5%, respectively) and PAC-based asthma (10.5% and 4.6%, respectively) when compared based on the proportion of their patients. For each of three asthma outcomes (i.e., ER visit for asthma, hospitalization for asthma, and asthma exacerbation), staff demonstrated better quality of care (i.e., lower ratio of incidents) than residents and they were statistically significant (95% CI) when considering all three outcomes as a whole.

Table 4

Asthma diagnoses and outcomes for staff and resident patients (2013 to 2015)

		Staff	Resident
Basic statistics	No. patients	3,304	798
	Median No. documents/patient (IQR)	3 (5)	2 (2)
Asthma diagnosis	No. patients of physician diagnosed asthma (%)	342 (10.4)	36 (4.5)
	No. patients of PAC asthma (%)	348 (10.5)	37 (4.6)
Outcome†	ER visit for asthma, ratio	0.067	0.306
	Hospitalization for asthma, ratio	0.012	0.083
	Asthma exacerbation (oral steroid use), ratio	0.211	0.333
	Any of above three outcomes (p-val = 0.034)	0.234	0.417

No. patients with a given condition/No. patients with CPAM physician-diagnosed asthma

Discussion

Clinical documentation between staff and resident physicians in asthma care has been examined in terms of document contents, asthma diagnosis, and asthma outcomes. Basic document statistics (i.e., the number of documents, tokens, asthma-related concepts, and note types) differed in number, and the individual words used in documents were not so similar (word-level similarity = 0.718). However, concepts used in their documents were very similar (i.e., topic similarity = 0.991; asthma-related concept similarity = 0.964), which may reflect the fact that both staff and residents share common terms to describe asthma episodes and events.

For asthma-related concepts in clinical documents (Figure 1), staff and residents demonstrated similar distribution except for the concept of dyspnea (consists of keywords: dyspnea, shortness of breath, breathing difficulty, respiratory distress, etc.). Interestingly, residents recorded a much larger portion of dyspnea (17.3%) than staff (3.2%), which is close to the proportion of residents’ recorded asthma concepts (17.5%). Staff and residents showed different usage of note types that contained asthma-related concepts (Figure 2), although the top two frequently used note types were the same (limited exam, multi-system evaluation). Over 90% of asthma-related concepts were found in three note types for staff (limited exam, multi-system evaluation, subsequent visit), while residents used additional note types (progress, admission), up to 90% of asthma-related concepts. We also investigated overall note type distribution regardless of asthma-related concepts and observed that progress notes were the second most frequently used notes for residents (23%) after multi-system evaluation (23%). Staff rarely used admission and observation notes, which may be one reason for note type discrepancies that contained asthma-related concepts. The overall trend of using sections for asthma-related concepts (Figure 3) was similar between two physician groups, but interestingly, residents used impression/report/plan sections most frequently, as compared to history of present illness, which was most commonly used among staff. Additionally, residents did not use diagnosis sections as often as staff. This may reflect the fact that residents are less likely confirmative in their asthma-related observations with regard to patient medical status.

There were notable gaps in clinical documentation between staff and resident physicians in asthma care. Our study shows that residents were less likely to diagnose asthma than staff even though the distribution of other asthma-related concepts appeared to be similar. Subsequently, asthma outcomes (ER visits, hospitalization, and oral steroid use) among patients of residents were relatively poor when compared with those of staff.

The recent growth of EHRs provides potential for comprehensive tracking of patient history, which allows evidence-based medicine to address treatment uncertainty and reduce clinical variances. However, EHRs may contain hidden biases and noise factors unrelated to disease burden [30; 31]. Without careful analysis of such biases, an informatics model built on such EHRs may not perform as expected. Along this line, differences in clinical documentation between staff and residents observed in our study must also be considered in order to ensure the accuracy of informatics models.

The similarity measure of asthma-related concepts used in our study depends on the type of concepts in the entire documents of each physician group, not necessarily individual numbers of each concept per physician. The high similarity reflects that both staff and residents use similar concepts, and therefore residents may not be required to undergo additional education to ensure proper use of additional terms or definitions. However, it does not show the actual presence/absence of individual concepts representing a patient’s medical status per physician. Further assessment of detailed presence/absence concepts of residents’ documentation representing patient medical status and divergence analysis compared to staff could be a strong potential resource, not only in the education of residents, but also in an effort to improve clinical documentation so as to reduce clinical variations.

Limitations of this study include lack of analyzing effects of potential confounding factors (e.g., patient’s demographic, environment, risk factors, and healthcare utilization) and the use of a relatively short period of time (three years) to analyze and compare asthma diagnoses and outcomes. We plan to consider these factors in the future and expand divergence analysis in asthma care at the individual resident level.

Conclusions

Clinical documentation variations were observed in asthma care between staff and resident physicians when utilizing note types, sections, and ascertaining asthma-related concepts. Additionally, residents were less likely to diagnose asthma than staff and were responsible for relatively poor outcomes in asthma care because a delay in asthma diagnosis may prevent asthmatic children from access to therapeutic and preventive interventions. The variations identified by our analysis may represent a potential bias, not only for clinical outcomes, but also for downstream applications using EHRs, and this needs to be considered in order to better interpret clinical research outcomes and the performance of informatics models for clinical decision support.