Comparative assessment of content overlap between written documentation and verbal communication: an observational study of resident sign-outs.

OBJECTIVE: Effective sign-outs involve verbal communication supported by written or electronic documentation. We investigated the clinical content overlap between sign-out documentation and face-to-face verbal sign-out communication.
METHODS: We audio-recorded resident verbal sign-out communication and collected electronically completed ("written") sign-out documentation on 44 sign-outs in a General Medicine service. A content analysis framework with nine sign-out elements was used to qualitatively code both written and verbal sign-out content. A content overlap framework based on the comparative analysis between written and verbal sign-out content characterized how much written content was verbally communicated. Using this framework, we computed the full, partial, and no overlap between written and verbal content.
RESULTS: We found high a high degree of full overlap on patient identifying information [name (present in 100% of sign-outs), age (96%), and gender (87%)], past medical history [hematology (100%), renal (100%), cardiology (79%), and GI (67%)], and tasks to-do (97%); lesser degree of overlap for active problems (46%), anticipatory guidance (46%), medications/treatments (15%), pending labs/studies/procedures (7%); and no overlap for code status (<1%), allergies (0%) and medical record number (0%). DISCUSSION AND
CONCLUSION: Three core functions of sign-outs are transfer of information, responsibility, and accountability. The overlap-highlighting what written content was communicated-characterizes how these functions manifest during sign-outs. Transfer of information varied with patient identifying information being explicitly communicated and remaining content being inconsistently communicated. Transfer of responsibility was explicit, with all pending and future tasks being communicated. Transfer of accountability was limited, with limited discussion of written contingency plans.

INTRODUCTION

Resident sign-outs (or handoffs) involve the formal transfer of patient care information, responsibility, and accountability between outgoing and incoming residents for maintaining the continuity of care. With the revised Accreditation Council for Graduate Medical Education (ACGME) guidelines, the number of sign-outs have significantly increased, with reports suggesting that medicine residents are involved in over 300 sign-outs per month., Given the complexity of information transfer during sign-outs, they are viewed as a threat to patient safety and contribute to medical errors.

Medical errors during sign-outs have been investigated based on the content and quality of verbal sign-out communication, and that of written and electronic sign-out documentation., A survey of internal medicine and surgical residents found that nearly 30% of sign-out communication was of poor or fair quality. The low quality of verbal communication has been attributed to missing, incorrect, conflicting, or irrelevant information exchanged by outgoing residents during sign-outs, or misinterpretation associated with overestimation of incoming residents’ understanding.,,

Studies evaluating sign-out documentation (written and electronic) found that resident documentation varied in their composition and organization of clinical content. Rosenbluth et al. noted that there was significant variability in both the structure and content of printed sign-out documents created by residents from different residency programs, with only 4 of 28 possible data elements being uniformly present. Additionally, sign-out documentation often missed key information related to clinical condition, overnight tasks, and anticipatory guidance.,, In a retrospective review of written sign-out documents, Miller et al. found that initial sign-out documents, often generated immediately after patient admission, missed roughly half of the 13 necessary components of a high quality sign-out. Additionally, the quality of information documented at sign-out, measured by information discrepancies, declined rapidly over a 6-h period after sign-out, with the number of information discrepancies increasing over time. A majority of these documentation discrepancies were related to medications and were found to persist over time.

To mitigate sign-out complexity and enhance information transfer resiliency, it has been recommended that clinicians use standardized sign-out documentation tools to support face-to-face verbal sign-out communication.,,, However, there is limited evidence on how the written/electronic standardized documentation supports verbal communication during sign-out. In other words, only a few studies have evaluated both written documentation and verbal communication for comprehensively characterizing the nature of content and structure of resident sign-outs. One such study, by DeRienzo et al., compared the content of sign-out documentation and verbal communication against an auto-populated required set of nine sign-out elements. They found that documentation errors were significantly higher for sicker patients. In another comparative study, Horwitz et al. found that a combination of oral and written sign-outs described patient clinical condition, hospital course, and tasks to be completed for only 62% of the patients. Weiss et al. used a similar comparison to a “gold standard” to evaluate the effectiveness of a written documentation for fostering sign-out communication. Their comparative evaluation showed that the written documentation served as a cognitive aid for sign-out communication and reasoning.

Although these studies provide insights on residents’ verbal communication and documentation practices, to our knowledge, there are no studies that directly compared sign-out documentation and verbal communication with each other. In other words, what is lacking is the empirical evidence regarding how much (and what) documented sign-out content is discussed during verbal communication. Additionally, there are no empirically validated content frameworks that can used to evaluate the overlap between written sign-out documentation and verbal communication.

In this article, we report on an exploratory study investigating the nature and degree of overlap between written sign-out documentation and verbal sign-out communication. In order to identify the nature of such an overlap, we first developed a content overlap framework to characterize what documented (written) sign-out content is communicated by the outgoing resident to the incoming resident during verbal sign-out. Using this framework, we ascertained the degree to which a set of sign-out content elements were: written and communicated (ie full overlap); written and partially communicated (ie partial overlap); and written and not communicated (ie no overlap). Such an understanding regarding the nature and varying degrees of overlap across sign-outs can highlight the typology of content that residents prioritize for written documentation and verbal communication. Study insights can be leveraged to inform the information organization, format, structure, and content underlying sign-out documentation tools.

METHODS

Study setting and participants

This study was conducted in the General Medicine (GM) service of a Midwestern hospital in the United States. The hospital is a 495-bed urban academic hospital with approximately 47 000 ED visits and 20 000 hospitalizations. The adjusted mean patient length of stay in the hospital was 5.7 days.

The GM service cares for patients with diagnoses including asthma, congestive heart failure, pneumonia, venous thromboembolism, diabetes mellitus with hyperglycemia, chronic obstructive pulmonary disease, cellulitis, and other non-surgical and non-critical conditions.

Participants in this study included 23 (n = 23) residents [8 PGY1 (Post Graduate Year), 8 PGY2, and 7 PGY3] who participated in the sign-outs for 44 (n = 44) patients. Although residents varied in their years of training, research has shown that there are limited perceived differences in the quality of their sign-outs or its impact on safety.

This study was a part of a series of studies investigating the nature and structure of communication during inpatient resident and nurse transitions in care.

Resident sign-outs

Resident sign-outs occurred at 5 pm daily, and involved the verbal exchange of information from the outgoing to the incoming resident. During these face-to-face sessions, an outgoing resident verbalized care-related information using information recorded on a printed sign-out document. This encrypted, Microsoft Word-based sign-out document was maintained external to the institution’s Electronic Health Record (EHR) and was organized in a standardized problem-based format with the following information: patient identifying information [name, date of birth, medical record number (MRN), room number, allergies, code and contacts], history of presenting illness (HPI), active problems, medications, and tasks to-do. An example sign-out document is provided in the Supplementary Figure S1.

Prior to the face-to-face verbal sign-out, residents typed in the requisite patient information, and printed a copy of the sign-out document for use during communication. We refer to this printed sign-out document as the “written” sign-out document.

Data collection

Data collection was conducted with a convenience sample of resident physicians who participated during a 2-month study period (February and March 2014). Based on a set of sign-outs collected as part of a larger study, we selected a convenience sample of 44 patient sign-outs (n = 44). This subset was selected based on the following inclusion criteria: (1) sign-outs for new patients, (2) unique pairs of residents, and (3) availability of written sign-out documentation.

Primary sources of data included audio recordings of resident–resident verbal sign-out communication and corresponding written sign-out documents. The institutional review board of the university approved this study, and consents were obtained from all participants.

Data coding and analysis

Coding of verbal and written sign-out transcripts

All audio-recorded sign-out communication and copies of written documentation were paired based on matching of patient name and MRN, prior to de-identification. During the de-identification process, all identifying information were scrubbed from the audio and written content. De-identified audio-recorded sign-outs were then transcribed by a professional medical transcription service.

Verbal communication per patient (we refer to these as verbal “transcripts”) was segmented into functional units called utterances. Utterances can be defined as the psychological analogs of a single unit of experience including statements, commands, and single words (eg “okay”)., Next, content analysis techniques were used to categorize the segmented verbal content. Content analysis involves the systematic analysis of textual content with reference to its meaning, context, and intent., We have previously used content analysis techniques to evaluate communication practices in a variety of settings.,,, Informed by these studies, we adapted a modified clinical content framework to fit within the context of GM resident sign-outs.,

Our clinical content framework consisted of the following nine elements: patient identifying information, code status, allergies, past medical history (PMH), active problems and assessment of active problems, anticipatory guidance (if/then statements regarding clinical contingencies), pending labs/studies/procedures, medications and treatments, and tasks. Given the different clinical conditions of GM patients, the PMH content element was further categorized by body systems: cardiovascular, pulmonary, gastrointestinal, endocrine, renal, neurological, hematological, integumentary, immune, and musculoskeletal systems. For example, PMH of Type 2 Diabetes Mellitus was classified as endocrine, and a PMH of manic depressive episodes was classified as psychiatric. A detailed description of this framework with examples from our data is provided in Table 1.

Table 1.

Clinical content framework that was used to categorize the verbal and written transcripts

Clinical content element	Description	Example
Patient identifying information	Patient demographic information related to DOB, sex, MRN (medical record number) location/room number, and service.	“A 80 year-old female”
Code status	Code status of patient including full code, DNR (do-not-resuscitate) and DNI (do-not-intubate).	“He is now DNR”
Allergies	Patient drug allergies	“Patient is allergic to penicillin”
Past medical history	Any past clinical diagnosis and surgeries, procedures etc.	“He has a history of Guillain-Barre syndrome”
Active problems and assessment of active problems	All active diagnoses and conditions being treated during current hospital encounter and their current status	“He presented yesterday with abdominal pain, nausea and vomiting” (Active problem) “His abdomen is soft and tender basically” (Assessment of active problem)
Anticipatory guidance	If/then statements	“If BP rises above 140 systolic and 90 diastolic give hydralazine”
Pending labs/studies/procedures	All ongoing/pending laboratory studies or procedures	“Do an EKG (electrocardiogram)”
Medications/treatments	Medications and treatments and their status (including administration, patient’s response to medications/treatments)	“I decreased all orals today and he hasn't asked for anything today” “He’s on Furosemide”
Tasks/to do	Incoming and outgoing clinician tasks to be performed during shift	“follow-up CT (computed tomography)” “Nothing to do”

Adapted from reference [25].

Similarly, the content of written sign-out documents was transcribed, segmented, and coded using the same clinical content framework.

Two physicians independently coded verbal communication transcripts with 95.6% agreement (Cohen’s K = 0.96) and written sign-out transcripts with 98.9% agreement (Cohen’s K = 0.98). Disagreements in both cases were resolved through discussion, and agreement reached 100%.

Sign-out content overlap framework

We developed an analytical framework to characterize the content overlap between written documentation and verbal communication during resident sign-outs. In order to determine the overlap, for each patient case, we first compared the coded written and verbal sign-out transcripts. The written sign-out transcript was used as the basis for comparison, as prior research has shown that written sign-out documentation often follows a pre-set, standardized structure.

To identify the sign-out content overlap, each content element (based on the clinical content framework; see Table 1) was compared across written and verbal transcripts for each patient. For example in the case of patient 1, we compared the functional units in the verbal transcript coded as “active problems and assessment of active problems” to the “active problems and assessment of active problems” in the written transcript. This helped us to ascertain how much (if any) overlapping information was present across verbal and written transcripts for the same patient for a particular content element.

Given that there were various degrees of overlap possible between the verbal and written sign-out content, we developed the following analytical framework for coding the content overlap. Each content element was coded with the following set of coding categories (see Table 2):

Table 2.

Framework used to determine the degree of overlap across verbal and written content

Code (overlap type)	Definition	Examples of analysis
		Verbal	Written
Present in both (full overlap)	Content is present in both written and verbal transcripts	Active problem/assessment of active problems: “Chest pain: positive nuclear stress test”	Active problem/assessment of active problems: “Chest Pain: Positive nuc test”
Additional verbal (full overlap)	Additional information presented in verbal transcripts than in written transcript	Pending labs/studies/procedures: Repeat TTE: showed vegetation persisted on tricuspid, blood culture	Pending labs/studies/procedures: Repeat TTE
Partial present in verbal (partial overlap)	Only partial content from the written transcript is present in verbal transcript	Medication/treatment: Methadone, Morphine	Medication/treatment: Methadone, Morphine, Naloxone
Present in verbal (no overlap)	Content present only in verbal transcripts (and absent from written)	Anticipatory guidance: “Give pain med when he wants”	Anticipatory guidance: Absent
Present in written (no overlap)	Content present only in written transcripts (and absent from verbal)	Task/to do: Absent	Task/to do: Monitor signs and symptoms of bleeding
Dissimilar (no overlap)	Content in verbal and written transcripts are different and dissimilar	Active problem/assessment of active problems: “Right leg cellulitis: stable”	Active problem/assessment of active problems: Orbital cellulitis
Absent (no overlap)	Contents in verbal and written transcripts were absent in the category being analyzed	Active problems: Absent	Active problems: Absent

Full overlap: Written documentation content was fully present in the verbal communication content (ie exact match); or in addition to the exact match, there was additional information present in the verbal communication.

Partial overlap: Written documentation content was not fully present in the verbal communication content (ie partial match). In other words, some written documentation was not verbalized during sign-out communication.

No overlap: Written documentation content was not present at all in the verbal communication content (ie there was no match).

It must be noted that for content elements related to patient identifying information, allergies, code status, and PMH, only full overlap and no overlap were determined as these elements are objective, and partial presence of such information was not meaningful. For example, a patient’s name or their allergies were either present or absent.

Table 3 provides several examples to illustrate the overlap framework that we used. Based on the framework, for each sign-out, each of the clinical content categories were categorized as “full overlap,” “partial overlap,” or “no overlap.” For example, for a patient case, written documentation included the following medications: nifedipine, statin, aspirin, and famotidine. During the verbal sign-out, nifedipine was only discussed. As a result, this was coded as a “partial overlap.”

Table 3.

Examples of determination of overlap across various clinical content categories

Clinical content element	Written documentation	Verbal communication	Type of overlap	Explanation
Active problem and assessment of active problems	HCAP: Fever	HCAP: He had elevated white count and fever	Full overlap	All content in the written documentation was presented during verbal (ie fever), and additional information was also presented (increase in white blood cell count)
Medications/treatments	Nifedipine Statin ASA Famotidine	Nifedipine	Partial overlap	Only one of the medications was presented in verbal communication (Nifedipine)
Anticipatory guidance	Culture if spikes	If he spikes a fever overnight culture him	Full overlap	The need for a culture was present in both written and verbal transcripts
Pending labs/studies/procedures	Waiting picc and BCx	(Absent)	No overlap	Pending labs/studies/procedures content documented in Written but not present in verbal transcript
Tasks/to do	F/U CT abdomen	F/U CT	Full overlap	Follow-up CT was present in both written and verbal transcripts as a pending task.

The comparative coding was performed by a physician researcher with extensive training and experience using the clinical content framework (see Table 2).

The entire process of data collection, coding, and comparative assessment of overlap is shown in Figure 1.

Figure 1.

The process of data analysis is shown. Written and verbal communication content was transcribed, segmented, and then coded using a sign-out clinical content framework. Then, for each of the clinical content categories, the degree of overlap between written and verbal content was determined.

Data analysis

Using the content overlap framework, we evaluated the degree of content overlap using an overlap metric—the percentage of sign-outs that had full, partial, or no overlap for each of nine clinical content elements. The overlap metric was computed as a percentage, based on the conditional probability of verbalizing content that was included in the written sign-out document (ie probability of verbalizing a clinical content element, given that it was written down). To compute the conditional probability, we divided the joint probability of a specific content element being fully, partially, or not-verbalized (see Table 2 for the content overlap framework) and that same content element being written, by the marginal probability of the content being written in the sign-out document. This degree of content overlap was expressed as the percentage of sign-outs (n = 44) that had a full, partial, or no overlap across the nine considered clinical content elements.

As an exploratory research question, we also investigated whether the presence of a written active problem was associated with corresponding written documentation related to anticipatory guidance, pending labs/studies/procedures, medication/treatments, and tasks. This analysis was based on an evaluation of the written sign-out transcripts per patient and was performed to assess the use and effect of a patient problem-based structure in organizing and developing the written documentation supporting resident sign-outs.

Results

The content overlap between the written documentation and verbal communication for each of the nine clinical content elements in cases of full, partial, and no overlap conditions are described below.

Full content overlap—signifying presence of all written documentation in verbal communication—was high for patient-identifying information related to patient name (present in 100% of the sign-outs), age (96%), and gender (87%); PMH related to the hematologic (100%), renal (100%), cardiologic (79%), and GI (67%) body systems; and resident tasks to-do (97%). (Note that the % denotes the percentage of sign-outs (n = 44) that had the considered type of overlap (in this case, full overlap).) Full content overlap was lesser for active problems and their assessments (46%), anticipatory guidance (46%), medications/treatments (15%), patient location (13%), and pending labs/studies/procedures (7%). Full content overlap was absent for patient MRN (0%), patient location (0%), code status (<1%), and allergies (0%) across all sign-outs. (As previously described, for the objective clinical elements patient identifying information, allergies, code status, and PMH, only full overlap and no overlap were determined; for these elements full overlap and no overlap are complementary and sum to 100%. For those clinical content elements that had a partial overlap, no overlap was 100%—% of full overlap—% of partial overlap (see Table 4).)

Table 4.

Overlap percentages for each of the clinical content

Clinical content elements	Full overlap (%)	Partial overlap (%)	No overlap (%)
Patient identifying informationa
Name	100	—	0
Age	96	—	4
Gender	87	—	13
MRN (medical record number)	0	—	100
Location	30	—	70
Service	0	—	100
Code status (mean of full code, DNR, DNI)a	0.7	—	99.3
Allergiesa	0	—	100
Active problems and assessment of active problems	46	13	41
Anticipatory guidance	46	0	54
Pending labs/studies/procedures	7	0	93
Medication/treatment	15	14	70
Tasks	97	0	3

The percentages in each column denote the % of sign-outs that had full, partial, or no overlap. Note that the overlap categories add to 100%. For example, for the gender clinical element, 87% of the sign-outs had a full overlap, and 13% of sign-outs did not have a full overlap (which also signifies the percentage of sign-outs that had no overlap). For clinical elements that had a meaningful “partial overlap,” the no overlap was 100%—% of full overlap—% of partial overlap. PMH was determined as relevant per patient and is provided separately (see Supplementary Table S1).

— denotes that partial overlap was not calculated.

The categories for which partial overlap calculation was not meaningful (ie either full overlap or not; eg name was both spoken and written; partial mentions were not meaningful).

Only two clinical content categories had a partial overlap—signifying the partial presence of written documentation in verbal communication—medication/treatments (14%) and active problems (13%). A summary of the overlap (full, partial, and no) for each of the clinical content elements are provided in Table 4.

With respect to the exploratory research question, we found that the presence of a written active problem and its assessment in a sign-out document was associated with the presence of medication/treatment information (89%), tasks to be performed (84%), anticipatory guidance (72%), and pending labs/studies/procedures (51%) in those sign-out documents.

Discussion

Much of the prior research on resident sign-outs has focused on evaluating either written documentation or verbal communication., In this exploratory study, we used communication analysis techniques to directly compare written documentation and verbal communication for characterizing the degree of sign-out content overlap. Towards this end, we developed an overlap metric that captured how much (and what) written documentation is communicated during resident sign-outs.

The purpose of this study is not to demonstrate what written information needs to be verbally communicated or which clinical content elements are relevant or important; rather, the purpose is to highlight what written information is prioritized and communicated during verbal sign-outs. Our findings highlight the core content elements that are documented and communicated during resident sign-outs supporting three essential functions underlying resident handoffs—transfer of information, transfer of responsibility, and transfer of accountability between outgoing and incoming residents.

The transfer of information related to patient identifying information (eg name, age, and gender) showed a high degree of content overlap—highlighting that such information was written down and verbalized across most of the patient sign-outs. As reported in previous research, this finding highlights the importance of situating and introducing the incoming resident to the patient, and reducing the potential for wrong-patient errors. Similarly, the transfer of information related to PMH specific to hematology, renal, cardiology, and GI also showed high degree of overlap. This is reflective of the chronic nature of the clinical conditions of patients treated in the GM service and the importance of discussing such information during sign-out.

Based on our data, the transfer of information including MRN, patient location, allergies, and code status during sign-outs across was limited; some of this information (eg MRN) was always present in the written document, but never verbalized, whereas others (eg code status) were often not written down nor verbalized. Despite empirical evidence indicating that code status and allergies are critical clinical information in patient care management, requiring explicit communication, our findings suggest that these were often not documented nor communicated, consistent with prior research., Such a lack of documentation and discussion of code status and allergies can potentially result in adverse outcomes, given that it is most likely that an incoming resident can assume that these elements are irrelevant to their patients’ care management as they were not explicitly communicated during sign-out.

Similarly, the transfer of information related to pending labs/studies/procedures, medications/treatments was also limited (15% and 7% of sign-outs, respectively). This may be likely due to the transient nature of such information, with these content elements getting updated frequently. Furthermore, these elements are often exhaustive, containing extensive information that may increase sign-out duration, if fully verbalized. Similarly, the transfer of information related to active problems and their assessments was also not comprehensive across sign-outs. Given that active problems and their assessments can provide a comprehensive overview of a patient’s current clinical condition, limited communication of these clinical elements can impact physicians’ prioritization and planning of care activities during their shift and proper identification of a decline in the patient’s state of health., For example, it has been reported that residents spend a third of their time seeking test results during their shift, which could be easily avoided if these clinical elements are effectively communicated during sign-outs.

The transfer of responsibility between residents was explicit and often comprehensive, specifically with respect to future (or pending) patient care tasks (ie “tasks to-do”). Such explicit verbalization and documentation of the “to-do” lists can improve the quality of the sign-out process and reduce the chances of adverse outcomes immediately after sign-outs., In other words, formal transfer of responsibility, highlights the importance that residents place in prioritizing their patient care plans/goals to tailor their task workflows.

Finally, the transfer of accountability between residents in the form of anticipatory guidance was limited. Anticipatory guidance during sign-out serves as a guide to incoming residents on what to expect in terms of clinical contingencies during their shift and how to handle any care event deviations.,

Evaluation of our exploratory research question highlights the relevance and impact of a patient problem-based structure used to organize the written documentation of sign-out content. The problem-based structure has been used to guide the design of resident sign-out tools in medical and surgical floor units. Our findings showed that the documentation of an active patient problem likely prompted the inclusion of related documentation regarding medications, treatments, tasks, anticipatory guidance, and pending labs/studies/procedures, specific to the patient problem. Given the widespread use of EHR-integrated sign-out tools, the objective problem-based sign-out documentation content can be automatically generated. Researchers have shown that the problem-based structure aids in keeping handoff documentation succinct, and clear––allowing for simple organization and prioritization of a patient’s active problems.

We acknowledge several limitations of our study. The study was conducted at a single academic medical center and was based on a relatively small sample of patient sign-outs (n = 44). However, the analysis was conducted at a very granular level, utilizing nine content elements, relevant to medicine resident sign-outs. We did not consider patient characteristics, patient complexity, time spent on sign-out, or the patient length of stay in the unit. However, only patients admitted to the GM service were considered, allowing for some degree of similarity across patients. With a small sample of residents and variations in sign-out training in academic medical programs, we were unable to study the effect of residents’ experience and training on content overlap. We used written documentation as the basis for comparative evaluation. It is likely that some of the clinical content elements (eg medications) were written down, but not discussed due to some changes in the patient’s clinical condition. We did not evaluate the impact of the content overlap on sign-out quality; however, our goal was to develop an evidence-based method to identify and evaluate the nature and degree of overlap between written and verbal sign-out content. Currently, we are conducting a larger, longitudinal study on sign-out content overlap accounting for differences in training of residents, experience, and patient differences.

CONCLUSION

We report on an exploratory study investigating the overlap between written and verbal communication content in resident sign-outs. Our findings highlight the focus of resident handoffs on transfer of responsibility—through the explicit communication of pending and future tasks. However, the limited transfer of information on several important variables including code status, anticipatory guidance, and active problems highlight potential patient safety challenges that can be addressed through re-thinking the structure of handoff tool design, formulating verbal communication protocols/checklists, and tailoring handoff entrusted professional activities in academic programs.

CONTRIBUTORS

Abraham conceptualized and designed the study. Abraham, Ihianle, Ward, and Kannampallil organized the data, conducted the analyses, and drafted the initial manuscript. Abraham, Ihianle, Ward, Kannampallil, and Arora were involved in interpreting the results, critically reviewing, revising, and finalizing the manuscript. All authors participated in all stages of the study and approved the final version of the manuscript.

Conflict of interest statement. None declared.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Click here for additional data file.