Evaluating narrative operative reports for endoscopic sinus surgery in a residency training program.

OBJECTIVE: The narrative operative report (NR) bears testimony to critical elements of patient care. Residents' NRs also provide insights into their comprehension of the procedure. NR documentation is an informal element of surgical residency training but data regarding quality of such training are scant. We aim to evaluate the NR within a residency training program.
METHODS: The quality of NRs for endoscopic sinus surgery (ESS) was evaluated through a retrospective analysis of 90 NRs for ESS. Thirty-four elements that the attending surgeon regards as "critical" variables, or quality indicators (QIs), that should be documented, were studied to evaluate quality. A "performance metric (PM)," defined as the average percent of QIs dictated/total word count, was determined. Subgroup analysis by the level of training was additionally performed.
RESULTS: Surgical indications, procedural steps, and immediate postoperative findings were accurately documented in 71%, 84%, and 82% of patients, respectively. The attending surgeon had the highest proportion of included key elements (89% ± 6.2%) followed by junior residents (87% ± 5.7%) and then senior residents (80% ± 14%) (P = .008). The attending surgeon also demonstrated the highest PM, followed by senior and then junior residents (P < .0001).
CONCLUSIONS: The quality of NRs was found to be high overall, but not "perfect" for either the attending or trainee surgeon. The PM among residents was expectedly lower than the attending surgeon. We propose that a synoptic reporting system that ensures inclusion of key elements may be helpful in training residents (and attendings) in creating comprehensive and efficient NRs. LEVEL OF EVIDENCE: 3.

INTRODUCTION

Narrative operative reports (NRs) have traditionally been completed through oral dictation by the surgeon after the procedure. These NRs document the indications for surgery, the procedures performed, pertinent findings, technical steps in execution, complications, if any, and the condition of the patient immediately after conclusion of the procedure. The NR is a critical documentation of patient care, as well as a record that impacts consequent patient care, payment and reimbursement, and data for analyzing quality control. NRs are currently the standard documentation method used for most surgical procedures within the United States.

However, the content of the NR is not usually standardized or regulated and there is limited research available that aims to do so.1 Furthermore, studies which have evaluated the quality of NRs in other surgical specialties have shown that NRs often omit critical aspects of the procedure, especially when dictated by surgical residents.2, 3, 4

Formal training in how to best dictate specific procedures is not yet part of most surgical residencies. Most surgeons likely informally guide residents in critical elements to record NR documentation as well as review the dictation, but this may not be standardized. NRs therefore may vary between attending surgeons and trainees.

Informed decision‐making postsurgery regarding future treatment may be impacted if critical details within the NRs are lacking.5 This is especially relevant for endoscopic sinus surgery (ESS), as in‐office debridement procedures are a standard of care postsurgery. For example, salient features such as the presence of a dehiscent carotid canal or spheno‐ethmoidal cells may warrant additional caution. Documentation of the presence of supraorbital ethmoidal cells alerts the surgeon to also intentionally address the cell along with the frontal ostioplasty during debridement procedures and follow‐up care. Additionally, many chronic rhinosinusitis subtypes require revision surgery.6, 7 Accurate documentation of any subtle dehiscence's in the skull base or lamina papyracea, or any complications, facilitates safe revision surgery.

In the age of electronic medical records (EMRs), quality indicators (QIs) are being tracked more closely and are being correlated with outcomes. Review of the NR can provide useful information in performance gaps and identify professional development needs for surgeons. This is especially effective for ESS, because these gaps can be further corroborated by in‐office endoscopic examinations and computed tomography scans, which provide critical information on the actual extent and meticulousness of surgery in recalcitrant patients. However, the quality of NRs for ESS in academic programs has not been previously examined. We conducted a review of our NRs to evaluate quality and efficiency, as well as effectiveness in informal instruction of residents in this skill.

MATERIALS AND METHODS

Following institutional review board approval, a retrospective chart review was performed and all patients who underwent ESS between 2014 and 2017 at Mayo Clinic Arizona were identified. Out of this cohort, a subset of patients who underwent bilateral maxillary antrostomy, anterior and posterior ethmoidectomy, and sphenoid sinusotomy were identified. These procedures are considered to be a standard part of residency training in Otolaryngology—Head and Neck Surgery. The most recent 30 NRs were selected in consecutive fashion for each level of physician dictating—including junior residents, senior residents, and an attending physician—for a total of 90 NRs. The sections of the NR relating to septoplasty, frontal sinusotomy, and/or inferior turbinate reduction were also analyzed if performed, but these procedures were not required for inclusion in the study. All NRs were created using an oral dictation service and characteristics of the narrator were determined (Table 1).

Table 1

Narrator Characteristics.

Characteristics	No. (%)
Level of training
Junior	6 (50)
Senior	5 (42)
Attending	1 (8)
Gender
Male	6 (60)
Female	4 (40)

Review of the literature has revealed that QIs for ESS have not been formally published. As such, we reviewed academic instructional material that the attending surgeon discusses at the outset of the Rhinology rotation consistently with all trainees. These instructional materials were used to construct a list of key metrics and a consensus agreement between the attending surgeon (DL), a chief resident (NLD), and a junior resident (AM) was achieved regarding which key metrics to include in our list of QIs. A finalized list of 34 QIs was generated (Table 2). The list encompasses documentation of indication(s) for the procedure, factors that impact safety of surgery, details of how the sinuses were dissected and any complications that occurred (as well as their management), and factors that may impact postoperative care and prognostication.8, 9, 10, 11

Table 2

Quality Indicators.

Preoperative

Indications for surgery

Primary diagnosis

Relevant comorbidities

Primary vs. revision

Summary of pertinent findings

Setup

Eye draped into field

Computerized image guidance setup

Patient positioning and padding

Technical Aspects/Steps

Nasal endoscopy

Appearance of nasal cavities prior to surgery

Injection of local anesthesia

Septoplasty

Location of deviation and spurs

Preservation of L‐strut

Lacerations or tears in mucosa

Maxillary antrostomy

Amount of uncinate removed

Identification of natural ostium

Angled scope used

Pertinent findings within sinus

Anterior ethmoidectomy

Partial or complete removal of bulla ethmoidalis

Identification of lamina papyracea in the lateral aspect of the bulla

Posterior ethmoidectomy

Identification of superior turbinate

Identification of lamina papyracea

Identification of skull base at dissection

Sphenoid sinusotomy

Description of approach to sphenoid sinus

Pertinent findings within sinus

Frontal sinusotomy

How frontal recess was identified

Which frontal cells removed

Pertinent findings within sinus

Inferior turbinate reduction

Type of reduction

Instrumentation used

Closure

Medialization of middle turbinate

Methods to reduce incidence of septal hematoma

Was hemostasis ensured

Complications

Postoperative Check

Immediate postoperative check

Were pupils checked and eyes palpated

Was stomach suctioned

The NRs were analyzed, using the data from Table 3, to determine the extent of QI inclusion (Table 4). Variables relating to patient identifiers, such as Medical Record Number, name, age, and DOB, were excluded, as this information is automatically populated into the note. The dictated NRs were obtained from the patients’ EMRs and data were entered into the secure web application database REDCap. Patient charts were also analyzed to ensure all relevant data were included in the NRs.

Table 3

Characteristics Reviewed.

	Overall	Attending	Junior	Senior	P Value
Preoperative
Indications for surgery	60 (71%)
Primary diagnosis	83 (92%)	29 (97%)	28 (93%)	26 (87%)	.3381*
Relevant comorbidities	11 (24%)	5 (25%)	4 (33%)	2 (14%)	.5191*
Primary vs revision	87 (97%)	29 (97%)	29 (97%)	29 (97%)	1.0000*
Summary of pertinent findings	88 (97.8%)	30 (100.0%)	30 (100.0%)	28 (93.3%)	.1293*
Setup	86 (96%)
Eye draped into field	88 (98%)	28 (93%)	30 (100%)	30 (100%)	.1293*
Computerized image guidance setup	81 (90%)	28 (93%)	29 (97%)	24 (80%)	.0748*
Patient positioning and padding	90 (100%)	30 (100%)	30 (100%)	30 (100%)	—
Technical aspects/steps
Nasal endoscopy	80 (89%)
Appearance of nasal cavities prior to surgery	75 (83%)	29 (97%)	28 (93%)	18 (60%)	.0001*
Injection of local anesthesia	86 (96%)	30 (100%)	29 (97%)	27 (90%)	.1602*
Septoplasty	56 (73%)
Location of deviation and spurs	72 (95%)	25 (96%)	25 (96%)	22 (92%)	.7178*
Preservation of L‐strut	42 (56%)	21 (84%)	13 (50%)	8 (33%)	.0013*
Lacerations or tears in mucosa	51 (67%)	17 (65%)	21 (81%)	13 (54%)	.1317*
Maxillary antrostomy	73 (89%)
Amount of uncinate removed	61 (84%)	18 (78%)	20 (77%)	23 (96%)	.1397*
Identification of natural ostium	73 (95%)	25 (96%)	28 (100%)	20 (87%)	.1050*
Angled scope used	76 (84%)	29 (97%)	27 (90%)	20 (67%)	.0035*
Pertinent findings within sinus	83 (92%)	28 (93%)	30 (100%)	25 (83%)	.0527*
Anterior ethmoidectomy	63 (73%)
Partial or complete removal of bulla ethmoidalis	64 (75%)	24 (83%)	18 (64%)	22 (79%)	.2400*
Identification of lamina papyracea in the lateral aspect of the bulla	62 (70%)	18 (60%)	20 (69%)	24 (80%)	.2406
Posterior ethmoidectomy	73 (82%)
Identification of superior turbinate	72 (80%)	23 (77%)	24 (80%)	25 (83%)	.8119*
Identification of lamina papyracea	69 (77%)	20 (67%)	23 (77%)	26 (87%)	.1869*
Identification of skull base at dissection	79 (88%)	29 (97%)	25 (83%)	25 (83%)	.1907*
Sphenoid sinusotomy	80 (89%)
Description of approach to sphenoid sinus	88 (98%)	29 (97%)	30 (100%)	29 (97%)	.5997*
Pertinent findings within sinus	72 (80%)	28 (93%)	25 (83%)	19 (63%)	.0126*
Frontal sinusotomy	75 (91%)
How frontal recess was identified	79 (95%)	27 (93%)	26 (100%)	26 (93%)	.3831*
Which frontal cells removed	65 (79%)	27 (93%)	18 (72%)	20 (71%)	.0732*
Pertinent findings within sinus	81 (98%)	29 (100%)	26 (100%)	26 (93%)	.1336*
Inferior turbinate reduction	70 (97%)
Type of reduction	70 (97%)	26 (100%)	23 (100%)	21 (91%)	.1118*
Instrumentation used	70 (97%)	24 (92%)	23 (100%)	23 (100%)	.1621*
Closure	67 (82%)
Medialization of middle turbinate	72 (91%)	26 (93%)	24 (89%)	22 (92%)	.8694
Methods to reduce incidence of septal hematoma	68 (85%)	24 (86%)	23 (85%)	21 (84%)	.9844*
Was hemostasis ensured	62 (69%)	28 (93%)	17 (57%)	17 (57%)	.0019*
Complications	66 (73%)	25 (83%)	26 (87%)	15 (50%)	.0018*
Postoperative check
Immediate postoperative check	72 (82%)
Were pupils checked and eyes palpated	62 (71%)	27 (90%)	21 (70%)	14 (52%)	.0063*
Was stomach suctioned	81 (93%)	28 (93%)	28 (93%)	25 (93%)	.9921

Analysis of Variance F‐test.

Table 4

Percent of QIs Included Overall.

	Total
Preoperative QIs	75.4 (86%)
Technical intraoperative QIs	70.3 (84%)
Immediately postoperative QIs	71.5 (82%)

QIs = quality indicators.

The overall inclusion percentage of QIs was calculated for preoperative information, intraoperative technical factors, and postoperative findings. The primary outcome measure was to determine the mean total percentage of QIs dictated as a function of level of training. Three groups, based on seniority level, were defined as the following: 1) attending physician, 2) senior resident (defined as PGY3 and above), and 3) junior resident (defined as PGY1‐2). A secondary outcome was the “performance metric (PM),” defined as the average percent of QIs dictated divided by the total word count, which was also compared between these three groups (Table 5).

Table 5

Word Count and Percent Correct by Level of Training.

	Total (N = 90)	Attending (N = 30)	Junior (N = 30)	Senior (N = 30)	P Value
Word count					<.0001*
N	90	30	30	30
Mean (SD)	1,498.4 (483.1)	1,313.6 (256.1)	1,855.3 (510.0)	1,326.3 (439.2)
Range	728.0–3,095.0	842.0–1,963.0	1,132.0–3,095.0	728.0–2,393.0
Percent inclusion					.0008*
N	90	30	30	30
Mean (SD)	85 (10)	89 (6.2)	87 (5.7)	80 (14)
Range	40–100	74–97	72–97	40–100
Performance metric					<.0001*
N	90	30	30	30
Mean (SD)	0.0612 (0.0164)	0.0699 (0.0127)	0.0497 (0.0128)	0.0640 (0.0166)
Range	0.0304–0.1048	0.0411–0.1048	0.0304–0.0753	0.0342–0.0973

Analysis of Variance F‐test.

Statistical Analyses

The statistician created REDCap electronic data forms. Each subspecialty was analyzed separately. The sample for each subspecialty included 30 patients per junior resident, senior resident, and attending. We calculated summary statistics for patient demographic information and the level of training of the surgeon narrating the report. The secondary outcome measure was the “PM,” defined as the percentage of elements reported, divided by the word count. Mean reporting efficiency was compared among the three groups by using one‐way analysis of variance. Pairwise comparisons of groups were made if the overall F test was significant. Secondary measures were assessed by using one‐way analysis of variance or Pearson's chi‐square test.

A sample of 20 patients per group has 80% power to detect a difference of 0.9 standard deviances. Vergis et al reported a standard deviation of 8.4% of the range of the scale for the percentage of elements reported.3 So, a sample of 20 patients per group would have good power to detect a mean difference of 3 percentage points of a 40‐point scale if (0.84 × 40 × 0.9) if the variations were comparable to that reported by Vergis et al.3 Power of reporting efficiency would be even higher if percentage of elements and word count both differ among groups. A sample of 30 was included for even greater statistical power. P values <.05 were considered significant. Statistical analysis was performed using SAS version 9.4 (SAS Institute, Inc.).

RESULTS

Ninety charts were reviewed in total and the characteristics of physicians dictating these NRs are displayed in Table 1. All patients underwent functional ESS in which the sinuses were addressed in a bilateral fashion. The frequency of reporting nontechnical aspects including indications for procedures and patient set‐up was 71% and 96%, respectively, for the overall group (Table 3). The frequency of reporting technical operative elements is displayed in Table 3. Overall, 84% of technical aspects were included in the NRs (Table 4). Reporting elements at the conclusion of the case including orogastric suction, complications, and palpation of orbits were documented in 93%, 73%, and 71%, respectively (Table 3).

When comparing inclusion of key indicators between levels of training, an analysis of variance showed that the level of training significantly impacted the percent of inclusion of key indicators (P .0008) (Table 5). Post hoc analyses using Tukey test for significance indicated that attending physicians had the highest mean inclusion at 89% compared to senior residents with 80% (P .0007). Tukey test for significance also determined that junior residents had a higher mean inclusion with 87% compared to senior residents (P .0169). However, there was no statistical significance when comparing mean inclusion between attending and junior residents (P .5711).

Efficiency of dictation, measured by word count, was also determined to be significantly affected by dictating author using an analysis of variance (P < .0001) (Table 5). Post hoc analyses using Tukey test for significance showed that word count was lowest in the attending group with mean word count of 1,313.6 compared to junior residents with 1,855.3 (P < .0001). Senior residents also had a smaller word count of 1,326.3 compared to junior residents (P < .0001). There was no statistical significance when comparing word count between attending and junior residents (P .9923). When analyzing the PM, the attending had the highest PM (0.0699) followed by the senior residents (0.0640) and then junior residents with the lowest (0.0127) (P < .0001).

DISCUSSION

In patients undergoing ESS, accurate clinical documentation can serve several important roles including future medical and surgical management, outcomes research, surgical reimbursement, and defending malpractice claims. To our knowledge, this is the first study within Otolaryngology—Head and Neck Surgery to analyze the quality of NRs. We chose ESS procedures that residents should become proficient at during the training period. The major conclusion from this study was that on average—across all levels of training—84% of QIs were included within NRs for ESS, indicating that the overall quality of NRs was excellent at an academic training center.

Notable elements were missing in approximately 16% of cases. Out of all levels of training, the attending surgeon was found to have the most complete NRs at 89% QI inclusion. This was only a small improvement compared to the junior residents (87%) but a large improvement compared to senior residents (80%). However, the comparison of the attending to the junior resident was not statistically significant.

The NRs of attending surgeons were shorter (eg, fewer words) compared to residents. Since documenting efficiently is especially important in today's era of EMRs and because verbose operative reports can be overly burdensome on physicians to review amid the abundance of lengthy documentation, we sought to measure the PM, calculated as percent inclusion of QIs divided by overall word count. We found that the attending surgeon had the highest PM with a value of 0.0699, followed by senior residents with 0.0640 and junior residents with 0.0497 (P < .0001). The ability to “say more with less” should guide residency training in narrative reporting. All dictations were ultimately signed off by the attending surgeon, and a more precise and structured approach may be beneficial in review and capture of all required elements. A structured, formal approach to teaching NR may therefore be effective for NR overall.

Our findings are consistent with the previously published literature from other surgical specialties. Stogryn et al demonstrated a perception of “mediocre” quality of narrative dictations among bariatric surgeons, based off a web‐based survey sent across Canada.12 Their quality audit reinforced these perceptions, as the 40 Roux‐en‐Y gastric bypass NRs they analyzed contained only 62% ± 6.6% of the variables deemed necessary for inclusion. This lack of completeness within NRs was also found in NRs for pancreatic surgery (72.5% completeness for pancreaticoduodenectomy), breast surgery (66% completeness), and upper limb surgery (71.1% completeness).13, 14, 15 That fact that our study found a higher average QI inclusion rate (85%) compared to these other studies may reflect the greater emphasis placed on documentation at our institution. Furthermore, junior residents have an informal template that they follow when learning how to dictate reports which could partially account for the fact that junior residents had a higher QI inclusion percentage than senior residents (87% vs 80%) (P .0169).

Implications

This study, as well as the others within the literature, has reinforced the inadequacy of NRs in including all notable QIs. We acknowledge that the major limitation of this study is how we define “critical” or QI elements. Since none have been defined for ESS in the literature, we arrived at a consensus between attending surgeon (DL), a chief resident (NLD), and a junior resident (AM) on which key elements to include. This process was informed using formal and informal academic material that the attending surgeon consistently reviewed with all trainees at the outset of the Rhinology rotation. The QIs listed in this study are simply a proposed list to study our own NR quality and quality of residency training, and may not be generalized universally. We present a single institution, single surgeon experience.

For any meaningful standardization, a consensus through professional societies or consortia is necessary. A standard agreed upon by consensus may be helpful in improving NRs across all levels of surgeons (attending and residents; academic and private practice settings) and in improving the validity of future studies. We do, however, believe that our study is of value in highlighting the need for such standardizing education for operative documentation. Training programs may benefit from developing individualized QI templates for residents to use while dictating to ensure all critical information is included in the operative report. As more outcomes research is performed, the list of QIs will evolve over time as certain variables will be validated as being important and worthy of inclusion, whereas others will be invalidated and removed.

It is imperative to emphasize the importance of complete operative report dictation throughout residency training, as junior residents were much more thorough (87%) compared to senior residents (80%) (P .0169). Potential methods to continue education include implementing ongoing quality control, establishing formal sessions on operative report dictation, or creating a mentorship program by attending faculty. This could minimize documentation errors and improve overall quality of care. A study by Hyde et al showed improvement in resident operative report completeness after implementing a formal plan for dictation education.16

Accurate documentation is not only essential for clinical care, but may impact reimbursement as well. If the report is incomplete or incorrect, it can lead to decreased or denied reimbursement. Novitsky et al found that reports dictated by residents led to incorrect coding in 14 cases (a 28% error rate). In addition to the potential adverse clinical outcomes of incomplete documentation, these errors resulted in reduced reimbursement by $18,200 (9.7%), which underscores the importance NRs have on surgical reimbursement.2

Furthermore, the growing use of synoptic reports (SRs) within EMRs pose as another method of documenting these QIs. The SR is a computerized, template‐based clinical documentation method that uses structured checklists to quickly capture data within standardized fields. The SR may further facilitate the collection of QIs and allow them to be easier to interpret and search, thereby optimizing communication between providers to improve clinical care.1, 3, 5, 14

Other specialties, such as Pathology, have widely adopted SRs. However, due to the high degree of variability—both patient‐to‐patient as well as between surgeons and institutions—SRs in surgery have been implemented at a slower rate. Nevertheless, many studies have shown the benefit of using SRs for surgical procedures.1, 3, 5, 14 Research by Gur et al showed significant improvement in data completeness with SRs (94.7% complete vs 66% in NRs). They found further benefit for the use of SRs within residency education due to the fact that the residents had to develop a better understanding of every procedure to fill out each SR.17

Limitations

As alluded to above, there are several limitations to this study. For one, we propose an initial nonstandardized list of QIs that should be included in operative reports for ESS and then retrospectively measure the rate at which these QIs are actually included in documentation. Some variables that we initially noted were difficult to analyze retrospectively because the importance of including them is dependent on whether or not they are present in a particular patient. For example, the presence of a concha bullosa or infraorbital cell should be included in a NR when applicable. However, if these anatomical variants are not present, it is not necessary to mention them. Other examples would be whether or not the cartilage was replaced during septoplasty, if a stent was placed within the frontal sinus, or if packing was needed during closure. It is pertinent to mention these if performed but not necessary if they were not performed. Furthermore, the accuracy of the operative note could not be determined retrospectively, as it is possible that the surgeon may have dictated a QI that was not actually performed during the surgery.

The number of variables deemed necessary for inclusion also was dependent on whether the surgery was primary or revision. For example, removal of the lower one‐third of the superior turbinate need not be dictated for certain revision surgeries, as that portion of the superior turbinate had been removed prior. As a result, many of the QIs had differing numbers of reports that were included for analysis. Complicating this fact, the number of primary or revision surgeries was not controlled for within each level of physician narrating the operative report.

Additionally, only a single attending's operative reports were surveyed. This was due to the fact the study institution had only one dedicated endoscopic rhinologist during the study period. It is quite possible that other surgeons and training institutions may be more (or less) comprehensive in their operative documentation. Therefore, future multi‐institutional studies involving different surgeons are needed to refine and validate the QIs used and the findings concluded in this study. However, our list of QIs is a starting step that can be used by others to further develop a consensus on critical components of an operative note. Furthermore, the objectives of the study were to ascertain the efficacy of NRs performed with diligence and how well we are doing at resident training in NRs. Our data support the use of standardized reporting for both attending surgeons and residents‐in‐training.

CONCLUSION

The dictated ESS operative report incompletely captures important clinical information. The quality of NR for ESS at our center was found to be high overall, reflecting that informal teaching of NR is effective. In our study, 84% of key indicators were captured using NRs. However, the NR was not “perfect” for either the attending or trainee surgeon. The “PM” among residents was expectedly lower than the attending surgeon's. Comparing inclusion of key indicators, attending physicians had the highest rate of inclusion and were the most efficient based on overall word count. An emphasis on comprehensive reporting of key indicators is necessary throughout residency training and a synoptic reporting system, which requires documentation of important indicators, may facilitate this mission and improve care for patients undergoing ESS. We present that a synoptic reporting system may provide guidance to trainees and attending surgeons for comprehensive and efficient NRs.