Resident Workshop to Improve Inpatient Documentation Using the Progress Note Assessment and Plan Evaluation (PNAPE) Tool.

Introduction: Physicians enter residency with varied knowledge regarding the purpose of progress notes and proficiency writing them. The objective of this study was to test whether resident knowledge, beliefs, and confidence writing inpatient progress notes improved after a 2.5-hour workshop intervention.
Methods: An educational workshop and note assessment tool was constructed by resident and faculty stakeholders based on a review of literature and institutional best practices. The Progress Note Assessment and Plan Evaluation (PNAPE) tool was designed to assess adherence to best practices in the assessment and plan section of progress notes. Thirty-four residents from a midsized pediatric residency program attended the workshop, which consisted of didactics and small-group work evaluating sample notes using the PNAPE tool. Participants completed a four-question online pre- and postworkshop survey to evaluate their knowledge of progress note components and attitudes regarding note importance. Pre-post analysis was performed with Chi-square testing for true/false questions, and Mann-Whitney testing for Likert scale questions and summative scores.
Results: A majority of pediatric residents completed the preintervention (n = 26, 76% response rate) and postintervention (n = 23, 68% response rate) surveys. Accurate response rate improved in 15 of 20 of the true/false items, with a statistically significant improvement in five items. Resident perceptions of note importance and confidence in note writing also increased. Discussion: A workshop intervention may effectively educate pediatric residents about progress note best practices. Further studies should assess the impact of the intervention on sustained knowledge and beliefs about progress notes and subsequent note quality.

Educational Objectives

By the end of this activity, learners will be able to:

Identify the purpose of a progress note.

Describe the Progress Note Assessment and Plan Evaluation (PNAPE) tool.

Apply the PNAPE tool as a framework to identify areas of progress note improvement.

Generate improved assessments and plans that adhere to the PNAPE framework.

Introduction

Generating clinical documentation in the electronic health records (EHR) is a required skill for physicians to ensure safe and effective patient care. Notes serve many purposes, including communication, documentation of critical thinking, medical decision-making, billing, and institutional coding. Documentation in EHR has become nearly ubiquitous with hospital incentives in the Health Information Technology for Economic and Clinical Health Act of 2009.[1] Despite the potential benefits, the introduction of the EHR has been associated with a decline in some aspects of important clinical documentation, including progress notes.[2-4]

Medical education must include training on effective clinical documentation.[5] Although teaching about documentation ideally occurs in the context of the clinical learning environment, faculty physicians may not be proficient EHR users.[3] Due to relatively recent implementation of EHR and the lack of appropriate training materials for this technology, new tools are necessary to enhance medical education on note writing. Some authors have demonstrated success improving note quality by pairing note templates with an educational intervention. For example, the Physician Documentation Quality Instrument (PDQI-9) is a validated tool that assesses progress note quality by scoring nine attributes on a 5-point Likert scale and was paired with an education intervention.[3,6] Others have created their own rubrics such as Talwalkar et al,[7] which was designed with international graduates in mind. This workshop employed unique, detailed rubrics for four different note types to enhance feedback specificity, though the number and length of rubrics may be cumbersome to some educators.[7] King and colleagues also developed a rubric for assessment of pediatric notes; however, this tool was validated only for history and physicals, which served a different purpose.[8] Other medical student workshops teach the basics of progress note format and content without consideration of the EHR or emphasis on note quality.[9]

Our group developed and validated a new rubric, the Progress Note Assessment and Plan Evaluation (PNAPE) tool (Appendix A). PNAPE is a binary checklist of specific note attributes designed to evaluate the assessment and plan of inpatient progress notes generated in the EHR.[10] Compared to other tools, PNAPE's binary checklist format is less subjective than the PDQI-9 5-point Likert scale and discrete items serve to provide specific, formative feedback about the notes to which it is applied. It also focused on assessing synthesis and management, which are increasingly important given that historical and objective components of the progress note are so readily accessible in the EHR. With this tool and new note template in mind, we created a workshop that aimed to improve resident knowledge, attitudes, and confidence in writing inpatient progress notes. The workshop format was selected to encourage the application of knowledge and allow for immediate peer feedback using simulated patient cases and group assessments.

Methods

Tool Development

Over the course of a year, a work group convened twice monthly to describe an ideal progress note assessment and plan after reviewing available literature and consulting with a departmental billing and coding expert. We then used an iterative process to develop and refine the 19-item PNAPE checklist (Appendix A) of specific note attributes important for EHR-generated progress notes.[10] In a forthcoming validation study, we found an interrater reliability of κ = .89 (95% confidence interval: .68-.97) and a moderate to strong positive association between mean PNAPE and PDQI-9 scores (Pearson's r = .63, p = .027).[11]

Curriculum Development

We utilized backwards design[10] as a model to develop this workshop, first identifying the desired results (learning objectives), determining the acceptable evidence (pre-post surveys assessing resident knowledge, beliefs, and confidence in note writing), and planning the learning experience (interactive workshop). Three sample cases typical of those seen on the pediatric hospital medicine service of increasing complexity were developed as part of the learning experience (Appendices B, C, and D), written by Kirstin A. M. Nackers, Michelle M. Kelly, and Daniel J. Sklansky, respectively, and were reviewed and revised by the workgroup.

The 2.5-hour long workshop was developed by key stakeholders, including faculty from the divisions of pediatric hospital medicine and pediatric critical care, resident representatives, and the institution's chief medical informatics officer.

Setting and Participants

The Pediatrics Residency at the University of Wisconsin Hospitals and Clinics Program is a mid-sized residency program of 45 categorical residents affiliated with the University of Wisconsin School of Medicine and Public Health. Thirty-four pediatric residents participated in this workshop, scheduled in August (early in the academic year) during required weekly block conference educational time.

Room Setup

We used a cluster room with tables arranged to facilitate teamwork in groups of three to four residents and one facilitator. Groups were intentionally arranged with residents of all three levels mixed to allow for sharing of perspectives and experiences. The room was equipped with standard AV equipment including a computer and projector. Participants were instructed to bring a tablet, mobile device, or laptop to use during the session. Paper copies of the PNAPE tool (Appendix A; three copies per participant) and cases (Appendices B, C, D; one copy per participant of each of the three cases) were brought to the session.

Schedule for the Workshop

Preworkshop questionnaire, 5 minutes

As participants arrived at the session, they were instructed to use their device to complete a four-question, anonymous online presurvey (Qualtrics) to assess knowledge, attitudes, and confidence about note writing (Appendix E). Knowledge was assessed with two questions instructing residents to select all correct items about the purpose of progress notes (nine items) and appropriate progress note content (11 items). Two 5-point Likert scale questions measured attitudes toward the importance of progress notes, and confidence in progress in note-writing skill.

Overview, background presentation, and large-group discussion, 15 minutes

Resident and faculty facilitators introduced themselves and reviewed the goals and objectives for the session. Participants were prompted to consider, “What is the purpose of an inpatient progress note?” and shared their thoughts in a facilitated large-group discussion. This discussion was steered to include facts about the progress note as a communication tool in the medical record, and its use for professional billing and coding. It also acknowledged that many trainees use the daily progress note as an aid for rounding and oral presentations and addressed the pitfalls of this approach, such as repetition and over-inclusion of information that is documented in other areas of the medical record (Appendix F).

Introduce PNAPE tool and note template to large group, 15 minutes

The PNAPE tool and its development was introduced to the participants (Appendix A). The new note template was also presented (Appendices G and H).

Cases 1–3, individual and small-group work, 100 minutes

The next portion of the activity was divided into three, 30-minute segments to review three patient cases, and included one 10-minute break. Participants were divided into small groups of three to four people, paired with a facilitator, and provided with a simple example progress note (Appendix B) to evaluate using the PNAPE tool (Appendix A). Participants reviewed and scored the note individually (10 minutes), discussed in their facilitated small group (10 minutes), and discussed with the large group (10 minutes). After a short break, this activity was repeated with a more complex patient scenario (Appendix C). Finally, participants had an additional 20 minutes to individually review a simulated patient chart (Appendix D) and generate a progress note compliant with the PNAPE tool. Once completed, participants traded notes with a partner and were given 10 minutes to assess and discuss one another's notes.

Wrap up and postworkshop questionnaire, 15 minutes

To conclude the session, the large group reconvened to summarize the key points for improved documentation of assessments and plans, adhering to the PNAPE framework. Participants also completed the same four-question anonymous online survey as a postsession assessment of their knowledge, attitudes, and confidence about note writing (Appendix E).

Workshop Evaluation

Residents completed a four-question online survey (Qualtrics) immediately before and after the workshop (Appendix E). Two multiple choice questions assessed participant knowledge about the purpose and recommended components of progress notes, for a total of 19 possible responses. The other two questions used 5-point Likert responses to assess resident beliefs about the importance of notes and confidence in their own ability to generate concise and complete notes. Pre-post survey data were imported into Excel and compared using Chi-squared for binary questions and Mann-Whitney for Likert scale questions. Pre-post differences were considered significant if p <.05. This project was deemed quality improvement and not under the purview of the University of Wisconsin-Madison Health Sciences Institutional Review Board.

Results

This workshop was delivered to a group of 34 University of Wisconsin Department of Pediatric Residents in the fall of 2018. Pre- and postworkshop questionnaires were completed by 26 (76%) and 23 (68%) participants, respectively. Prior to the workshop, many residents correctly identified several purposes of progress notes (Table), including billing for the visit, communication with other providers, medicolegal documentation, and documenting important events and the plan for the day. Most residents already knew some key components of progress notes, including patient age, admission problem, problem requiring ongoing hospitalization, plan for each problem, and discharge criteria. Notably, they were unsure of the need to include past medical history, recent diagnostic results (laboratory values, culture results, and radiology reports), and 24-hour vital signs (Table).

Table.

Pre- and Postworkshop Survey Correct Responses to Knowledge Questions About the Purpose and Components of Progress Notes

Overall, residents improved their knowledge regarding progress notes, as demonstrated by an improvement in the proportion of total accurate responses from 78% to 91% (p <.05). Accurate responses improved in 15 out of the 20 knowledge items regarding the purpose of progress notes and key note components, and were statistically significant for five individual items regarding note components (p <.01 for each; Table).

Resident belief in the importance of progress notes and confidence in writing a concise and complete note both increased from medians of 4 out of 5 before the workshop to 5 of 5 after (p <.01, p <.05, respectively).

Discussion

Resident knowledge about the purpose of inpatient daily progress notes and the identification of necessary content improved after participation in an interactive workshop. Specifically, improvements were seen in self-reported resident belief in the importance of progress notes and confidence in writing concise and complete notes. It is worth noting that pre- and postworkshop knowledge assessment demonstrated statistically significant improvement for questions regarding best practices for note elements, but not questions regarding note purpose (Table). It was possible that residents enter training with relatively strong knowledge about the purpose of notes, making improvement more challenging, but with less knowledge of best practices in note writing. Implementing this workshop early in the academic year can help make expectations for progress notes uniform and explicit rather than relying on advice and potentially out-of-date note templates handed down from individual senior residents to interns during clinical care.

The workshop employed the novel, validated PNAPE tool that featured a simple, logical, binary checklist-style format and focused on progress note assessments and plans in EHR-based notes. In constructing the PNAPE tool, we identified best practices that would be important in some, but not all situations. For example, not all patients will have new symptoms and events requiring synthesis (item 8) or multiple problems to prioritize or consider omitting from the note (items 9 and 10, respectively). We also noted that for some neonatal admissions, previously healthy would be implied by a lack of other history such that for those particular children, it would be acceptable not to comment on the past medical history. We recognize that there was a bias toward higher scoring inherent in counting non-applicable as yes, however this would not affect pre-post scores from the workshop and seemed necessary in order to collect information about key components of note quality for the majority of patients.

Involving multiple resident and faculty stakeholders in the creation of the workshop was helpful to ensure the generalizability of the content across our program. Resident participation in presenting the workshop also helped to engage and earn credibility from participants. The PNAPE tool helped focus the learning activity on specific, actionable opportunities to improve progress notes. Many of the workshop presenters have also used the PNAPE tool to provide formative feedback to residents during their ward rotations. The binary checklist items were specific and objective enough for learners to use the tool independently to review and improve their own notes.

Setting the order of the example cases was one challenge while preparing the workshop. We didn't want the last case to be too complicated for residents to complete independently, but it was a bit unusual to have a backwards progression in complexity. Planning the appropriate amount of time for each case was also challenging, as individual learners required variable amounts of time to complete the individual and small-group work. Some institutions may struggle recruiting senior residents to help present the workshop, scheduling a sufficient block of educational time, and/or engaging the learners in a topic that may be viewed as nonclinical.

However, note writing in the EHR is here to stay, and we as educators need to grapple with how to best train the workforce in practices that optimize utility and minimize the burden of electronic notes for the numerous parties who create and use them. Educators may employ this workshop, with or without additional institution-specific components, to improve resident understanding and confidence in note writing. The cases used in this workshop could be changed out to suit different environments, but the didactic material, workshop format, and focus on learning through assessment can be ported to any setting.

The study of our workshop had important limitations. It occurred at a single medium-sized pediatric residency program, did not assess note quality outcomes (assessed in a companion study), and did not follow knowledge and confidence retention over time. While the workshop was conducted with pediatric residents, the tool was not pediatric specific. The PNAPE tool was based on best practices for generating thoughtful, accurate, synthesized, concise assessments and plans, such that it should be portable to any inpatient progress note. The note template (Appendices G and H) may not be applicable to surgical specialties or those not billing based on documented evaluation and management. Future studies should affirm the results in a sample of nationally diverse programs of both academic and community-based residency programs, as well as specialties outside of pediatrics. Additionally, PNAPE was a new tool; a separate manuscript describing its validation is forthcoming.

Following successful initial implementation, this workshop has been integrated into the annual curricula for interns at the University of Wisconsin pediatrics residency program. Anecdotally, the original workshop described here and the first repetition of it at a problem-based learning block conference for interns in 2019 were well received on qualitative program conference feedback forms. One learner provided the following unsolicited feedback, “My progress notes are more organized, focused, succinct, readable, and useful after implementing your guidelines and using the templates. They also take less time to write when I know exactly what five or so things to include in an assessment, which is notable given than I'm on [a service] with so many complex patients.”

Next steps include dissemination of this intervention to other learners, including medical students and residents in other specialties at our institution. Future work should assess whether resident knowledge, beliefs, and confidence are sustained over time and differ by level of training, and to test the efficacy of this intervention on progress note quality and timeliness of submission.

PNAPE Tool.docx

PNAPE Case 1.docx

PNAPE Case 2.docx

PNAPE Case 3.docx

PNAPE Pre- and Postsurvey.docx

PNAPE Workshop Slides.pptx

PNAPE Progress Note Template.docx

PNAPE Progress Note Template with Epic SmartPhrases.docx

All appendices are peer reviewed as integral parts of the Original Publication.