Patient feedback in the emergency department: A feasibility study of the Resident Communication Assessment Program (ReCAP).

OBJECTIVE: Resident physicians must develop competence in interpersonal and communication skills, but workplace-based assessment of these skills remains challenging. We explored the feasibility of the Resident Communication Assessment Program (ReCAP) for eliciting patient feedback about resident physician communication in the emergency department (ED).
METHODS: This study is a prospective, observational study conducted in the ED of a university-based hospital from December 2018 through April 2019. ReCAP is a program that interviews patients prior to discharge from the ED using the Communication Assessment Tool (CAT). CAT consists of 14 Likert style questions and 3 open-ended questions for patient feedback about residents' communication. Open-text, narrative responses from patients were coded using a modified version of the Completed Clinical Evaluation Report Rating tool.
RESULTS: We collected data from 42 subjects who completed the CAT, and provided 32 open-text, narrative responses about 20 resident physicians. Patient responses were overwhelmingly positive with 551/588 (94%) CAT responses scoring "Very Good," the highest category. Open-text, narrative comments analyzed using CCERR were unbalanced, favoring residents' strengths rather than areas for improvement. Patient comments offered more examples of strengths than weaknesses, and few subjects provided recommendations to improve resident performance.
CONCLUSION: ReCAP represents a feasible method for eliciting patient feedback about resident communication skills in the ED. The CAT can be used to structure brief patient interviews by trained staff but generally elicits only positive feedback. Further studies are needed to identify more discriminatory assessment tools.

INTRODUCTION

Background

Effective communication is highlighted as essential by the American College of Graduate Medical Education (ACGME) and the Royal College of Physicians and Surgeons of Canada (RCPSC). Communication in the emergency department (ED) is demanding of trainees, requiring them to build rapport with patients quickly, understand and address a diverse array of patient concerns, and manage systems‐level expectations. , However, teaching and assessing communication skills of trainees in the workplace environments presents a significant challenge for both educators and residents due to time constraints and a lack of criterion‐referenced assessment tools.

Importance

Although direct observation remains the current gold standard, opportunities for faculty members to observe resident–patient interactions are infrequent. Although emergency medicine attendings are in‐house at all hours, they are limited by the sheer volume of patients requiring care, and direct observation can limit efficiency and throughput. Bedside observations are limited, with emergency medicine clinical faculty spending as little as 1% of their time on‐shift supervising interactions with non‐critical patients.

Goals of this investigation

Patients constantly observe residents and form impressions of their behavior, and these impressions are critical for resident education. We believe patients are ideally suited to judge the quality of their resident's communication. The Resident Communication Assessment Program, also known as “ReCAP,” was introduced at Stanford University Hospital in 2015 and piloted in the departments of Otolaryngology, Plastic Surgery, Neurology, Orthopedic Surgery, and Internal Medicine. This program seeks patient perspectives about resident communication to provide trainees with feedback, focusing on quality and satisfaction with these interactions. Our study aims to explore the feasibility of using ReCAP to elicit patient feedback about resident communication skills in the ED. ReCAP uses the Communication Assessment Tool, which has been studied in several inpatient and outpatient settings , , and was adapted for use in emergency medicine in 2018.

METHODS

Study design and setting

This study was a prospective, observational study conducted in the ED in a university‐based, tertiary care hospital. Our ED has 58 beds and an annual volume of 80,000 patients.

Selection of participants

Our sample included patients who visited the Stanford ED and were pending discharge. Inclusion criteria were all stable patients over the age of 18 with normal mental status. Patients with altered mentation including psychosis or dementia, those unable to verbally communicate, and those with a history of violent or verbally aggressive behavior toward staff during current or prior visits were excluded as determined by the bedside nurse. Patients admitted to the hospital were excluded to mirror common patient satisfaction survey efforts targeted at discharged ED patients, such as Press‐Ganey. Information from patients who declined to participate or were excluded was not recorded. In‐person language interpreters and phone translation services were available for non‐English speaking patients. Patients were recruited as a convenience sample based on the limited availability of volunteer patient interviewers. Patients were interviewed on weekends between the hours of 8:00 am and 6:00 pm from December 2018 to April 2019. Patients were only considered for enrollment after residents flagged their patients for discharge from the ED. Residents were not notified when volunteers were in the ED or interviewing patients. Patients seen by emergency medicine residents from all 4 years were interviewed. Approval was obtained from the Stanford University Institutional Review Board (file 48206) prior to patient sampling, and patients verbally consented for participation.

Patient interviewers

Our patient interviewers were graduate and post‐graduate students recruited through the Stanford hospital's volunteer resources program. All interviewers had at least 2 years of experience interviewing patients in other inpatient iterations of ReCAP and assumed leadership positions within the program. They received 6 hours of didactic training in privacy compliance, service recovery, empathetic interviewing, personal safety training and supervised practice interviewing with patients. Interviewer training included a review of our study goals, study protocol, and data collection methods. Interviewers did not participate in data analysis.

Assessment tool

ReCAP uses the CAT, a widely adopted CAT that has validity evidence in a number of clinical settings. , , We chose to use this tool because it was the most comprehensive and appropriate communication assessment for post‐graduate trainees as it closely mirrors the ACGME milestones for emergency medicine.

The CAT instrument consists of 14 questions on attributes of communication using a symmetric 1–5 Likert scale. We adapted the original tool by adding 3 open‐text questions for narrative comments to elicit additional patient feedback. Given this modification, the tool was piloted in our patient population for feasibility, length of interview, and general content analysis by several of our study investigators. These included 2 faculty physicians with expertise in resident assessment (AA and MG), a faculty psychometrician (SSS‐S), and 3 resident physicians (CM, KM, and HM). These investigators convened to review field notes and experiences from their use of the modified CAT prior to implementation, with consensus agreement to use the open‐text questions without further modification.

Content analysis

We coded our open‐text responses using the Modified Completed Clinical Evaluation Report Rating (CCERR). , This method of analysis codes open‐text data using 6 questions answered on a 5‐point Likert scale anchored at 3 points (1‐Not at all, 3‐Acceptable, 5‐Exemplary) to assess critical features of written feedback given to resident physicians. Our psychometrician (SSS‐S) is an expert in the use of the Modified CCERR for workplace‐based assessment of physicians‐in‐training; her previous work provides validity evidence for our selection of the tool. She trained 2 study authors in the use of the Modified CCERR (CM and HH) and they rated all responses. Instances of disagreement between raters (41% of responses) were adjudicated by a third, trained rater (EC) and reviewed for internal consistency by our psychometrician.

RESULTS

ReCAP‐emergency medicine data were collected from 42 subjects, which provided patient feedback ∼43% (20/46) of our resident physicians. Of the 20 residents who received patient feedback in this study, 5 = post‐graduate year (PGY)1, 6 = PGY2, and 9 = PGY3. All subjects were English speaking; 18 (43%) patients were male, 13 (31%) female, and 11 (26%) did not report their sex. The average interview length was 10 minutes.

The Bottom Line

Patients are potentially important sources of resident performance data. This study demonstrates the feasibility of obtaining patient evaluation of resident communication in the emergency department.

As presented in Table 1, patients rated the communication of their resident physicians very positively, with (94%) selecting the highest category of “Very Good,” and (4%) reporting “Good,” the second‐highest group. We received 32 responses to the open‐text questions. The coding breakdown of these responses according to the modified CCERR tool are shown in Table 2. Seventy‐two% (23/32) of patient responses were scored as “Somewhat” explaining examples of residents’ strengths, and 9% (3/32) earned a score of “Good.” Responses were also generally supportive of residents, and 71% (24/32) scored at least “Good” for this rating. However, 88% (28/32) of comments were scored as “Not at all” balanced in regards to residents’ strengths and weaknesses, 84% (27/32) were scored as “Not at all” offering examples of weaknesses, and only 9% (3/32) of responses offered any recommendation for improvement. Fifty‐three% (17/32) were rated as “Somewhat” and 31% (10/32) were rated as “Good” in terms of justifying the CAT ratings. In our reporting of patients’ open‐text responses, we identified patients using the code “P#.”

TABLE 1

Modified Communication Assessment Tool (CAT) questionnaire

“How well did the resident physician…”	Very poor	Poor	Fair	Good	Very good	N/A
Greet you in a way that made you feel comfortable?	0(0)	0(0)	0(0)	2(5)	40(95)	0(0)
Treat you with respect?	0(0)	0(0)	0(0)	2(5)	40(95)	0(0)
Show interest in your ideas about your health?	0(0)	0(0)	0(0)	2(5)	40(95)	0(0)
Understand your main health concerns?	0(0)	0(0)	1(2)	0(0)	41(98)	0(0)
Pay attention to you (look at you, listen carefully)	0(0)	0(0)	1(2)	2(5)	39(93)	0(0)
Let you talk without interruptions?	0(0)	0(0)	1(2)	2(5)	39(93)	0(0)
Give you as much information as you wanted?	0(0)	0(0)	0(0)	3(7)	39(93)	0(0)
Talk in terms you could understand?	0(0)	0(0)	0(0)	2(5)	40(95)	0(0)
Check to be sure you understood everything?	0(0)	0(0)	3(7)	2(5)	36(86)	1(2)
Encourage you to ask questions?	0(0)	0(0)	1(2)	2(5)	38(90)	1(2)
Involve you in decisions as much as you wanted?	0(0)	0(0)	0(0)	1(2)	40(95)	1(2)
Discuss next steps, follow‐up plans.	0(0)	0(0)	0(0)	1(2)	41(98)	0(0)
Show care and concern.	0(0)	0(0)	0(0)	3(7)	38(90)	1(2)
Spend the right amount of time with you.	0(0)	1(2)	0(0)	1(2)	39(93)	1(2)

Data are reported as n(%). Total n = 42.

TABLE 2

Modified Completed Clinical Evaluation Report Rating (CCERR) tool

	Not at all	Somewhat	Good	Very good	Excellent
Comments are balanced providing both strengths and areas for improvement.	28(88)	2(6)	2(6)	0(0)	0(0)
Comments justify the ratings provided.	5(16)	17(53)	10(31)	0(0)	0(0)
Clearly explained examples of strengths using specific descriptions are provided in the comments.	6(19)	23(72)	3(9)	0(0)	0(0)
Clearly explained examples of weaknesses using specific descriptions are provided in the comments.	27(84)	5(16)	0(0)	0(0)	0(0)
Concrete recommendations for the trainee to attain a higher level of performance are provided.	29(91)	3(9)	0(0)	0(0)	0(0)
Comments are provided in a supportive manner.	4(13)	4(13)	21(65)	3(9)	0(0)

Data are reported as n(%). Total n = 32.

When providing feedback, most patients focused on residents’ strengths, some on weaknesses, and few on both. However, in comments that were balanced, the responses seemed high yield. For example, “Very good at her job. Knowledgeable, kind, and friendly. Could be less rushed, but unsure if [that is] her fault” (P#19). With facilitation, this comment may alert the resident to the idea that controlling their manner of speaking or body language can help the patient feel more at ease.

A typical comment scored as commenting on examples of strengths with “Somewhat” clear examples would be as follows: “Very kind and friendly. Knew what she was doing. Very thorough, spent a lot of time” (P#13). This patient is explicit in describing which of the residents’ qualities they appreciated, although they were not specific about what the resident did that made them feel that way. Some comments described examples of resident behavior in greater detail: “[The resident's name] did an excellent job listening to patient's concerns and taking them into account” (P#22) and “He cared about me and told me what he was doing before doing it” (P#33). These both reward specific resident behaviors that were notable to patients. Although the impact of this feedback on resident performance is unclear, it seems feasible that this feedback would reinforce positive behaviors.

The following patient offered the sole comment that contained no positive elements; they stated that the resident did “not [come] visit me because I have pain" (P#29). Acknowledging a potential language barrier in this comment, this patient's understanding of their interaction with the resident is that they were neglected specifically because of their pain. When delays in care occur, especially those related to circumstantial issues such as critically ill patients or high patient volumes, this needs to be communicated in a way that is meaningful to patients. This comment is valuable because it captures an aspect of patient care that another evaluator may not necessarily recognize.

DISCUSSION

ReCAP appears to be a feasible method for eliciting timely patient feedback regarding emergency medicine residents’ communication skills. ReCAP uses a standardized and widely accepted CAT , , administered by trained interviewers prior to patient discharge. These efforts were undertaken with the intention of reducing potential sources of bias, providing a more uniform medium for assessing residents, and facilitating feedback collection and delivery. This offers advantages over other written feedback modalities such as Press‐Ganey, which are completed asynchronously and returned weeks or months later. With the assistance of an interviewer, patients were able to express their thoughts regarding their care in real‐time, and their comments appeared genuine and supportive of residents’ education. However, requiring a trained in‐person interviewer may be cost prohibitive to some programs. We have been able to use the yield of ReCAP thus far to expand our understanding of what we can expect from patients: comments that are resident‐focused, supportive, and often provide examples of things done well. We have demonstrated a viable method for obtaining patient feedback that emergency medicine residency programs could use to supplement a facet of emergency medicine training that has been historically challenging for educators.

Patient feedback also had limitations. Recorded responses were often brief, and patients were regularly unable to suggest how residents could perform better. However, during periodic quality control of patient interviews by a member of our research team (CM), it was noted that patients often explained their reasoning at length when deciding how to answer the ReCAP questions. This suggests an opportunity to improve the data: expanding the open‐text, narrative component of the interview for each CAT item. Although the Likert scale responses were less useful for differentiating residents’ performance, patients frequently verbalized specific details and examples of resident behavior in conversation with their interviewers. There was no structure for recording these responses, and seemingly valuable information was omitted. It was apparent that these survey items were thought‐provoking, and that an iteration of ReCAP allowing for open‐text feedback to all of the CAT questions may yield high quality patient responses.

Limitations

Patient responses were generally positive and did not discriminate between residents at different skill levels. The open‐text comments offered minimal actionable feedback for skill improvement. Although this issue might be solved by modifying our interview process or interviewer training, it may be that patients consented for a study are more likely to offer positive feedback only. This calls into question the return on investment for the current version of the ReCAP program. Patients may also be more likely to offer positive comments when asked in person instead of asynchronous, anonymous, electronic feedback. Furthermore, residents may recognize interviewers in the ED, confounding the data. Being a pilot study, all data were obtained at a single hospital as a convenience sample, which limits generalizability of the data. Admitted patients were not included to mimic ED reporting such as Press‐Ganey, but may provide a different perspective and serve as a valuable cohort for future studies.

Nonetheless, ReCAP examined patient feedback about the communication skills of emergency medicine residents and highlighted that patients could provide positive and supportive comments while describing specific aspects of their care that they liked or disliked. This feasibility study provided insights that can be gleaned to support patient feedback as an aid in the development of future communications curricula.

AUTHOR CONTRIBUTIONS

CM, HM, EC, MAG, and SSS‐S designed the study and selected the tools used to assess patient responses. CM and MK trained patient interviewers and organized the patient interview process under guidance from MAG. CM, HM, and EC organized and interpreted data, including qualitative analysis, under guidance of SSS‐S and MAG. CM and SSS‐S drafted the article. All other authors contributed to revision. CM takes the final responsibility of the paper.

ETHICAL APPROVAL

This study was approved by the Institutional Review Board at Stanford University on October 28, 2018 (file 48206).

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.