Medical Student Perceptions of Learner-Initiated Feedback Using a Mobile Web Application.

Feedback, especially timely, specific, and actionable feedback, frequently does not occur. Efforts to better understand methods to improve the effectiveness of feedback are an important area of educational research. This study represents preliminary work as part of a plan to investigate the perceptions of a student-driven system to request feedback from faculty using a mobile device and Web-based application. We hypothesize that medical students will perceive learner-initiated, timely feedback to be an essential component of clinical education. Furthermore, we predict that students will recognize the use of a mobile device and Web application to be an advantageous and effective method when requesting feedback from supervising physicians. Focus group data from 18 students enrolled in a 4-week anesthesia clerkship revealed the following themes: (1) students often have to solicit feedback, (2) timely feedback is perceived as being advantageous, (3) feedback from faculty is perceived to be more effective, (4) requesting feedback from faculty physicians poses challenges, (5) the decision to request feedback may be influenced by the student's clinical performance, and (6) using a mobile device and Web application may not guarantee timely feedback. Students perceived using a mobile Web-based application to initiate feedback from supervising physicians to be a valuable method of assessment. However, challenges and barriers were identified.

Introduction

Feedback in clinical education has been defined by van de Ridder et al[1] as “specific information about the comparison between a trainee’s observed performance and a standard, given with the intent to improve the trainee’s performance.” In the clinical setting, medical students receive feedback in the form of both formative and summative assessment[2]; however, the quality, quantity, timeliness, and effectiveness of feedback can vary significantly across clerkships and clinical experiences. Efforts to better understand methods to improve feedback are an important area of educational research.

Summative feedback is traditionally provided at the completion of a clerkship, with the intent of evaluating the student’s achievement of course objectives, assigning grades, and recommending promotion.[3] In contrast, the aim of formative feedback is performance improvement.[4,5] Formative feedback is more effective when provided in a timely fashion as it allows students to recall their performance and apply it to future learning.[5,6] Timely formative feedback is often not adopted by evaluators, leaving students uninformed of potential areas of improvement and progress toward competency.[7]

Historically, feedback is described as educator-driven with a focus on content delivery from the supervisor to the trainee.[8-10] Specific to medical education, a hierarchical approach was often adopted as those directly supervising clinical activities are optimally positioned to provide feedback to the learners.[11] In his classic paper on feedback in clinical education, Ende[7] states “the hierarchy of the teaching hospital-attending, resident, intern, and student-allows for an orderly flow of information.” He further describes that anyone responsible for a “subordinate’s” evaluation is obligated to provide feedback. The feedback sandwich is another example where the delivery of feedback typically occurs from teacher to learner. Feedback provided by the evaluator is framed in the following manner: strengths or positive aspects, corrective action or area of improvement, strengths or positive aspects.[4,12] This approach to feedback tends to emphasize a one-sided conversation with minimal contribution from the student.

In contrast to the traditional unidirectional feedback from teacher to learner, learner-centered models support a more active role of the student in the feedback process. Telio et al[8] described an education alliance framework in which a collegial relationship between the teacher and learner allows for mutual understanding of performance goals, negotiating shared agreement on action plans, and cocreating educational opportunities to use feedback in practice. Rudland et al[13] described a learner-centered model where the student assumes a principle role in seeking feedback. This model places students in the center of the feedback, requiring them to actively engage in the process by seeking clarification when feedback may lack specificity and evaluating the feedback against their own perception. French et al[14] developed the ask-tell-ask feedback model in which the learner must self-assess performance and identify strengths and areas of improvement prior to receiving feedback.

Described benefits of learner-centered and student-driven feedback models include promoting student accountability, enhancing self-regulated learning, and more readily supporting adult learning principles.[13,14] However, these models are not without challenges. Adopting a new feedback process requires time and commitment for faculty development.[14] A collaborative effort between the learner and teacher is necessary to facilitate learner progression.[15] The educator-student relationship is complex with students describing their interactions with supervisors as a major determining factor in the feedback process.[16] Students may lack the confidence and self-esteem to initiate engagement in the feedback process or take responsibility for performance improvement.[13] Finally, it is important to note that feedback elicited from an external source may be in conflict with a learner’s internally generated feedback.[16] Failure to change learning behavior is a consequence of ineffectively adopted learner-centered and student-driven feedback models.

Studies in the surgical literature have demonstrated that a student-focused curriculum, with an emphasis on self-direction, initiative, and engagement, is perhaps the best approach to improve the quality and quantity of feedback.[17,18] Furthermore, the development of lifelong learning skills requires trainees to actively participate in the assessment process, which is imperative as medical education has adopted a competency-based framework.[17] Thus, feedback that is learner-initiated and occurs in a timely fashion may provide trainees with the skills and knowledge to successfully achieve milestones in a competency-based curriculum.[19]

The role of mobile technology to facilitate competency-based feedback is an emerging concept in medical education.[20,21] VSTAR Compass, developed by the Vanderbilt University School of Medicine, is a Web application designed for use with a mobile device with the intent to empower students to request feedback from supervising physicians.[22] Immediately after an observed clinical experience, the student can request feedback using a mobile device. An email is automatically generated to the evaluator with this request. The supervising physician completes and submits an online formative assessment that is viewable by the student, course director, and portfolio coach (Figure 1).

Figure 1.

VSTAR Compass feedback.

Prior to piloting the use of VSTAR Compass during a clinical elective in anesthesiology, we desired to better understand medical students’ perceptions of feedback as well as their thoughts on the implementation of a mobile Web application to gather learner-initiated feedback. Thus, we used a qualitative research method of collecting data with focus groups. Conducting focus groups is an ideal approach for exploratory research, where the goal is to gain a better understanding of participants’ thoughts, understandings, emotions, actions, and circumstances. Furthermore, focus groups are frequently used early in a research project to lay the foundation for subsequent research.[23] Because we ultimately plan to compare the quality and quantity of learner-initiated feedback with that of our current educator-driven formative assessment process, obtaining focus group data on feedback methods is a logical initial step.

This study represents preliminary work as part of a plan to investigate the effectiveness, impact, and barriers of a student-driven system to request feedback from faculty. The intent of the study was to identify perceptions of medical students about faculty feedback and soliciting faculty feedback using a mobile device and Web-based application. We hypothesize that medical students will perceive learner-initiated, timely feedback to be an essential component of clinical education. Furthermore, we predict that students will recognize the use of a mobile device and Web application to be an advantageous and effective method when requesting feedback from supervising physicians.

Methods

After receiving approval from the Vanderbilt University Institutional Review Board, fourth year medical students taking a 4-week anesthesia clerkship were recruited via email prior to their first day of the course. Students’ participation was on a voluntary basis. Four focus groups were conducted with each group consisting of 4 to 6 medical students. A total of 18 students participated. The moderator of the focus groups was not involved with the clinical course ensuring that responses would have no bearing on student assessment or grade. The same person moderated all focus groups.

We desired to glean from student’s past experience regarding collecting formative feedback from residents and attendings to better understand how the mobile application would be used. Understanding the meaning of the learner’s previous experience was of paramount importance in order for them to relate it to how they perceived this feedback method going forward. At the beginning of the focus group, students received a demonstration of the VSTAR Compass mobile Web application. Six open-ended questions (Table 1) were asked of the group. In an attempt to collect first impressions of a novel application, questions were designed to elicit student perceptions regarding learner-initiated feedback from supervising physicians in the clinical setting, the potential impact of timely feedback on clinical performance, and possible barriers to learner-initiated feedback using a mobile Web application. Each group participant was given the opportunity to share their thoughts and experiences with the larger group, and each response was recorded and transcribed at a later time.

Table 1.

Focus group questions.

1. How do you currently receive feedback about your clinical performance?

2. What are your thoughts about requesting feedback from a faculty physician?

3. What are your thoughts about requesting feedback from residents?

4. Do you think timely feedback can improve your clinical skills or performance?

5. What are your thoughts about using a mobile device to request real-time feedback?

6. What are potential barriers to learner-initiated feedback using a mobile Web application?

Results

Answers were transcribed and the de-identified transcripts were entered into Atlas.ti (version 7.0; Atlas.ti Scientific Software Development GmbH, Berlin, Germany) for data management and analysis. Answers and discussions associated with each question were analyzed and categorized into themes. Predominate themes and representative quotes from participants are summarized in Table 2.

Table 2.

Themes and representative responses from focus groups.

Theme	Representative responses
1. Medical students often have to solicit feedback	“I typically get feedback at the halfway point, like two weeks in but I ask for it. If I don’t ask for it, then I won’t get feedback until the end of the rotation”
	“You have to seek it out and it is nonspecific.”
2. Timely feedback is perceived as being advantageous	“The evaluations we get at the end are vague and you don’t find out about it until the end of the rotation or towards the end and you can’t change anything then”
	“By the time you typically receive feedback, you are done with the rotation. It does not matter anymore because you can’t change anything”
3. Feedback from faculty (compared with feedback from residents) is perceived to be more effective	“Faculty provide better quality feedback. I think because they have a different perspective than residents”
	“I’d say attendings aren’t under as much stress as residents. Residents can lack perspective. They are being observed themselves”
4. Requesting feedback from faculty physicians poses challenges	“I struggle with asking attendings. It is easier to ask residents for feedback”
	“If the attending giving feedback is intimidating or not approachable, it makes it hard to ask”
5. The decision to request feedback may be influenced by the student’s clinical performance	“If I have a bad day, I may not request feedback”
	“My mood or how I performed that day may be a barrier”
6. Using a mobile device and Web application may not guarantee timely feedback	“Feedback will not be timely if they (supervising physicians) don’t check their email”
	“What if they don’t check their emails after you request the feedback? If there’s a time limit on when it gets filled out, you may not get very many”

Theme 1: Medical students often have to solicit feedback

Across all focus groups, participants expressed the importance of receiving feedback. A predominant theme emerged as students either stated they routinely solicit feedback or described specific clinical scenarios in which feedback was given only after requesting it. Statements coded to this theme include the following:

I typically get feedback at the halfway point, like two weeks in but I ask for it. If I don’t ask for it, then I won’t get feedback until the end of the rotation.

You have to seek it out and it is nonspecific.

I don’t remember ever getting feedback from residents that was not me initiating it first.

If I don’t do something correctly, like inserting an IV, I try to ask what I could have done differently. Otherwise I won’t know for the next time.

Theme 2: Timely feedback is perceived as being advantageous

Students within all focus groups perceived timely feedback as an opportunity for improvement. End-of-course assessments as a primary source of feedback may preclude students from improving their clinical performance. Statements coded to this theme include the following:

The evaluations we get at the end are vague and you don’t find out about it until the end of the rotation or towards the end and you can’t change anything then.

By the time you typically receive feedback, you are done with the rotation. It does not matter anymore because you can’t change anything.

It’s difficult to improve if you receive feedback 6 weeks later.

Immediate feedback is helpful.

Evaluations at the end of the course tell you how you did, not what you could do to improve. It’s already too late to make changes.

Theme 3: Feedback from faculty (compared with feedback from residents) is perceived to be more effective

Within all focus groups, students discussed the effectiveness of feedback provided by faculty compared with feedback provided by residents. Many students believed that ineffective feedback from residents was due to either the resident stress level or inexperience with providing quality feedback. Specific statements coded to this theme include the following:

Faculty provide better quality feedback. I think because they have a different perspective than residents.

I’d say attendings aren’t under as much stress as residents. Residents can lack perspective. They are being observed themselves.

The best form of feedback is from attendings.

Attending feedback has more quality than quantity.

I think you get better feedback from attendings, its more valuable.

Theme 4: Requesting feedback from faculty physicians poses challenges to medical students

Medical students expressed difficulties associated with requesting feedback from faculty physicians. Specific comments coded to this theme include the following:

I struggle with asking attendings. It is easier to ask residents for feedback.

If the attending giving feedback is intimidating or not approachable, it makes it hard to ask.

I worry about being annoying.

It’s better if there is a clear cut time to meet. Otherwise, it’s difficult.

Some attendings don’t interact with students at all, making it impossible to ask for feedback.

Some are intimidating or not approachable, it makes it hard to ask.

Theme 5: The decision to request feedback may be influenced by the student’s clinical performance

Within each focus group, students pointed out that personal factors may affect whether or not to request feedback. Statements coded to this theme include the following:

If I have a bad day, I may not request feedback.

My mood or how I performed that day may be a barrier.

I could see that some people would just not ask for feedback in some cases.

What if the interaction is bad? I worry about the effectiveness of the feedback if the student picks and choses who to have evaluate them.

Theme 6: Using a mobile device and Web application may not guarantee timely feedback

Medical students within each focus group expressed concern that this method of feedback may not be timely:

Feedback will not be timely if they (supervising physicians) don’t check their email.

What if they don’t check their emails after you request the feedback? If there’s a time limit on when it gets filled out, you may not get very many.

They may not remember me by the time they check their email.

I worry about survey fatigue. They may just start to ignore the emails.

Discussion

A feedback system that supports students in achieving their learning goals is a vital component of a learner-centered curriculum.[19] Based on focus group responses, using a mobile Web application to facilitate timely, learner-initiated formative feedback has the potential to enhance clinical education and enable achievement of student learning goals. However, medical students also identified potential barriers and challenges.

Analysis of focus group data revealed 6 main themes. The first theme notes that students often have to solicit feedback from supervising physicians. When it comes to receiving effective feedback, it is well known that learners are often dissatisfied.[24-26] Although soliciting feedback requires the learner to play an active role in the process, current educational literature describes a shift from the traditional instructor-initiated feedback model to a learner-centered feedback paradigm.[27,28] The authors of a recently published study investigating resident perceptions of learner versus attending-initiated feedback reported residents to be equally satisfied with the quality of feedback received from either method.[29] Thus, medical schools should educate students regarding the importance of assuming a proactive role in the learning process and empower them with the tools to be an active learner. The VSTAR Compass application will enable students to successfully solicit feedback, allowing them to be proactive learners in their education.

The second identified theme from focus group data is timely feedback is perceived as being advantageous. An important purpose of formative feedback is to enable the learner to make needed changes by providing timely feedback, ideally as soon as possible after the encounter allowing for sufficient time to implement required actions.[30] End-of-course assessments as a primary source of feedback may preclude students from having an opportunity to improve their clinical performance or skills.[9,27] In the absence of ongoing, formative feedback from supervising faculty, learners tend to generate their own feedback. Mistakes and misconceptions may go uncorrected and clinical competence may not be achieved.[3] Thus, using VSTAR Compass to request timely feedback throughout a clinical experience may guide our medical students to concentrate on improvement of specific knowledge or skills, allowing for achievement of optimal performance.

The third and fourth themes relate to student and attending physician interaction. Compared with feedback provided by residents, feedback from faculty is perceived to be more effective. However, focus group data revealed that requesting feedback poses challenges to medical students, particularly if the student perceives the faculty physician as intimidating or unapproachable. This is a valid concern related to the risks associated with direct feedback seeking and is supported in the literature.[31] A study by Milan et al[26] investigated formative feedback during clerkships from the student perspective. Students reported that faculty “approachability” has a considerably greater impact than faculty “expertise” when electing to elicit feedback. Medical students in our focus groups expressed concerns regarding interrupting faculty during their busy schedule. Research has demonstrated that individuals tend be more inclined to seek feedback from those who are perceived as being more accessible.[31] Furthermore, some students described inconsistent and often limited interactions with faculty compared with residents. In addition, there appears to be a comfort level involved with requesting feedback from residents compared with faculty physicians. Learners are also more receptive to feedback from those whom they trust.[27] VSTAR Compass has the potential to empower students with a feedback system that eliminates the risk of disruption while supporting the ability to obtain feedback from both faculty and resident physicians.

Theme 5 emerges surrounding student performance. The decision to request feedback may be determined by the learner’s perception of his or her performance. Feedback can present risks to learners if the information threatens their self-image.[32] In this situation, the learner may altogether avoid feedback opportunities.[33] A recent study by Bok et al[28] investigated feedback-seeking behaviors in clinical clerkships. The authors identified factors that influenced feedback-seeking behavior. Students with a learning orientation focused on knowledge and skill development with the aim of achieving clinical competence. Students with a performance goal orientation are motivated to avoid the effects of potential negative feedback, such as inferior clinical competence. Students with a focus on professional development and clinical competency are likely to be more inclined to request feedback regardless if the feedback is positive or negative. In contrast, students whose primary focus is the impact of performance on their final course grade will be less inclined to request feedback. As we pilot VSTAR Compass, it is imperative to understand that the learner’s perception of the value and risk associated with the feedback could be a potential obstacle. Students deliberately requesting feedback after what they perceive is a successfully clinical experience or from supervising physicians who are likely to give them positive feedback will introduce a selection bias into the feedback process.

The sixth and final theme that emerged is using a mobile device and Web application may not guarantee timely feedback. Innovation in mobile technologies for workplace-based assessment and feedback has occurred in recent years.[20,34] However, the use of technology does not guarantee an improved feedback process. The online link containing the assessment form generated by the VSTAR Compass feedback request expires after 7 days. Therefore, students may receive feedback up to a week after completing their clinical experience. Alternatively, students may not receive feedback if the assessment form is not completed prior to expiration of the link. Students correctly pointed out that the only method to ensure timely feedback is for the assessment to be completed face-to-face in real time, which may further preclude students from requesting feedback due to the barriers previously discussed.

Information gained from the focus groups is a valuable resource in developing and testing an intervention to improvement feedback received by medical students. This information has several implications. Buy-in from both learners and teachers is required when moving to a new technical assessment format.[22] For successful implementation, we believe that it is necessary to provide training to faculty and residents regarding this process of feedback. In addition, investigating beyond the utility of a mobile Web application and medical student perceptions is imperative. It is unknown whether using this method of feedback will be associated with a change in the quality and/or quantity of feedback. As this feedback process is implemented, usage data will be collected. For example, the number of feedback requests will be compared with the actual number completed. Data must also be obtained from learners regarding the benefit of timely feedback and its actual impact on incorporating the feedback and improving performance during the clinical experience.

With the implementation of this student-driven feedback system, we must remain cognizant of potential issues. Students may avoid requesting face-to-face feedback and miss the opportunity to gain insight into critical thinking and reasoning in the clinical setting. VSTAR Compass is not intended to replace clinical teaching where formative feedback is often given. It is a process to request and provide summary feedback at the conclusion of a clinical experience or interaction. However, this mode of feedback solicitation exists as an adjunct to existing forms of summative and formative evaluation already in place not as a stand-alone form of assessment.

Students could find it difficult to request feedback from faculty or residents for a number of reasons, including the busy nature of clinical schedules, presence of less approachable supervisor, or lack of confidence in their own performance. Although it is gradually changing, the authoritative culture of medical education may still exist in certain settings. VSTAR Compass is a tool that allows students to request feedback in a novel format that could empower students to become active learners rather than a tool to simply circumvent the potential stress of face-to-face feedback.

As the completed feedback is viewable by the portfolio coach and course director in addition to the student, another potential issue with this feedback system is privacy. The School of Medicine at our institution uses a unique academic advising structure. Medical students are assigned a portfolio coach on matriculating. Throughout the 4 years of medical school, students meet with their coach on a quarterly basis. Coaches partner with students to develop skills for self-assessment, professionalism, and the concept of lifelong learning. Students are encouraged to appraise data about their performance and to create goals for learning. To accomplish this, coaches must have access to student portfolios that contain course grades, feedback evaluations, self-assessments, and evidence of milestone progression. Thus, VSTAR Compass feedback is vital to course directors and portfolio coaches as they are devoted to teaching and empowering students with lifelong learning skills.

There are several limitations to our study. This study was conducted with fourth year medical students enrolled in a specific elective within a single institution. Within an institution, the process and perceptions of feedback can vary between medical specialties and departments. The culture of requesting and providing feedback also varies between institutions. Thus, our results may not necessarily apply to all learning environments. In addition, our results may reflect the biases of the individuals who volunteered for the study. Many of the students expressed an interest in anesthesiology and may be more inclined to value formative feedback during this specific clinical rotation. Finally, the students’ verbalized perceptions may not accurately reflect their actual feedback-seeking behaviors. Although the focus group method is effective in identifying themes, objective data pertaining to the actual use of a mobile device and Web application to request feedback are necessary.

Future considerations that may improve the usability and perception of VSTAR Compass should include development of a thorough orientation session for students. Implementation of faculty development surrounding the benefits of timely, student-initiated feedback should also be considered. Creation of postutilization focus groups could provide vital information regarding the development of VSTAR Compass and may achieve increased usability and feasibility. Finally, it will be imperative to compare the quantity and quality of formative feedback obtained using this innovative tool with feedback from our traditional evaluator-initiated method.

Conclusions

Prior to implementing an innovative method for receiving formative feedback, we sought to understand the perceptions of fourth year medical students concerning the use of VSTAR Compass, a Web application that has the potential to provide trainees with timely and focused assessment of their performance in the clinical learning environment. This area of investigation has the potential to provide valuable information for medical educators. Based on responses, most students felt that self-directed, learner-initiated feedback from supervising physicians would be a valuable method of assessment. Furthermore, they felt this mechanism might allow evaluators to provide a real-time competency-based evaluation as they progressed through clinical courses as well as allow for performance improvement based on the specific feedback received. Although most believe that this mechanism of soliciting feedback would be an improvement to current methods, concerns were identified with respect to difficult interactions with supervising physicians and causing interruption in the schedule of already busy faculty members.