Virtual ARCP assessment and trainee feedback meetings: facilitating the best experience and practice.

As a result of COVID-19, there has been an exponential increase in the use of remote technology for many local, regional and national meetings that would previously have been held on a face-to-face basis. Remote meetings have ensured that essential clinical, educational and strategic work can continue but it is not 'business as usual', although colleagues accept this form of communication as the new norm. In medical education and assessment, the Annual Review of Competence Progression (ARCP) meetings and other formative educational meetings are being conducted remotely. This form of communication has some advantages but may also present possible barriers for feedback and development particularly for 'trainees in difficulty' when there are concerns about progression, and when an unsatisfactory outcome has been awarded. It is also worth remembering that there may be generational differences with the ease of use of virtual meeting platforms. We present some of the important factors for optimising the panel of virtual ARCPs and discuss methods to improve feedback given remotely for trainees.

Introduction

In just a few months, the coronavirus (COVID-19) pandemic has changed our way of life and how we conduct local, regional and national meetings. There has been a significant rise in the use of virtual meeting technology using platforms such as Zoom, Microsoft® Teams and GoToMeeting.® Most platforms have a secure environment and also enable recording of the meetings when needed. Some of the significant differences between virtual and face-to-face meetings include reduced non verbal communication clues, including body language and gestures as well as limited casual interactions that are so important for building rapport and trus.1, 2 Furthermore, the inability to meet in person either before or after meetings can have a negative effect on morale, as well as networking, socialising with colleagues and catching up on news. These and other issues suggest that virtual technology is unlikely to completely replace the traditional face-to-face meetings particularly where team interaction is important such as for multidisciplinary meetings. The ‘normalisation’ of online technology to resemble face-to-face meetings includes positioning cameras to ensure eye contact with other participants.

The Annual Review of Competence Progression (ARCP) meeting is an important summative assessment of a trainee’s portfolio, logbook and achievements matched to the curriculum for their stage of training. Non adverse outcome ARCP (outcome 1) is required to progress to the next year of training. In a large study of over 11,000 ARCPs, 20% of trainees were awarded at least one adverse ARCP outcome (2- targeted training with no additional time needed, or 3 additional time required) during their training. In the event of an unsatisfactory ARCP, most panels arrange to meet with the trainee, either on the day or subsequently to discuss the outcome and provide guidance and support. It should be noted that the ARCP and any feedback are separate processes. During the current COVID-19 pandemic, this meeting is likely to be remote in its delivery. The yearly educational review meeting may also be held remotely. In this article we discuss some of the important considerations to help ensure the best experience for trainees and trainers, particularly when the ARCP outcome is unsatisfactory.

Before an ARCP panel formally meets

Prior to the ARCP panel meeting, the trainee should have been briefed by their educational supervisor (ES) as to the likely outcome, particularly if this is likely to be adverse. It is also essential for panel members to be fully prepared before any meeting but particularly when conducted virtually. This includes familiarisation with the ARCP process itself, reading the trainees’ ePortfolios (ISCP for surgery) and preparing notes if necessary to aid discussion in areas that require clarification or expansion.

To assist the ARCP panel, trainees for their part should ensure that correct and accurate evidence to be considered is uploaded well ahead of the ARCP date, particularly when panel members are required to review large volumes of evidence in advance of the meeting, and preferably not on the night before. and not the night before. The number of trainees being assessed is likely to influence the way in which the panel conducts the process. For example, for panels required to assess larger numbers (10+), the ARCP panel chair may delegate certain tasks or evidence review to each panel member, but with fewer trainees, panel members can usually review the evidence for all trainees.

Panel members should check to make sure that their broadband is working well, and a ‘test run’ held at some point before the formal ARCP, is often a valuable exercise. Issues with bandwidth during virtual meetings can result in freezing of computer screens, frustration or even anger, resulting in reduced performance of the whole panel.2, 6, 7

Adapting to the virtual meeting environment

From the outset, the ARCP chair should give an initial panel briefing to establish how process is to be run, ensure all the relevant regulatory requirements for panel members have been met, discuss timings and most importantly ensure that all members feel equally valued. Furthermore, it is important to check that panel members are familiar with what to expect and it cannot be assumed that all are comfortable with the online platform being used. It may be that for those panel members who are less confident or experienced at virtual meetings, the chair could facilitate a more balanced discussion by inviting opinions from them. Similarly, the chair should be aware of any panel members who may have a tendency to dominate the discussion. As with face-to-face meetings, etiquette with virtual meetings should be even more tightly controlled allowing only one panel member to speak at a time. When bandwidth saturation results in slowing or freezing of video, we recommend that only the chair and one other speaker have their cameras on to improve sound quality. However, ideally it is better to be able to see others on the virtual platform, just as would occur in a ‘normal’ meeting. On many platforms, the hand up function is useful to alert the chair. Another option is to turn ones camera on so as to inform the chair that you wish to speak in a similar way to hand raising during face-to-face meetings. This also encourages panel members to remain focussed as they are less likely to be distracted.

Using checklists and minimising distraction

A printed version of the ARCP checklist provided by the local Health Education England (HEE) region is a safe and reliable way to make sure all uploaded evidence has been considered and documented for each trainee (Fig. 1 ). One or more panel members can be delegated beforehand to complete this. Checklists to assist and confirm that all evidence has been considered can be invaluable not only as evidence for potential appeals but also to maintain focus particularly when distracted or interrupted.8, 9, 10 It would be naïve to think that panel members do not become distracted when not actively participating in discussions, particularly when attenting virtual meetings. It is all too easy to simultaneously multitask by using a smart phone,11, 12 clicking on an internet link or catching up on emails while ‘out of view’ of other panel members. However it should be remembered that performance is undoubtedly reduced when distracted and could even be a source of missing some vital discussion points. Many will be familiar with the ‘what do you think Dr X?’ question which comes unexpectedly, giving a startle effect and often embarrassment about how to answer.

Fig. 1

A standard checklist used to ensure that process is followed during ARCP.

Background noise can also reduce both concentration and performance so for this reason we recommend the use of headset when attending virtual meetings as well as turning off the microphone when not speaking. Finally, it is a good idea for one panel member to be the designated ‘note taker’ which can be referred to for areas of concern or further discussion, as well as ensuring that any required actions are minuted. This is particularly important for adverse outcomes and can be used to help guide post ARCP discussions with trainees.

Optimising our performance

In some schools and training schemes (such as core training, radiology, orthopaedic surgery), with many trainees to assess, the ARCP process can last for more than one day. Maintaining focus, concentration and giving the same attention to the final trainee as the first demands that panel members are optimised to give the best performance possible. It is therefore important to incorporate regular breaks throughout the day, especially since concentrating during remote meetings can be harder than face-to-face discussions for the reasons mentioned above. Unlike discussions in a room, it is harder to look for and recognise signs of fatigue in panel members.

We recommend a minimum 10-minute break every 90 minutes, with a longer break of at least 20 minutes after three hours. During this time, it is important to walk away, rehydrate and spend a short time relaxing in whatever way works for the individual. During the summer months, it is easy to become dehydrated, which can result in cognitive impairment, changes in mood, and loss of interest.15, 16 Regular comfort breaks make longer virtual meetings much more enjoyable and improve efficiency, paradoxically leading to an earlier finish time! Finally, a review or discussion summary by the ARCP panel chair is a good way of ensuring that all relevant aspects have been covered while providing an opporutnity for panel members to raise any additional points, concerns or highlights. Table 1 gives a summary of recommendations for improving the virtual ARCP panel experience.

Table 1

Some recommendations for improving virtual ARCP panel meeting experience.

Adequate preparation beforehand to include reading ePortfolio (ISCP for surgery)

Familiarity with technical platform before meeting starts

One person only to speak at a time

Consider using the ‘hand up’ function to avoid talking over each other

Ensure clear communication (remember limited non-verbal communication)

Check to panel understand what is being said

Muting of microphone when not speaking

Use of a headset to reduce background noise

One panel member to review and complete checklist; another note taker

Summary at the end of important points

Reflections for future improvement

Virtual feedback and educational review meetings

There are differences in practice across training rotations and schools for providing trainee feedback following the formal, summative ARCP process, and the conduct of yearly educational reviews. As previously mentioned, the ARCP outcome should not come as a shock for trainees who should have been briefed as to the likely result by their ES before the formal panel meeting as documented in the Gold Guide. Even with an ARCP outcome 1, trainees appreciate receiving feedback from educators and trainers about their performance.

With adoption of virtual meeting s over conventional face-to-face sessions during the COVID-19 pandemic, not unsurprisingly, there are few published studies that have evaluated this method of feedback. In one study of medical student feedback, no significant difference was found in overall experience, stress levels or helpfulness between these two modalities. However, students who met in-person with their trainers rated communication effectiveness significantly higher than remote meeting. One advantage of remote meeting was convenience and reduced travel. Given that many training schemes involve rotations to hospitals that can be considerably far apart, some UK trainees have complained about travelling several hours to attend an ARCP only to spend less than 15 minutes receiving their outcome with little feedback. Virtual ARCPs could therefore be of considerable benefit to some trainees, as well as to panel members who might also find it easier than travelling long distances to attend.

How can we improve a virtual feedback meeting experience?

We will not consider the various feedback techniques that can be used, but focus on the use of remote technology. This should ideally be supplemented with a face-to-face meeting if at all possible. As with giving any feedback, it should be done in an unhurried manner, with the minimum stress for the trainee. Hierarchy should ideally be reduced at the outset with introductions and some non-related discussion to try and put the trainee at ease. In our opinion, one of the biggest barriers for virtual feedback to be as good as in person is the relative lack of non-verbal communication, body language, and other clues that are readily visible during a face-to-face meeting. Therefore it is important to ensure that the trainee understands what is being said, and the careful and precise use of verbal communication is important. Recording feedback might also be valuable although some trainers could feel uncomfortable for fear of it being used against them. A summary of recommendations for improving virtual feedback is provided in Table 2 . Asking trainees to reiterate what has been discussed and recommended as well as written documentation of the meeting may be useful. There may be generational differences when using online platforms, with trainees often more familiar with technology than some of their trainers and this may have some implications in both giving and receiving virtual feedback. When in doubt, discussion with colleagues and/or a trial session beforehand can be beneficial.

Table 2

Recommendations for giving effective trainee feedback remotely.

Introductions to help the trainee feel at ease

Recognising that virtual feedback can be technically difficult remotely

Making clear verbal communication

Checking that the trainee understands what is being said

Consider repeat back or rephrasing

Using the ‘hand up’ function if several panel members

Summarising the meeting

Providing some written bullet points for consideration/reflection

(Recording the meeting if agreeable)

Conclusion

The use of non face-to-face meeting methods for providing educational assessment and trainee feedback has dramatically increased during the COVID-19 pandemic. While online platforms have some advantages over traditional face-to-face meetings including convenience and reduced travel time, they have limitiations including reduced non verbal communication and an inability to socialise and network with panel members. The lack of non verbal communucation is also a potential issue when providing trainee feedback, but there are some relatively simple methods that can be used to enhance these online meetings.

Conflict of interest

We have no conflicts of interest.

Ethics statement/confirmation of patient permission

N/A.

Cited Reference

21.