Radiology trainee and attending satisfaction with virtual readouts during the COVID-19 pandemic.

RATIONALE AND
OBJECTIVES: In response to COVID-19, our institution implemented three virtual readout systems: a commercial HIPAA compliant web-based video conferencing platform used for screen-sharing (Starleaf), an interactive control sharing system integrated into PACS allowing simultaneous multi-user mouse control over images (Collaborate), and the telephone. Our aim was to assess overall satisfaction with and perceived effectiveness of these virtual readout methods to optimize best practices for the future.
MATERIALS AND METHODS: An IRB-exempt survey was electronically distributed to 64 trainees and 76 attendings at one tertiary-care institution via Survey Monkey. Questions focused on overall satisfaction, perceived effectiveness, technical difficulties, and continued future use of the three virtual readout strategies. Answers were collected with Likert scales, tick boxes, and open-ended questions.
RESULTS: 32/64 trainees (50%) and 32/76 attendings (42%) completed the survey. Trainees and attendings were more satisfied with screen sharing (Starleaf) and perceived it more effective than control sharing (Collaborate) or the telephone (p < 0.0001). Respondents experienced more technical difficulties with control sharing versus screen sharing (p = 0.0004) with a negative correlation between level of technical difficulties and satisfaction with screen sharing (r = -0.50, p < 0.0001) and control sharing (r = -0.38, p = 0.0006). Trainees and faculty supported a combination of in-person and virtual readouts in the future (p < 0.0001).
CONCLUSION: Platforms mirroring in-person readouts, such as Starleaf, are preferred by both trainees and attendings over non-screen sharing platforms such as the telephone. However, technical stability determines satisfaction between similar platforms. Both trainees and attendings support incorporation of virtual readout methods in combination with traditional in-person readouts in the post-COVID-19 era.

Introduction

The COVID-19 pandemic forced radiology residency programs to restructure their learning environments in order to comply with the mandated social distancing measures.1., 2. Specifically, traditional in-person readouts were disrupted, and new virtual readout methods were subsequently developed. Several papers published during the earlier stages of the pandemic detailed the different virtual readout methods employed, which included videoconferencing and screening sharing software,3., 4., 5, 6., 7., 8. however few focused on trainee and faculty satisfaction with these practices. One paper investigated resident satisfaction with virtual learning platforms, including virtual readouts, and found that residents were less satisfied with virtual readouts compared to other components of their virtual curriculum, but did not go into further detail.

At the start of the pandemic, our department established three different virtual readout systems, including a commercially available HIPAA compliant web-based video conferencing screen sharing platform (Starleaf; Watford, UK), an interactive control sharing system integrated into the institutional PACS allowing simultaneous multi-user mouse control over images (Collaborate; Change Healthcare, Nashville TN), and telephone readouts. Both the screen sharing and control sharing methods permitted the other user(s) to see the presenter's PACS images. However, screen sharing allowed only the presenter to control the mouse and manipulate PACS images, while control sharing allowed both the presenter and the trainee to control the mouse to scroll through images and point to areas of interest, theoretically allowing for a more interactive learning experience. The telephone method only allowed for a verbal description of imaging findings, presumably preferred for a quick discussion of simple points decreasing the added time needed to load screen and control share applications.

The purpose of this study was to assess both radiology trainee and attending overall satisfaction with and subjective perceived effectiveness of three different methods of virtual readout implemented since April 2020 at a single large residency program and to determine the best virtual readout platform for optimal trainee education during the COVID-19 pandemic, as well as the opportunity to continue using these methods in the post pandemic era. Additionally, while there is already much agreement that in-person readouts are generally preferred, there seems to be disagreement institutionally regarding the utility of virtual readout platforms, therefore, this study was also meant to elucidate a general consensus in order to develop best practices for readouts going forward.

Materials and methods

This study was submitted to the Medical Board of Ethics at our institution and was approved for institutional review board exemption.

Virtual readout platforms

Our diagnostic radiology residency program currently trains 46 radiology residents (34 diagnostic radiology residents, 9 integrated interventional radiology residents, 3 interventional radiology residents) and 18 fellows, and is affiliated with a large tertiary-care academic institution. In response to the COVID-19 social distancing mandates, our department implemented three different virtual readout methods starting in April 2020. These included: a HIPAA compliant web-based video conferencing screen sharing platform (Starleaf, Watford, UK), a screen and control sharing tool integrated into the PACS (Collaborate, Change Healthcare, Nashville, TN); and telephone readout sessions. The questionnaire surveys covered the time period between April 2020 and April 2021.

Starting in July 2020, some sections in our department returned to a hybrid mixture of in-person and virtual readouts, including the Body and Musculoskeletal sections, with individual sections slowly increasing their percentages of in-person readout to April 2021. This was done on a section-by-section basis as well by individual attending preference. Other sections remained completely virtual, including our Neuroradiology and Emergency department sections. Therefore, the new first year residents starting in July 2020 experienced both virtual and in-person readouts. Thus, all residents experienced both in-person and virtual readout sessions at some point during the survey period and could make direct comparisons between them.

Screen sharing (Starleaf)

Starleaf is a commercially available cloud-based HIPAA compliant videoconferencing system which allows for a single participant to share his/her screen (Fig. 1 ). The platform also allows for multiple users to participate in a single session. Multiple participants can be seen using separate webcams if desired and can converse through microphones and speakers. The participant sharing his/her screen is the only one able to control the mouse to scroll through images and point out pertinent findings. Since the creation and sharing of ad-hoc meeting invitations can be cumbersome, an institutional “virtual reading room” web application was created (Fig. 1A). This application, accessible via a web browser through a link built into the PACS workstation, provides a list of links that can launch “virtual reading rooms” named after the physical reading rooms, with larger reading rooms connected via multiple virtual links. The links associated with the meetings are dynamically regenerated every 24 h using the Starleaf application programming interface (API). This approach provided an additional layer of protection against the meeting links being inadvertently discovered by an unauthorized person.

Fig. 1

Screen sharing (Starleaf). A. Multiple Starleaf sessions accessible through the institutional intranet. The virtual reading rooms were divided by service, with a up to 4 different rooms per service. B. After a participant selects a virtual reading room, a screen loads for the participants to enter his/her name as well as select the option to connect to webcam and microphone. C. Main screen of the virtual reading room. The menu bar at the top has several features including: list of the number of participants in the session, option to launch a text chat box, ability to share screen, webcam and microphone settings, and ability to record a session.

Control sharing (Collaborate)

Collaborate is an integrated software tool accessible through PACS which allows for both participants to work on a shared window simultaneously (Fig. 2 ). Each participant has the ability to take control and use the mouse and PACS functions within the same shared PACS window, allowing for an interactive session. Each participant can scroll, zoom, pan, window/level and annotate the images using the same tools in the PACS viewer, and these manipulations are viewable in real time by the other participant. While the shared mouse is being used by one participant, the other person can see the mouse but cannot control it for his/her monitor or other linked computer monitors. This platform does not have a built-in microphone or webcam, therefore discussion needs to be done using either the telephone or another system with a microphone. To initiate the control sharing feature on PACS, the participant first launches the study they wish to collaborate on and clicks the “Collaborate” link on the PACS menu bar (Fig. 2A). From there, a menu pops up listing everyone currently logged into a PACS workstation (Fig. 2B). The participant must select the additional collaborator from the menu to send an invitation. The invited collaborator will then see the invitation pop up in the corner of his/her PACS screen (Fig. 2C). After clicking accept, a new integrated PACS window is launched that includes both participants (Fig. 2D). After the current study is finished, the Collaborate platform is exited. The initial version of this platform required a new invitation to be sent for every new study to be reviewed. After providing feedback to the vendor, a software patch was released to enable multiple studies to be reviewed in a single Collaborate session.

Fig. 2

Control sharing (Collaborate). A. Menu bar on every PACS workstation has a link to the Collaborate platform. B. Once the participant clicks the link, a menu launches listing every radiology trainee and attending in the system. The participant chooses a single collaborator to send an invitation. C. The invitation populates on the collaborator's PACS station to be accepted. D. Once the invitation is accepted, a shared screen launches with the case on the original participant's screen. The shared screen contains all the PACS features and allows both participants to manipulate the images.

Telephone

Trainees and faculty discussed cases over the telephone while viewing images separately on their own workstations.

Data collection and analysis

A 10-question survey was distributed to all current radiology residents, fellows, and attendings between April to June 2021, after the virtual readout platforms were employed for one year. The survey was first distributed electronically via email using SurveyMonkey (San Mateo, CA) in April 2021, followed by two additional follow-up emails in May and June 2021.

Survey questions (Appendix A) focused on satisfaction with and perceived effectiveness of the three different virtual readout strategies in comparison to in-person readouts, as well as the desire to continue using these methods in the future, including in the post-pandemic era. Only subjective perceived effectiveness of the platforms relative to in-person readouts was evaluated given the difficulty in controlling for confounding variables influencing quantitative measures of effectiveness, such as standardized or in-house exams. For instance, the senior residents had years of prior training based on traditional in-person readouts while more junior residents did not. Additionally, there is a lack of statistical power in the number of junior resident scores to compare with previous years to assess scores before and after the use of virtual readouts. Thus, no metric on actual teaching effectiveness can be assessed at this time. Technical difficulties with each of the virtual readout methods and suggestions for improvement were also assessed. Specifically relating to the control sharing platform, we assessed whether the ability for both users to manipulate the PACS images was an important feature for teaching and learning. Question types included answers with 3 to 5-point Likert scales, tick boxes, and open-ended questions.

Data were exported from Survey Monkey and analyzed via Microsoft Excel and Matlab (Mathworks, Natick, MA). Answers made on a 5-point scale were converted to numbers from 1 to 5 for most analyses, and for computing mean and/or median grades. Some questions analyzed on a 5-point scale included an additional question about whether or not the respondent had used the specific platform, which was not ultimately used for generation of mean and/or median grades. Answers made on a 3-point scale were converted to numbers from 1 to 3 for most analysis, and for computing mean and/or median grades. The quantitative data were analyzed using proportions, Kendall rank correlation, Kruskal-Wallis and Friedman tests. Pairwise comparisons were performed using Tukey post-hoc procedure after the overall significant difference was established. p-Values ≤ 0.05 were considered statistically significant. Data from the open-ended questions were reviewed and analyzed by three study authors. The main themes discussed in the comments are presented in this study.

To assess possible effects of virtual readouts on resident productivity, additional data was gathered via CPT codes regarding the percentage and number of reports generated by residents between April 1, 2020-July 1, 2020 (a period during the COVID-19 pandemic when all nonessential medical procedures were placed on hold) and between April 1, 2020-March 31, 2021, to assess overall studies performed during the time period of our study. This was compared to the percentage and number of reports generated by residents before the COVID-19 pandemic time periods of April 1, 2019-July 1, 2019, and April 1, 2019-March 31, 2020. Absolute numbers were not compared given the ~30% decline in radiology volume during the months of April 1, 2020-July 1, 2020, due to the hold on all nonessential radiology studies.

Results

A total of 32 out of 64 trainees (50%) completed the survey, which included 26 out of 46 residents (57%) and 6 out of 18 fellows (33%). The distribution of responses by level of training was: 8 (13%) first year residents, 7 (11%) second year residents, 6 (9%) third year residents, and 5 (8%) fourth year residents. A total of 32 out of 76 radiology attendings (42%) completed the survey. Two respondents did not specify level of training. Thirty-three (50%) women and twenty-eight (42%) men responded, while five (8%) preferred not to disclose their gender. A summary of the survey results is listed in Table 1 .

Table 1

Virtual readout survey results.

Question	n	%
Level of training
R1	8	13%
R2	7	11%
R3	6	9%
R4	5	8%
Fellow	6	9%
Faculty	32	50%
Total	64
Gender
Female	33	50%
Male	28	42%
Prefer not to disclose	5	8%
Total	66
Level of satisfaction with virtual readout strategies
Screen sharing (Starleaf)
Did not use	5	8%
Extremely satisfied	15	23%
Very satisfied	12	18%
Moderately satisfied	19	29%
Slightly satisfied	13	20%
Not at all satisfied	2	3%
Total	66
Control sharing (Collaborate)
Did not use	7	11%
Extremely satisfied	7	11%
Very satisfied	9	14%
Moderately satisfied	14	22%
Slightly satisfied	22	34%
Not at all satisfied	6	9%
Total	66
Telephone
Did not use	6	9%
Extremely satisfied	5	8%
Very satisfied	14	21%
Moderately satisfied	18	27%
Slightly satisfied	19	29%
Not at all satisfied	4	6%
Total	66
Learning as effective with virtual readout as in-person readouts
Screen sharing (Starleaf)
Did not use	4	6%
Strongly agree	11	17%
Agree	20	30%
Neutral	6	9%
Disagree	17	26%
Strongly disagree	8	12%
Total	66
Control sharing (Collaborate)
Did not use	6	9%
Strongly agree	9	14%
Agree	16	25%
Neutral	8	12%
Disagree	15	23%
Strongly disagree	11	17%
Total	66
Telephone
Did not use	4	6%
Strongly agree	4	6%
Agree	11	17%
Neutral	12	19%
Disagree	24	37%
Strongly disagree	10	15%
Total	66
Technical difficulties (Ex. System crashes, trouble loading platform, slowing/freezing of software, etc.)
Screen sharing (Starleaf)
Did not use	5	8%
Strongly agree	4	6%
Agree	7	11%
Neutral	13	20%
Disagree	22	33%
Strongly disagree	15	23%
Total	66
Control sharing (Collaborate)
Did not use	9	14%
Strongly agree	7	11%
Agree	18	27%
Neutral	9	14%
Disagree	16	24%
Strongly disagree	7	11%
Total	66
Telephone
Did not use	5	8%
Strongly agree	1	2%
Agree	5	8%
Neutral	9	14%
Disagree	27	42%
Strongly disagree	18	28%
Total	66
Do you agree that the ability for both users to manipulate the PACS images on Collaborate was important for learning/teaching?
Strongly agree	24	45%
Agree	14	26%
Neutral	11	21%
Disagree	2	4%
Strongly disagree	2	4%
Total	53
In the future, would you agree to support the use of some form of virtual read-out combined with in-person readouts?
Yes	41	77%
No	4	8%
Neutral	8	15%
Total	53
Support of these virtual programs in the future combined with in person readouts
Screen sharing (Starleaf)
Did not use	3	6%
Strongly support	25	48%
Support	17	33%
Neutral	3	6%
Do not support	3	6%
Strongly do not support	1	2%
Total	52
Control sharing (Collaborate)
Did not use	4	8%
Strongly support	13	25%
Support	16	31%
Neutral	9	17%
Do not support	8	15%
Strongly do not support	2	4%
Total	52
Telephone
Did not use	1	2%
Strongly support	11	21%
Support	20	38%
Neutral	8	15%
Do not support	9	17%
Strongly do not support	3	6%
Total	52

Trainees and attendings combined were significantly more satisfied with the screen sharing platform compared to the control sharing platform (3.41 vs. 2.81; p = 0.001) and compared to the telephone (3.41 vs. 2.95; p = 0.0003). There was no significant difference in satisfaction between the control sharing and the telephone methods (2.81 vs. 2.95; p = 0.93).

Overall, trainees and attendings experienced significantly more technical difficulties with control sharing than with screen sharing (3.04 vs. 2.39; p = 0.0005) and the telephone (3.04 vs. 2.07; p < 0.0001). There was no significant difference in technical issues experienced between screen sharing and the telephone (2.39 vs. 2.07; p = 0.2).

With regards to these virtual methods in comparison to in-person readouts, screen sharing and control sharing were both perceived to be significantly more effective than the telephone (3.14 vs. 2.59; p = 0.0001 and 2.95 vs. 2.59; p = 0.007 respectively). There was no significant difference in perceived effectiveness between screen sharing and control sharing in comparison to in-person readouts (3.14 vs. 2.95; p = 0.5). Means for satisfaction, perceived effectiveness, and experiences of technical difficulties are summarized in Table 2 .

Table 2

Mean responses by all participants.

Variable	Mean	Variable comparisons	p-Value
Satisfactiona
Screen sharing (Starleaf)	3.41	Starleaf vs. Collaborate	0.001
Control sharing (Collaborate)	2.81	Starleaf vs. Telephone	0.0003
Telephone	2.95	Collaborate vs. Telephone	0.9
Perceived effectivenessb
Screen sharing (Starleaf)	3.14	Starleaf vs. Collaborate	0.5
Control sharing (Collaborate)	2.95	Starleaf vs. Telephone	0.0001
Telephone	2.59	Collaborate vs. Telephone	0.007
Experienced technical difficultiesb
Screen sharing (Starleaf)	2.39	Starleaf vs. Collaborate	0.0005
Control sharing (Collaborate)	3.04	Starleaf vs. Telephone	0.2
Telephone	2.07	Collaborate vs. Telephone	<0.0001
Support of virtual platform in futurec
Screen sharing (Starleaf)	4.26	Starleaf vs. Collaborate	0.07
Control sharing (Collaborate)	3.60	Starleaf vs. Telephone	0.0009
Telephone	2.54	Collaborate vs. Telephone	0.3

Bolded text indicates statistical significance

Means calculated from the following 5-point scale: 5 = Extremely Satisfied, 4 = Very satisfied, 3 = Moderately satisfied, 2 = Slightly satisfied, 1 = Not satisfied at all.

Means calculated from the following 5-point scale: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree.

Means calculated from the following 5-point scale: 5 = Strong support, 4 = Support, 3 = Neutral, 2 = Do not support, 1 = Strongly do not support.

There was somewhat greater satisfaction and significantly more positive perception of effectiveness with the telephone by attendings compared to trainees (3.18 vs. 2.68; p = 0.07 and 2.86 vs. 2.26; p = 0.05 respectively). Otherwise, there were no other significant differences between trainee and attending experiences with the three virtual readout platforms (Table 3 ).

Table 3

Comparison of responses by trainees and attendings.

Variable	Trainee (mean)	Attending (mean)	p-Value
Satisfactiona
Screen sharing (Starleaf)	3.34	3.43	0.83
Control sharing (Collaborate)	2.91	2.60	0.32
Telephone	2.68	3.18	0.07
Perceived effectivenessb
Screen sharing (Starleaf	3.09	3.14	0.83
Control sharing (Collaborate)	2.81	3.04	0.54
Telephone	2.26	2.86	0.05
Experienced technical difficultiesb
Screen sharing (Starleaf	2.38	2.46	0.72
Control sharing (Collaborate)	3.06	3.08	0.99
Telephone	2.13	2.04	0.71
Ability for both users manipulate mouse on Collaborate aids in learning/teachingc	4.07	3.95	0.95
Support the use of some form of virtual read-out combined with in-person readoutsc	2.55	2.86	0.13
Support of virtual platform in future combined with in-person readoutc
Screen sharing (Starleaf	4.17	4.41	0.46
Control sharing (Collaborate)	3.55	3.69	0.54
Telephone	3.03	4.32	0.0004

Bolded text indicates statistical significance

Means calculated from the following 5-point scale: 5 = Extremely Satisfied, 4 = Very satisfied, 3 = Moderately satisfied, 2 = Slightly satisfied, 1 = Not satisfied at all.

Means calculated from the following 5-point scale: 5 = Strongly agree, 4 = Agree, 3 = Neutral, 2 = Disagree, 1 = Strongly disagree.

Means calculated from the following 3-point scale: 3 = Agree, 2 = Neutral, 1 = Disagree.

When the trainee and attending groups were analyzed separately, for trainees there was only a significant difference in satisfaction between screen sharing and the telephone (3.35 vs. 2.68; p = 0.0006) but not between control sharing and screen sharing (p = 0.1) or the telephone (p = 0.2). For attendings, there was a significant difference in satisfaction between screen sharing and control sharing (3.57 vs. 2.65; p = 0.005) and no significant difference between screen sharing or control sharing compared to the telephone (p = 0.3 both), which differs from trainees. In terms of perceived effectiveness compared to in-person readouts, the trainee group showed that both screen sharing and control sharing were perceived as more effective than the telephone (3.06 vs. 2.26; p = 0.0004 and 2.81 vs. 2.26; p = 0.03 respectively). In the attending group, there was no significant difference in perceived effectiveness between the virtual platforms (p = 0.12). Both trainees and attendings had significantly more technical difficulties with control sharing compared to the other virtual platforms (p = 0.0002 and p = 0.0007 respectively), with no significant difference between screen sharing and the telephone (p = 0.64 and p = 0.26 respectively).

There was a significant negative correlation between the level of technical difficulties and satisfaction for screen sharing (r = −0.50, p < 0.0001) and control sharing (r = −0.38, p = 0.0006), but not for the telephone (p = 0.13). Additionally, there was a significant negative correlation between the level of technical difficulties and perceived effectiveness of screen sharing (r = −0.28, p = 0.009) and control sharing (r = −0.25, p = 0.02) relative to in-person readouts, but not for the telephone (p = 0.8).

Regarding the control sharing platform, most agreed to strongly agreed that the ability for both users to manipulate the PACS images was important for learning and teaching (p < 0.0001). There was no significant difference in responses between trainees and attendings (p = 0.95) (Table 3).

Of the 33 comments regarding screen sharing, 11 (33%) mentioned that the resolution quality was not ideal. Six out of 31 (19%) comments about control sharing expressed dissatisfaction that only one participant could use the mouse at a time and 11 comments (35%) stated that control sharing was often slow, crashed, or did not work. The lack of visual learning with the telephone was specified in 21 of the 34 comments (62%). Representative positive and negative comments are detailed in Table 4 .

Table 4

Representative comments by trainees and attendings.

	Positive	Negative
Screen sharing (Starleaf)	Able to review multiple studies at once without needing to start new session for each study.When used continuously with a webcam, there was continuous oversight of workflow and real time discussion of cases, which re-created the environment of in-person readouts.Works well and allows nearly immediate access to trainee or attending.	Poorer resolution, harder to see smaller findings/suboptimal image quality.Spectator readout.Only one participant able to move the mouse.
Control sharing (Collaborate)	Easier to see images and navigate since it is on PACS.Both participants can use the mouse to navigate.	Cannot concurrently use the shared mouse.Had trouble getting Collaborate to work on many occasions.Slow, often did not work.Can only open one study at a time.
Telephone	Helpful for quick conversation.Great for efficiency for simple cases.Used only for select cases that don't need much interactive review.	Cannot share screen, which is okay for easier things, but makes showing findings more difficult.Telephone readouts only really work with more senior residents, who have basics down. Many more junior residents need specific findings pointed out directly which is tricky on the phone.Terrible for cases that are complex and of high learning value.

Of the 35 comments provided by respondents regarding suggestions for improvement, a large portion of respondents (40%) expressed a strong preference for in-person readouts, however there was not an overwhelmingly negative attitude towards the virtual readout platforms overall. Five respondents (14%) voiced positive feedback regarding the screen sharing system. Eight respondents (23%) provided feedback to improve the platforms and did not convey a strong opinion against virtual readouts. Two respondents (6%) expressed strongly negative feedback regarding the control sharing function and only one respondent expressed strongly negative feedback regarding the telephone readouts. There was no strongly negative feedback written about the screen sharing virtual platform. Positive responses regarding screen sharing were that multiple participants could log into the same virtual room and participate in the readout. Several respondents liked the image resolution on the control sharing platform and the convenience of the telephone for quick questions or straightforward cases. Several comments mentioned the importance of having a webcam in order to see each other and receive non-verbal cues similar to in-person readouts, which was an optional feature built into the screen sharing application.

The majority of respondents supported the use of some form of virtual readout method in conjunction with in-person readouts in the future. The number of responses in support of incorporation of virtual platforms in the future was significantly higher than the number of responses against it (p < 0.0001), with no significant difference between trainees and attendings (p = 0.13). Out of the three virtual platforms, respondents significantly supported the use of screen sharing over the telephone (4.26 vs. 3.54; p = 0.0009) and showed some support for screen sharing over control sharing in conjunction with in-person readouts in the future (4.26 vs. 3.60; p = 0.07). There was no significant difference in support between control sharing and the telephone (p = 0.34). There was no significant difference in support between trainees and attendings regarding screen sharing (p = 0.46) and control sharing (p = 0.54), however attendings more strongly supported the use of the telephone in future readouts compared to trainees (4.32 vs. 3.03; p = 0.0004) (Table 3).

Additional data regarding the percentage and number of reports generated by residents during and before the COVID-19 pandemic was obtained to assess possible effects of virtual readouts on resident productivity. From April 1, 2020-July 1, 2020, during the pandemic when all non-essential medical procedures were on hold, 53,587 exams were read in total with 27,039 (50.5%) exams having a resident read. During the entire period from April 1, 2020- March 31, 2021 (the time period assessed in this study), 309,603 exams were read in total with 151,285 (48.9%) exams having a resident read. From April 1, 2019- July 1,2019 before the pandemic, 94,360 exams were read in total with 42,593 (45.1%) exams having a resident read. Between April 1, 2019-March 31,2020, 357,140 exams were read in total with 168,836 (47.3%) having a resident read.

Discussion

The COVID-19 pandemic has changed the way in which radiology residency programs structure their learning environments, particularly in terms of the traditional in-person readouts, which now include numerous virtual platforms. Our study provides both trainee and attending feedback regarding three different virtual readout methods implemented at our institution, including a commercially available HIPAA compliant web-based videoconferencing screen sharing platform (Starleaf), a control sharing integrated PACS platform (Collaborate), both of which most closely resemble in-person readouts by allowing for combined visual and verbal direction with control sharing further allowing for shared interaction with the images, and telephone readout sessions.

Trainees and attendings were significantly more satisfied with the screen sharing platform. This was likely in part related to the fact that screen sharing had significantly fewer technical issues than control sharing and also allowed users to concurrently review images unlike with the telephone. The key features to a traditional readout described by Li et al. include direct two-way communication between the trainee and attending, ability to view images simultaneously, potential for all parties to scroll through images, and an ability to annotate the images, and the ideal virtual system would incorporate these key features. Screen sharing includes most of these features in some form, while the telephone does not since it does not allow for simultaneous review of images and makes it much more difficult to point out findings. Control sharing also allows for screen sharing, however lacked a build in microphone and webcam. While control sharing permits both participants to view and scroll on the same PACS screen, its technological issues limit complete assessment of the benefit of this increased interactivity.

Although there was no statistical difference in perceived effectiveness of screen sharing versus control sharing, both visually based systems were considered significantly more effective compared to the telephone. This likely reflects a preference for traditional in-person readouts, most closely resembled by the screen sharing and control sharing platforms. It is important to note that only subjective perceived effectiveness of the platforms relative to in-person readouts was able to be evaluated and no metric on actual teaching effectiveness could be assessed at this time.

As alluded to above, technical difficulty with a virtual platform negatively impacts its ability to be successfully implemented for readouts. In our quantitative data, there was a significant negative correlation between technical difficulties and satisfaction as well as with perceived effectiveness of both screen sharing and control sharing. In our qualitative data, this was reflected in comments regarding the low image resolution in the screen sharing program and the inability for both participants to simultaneously use the mouse in the control sharing tool. Additionally, control sharing was often slow or did not work appropriately. Respondents also did not like that only one study could be opened at a time on control sharing. Although multi-study control sharing sessions were added via a software patch, most remote readouts during the first COVID surge occurred before this fix was available. Respondents liked the ability for multiple participants to log into a virtual reading room at the same time on the screen sharing platform to review cases together. Several respondents mentioned the need to always have a webcam in order receive non-verbal cues during virtual readouts and better gauge understanding, similar to the finding detailed by Matalon et al.. Although most participants felt that the ability for both users to manipulate the PACS images in the control sharing tool was important for teaching and learning, there was still an overall dissatisfaction with this platform, presumably due to the higher degree of technical issues.

Trainees and attendings generally agreed regarding satisfaction with and perceived effectiveness of the different virtual readout platforms, though there was a trend towards greater satisfaction with the telephone by attendings compared to trainees. This result is likely related to the faster speed and greater ease of use with the telephone compared to the other methods. Feedback regarding the telephone however was predominately negative in terms of education since findings could not be visually pointed out like with in-person readouts. In a time where case volume continues to rise, speed and efficiency are important factors to consider, while also not compromising education. The greater degree of technical issues encountered on screen and control sharing tools relative to the phone likely disrupts workflow. Therefore, the telephone could provide benefit and increase efficiency in certain circumstances, such as when confronted with very high exam volumes, for addressing straightforward questions, or when reading out with more senior trainees who may not require as much direct supervision and discussion.3., 10. Though based on our results, the telephone should not be the primary method for virtual readout.

In general, respondents still prefer in-person readouts over virtual readouts. While virtual methods should not replace in-person interactions, there is a strong support for incorporation of virtual platforms in the future. Respondents most strongly supported the continued use of the screening sharing application in the future, in keeping with the overall satisfaction and perceived effectiveness with the platform. Attendings significantly supported the use of the telephone in future readouts, likely due to improved efficiency compared to trainees, who valued visual learning more.

Virtual readouts did not impact resident productivity. A quantitative comparison of efficiency between virtual and in-person readouts is difficult to assess on a numerical level because in the first 4–5 months of implementing virtual readout (and when nearly all readouts were virtual), there was a considerable drop in radiology volume due to suspension of nonessential hospital visits. However, the percentages of cases relative to overall volume was essentially unchanged to slightly higher during those months compared to the prior year before the pandemic, suggesting that there was no decreased efficiency in readout due to virtual methods. Once the suspension of nonessential hospital visits was lifted, some sections resumed readouts in a hybrid manner while others continued with virtual readouts only. During the time period which included these months, there was also no difference in percentages of resident reports relative to total number of studies, further suggesting that there is no significant effect of virtual readouts on resident efficiency.

Limitations of this study include a small sample size at a single institution, which may limit generalizability of the findings across all institutions, however, given the presumed heterogeneity in implementation and duration of virtual readouts across institutions, we feel that our single institution paper allowed for specific and granular comparisons across distinct virtual platforms that would be impossible over multiple training programs. In addition, the smaller sample size did not allow for more detailed analysis of satisfaction of the virtual readout methods among differing resident training levels or years of experience among attendings. Future multi-institutional research would be needed to assess for differences between training levels. Furthermore, radiology subspeciality differences in virtual readout method preferences were not assessed in this study. It is possible that some sections would prefer or view different platforms more favorably based on specific needs. Future studies are needed to elucidate these differences. Selection bias may impact our results as potentially trainees and attendings who were more or less enthusiastic about virtual readouts may have been more likely to participate. Additionally, certain survey questions received fewer responses than others, which may also contribute to selection bias. Our study chose to evaluate subjective trainee and attending feedback metrics, though incorporating more objective metrics could help strengthen our results. For example, only perceived effectiveness of the virtual platforms relative to in-person readouts was able to be assessed and no metric on actual teaching effectiveness could be assessed. Given the relatively short time frame of virtual only readouts and lack of power in the number of junior residents where the potentially negative impacts of virtual readouts might be felt most strongly, no statistical analysis could be made on quantitative measures of learning. Nevertheless, attendings, whose evaluation of residents make up an important component of trainee core competency progress assessments, felt that virtual readouts were comparable to in-person readouts. Future multi-institutional studies will be required to assess if teaching was comparably effective between virtual versus in-person readouts in a quantitative manner. We were also unable to track the number of cases read using each virtual platform. Furthermore, given the technical issues faced with the control sharing platform, analysis of its true utility, specifically relating to increased interactivity between the attending and trainee during virtual readouts, could not be adequately assessed. A follow-up study with technical improvements to the control sharing tool may be needed to fully describe the impact of the added functionality on satisfaction and perceived effectiveness on teaching versus other virtual readout platforms. The return to a hybrid readout model starting in July 2020 introduces some confounding variables due to section inhomogeneity in virtual and in-person readouts. However, since the purpose of this study was to assess overall satisfaction and a relative sense of effectiveness of the virtual platforms, the hybrid system is not believed to impede the overall goals of the study because this allowed for a direct comparison between virtual and in-person readouts in the sample population at the same time. Thus, the results showing similar perceived teaching effectiveness of screen and control sharing platforms versus in-person readouts as well as overall satisfaction with virtual readouts is not due to recency effect but is mostly a contemporaneous comparison. Finally, the virtual readout platforms studied in this paper are representative of our institution and may not necessarily match exactly with the platforms at other institutions somewhat limiting generalizability of the study results.

The results of our study can provide guidance for radiology residency readouts in the post-pandemic era. It is evident that the most important features of any virtual readout system are stability and functionality. We found that our screen sharing application served as a stable platform and encompassed most of the key elements of a traditional in-person readout. While solely virtual readouts are not ideal, they can be useful in certain circumstances, such as when the resident and attending are in separate facilities. These new technological advances that came with the COVID-19 pandemic can be integrated into radiology residencies and help improve overall education and work efficiency. A hybrid virtual/in-person model for readouts may be the future of radiology education.

Conclusion

Virtual platforms that most closely resemble in-person readouts are overall preferred by both trainees and attendings. These platforms include a combination of audio and visual cues.

Between similar platforms, technical stability determines overall satisfaction and perceived effectiveness of a virtual readout system. Both trainees and attendings support incorporation of virtual readout methods in combination with traditional in-person readouts in the post-COVID-19 era. Our study alludes to how these new virtual platforms may be effectively incorporated into radiology residency education in the future.