Rethinking Clinical Trial Radiology Workflows and Student Training: Integrated Virtual Student Shadowing Experience, Education, and Evaluation.

There is consistent demand for clinical exposure from students interested in radiology; however, the COVID-19 pandemic resulted in fewer available options and limited student access to radiology departments. Additionally, there is increased demand for radiologists to manage more complex quantification in reports on patients enrolled in clinical trials. We present an online educational curriculum that addresses both of these gaps by virtually immersing students (radiology preprocessors, or RPs) into radiologists' workflows where they identify and measure target lesions in advance of radiologists, streamlining report quantification. RPs switched to remote work at the beginning of the COVID-19 pandemic in our National Institutes of Health (NIH). We accommodated them by transitioning our curriculum on cross-sectional anatomy and advanced PACS tools to a publicly available online curriculum. We describe collaborations between multiple academic research centers and industry through contributions of academic content to this curriculum. Further, we describe how we objectively assess educational effectiveness with cross-sectional anatomical quizzes and decreasing RP miss rates as they gain experience. Our RP curriculum generated significant interest evidenced by a dozen academic and research institutes providing online presentations including radiology modality basics and quantification in clinical trials. We report a decrease in RP miss rate percentage, including one virtual RP over a period of 1 year. Results reflect training effectiveness through decreased discrepancies with radiologist reports and improved tumor identification over time. We present our RP curriculum and multicenter experience as a pilot experience in a clinical trial research setting. Students are able to obtain useful clinical radiology experience in a virtual learning environment by immersing themselves into a clinical radiologist's workflow. At the same time, they help radiologists improve patient care with more valuable quantitative reports, previously shown to improve radiologist efficiency. Students identify and measure lesions in clinical trials before radiologists, and then review their reports for self-evaluation based on included measurements from the radiologists. We consider our virtual approach as a supplement to student education while providing a model for how artificial intelligence will improve patient care with more consistent quantification while improving radiologist efficiency.

Introduction

There was a significant decrease in available clinical experience options for students during the COVID-19 pandemic [1]; however, the demand for clinical shadowing opportunities remained. With this in mind, we developed and launched an online curriculum in November 2020 aimed to teach interested students the skills needed to be a radiology preprocessor (RP).

Via this online curriculum, RPs are taught to perform highly specific tasks to guide existing and evolving automated lesion identification and measurement tools in advance of radiologist interpretation on patients enrolled in clinical trials. RPs may include radiologic technologists, undergraduate and medical students, residents, research scientists, and fellows of medical specialties outside of radiology (e.g., oncology, critical care, or infectious disease). In addition to developing radiology knowledge and abnormality recognition skills on imaging, RPs also become adept at leveraging automated tools in picture archiving and communication system (PACS) to help prepare tumor quantification data while being actively integrated in radiologist workflows in a completely virtual manner.

Knowledge gained by RPs is initially evaluated through cross-sectional anatomical quizzes and continual self-grading by reviewing radiology reports and calculating miss rates. We have been engaging RPs (similar to how some institutions have integrated medical students on their radiology rotations) in a “look ahead” workflow described by Huang [2] by having RPs open computed tomography (CT) exams in advance of the radiologists and annotate findings such as metastatic target lesions and/or incidental/critical findings.

Our RP program was previously an integrated in-person radiologist department where RPs were introduced into the current workflow. This integrated RP-radiologist workflow approach, otherwise referred to as the RP workflow, has shown to improve report value, patient care in clinical trials, expedited worklist triage with earlier notification of incidental critical findings, and radiologist productivity [3]. The workflow consists of RPs opening exams in advance of radiologists, identifying target and/or other previously measured index lesions and measuring them. The RP then opens the radiologist report at the end of their shift to review what lesions the radiologist included. Lesions and pathology not included are reviewed and recorded as missed findings. Figure 1 depicts the RP workflow in further detail.

Fig. 1

Illustrates the integrated and cyclic nature in which RPs open CT exams as soon as they become available in PACS, reviewing and annotating images in advance of radiologists. This includes identifying and measuring target lesions (in the case of clinical trials) and/or index lesions (selected based on radiologist preference, since most are not aware of verified target lesions). The RP then closes the exam, which indicates the exam has been reviewed/annotated. The radiologist then interprets the exam as usual. Depending on radiologist preference, they can “prompt” the RP measurements to show up in PACS and insert the measurement metadata (measurement, series, image number). After image interpretation and report dictation, the RP determines the number of missed findings based on annotations the radiologist accepted and makes notes about the pathology in their data sheet to track learning progress. We have begun research applying this cyclic nature of improvement for machine learning workflows that will continually improve over time

Many radiologists outside of academic centers generate reports in a traditional isolated environment without resident-staff interaction or medical student teaching. The NIH clinical center has this type of isolated workflow. Due to the interactive nature of the RP workflow, radiologists’ workflows become more collaborative and team-oriented while simultaneously providing learning opportunities and unique clinical experience for RPs. This is done through collaborative phone calls, text notifications when scans are completed, and a data sheet where radiologists can leave notes on missed findings for further RP learning and self-evaluation.

Many patients at the NIH clinical center are enrolled in cancer and other clinical trials, receiving serial cross sectional (mostly CT) imaging to evaluate tumor response to therapy, highlighting the importance of consistent tumor measurements over time to provide oncologists with essential information to treat their patients.

We published our first year of experience with this RP workflow that demonstrated improved patient care while enhancing radiologist productivity and efficiency in clinical trials [4]. Others have reported similarly improved performance with radiologist extenders/assistants [5, 6]., Following approximately 20 hours of training and review of 40 chest, abdomen, and pelvis (CAP) CT exams at our clinical center, RPs achieve a level of measurement precision that meets radiologists’ satisfaction, allowing radiologists time to focus on more complex tasks based on higher level training (i.e., image interpretation, report dictation per training, and standard of practice) in increasingly burdened workflows [7]. Since most PACS automated lesion identification and measurement tools are not yet 100% precise, RPs can help improve the success of these tools by refining measurements and relating same lesions over time. This enables eventual automated lesion recognition, clinical trial response calculations, and graphing capabilities.

Given the successes of the previous in person RP workflow and challenges of limiting students in the medical center setting during the pandemic, we developed an online RP curriculum which includes presentations on basic principles in diagnostic radiology, as well as anatomy and pathology in various imaging modalities. To assess RP baseline knowledge and learning progression, the curriculum also has online quizzes with instructional videos that explain cross-sectional anatomy and clinical trial quantification basics. The curriculum emphasizes gaining fluency with various PACS platforms, lesion management, annotation tools from industry, and other open-source applications, which are all possible in remote settings, negating the need for physical presence in the radiology department or medical center. The course quickly matured with numerous participating centers volunteering to help across the USA and Europe due to enthusiastic support of online training—a new norm during the COVID-19 pandemic.

The goals of our publicly available online RP curriculum are to educate interested students while simultaneously improving patient care. Our curriculum educates interested students by teaching them the requisite knowledge and skills to identify imaging abnormalities and leverage evolving automated PACS tools for tumor quantification data preparation. To assess the educational objectives of our curriculum, we evaluate decreasing RP miss rates of CT cases reviewed by RPs over time.

Materials and Methods

Several months into the 2020 pandemic, we made our RP course publicly available online [8]. Once the course was public, we contacted several academic and research institutions and industry who immediately offered dozens of presentations which are available on our curriculum website. The authors believe that the immediate interest was partially due to cancer centers and academic institutions realizing there was a “new norm” when it comes to student education. The possibility of a completely virtual education was especially appealing in that most students were prohibited to be on site at most medical facilities. We compare remote workflow with educational objectives, such as RP miss rates, that are continuously evaluated in the workflow.

RP Radiologist Immersive Workflow

The clinical center implementing the aforementioned RP workflow experience has been using a hybrid/native PACS/Radiology Information System (RIS) solution (Vue PACS 12.1, Philips, Netherlands) that includes automated lesion identification and measurement tools [9]. The RP workflow is integrated in concert with the radiologist using collaborative communication through a PACS custom worklist (a unique protected health information (PHI)-compliant exam identifier created by permitted PACS users) and other tools that reflect exam review and inputs. RPs identify and measure lesions, some of which are previously measured (especially oncologist verified target lesions for cancer clinical trial patients) before the radiologist opens the exam.

The hybrid nature of PACS enables a worklist notification scheme that allows RPs to identify exams requiring quantification while also notifying radiologists of exams that have been processed. The immediate inputs from the radiologist accepting or rejecting annotations serve as real-time feedback to the RP as part of the educational experience. When radiologists see additional teaching opportunities such as missed critical findings, they note those for RP review.

When radiologists open preprocessed exams in the clinical trial workflow, they import accepted measurements and other inputs into their reports in the form of hyperlinks in interactive multimedia reports [10], minimizing the need to identify or measure target lesions while also saving interpretation time. See Fig. 2 for an example screenshot of the PACS bookmark table where stored measurements are also used as communication between RPs, radiologists, and clinical teams.

Fig. 2

Shows an example bookmark table within PACS (VuePACS, v 12.2 Philips Medical, the Netherlands) illustrating primary investigator approved target lesions (in concert with quantitative core lab radiologist). RPs are trained to open this table after their first review of the entire exam (to avoid bias) to guide their identification and selection of oncologist verified target (and other) lesions; in this case, there are only two target lesions (nodules in the right lung). This saves radiologists from having to differentiate target lesions from unspecified (neither target nor non-target) lesions. This effort starts a cycle of radiologist reports being more concordant with clinical trial investigators target lesions selection; hence, more valuable reports to investigators

Analogous to evolving automated AI triage platforms [11], radiologists are also immediately notified of potential critical findings [12] which allow for earlier notification to ordering providers and thus improve patient care.

Following interpretation and dictation, RPs record the number of missed findings per study into their study-specific spreadsheet database also monitored by radiologists. The RP miss rate percentage is then subsequently calculated, with the miss rate percentage being defined as the percentage of exam missed findings. Using this parameter, each individual RP performance is compared every 20 exams (optimal grouping for improved data visualization) and plotted with established missed rates by previous RPs that can serve as benchmarks of progression and proficiency. We chose our inclusion criteria to encompass RPs with at least 80 CAP CTs where rates were recorded (see Fig. 3 comparing four RP miss rates including an average). The comparison of missed findings serves as the major evaluation mechanism that provides a relative “grade” in that students can continually monitor their progress by comparing themselves to their peers.

Fig. 3

Average RP miss rate percentage decreases with more experience. This figure illustrates improved precision of identifying and measuring metastatic lesions and other findings on CAP CT exams among four RPs. Improved precision is objectively evaluated by decreasing RP miss rate percentage as they gain more CT exam review experience. We believe this is an important indication that RPs get more consistent/concordant with radiologists’ threshold for acceptable tumor measurements as they gain more experience with continual evaluation

The CT exams we included in this study were CAP exams that were either non-contrast or following intravenous contrast. Additionally, the included CTs were either baseline or follow-up exams in clinical trials. For the purposes of this study, we excluded triple phase CT CAP exams due to the complexities of interpreting these exams for our RPs who may not gain the requisite formalized radiology training (most participate between 1 month and 1 year).

As an objective measure of educational effectiveness, RP miss rates are calculated directly from each RP’s missed findings which include the following: when the RP did not annotate a new lesion (e.g., new lung nodule or liver metastasis) or the RP did not measure two enlarged lymph nodes in a region (e.g., mediastinum or retroperitoneum per RECIST 1.1). It is well known that lesion measurement consistency varies among radiologists (intra- and inter-observer variability), so this was taken into account [13]. For example, measurement discrepancies were only included in the calculations of RP miss rates if measured anatomy was not pathologic or otherwise did not adequately represent lesion size. False positives (measurement of fatty replacement near the falciform ligament, annotation of vascular anomalies that were incorrectly assessed as lesions) were recorded for teaching purposes only, and RPs review radiologist reports at the end of each shift for immediate inputs and instant feedback. Recording the miss rate allows RPs to learn what they missed and indicates potential areas for improvement that may require further reading on the associated pathology to reinforce the learning process.

This educational quality improvement initiative is considered non-human subject research (NHSR per NIH), hence does not require IRB review/exemption. Miss rate data collected and example listed exams are anonymized/not identifiable with exam dates blocked in Fig. 2.

Results

Our RP course presentations were made publicly available in November 2020, and within 20 days of posting, the RP course website had over 700 visits. This was in part thanks to numerous centers voluntarily providing content, further aiding the efforts of the RP course. See Fig. 3 of four recent RPs that have preprocessed at least 80 CT exams, indicating notable RP improvement with more CT exams preprocessed as evidenced by decreasing miss rate percentage.

We plotted the miss rate percentage in groups of 20 exams for optimal visualization purposes and averaged across four participating RPs. Of the four RPs who contributed to this analysis, three of them worked onsite alongside radiologists at our clinical center, while the RP listed as “RP 4” worked remotely due to the COVID-19 pandemic. In addition to confirming teaching efficiency, as more RPs participate and record miss rates over time in various modalities, this data can serve as benchmarks for future RPs to aim for.

These measurements and metadata can be exported from PACS and imported into cancer databases or for research investigators for data management and analysis [14].

Radiologist are aware of potential bias associated with viewing RP annotations prior to interpreting images and have since learned to turn off annotations upon initial review of images. Once initial image interpretation is completed, radiologists in the RP workflow will then “toggle on” or prompt annotations when ready to review and import approved measurements into their reports. This is similar; however, we believe not as important for avoiding RP bias, as the RPs have the ability to turn off any previous annotations. RPs often identify previously measured lesions at the end of an exam through the bookmark table for learning purposes.

Lastly, in addition to the objective virtual student comparison (two student authors of this paper) to in-person miss rates (one student author) as mentioned above, the following are a few subjective comments provided by RP alumni, with more available on our RP curriculum website on the “RP Alumni” page [15]:

The experience going through hundreds of body CTs, finding and measuring target lesions enriched my knowledge in anatomy and pathology, while providing the confidence and foundational knowledge of radiology giving me a head start in radiology residency. Experience navigating advanced automated PACS tools was especially helpful as a foreign medical graduate pursuing residency in the US.

This course let me step into the shoes of a radiologist. I was able to look at scans, understand what I saw, and physically mark my impact in bookmark tables, all while in a controlled active learning environment where my mistakes would be managed (and get fewer over time).

This RP course helped me develop not only an understanding and appreciation of anatomy within the human body but also familiarity with medical conditions associated with certain imaging findings. I gained a unique perspective of radiology as an active participant rather than a passive observer!

Discussion

We describe a radiology preprocessing curriculum with multicenter inputs that includes basic modality familiarity, PACS measurement tools, and cross-sectional anatomy. The major motivation to transition the RP curriculum to an online and publicly available platform was the COVID-19 pandemic in 2020 limiting trainee access to medical centers. RPs now train, work, and interact with radiologists in a completely virtual manner without patient or staff contact.

Once the curriculum was online and publicly available, there was immediate interest in participation from several centers by providing various presentations to include clinical trial information and cross-sectional anatomy. We deduced improved RP precision in the research setting by the decreasing number of annotation misses, mostly involving metastatic lesion measurements. We attribute this to RPs learning from previous mistakes and improving with the help of radiologist feedback. Additionally, we observed that the remote RP’s miss rate decrease was similar to that of the in-house RPs’; however, our limited experience may not be generalizable in other centers.

Our comprehensive online curriculum is geared toward training and preparing our RPs for the specific tasks of tumor quantification and identification of critical findings, allowing for seamless integration of our RPs into radiology workflows. We believe this workflow may also be effective in non-clinical trial workflows similar to how scribes assist private practice radiologists. Many private practice radiologists do not have residents or students in their workflows, which is quite different than the learning environment of radiology training where there are daily educational sessions and didactics to include read out sessions with residents and fellows. Private practice workflows without residents often have technologists prepare information such as ultrasound technologists providing most measurements for synoptic reports in RIS (e.g., TI-RADS [16]) and CT technologists annotating measurements and abnormalities [17] for cross sectional images. Our RP workflow and online curriculum may provide a unique niche for interested students to learn basic concepts of radiology, apply acquired knowledge to real cases, and discuss imaging findings with radiologists while improving workflow efficiency. Furthermore, the RP workflow and online curriculum would translate well to non-clinical trial workflows because RPs would have the abilities to efficiently assess imaging studies and identify abnormal findings while abiding by the physical workspace limitations related to the COVID-19 pandemic.

RPs learn cross-sectional anatomy and how to leverage automated tools to identify and measure tumors as well as other quantifiable abnormalities on imaging. Additionally, RPs learn how to identify actionable and critical findings commonly seen in clinical trials based on established definitions [18]. Once training begins, RPs have direct interactions with radiologists, including immediate feedback in the workflow, and review of involved anatomy and missed findings following submission of the final radiology reports at the end of each reading shift. Additionally, the authors (a majority were students) believe that our approach follows educators’ codes of ethics in that there was responsibility to the profession (improving patient care via more consistent and accurate measurements), responsibility to the students (students gained critical clinical knowledge through continual real time experience), and responsibility to leveraging advanced technology.

Clinical exposure to students interested in medicine is important for several reasons. For example, when exposed early to clinical demands, many students decide areas to focus on in school. Some decide to go to medical school or may change their mind and seek other professional education such as an engineering degree or masters. Clinical exposure also provides essential experiences to students in order to not only prepare them for medical school but also to help in their acceptance and academic course. These experiences are extremely valued by students.

The RP miss rate increased for two of the four RPs between cases 21–40 and 41–60 which we believe could be due to various radiologists increasing their threshold of measurement tolerance; however, we could not find objective reasons. The RP that trained and was completely virtual (#4) may have had higher miss rates at first due to lack of physical interaction with radiologists, in addition to having constant remote access issues, relying on WiFi rather than hard wired connectivity the other RPs had when in person.

The increased interest and active engagement of students into the RP workflow reinforces learning while providing clinical experience that students seek in radiology. This integrated approach with active learning is a well-established improvement over shadowing or passive observation [19]. The fact that RPs can compare their workflow performance against prior or current RPs using miss rate percentage anonymously as a benchmark provides incentives for self-improvement. We look at this comparison among peers similar to video game leaderboards that apply gamification incentives. Some of the authors were also involved in developing a gamification model for radiologist residents in last year’s SIIM hackathon [20].

Additionally, we have kept in touch with previously trained RPs that have shared positive experiences regarding the curriculum and how the clinical experience helped them transition into their current careers with an example former RP stating:

The RP course gave me a strong understanding of anatomy and the basics of interpreting radiologic imaging, in addition to a general understanding of radiology as a specialty. This has been extremely useful in medical school, especially since our anatomy courses integrate imaging into their curriculums.

Additional RP Alumni statements can be found on our RP Alumni page [21].

Furthermore, this curriculum can serve as a guide for students interested in imaging research or computer science careers, while improving radiologist efficiency by not having to spend excess time measuring lesions. We also discuss several advantages of the collaborative RP-radiologist workflow, including improved measurement concordance of target lesions in clinical trials and worklist prioritization of critical findings on CT imaging, thus enhancing patient care. Additionally, and intuitively, it has been demonstrated that RPs and automation perform better on follow-up exams when lesions have been measured on prior exams (knowing where to look) [22].

By sharing our preprocessing curriculum including various approaches and experiences, we believe similar workflows may help radiologists transition into “human guided artificial intelligence (AI) workflows” as machine learning (ML) algorithms continue to improve, eventually becoming commonplace. This should also be helpful when preparing measurement data in clinical trials, relieving data management burden as reports become more synoptic and allowing radiologists to focus on images while providing more comprehensive reports. This is especially helpful for radiologists given the increasing trend toward more quantitative and synoptic reporting [23], while radiologists remain primarily engaged with the images and produce the final report with minimal clerical effort.

The RPs integrated in these workflows benefit from clinical exposure in a safe manner (virtually, abiding by social distancing regulations during the COVID-19 pandemic). Additionally, the RP curriculum helps bridge an educational and experiential gap between clinically focused students and practitioners, with coders and engineers developing our AI workflow algorithms. We believe the improved measurement concordance and relation to radiology reports will serve as high quality labels that should help advance ML/deep learning (DL) training algorithms [24].

Lastly, we look at this unique workflow as a rethinking of quantification in clinical trials; e.g., rather than “post-processing” in core quantification labs, measurements are made in advance of radiologist review, improving the concordance with oncologist verified target lesions.

Limitations

The authors realize there are several limitations to our study. Due to the pressures of time and increasing workloads, radiologists may become complacent in augmented workflows, quickly accepting and hyperlinking RP annotations into their reports without careful verification. Also, radiologists may be distracted by RP annotations upon interpreting images, similar to a “satisfaction of search.” We therefore introduced “prompting” into the workflow based on distraction initially noted and experience from other countries [25] which allows radiologists to hide RP annotations in their initial assessment, avoiding complacency, bias, and distraction. This is done through a keyboard shortcut and the radiologist first reads the exam with no annotations. Through the first read, if the radiologist identifies a discrepancy, they will reveal the annotations, and if it is previously recorded by the RP, it is implemented into the report without the need for annotation by the radiologist.

Another limitation is the difficulty of RP-radiologist communication in a virtual format compared to the in-person format. We believe the relatively higher miss rate from our first remote RP may be due to lack of personal interaction (not possible in 2020 or 2021). The exclusively virtual workflow lacks the live in-person feedback and interactions that are important for RP learning and development. Additionally, there were some initial technological issues regarding remotely accessing PACS and RIS with a required virtual private network (VPN) and only WiFi router access.

Additionally, with slower connections, scrolling through cross-sectional exams with delay of image display can result in missed findings. Further, the RP miss rate percentage may vary based on timing and training. For example, radiologists may apply different thresholds regarding what is considered a missed finding or discrepancy according to RP education level, being more stringent with RPs with higher education levels. Lastly, our referral patient population during the pandemic may have varied compared to other years due limited travel, especially international referrals that used to be more common.

We believe our continually developing online curriculum may provide a long-term resource with continual updates based on inputs for RP alumni as a ready reference throughout their careers.

Conclusions

Our initial multicenter experience with an online RP curriculum including virtual yet immersive workflows may help research centers involved in clinical trials meet an increasing demand from students seeking virtual clinical experience during times of social distancing while immersed in radiologists’ increasingly complex workflows. The time saved from not measuring lesions may provide radiologists with an additional incentive to mentor students in a collaborative reporting environment rather than working in isolation (non-resident workflows). Also, consistent training and workflow approaches can result in a fun, active, educational, and clinical experience for students of all levels while also providing more comprehensive radiologist reports, resulting in improved patient care and radiologist efficiency.

We believe the improved RP measurement precision over time based on decreasing miss rate percentage with increasing CT exam review experience is promising objective evidence of how our RP curriculum may promote learning with the potential to help teach students interested in radiology. The remote RP miss rate was comparable to in-house RPs given limitations of PACS access; however, our pilot experience may not be representative for all settings and additional assessment with more comparisons would be prudent. We also believe these types of workflows represent the next steps of AI implementation into radiologist workflows—a human-driven approach rather than exclusive use of AI algorithms to augment radiologists with increasing quantification in reports.