Programmed Cell Death Ligand 1 Pathologist Training in the Time of COVID-19: Our Experience using a Digital Solution.

The COVID-19 pandemic presented numerous challenges to the continuity of programmed cell death ligand 1 (PD-L1) assay training events conducted by our organization. Under typical conditions, these training events are face-to-face affairs, where participants are trained to assay algorithms on glass slides during multi-headed scope sessions. Social distancing measures undertaken to slow pandemic spread necessitated the adaptation of our training methods to facilitate assay training and subsequent continuation of clinical trials. The present report details the creation and use of the Roche pathology training portal (PTP) that allowed for remote training to diagnostic assay algorithms. The PTP is a web-based system comprised of a learning management system (LMS) coupled to an image management system (IMS). Whole slide images (WSIs) were produced using a DP200 instrument (Roche, Pleasanton, CA) and these scan files were then uploaded to an IMS. Courses were created on the LMS using annotated WSIs that were shared with enrolled pathologists worldwide during assay training events. These courses culminated in assay certification examinations, where pathologists evaluated test-case WSIs and evaluated these cases within the LMS. Trainee submissions were analyzed for pass/fail status by comparing user data entries with consensus scores on these test-case WSIs. To date, 47 pathologist trainings have occurred and of these, 44 have successfully passed the associated assay certification exam on the first attempt (93% 1st-try pass rate). The PTP allowed roche to continue training sites during the COVID-19 pandemic, and these early results demonstrate the capability of this digital solution regarding PD-L1 diagnostic assay training events. Copyright:

INTRODUCTION

The pathway to successful medical device development is long and arduous, requiring years of precisely planned clinical trials and analysis of patient outcome data. The data gathered from clinical trials are in FDA submissions for drugs that will provide effective oncologic treatment options for patients. Therefore, the integrity of the clinical trial data depends upon accurate interpretation and assessment of tissue biomarkers such as programmed cell death ligand 1 (PD-L1).

One distinct challenge during COVID-19 is ensuring that the rigorous standardized training of external pathologists who will be interpreting clinical trial samples is available to all clinical sites. In the past, external pathologist training events occurred during on-site visits referred to as site initiation visits (SIV). These SIVs were previously face-to-face training events that followed a mandatory training agenda. Our standard training format consists of an introductory presentation on the biomarker, a set of microscope glass training slides, a set of annotated glass self-study slides, a glass slide self-assessment test, and a glass slide assay certification test. Biostatisticians analyze the results of the certification test and notify the lead pathologist of the pass/fail status of the individual pathologists.

Digital pathology has transformed the practice of pathology. Whole slide images (WSIs) have been used in the training of pathology residents, in the frozen section suite, and in pathology consultations.[1] WSIs have been shown to be noninferior to light microscopy for primary surgical pathology diagnosis.[2345] Similarly, diagnostic immunohistochemistry (IHC) assay evaluation using WSIs have proven to be comparable to evaluation using glass slides.[67] In addition, we have demonstrated that individuals can be trained to IHC diagnostic assay algorithms using WSIs, and then subsequently evaluate IHC-stained glass slides successfully.[8] The transition from using glass slide training sets for clinical trial biomarker training to using WSIs, therefore, is the next logical step.

The COVID-19 pandemic restricted international and domestic travel, thus limiting the number of clinical sites able to participate in a face-to-face training session. An additional complicating factor is that some countries restricted the entry and exit of glass slides. The question of how to maintain ongoing clinical studies, perform refresher pathology training, and initiate life-saving clinical trials globally during the pandemic became a critical issue for companion diagnostic assay development. Laboratories around the world have adopted digital solutions to help stay functional during the pandemic.[91011] The solution we employed to address the daunting task of continuation of clinical trial training came in the form of the pathology training portal (PTP) that uses WSIs for biomarker training and testing.

MATERIALS AND METHODS

The PTP is a web-based system comprised the Moodle™ learning management system (LMS; Moodle, West Perth, Australia) coupled to the VENTANA Vector Image Management System (IMS; Roche, Pleasanton, CA). Using the VENTANA DP200 slide scanner (Roche, Pleasanton, CA), WSIs were generated from cases comprising the consensus-scored glass slide training set. WSIs were of the .bif file type, scanned at either ×20 (0.465 μm/px) or ×40 (0.250 μm/px) with a single z-layer. WSIs were organized in the IMS to allow for web access of the slide viewer.

System validation of the PTP occurred before execution of any SIV training event. Published guidelines were used to design studies demonstrating that the PTP was capable of displaying WSIs of sufficient quality for evaluation and satisfied user requirements for operation.[12] Multiple studies were conducted on various indications and encompassing a variety of scoring algorithms that consistently exhibited high intra-observer agreement rates (>90%) between pathologists' glass and digital reads. Trainee's monitors/displays were not strictly regulated during these preliminary studies. Despite display differences, trainees participating in these preliminary studies expressed the opinion that the WSI displayed matched the glass slide counterpart in terms of color quality, tissue clarity, and tissue area captured.

For each of the diagnostic assays, individual courses were created in the PTP using WSIs of the glass slides in each module of each training set (i.e., annotated training set, annotated self-study set, a self-assessment test, and the certification test). Courses were gated in such a fashion as to compel the trainees to navigate through the assay training modules in a progressive step-wise order. Prior to training events, Roche pathologists examined and annotated the WSIs of the training set and self-study set. These annotations highlighted specific staining characteristics and teaching points. Annotated WSIs ensured standardization of the training material presented to the trainees in every training event. Test sets had previously been consensus scored by at least 2 pathologists on glass slides, and scores for the corresponding WSIs were confirmed by the lead pathologist for a given assay before conducting the SIV. Roche enrolled trainees in the specified course just before the training event, and they were unenrolled following the conclusion of certification exam reviews.

Training events were held with each individual site via teleconference. The initial training event for each site consists of a powerpoint presentation introducing the diagnostic assay and associated scoring algorithm to the trainees. Following the diagnostic assay presentation, a Roche pathologist led the trainees through the first module of the training event, the WSIs training set. During the WSI review of the training set, the Roche pathologist demonstrated the use of the PTP. The trainees practiced navigation of the PTP during this initial session. Upon conclusion of the initial WSI training set, the trainee pathologists were sufficiently familiar with both PTP operation as well as the algorithm and could then progress through the annotated self-study set, followed by the self-assessment test. A discrepant case review was held after the self-assessment test before unlocking the certification test for the trainee. A similar discrepancy review was held between the trainer and the trainee following the completion of the certification exam. All evaluations of WSIs on the system were performed without any use of digital algorithms.

Questions pertaining to test WSIs were created within the LMS. Moodle™ LMS allows for the creation of embedded answer (CLOZE) type questions which present to the trainee as multi-part questions. Links to WSIs associated with a case were embedded in each individual question. Activation of these links would launch the WSI viewer via the web for evaluation. The CLOZE question format facilitated the capture of multiple pieces of data pertaining to a case, including WSI acceptability, stain acceptability, % score, case status, and general case comments [Figure 1]. Trainees were asked to complete all data fields pertaining to a case and submit these responses via the LMS when their evaluation was complete. The data fields captured correspond to the data fields captured when reviewing glass slides (e.g., background/morphology acceptability, raw score, case status). Reports were generated in the LMS that allowed the tabulation of these data entries for further analyses.

Figure 1

Example whole slide images and associated questions. After examination of the whole slide images, the trainee is asked to evaluate the case for a number of attributes. Data entries are stored within the pathology training portal and can be accessed by authorized personnel anytime thereafter

Trainee scores on certification tests (minimum of 40 cases) were analyzed for positive percent agreement (PPA), negative percent agreement (NPA), and overall percent agreement (OPA). These agreement rates compared a trainee's status assessment on a case against the consensus case status. A score of 85% PPA and 85% NPA was required for a trainee to pass a certification test. In the event that a trainee failed a certification examination, the LMS would enforce a washout period of 2 weeks before another attempt of the certification examinaion could begin. In addition, the order of questions in both the self-assessment test and certification test was randomized in the LMS so that each trainee was presented the cases in a different order and a subsequent attempt would also display the cases in a different order. The cases themselves remained unchanged between the first and subsequent attempts.

RESULTS

The PTP was used as the primary training platform for 14 PD-L1 (SP263) SIVs occurring between April 2020 and January 2021 [Table 1]. These training events were conducted remotely and all teaching was performed solely through the PTP. External pathologists viewed and scored WSIs associated with certification test cases upon the PTP, and submitted their results when finished. No digital algorithms were employed during these training events to aid pathologist evaluation of the WSIs. Trainees' case scores were compared to consensus scores to determine pass/fail status based on case agreement rates (i.e., PPA, NPA, and OPA). Case consensus scores were obtained using at least two independent reads by internal subject matter experts. The digital training events encompassed a variety of indications and a variety of scoring algorithms [Table 1].

Table 1

Summary of training events to date using the pathology training portal

Site	Date	Assay	Number of pathologists	Certification exam result
China	April/May 2020	SP263 NSCLC TC	3	2 1st attempt pass, 1 2nd attempt pass (90% OPA on initial attempt)
China	April/May 2020	SP263 NSCLC TC	2	1 1st attempt pass, 1 2nd attempt pass (90% OPA on initial attempt)
China	May 2020	SP263 NSCLC TC	2	2 1st attempt pass
China	August 2020	SP263 NSCLC TC	2	2 1st attempt pass
China	August 2020	SP263 NSCLC TC	2	2 1st attempt pass
China	August 2020	SP263 NSCLC TC	2	1 1st attempt pass, 1 2nd attempt pass (87.5% OPA on initial attempt)
China	August/September 2020	SP263 NSCLC TC	1	1 1st attempt pass
China	September 2020	SP263 NSCLC IC	2	2 1st attempt pass
China	September 2020	SP263 HCC TC and IC	3	3 1st attempt pass
Europe	September 2020	SP263 NSCLC TC	1	1 1st attempt pass
Europe	September 2020	SP263 NSCLC TC	1	1 1st attempt pass
China	October 2020	SP263 HCC TC and IC	3	3 1st attempt pass
USA	October 2020	SP263 gastric TC and IC	2	2 1st attempt pass
USA	October 2020	SP263 gastric TC and IC	2	2 1st attempt pass
Europe	November 2020	SP263 gastric TC and IC	3	3 1st attempt pass
Europe	November 2020	SP263 gastric TC and IC	2	2 1st-attempt pass
China	January 2021	SP263 cervical TC and IC	6	6 1st attempt pass
Europe	January 2021	SP263 cervical TC and IC	5	5 1st attempt pass
China	January 2021	SP263 ESCC TC and IC	3	3 1st attempt pass

OPA: Overall percent agreement, ESCC: Esophageal squamous cell carcinoma, TC: Tumor cells, IC: Immune cells, HCC: Hepatocellular carcinoma, NSCLC: Nonsmall cell lung cancer

All 19 SIVs concluded successfully and met the deadline necessary for their respective clinical trial to begin and/or continue. The time spent by the trainer performing an actual training is similar to that of a standard, face-to-face clinical trial training, although there is an additional time requirement to train users on PTP operation and navigation to acclimate these individuals to the system. Thus far, 47 pathologist trainings have occurred, and 44 successfully passed the associated assay certification exam on the first attempt (93% 1st-try pass rate).

DISCUSSION

COVID-19 restrictions have posed a unique challenge to the training of pathologists in the context of clinical trials. In the absence of digital pathology training tools, these trials could have ended prematurely, depriving certain patient populations from potential treatment chances. In this study, we have shared our experience with using PTP as the solution that enabled the continuity of PD-L1 training for the pathologists during these times. There are many known benefits to utilizing WSIs, including the negation of travel between sites, consistent annotations, and lack of breakage or fading,[13] all of which proved true in the case of the PTP. Glass slide test sets must be shared between trainees whereas the PTP allows for simultaneous training and testing of the pathologists with automatic randomization of cases between readers. Embedding annotations in the training set and self-study set ensure that identical teaching points are reviewed with every trainee. WSIs are less prone to fading than their glass slide counterparts. Shipping glass slide test sets risks glass slide breakage or even the loss of the entire test set during the mailing process. WSIs within the PTP are not at risk for such issues.

User activities were captured upon the system and can be audited at any time to determine a number of attributes. Trainee data entries pertaining to test questions were recorded on the system, as well as test start date and completion date [Figure 2]. Additionally, the amount of time the trainee spent on each case within the test was recorded. This built-in feature provides some additional insight into which cases, and potentially what kind of features, give trainees the most difficulty during training and testing. This information can improve subsequent trainings by highlighting features that demand additional focus.

Figure 2

Results summary page. Compiles answers users entered during testing back into the original case order. The green box highlights the column giving the overall grade for each user. This view allows the trainer to see which cases were particularly challenging across readers

Case order is randomized when a test is attempted, but the LMS compiles these cases back into their initial order on review screens [Figure 2]. This capability allows the trainer to assess case difficulty at a glance, and also load the WSI associated with the case immediately to better understand what about the case may have been challenging. Trainers can also review test submissions and note via the “Comments” section or the flagged questions, which cases were particularly troublesome. Finally, trainers can set tolerances on numerical entries captured in the CLOZE questions to have the LMS flag out-of-tolerance entries on a submission review page [Figure 3]. Identifying cases that exhibit a high degree of discordance illustrates additional training points the trainer may like to focus on in future trainings.

Figure 3

Submission review page. This review page details a user's data entries for each case in a test. Entries for each part of a multi-part question are displayed for the trainer to review and assess a trainee's understanding of the assay scoring algorithm

The PTP has a function that allows for the automatic calculation of various metrics surrounding quizzes or tests on the system. A report was created on the LMS that calculated user data entries on test cases (e.g., scores or case status) for PPA, NPA, and OPA agreement rates to consensus scores. This allowed our pathologists to quickly assess whether an individual had passed a certification test for a particular assay. This function allows for rapid conveyance of pass/fail information to pathologists waiting to begin clinical trial case evaluation.

Conducting SIVs using the PTP is not without a few pitfalls. Sites must have adequate internet capabilities so that images do not remain pixelated or fail to load. Certain institutions had difficulty accessing the WSIs within the PTP presumably due to institutional network firewalls. However, all were ultimately able to access the WSIs. A number of studies have investigated the effect that different monitors can have when reviewing WSIs.[14] Some studies indicate that medical-grade displays perform better in WSIs evaluation,[15] where others saw little difference between medical-grade displays and “consumer-grade” monitors.[16] While we did stipulate that monitors should have at least minimally-acceptable characteristics (1920 × 1080px resolution, 24bit color, 17” diagonal viewing area, LED backlighting), we did not tightly control for this aspect of the training events. We recognize the importance that a display can have in factors such as color handling[17] and high power field area,[18] and agree that monitor considerations should not be overlooked. However, perhaps the respectable pass rate of pathologists using the PTP (93% 1st attempt pass rate) lends credence to the notion that modern monitors are often of a quality that is adequate for IHC diagnostic assay training and testing.

During COVID-19, the time allotted for training has been increased to incorporate sufficient training to PTP operation and navigation. One study comparing WSI and light microscopy manual slide review reported 5.20 min for WSI as compared to 4.95 min for glass slide review of the same case.[2] The training agenda must account for the subtle differences in review time when scheduling. Like any technological adoption, there exists a learning curve to familiarize oneself with the new technology, and the PTP is no different. However, this modest time increase at the outset of training is largely offset by the amount of time saved in personnel travel, and travel of the physical glass slide sets. Moreover, anecdotal evidence collected from the trainee pathologists following the use of the PTP was generally positive.

A 1st-attempt pass of 93% was achieved for 47 training events [Table 1]. Those trainees that failed the initial exam did not do poorly (90% OPA, 90% OPA, and 87.5% OPA), but fell just short of the passing criteria of 85% PPA and 85% NPA. Trainee pass/fail rates for various biomarker training events involve a myriad of factors which include but are not limited to the complexity of the scoring algorithm, the pathologist experience with clinical trial testing events, the adequacy of the consensus-scored test set, as well as the experience of the trainer. In training events performed thus far, case status discordance is predominantly attributable to factors such as cytoplasmic staining, staining in a cell type excluded by a specific scoring algorithm, or cases near the specified cutoff for the biomarker. Identification of these challenges allows trainers to focus on these points in subsequent SIVs, thereby improving the overall training paradigm.

CONCLUSION

Our study findings demonstrate that our training portal was capable of providing robust, standardized biomarker training to clinical trial sites globally. Multiple studies demonstrate that the PTP training events are equivalent to those that a trainee might receive in person with glass slides. The PTP provides a number of unique benefits that have improved training event efficiency, including automatic case randomization, automatic PPA/NPA calculation, and user activity tracking. Without the PTP, multiple ongoing clinical trials during the travel-restricted COVID-19 era would have been halted, thus affecting hundreds, if not thousands, of cancer patients worldwide.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.