The use of digital pathology and image analysis in clinical trials.

Digital pathology and image analysis potentially provide greater accuracy, reproducibility and standardisation of pathology-based trial entry criteria and endpoints, alongside extracting new insights from both existing and novel features. Image analysis has great potential to identify, extract and quantify features in greater detail in comparison to pathologist assessment, which may produce improved prediction models or perform tasks beyond manual capability. In this article, we provide an overview of the utility of such technologies in clinical trials and provide a discussion of the potential applications, current challenges, limitations and remaining unanswered questions that require addressing prior to routine adoption in such studies. We reiterate the value of central review of pathology in clinical trials, and discuss inherent logistical, cost and performance advantages of using a digital approach. The current and emerging regulatory landscape is outlined. The role of digital platforms and remote learning to improve the training and performance of clinical trial pathologists is discussed. The impact of image analysis on quantitative tissue morphometrics in key areas such as standardisation of immunohistochemical stain interpretation, assessment of tumour cellularity prior to molecular analytical applications and the assessment of novel histological features is described. The standardisation of digital image production, establishment of criteria for digital pathology use in pre-clinical and clinical studies, establishment of performance criteria for image analysis algorithms and liaison with regulatory bodies to facilitate incorporation of image analysis applications into clinical practice are key issues to be addressed to improve digital pathology incorporation into clinical trials.

Introduction

Cellular pathology facilitates diagnosis and clinical trial treatment stratification. Genomic approaches divide traditional entities into smaller subcategories requiring large multicentre, often multinational, interventional studies. Such studies require high diagnostic standards and high reporting uniformity.

Digital pathology refers to the use of computer workstations to view digital whole slide images (WSIs) obtained from high resolution scanning of glass microscope slides 1; uses include teaching, research or primary diagnostic reporting. Digital pathology devices or image analysis algorithms used in diagnostic reporting usually require regulatory approval such as FDA (USA) or CE IVD (Europe).

Digital pathology and image analysis could ensure greater accuracy, reproducibility and standardisation of study inclusion criteria and outcomes. Automated image analysis can help to extract measurements and features that are known to be relevant. Quantification of immunohistochemical staining (IHC) is one example where automated methods are already being incorporated with some success into clinical practice 2. In general, image analysis can provide a more reproducible quantification of morphology of individual cells or relevant tissue components such as glands. Increasingly, deep learning based approaches replace traditional image analysis algorithms. By training complex computation models directly from data it is often possible to build algorithms that surpass the capabilities of traditional image analysis methods. Examples include the scoring of PD‐L1 3, quantification of immune infiltrates to predict outcomes in testicular tumours 4, detecting sentinel lymph node metastases 5 and superior prediction of colorectal cancer outcome compared to standard morphological assessment 6. The interpretation of human versus deep learning studies in pathology can be affected by methodological considerations. For example, in the Bejnordi et al study, pathologists were required to review 129 consecutive cases in a limited time frame which does not reproduce typical pathology reporting (where cases are mixed and difficult cases are not subjected to a time limit) 7.

Machine learning holds the promise of providing us with capabilities that not only mimic but also enhance the visual analysis pathologists perform. Beck et al 8 used a range of features provided by a commercial image analysis platform to identify novel stromal features associated with survival in breast cancer. Deep learning‐based artificial intelligence (AI) has identified novel markers of distant metastasis in colorectal cancer 9 and novel indications of morpho‐molecular associations 10. Correlating morphological patterns with genetic stratifiers using traditional machine learning or deep learning demonstrates substantial promise 11, 12. Ultimately, it may be possible to overcome limitations of existing categorical grading systems with such tools.

Image analysis is a complex task that may involve steps such as pre‐processing, accurate delineation of objects of interest, or the measurement of certain shape or texture features. Pre‐analytical considerations such as standardisation of immunohistochemical staining for reproducible results must be addressed, even for relatively simple tasks such as quantifying areas of tumour demonstrating positive staining for a cytokeratin marker using a colour threshold. Overview articles 13, 14, 15 provide an introduction to different image analysis techniques.

It is possible to utilise both commercially available dedicated image analysis programmes designed for digital pathology applications [Visiopharm, Definiens Tissue Studio, Indica Labs HALO (Corrales, New Mexico, USA)], or alternatively an open source software programme such as QuPath Open Source Software for Quantitative Pathology or ImageJ (Image Process and Analysis in Java) 16, 17, 18. Deep learning methods are increasingly incorporated into commercial and open‐source image analysis applications. Radiological imaging applications used to assess lung and liver nodules already have FDA clearance for clinical use 19; additionally, equivalent performance to human experts has been demonstrated in the interpretation of optical coherence tomography scans 20.

Image analysis has the potential to identify, extract and quantify features in greater detail in comparison to pathologist assessment, which may produce improved prediction models 6 or perform tasks beyond manual capability such as generating tumour‐infiltrating lymphocyte (TIL) maps 21 and glandular maps 22. There is now much interest in using these technologies in clinical trials 23, 24.

An NCRI CM‐Path Quality Assurance in Clinical Trials Workshop was held on 21 March 2017 with representation from key members of industry, regulation and pathology. Regulation, training, oversight, laboratory processes and scoring/reporting were divided up into subgroup teams, which were presented for discussion by the QA panel. These topics are covered in separate articles 25, 26. The scoring/reporting discussions included the use and validation of digital pathology and image analysis technologies in clinical trials. Currently there are limited published examples of their use in clinical trial practice, and methodological standardisation with minimal standards for publication requires further development.

In this article, we provide an overview of the utility of these technologies in clinical trials and discuss potential applications, current challenges, limitations and remaining unanswered questions that require addressing prior to routine study adoption (Table 1).

Table 1

Key issues that require consensus for the adoption of digital pathology and image analysis

Standardisation of operating procedures and training in digital image production

Adoption of digital pathology infrastructure for trials

Resource allocation for digital central review

Data governance and research approval

Criteria for standards in digital pathology in pre‐clinical, clinical and interventional studies

Evidence base and performance criteria for image analysis algorithms

In the UK, development of liaison between the Medicines and Healthcare products Regulatory Agency (MHRA), the British Standards Institution (BSI), the National Institute for Health and Care Excellence (NICE) and other regulatory bodies

Clinical trial adoption of this technology is in its infancy, and although the potential applications are exciting to all stakeholders involved, a structured and integrative approach to effectively and safely incorporate developments into practice is required.

Central review of pathology in clinical trials

The value of central review

The negative impact of inter‐observer variation in pathologists assessing lymphoma prompted the adoption of central case review in the 1960s 27. Widespread central pathology case review occurs in clinical trials; and is particularly valuable where rare and morphologically challenging diagnostic entities occur. Here, reporting pathologist experience can substantially influence reporting 28.

Currently, most central reviews occur after implementing patient management decisions for quality control prior to publication, rather than in ‘real time’ for trial entry. Central review requires additional slides to be produced from tissue blocks, which risks exhausting tissue required for direct patient care. In the event of review slide loss, it may not be possible to produce facsimile slides from limited remaining tissue.

Shortage of skilled trials pathologists is becoming a key issue in the conduct of clinical trials within the UK 29 and digital pathology has the potential to ameliorate this issue by linking distant sites and expanding access to expert pathologists. Rapid dissemination of identical images to multiple centres allows simultaneous case review, reducing turnaround times and ensuring consensus opinion before therapeutic allocation. Case reclassification at the end of a study could indicate suboptimal patient treatment and negate the significance of investigational findings. Duplicate tissue sections have allowed simultaneous peripheral and central review reporting in a multinational interventional trial in nephroblastoma 30 to address reporting variance before therapeutic allocation. In the study conducted by Vujanić et al, 9 of 248 simultaneously reported cases underwent diagnostic change resulting in alternative post‐operative therapy. Even minor error in diagnostic accuracy can affect the statistical significance of trial outcomes. Poor biomarker validation and positive publication bias can lead to expensive negative randomised control trials 31.

The central review of 552 radical prostatectomy specimens submitted for EORTC trial 22 911 32 showed low concordance between local assessment and central review for certain key parameters (evaluation of extra‐prostatic extension and surgical margin status). Central review can facilitate identification of discordant diagnostic parameters, but this may occur after recruitment of numerous patients.

Digital pathology and central pathology review

Digital pathology can permit simultaneous case review and allow rapid access to international experts in entities of interest. Reduction or removal of the need to physically transfer slides and tissue blocks, avoiding damage or loss, is a defining logistical advantage of the digital approach 33. Although multinational commercial platform providers can overcome technical limitations provided by trials carried out in separate countries, differing requirements between nations regarding appropriate data governance could impede disseminated review of digital material. The European General Data Protection Regulations (GDPR) which came into force on 25 May 2018, replacing the Data Protection Act (1998), have strengthened and unified data protection for individuals within the European Union (EU), whilst addressing the export of personal data outside the EU 34.

The requirement to physically transfer slides may be retained by centralised genomic analysis and biobanking requirements, or by participating units lacking appropriate infrastructure to transfer slides for digitisation. Slide hosting repositories that can accommodate large slide images (typically 0.5‐4Gb), satisfy data protection legislation, meet trial protocol requirements and ensure that patient‐identifiable image‐associated metadata have been removed, provide a considerable resource burden. A variety of proprietary file formats and platforms pose interoperability challenges. Clinical trial protocol specification of image formats is a requirement in the absence of widespread vendor‐agnostic file formats.

Cost implications

Pathologists who have not transferred to digital pathology need to be trained in line with RCPath guidelines 35, and costing of appropriate pathologists and their training should be factored into the trial business plan. Additionally, construction of standardised operating procedures (SOPs) covering scanning equipment, database construction, anonymisation of images (where appropriate) and their transfer is required: potentially with input from a pathology working group. Appropriate training for digital image reporting and slide scanning is required, which should include appropriate information governance. SOPs for equipment should define acceptable scanning platforms, required file formats and required scanning resolution. If scanning and reporting for the trial can be done by laboratories operating under ISO 15189: 2012 or working to good clinical practice (GCP) standards, this is preferable. Should scanners with appropriate regulatory approval (CE IVD or FDA) be available, these should be used in preference to those without such quality marks.

Digital images can form image libraries allowing streamlined research as new innovations become available. H&E stained slides and notably immunofluorescence‐labelled slides degrade over time and require additional preparation (potentially including further slide production from tissue blocks).

Maintenance and sustainability of data storage are thus key requirements. Ensuring strict adherence to data governance frameworks and undertaking documentation that exceeds current regulatory requirements will hopefully ensure the availability of valuable material to future research projects.

Training

Pathologist training for participation in clinical trials often involves face‐to‐face training. Digital pathology can facilitate flexible distance learning incorporating a wider audience than traditional face‐to‐face learning. Digital training programmes potentially reduce the requirement for expert pathologists leading the study to deliver face‐to‐face guidance. The requirement of training may be to improve standardisation of a feature routinely evaluated in diagnostic work, or alternatively to evaluate a novel feature not routinely reported: this may require a more comprehensive educational program.

Investigators can be trained by remote use of annotated digital material, either as part of an interactive web seminar or as an online training programme undertaken at the convenience of the participating pathologist. Sample images for reporting can be incorporated as an additional quality assurance measure. This is particularly valuable where pathologists are required to report a novel parameter in relatively few patients recruited from a single centre. The advantage of using direct visual feedback guided software for a novel parameter has been demonstrated in the assessment of percentage of TILs in breast cancer, which predicts response to neoadjuvant therapy 36. This is not a standard parameter for reporting in routine practice, and participating pathologists require training to report to an agreed standard. Improved concordance between pathologists reporting percentage of TILs was demonstrated after the use of training software utilising digital images of breast tumour tissue microarrays (TMAs) 37.

Digital pathology training software allows direct demonstration of compliance with clinical trial regulatory requirements and training logs, which can additionally be used by pathologists for revalidation and appraisal purposes. Clinical trials training applications can be reformatted to drive quality improvement in routine practice.

Image analysis

Analysis of IHC

There is substantial interest in the use of image analysis programmes to assist standardisation of reporting and introduction of novel parameters where accurate assessment by pathologists is unfeasible. Standardisation of ER and PR staining interpretation in breast cancer has received considerable interest as inter‐observer variation can directly impact patient care. Although expert pathologist concordance in stain interpretation was greater in comparison to machine assessment in a large study of pooled TMA cores 38, equivalent concordance was seen for HER‐2 assessment. Interestingly, a small number of extreme discrepancies were identified by digital image analysis: digital image analysis screening could be initially used to highlight isolated reporting discrepancies.

Automated image analysis also allows simultaneous scoring of ER and PR restricted to tumour‐rich areas only, as reported in 39 and additional studies demonstrate non‐inferiority to manual HER‐2 scoring in breast cancer 40.

FDA and CE IVD‐cleared algorithms exist for the assessment of a small number of pathological features, such as ER, PR, HER‐2 and Ki‐67 expression in breast cancer. Where image analysis algorithms are used in clinical trials, an appropriate evidence base for the algorithms must be available. A separately published validation dataset describing the performance of the algorithm in measuring the variable assessed would be regarded as a minimum requirement.

Pre‐analytical features

Pre‐analytical staining variation and background staining are key problems in image analysis development. Considerable variation in immunohistochemical staining and standard H&E staining can exist between laboratories (although image analysis can control for variation in staining intensity) 41. When constructing image analysis applications, demonstrating reproducibility in comparison to standard histopathological assessment is an important requirement. Algorithm – pathologist correlation is the most commonly used method: the use of multiple pathologists is always preferable to reflect routine practice. Clear descriptions must be provided of quality control measures and validation steps for every trial where image analysis is used. This should include a careful description of algorithm validation. Measures of reproducibility should be provided in publications such as pathologist‐algorithm correlation and measures of inter‐pathologist variability.

Regions of interest (ROI) selection methodology should be transparently described (if applicable) and whether analysis was on ROI, hot spots, WSI or was based on a pre‐selected sample (e.g. TMA spot) in order for it to be reliable and reproducible. Selection of ROI or hot spots can be completely automated, completely manual, or a combination of both. The approaches are subject to alternative potential errors, which can impact on the study design. For example, acquiring data or applying scoring systems optimised for visual interpretation and performing an AI protocol without appropriate validation is inappropriate.

Assessing tumour percentage content

There is considerable inter‐observer variability between pathologist assessment of percentage tumour cells within lung 42 and colorectal 43 cancer biopsies. Inaccurate estimations of tumour cellularity have the potential to result in inaccurate reporting of key actionable variants in genes such as EGFR, RAS and BRAF. Utilisation of the TissueMark™ (Philips Pathology, The Philips Centre, Guildford, Surrey, UK) platform to automatically annotate tumour boundaries with assessment of percentage tumour cells indicated superior performance compared to manual assessment 44. Standardisation of tumour sampling allows more refined molecular categorisation of tumours and facilitates stratified therapeutic approaches.

Assessing novel features

The use of digital pathology to provide novel information from histological samples is an exciting area of development. Rapid quantification and high reproducibility can be achieved by using digital image analysis, such as with localisation and quantification of immune cell infiltrates. The quantity and localisation of T cells is the underlying basis for the Immunoscore™ (Laboratory of Integrative Cancer Immunology INSERM, Paris, France) technique in colorectal adenocarcinoma: patients with a low CD3+ and CD8+ T cell density in the tumour centre and invasive margin are at an increased risk for disease relapse 45, 46. This marker could be used to stratify adjuvant therapy in prospective randomised control trials in Stage II disease, where the absolute benefit of adjuvant cytotoxic therapy is small 47. Novel digital morphometric signatures previously not established in the pathology community can also be mined and linked to clinical outcome, as shown recently in the case of breast cancer 8 and stage II colorectal adenocarcinoma 9.

Machine learning methods

Machine learning methods can facilitate accurate quantitative assessment of digital images at a performance level potentially exceeding that of human observers. The identification of cancer cells, stromal tissue or inflammatory cells 48 typically requires accurate ground‐truth training datasets and large numbers of cases to provide optimal automated assessment. Large openly available pooled digital datasets and the establishment of Grand Challenges for computational biologists provide a means to benchmark and evaluate image analysis algorithms. The CAMELYON challenge to identify lymph node metastases demonstrates the advantage of this approach 5, 7. Large‐scale interventional studies provide a relatively accessible source of suitable cases for retrospective analysis, with clear management and outcome data. This allows assessment of multiple image features by multiple image analysis programmes with subsequent univariate and multivariate analysis to identify the most relevant novel predictive and prognostic parameters or correlation with molecular markers.

Retrospective analysis of 768 pre‐treatment biopsies taken as part of a large RCT investigating neo‐adjuvant therapeutic approaches in breast cancer (neo‐tAnGo) identified median lymphocyte density as being independently predictive of complete pathological response to therapy on multivariate analysis 36. This study employed machine learning methods to classify cells as carcinoma, stroma or lymphocytes, with clear description of how quality metrics involved in analysis were determined, such as the use of automated ROI. Notably, the study was able to quickly evaluate initially promising parameters and demonstrate lack of a predictive utility in an adequately powered study. The adherence to good scientific practice by providing detailed description of techniques used and making the source code and images utilised available allowed transparency and facilitated reproducibility. Establishing such clear and transparent standards in image research practice is to be commended: maintaining algorithms and source images open to scientific scrutiny as opposed to keeping methodology closed to safeguard commercial interests can be anticipated as a key issue for the developing Digital Image Analysis community.

The discussion of applying advanced data analysis technologies to clinical trial data is by no means limited to image analysis tasks. Scott Gottlieb, Commissioner of Food and Drugs, remarks 49 that ‘AI holds enormous promise for the future of medicine, and we're actively developing a new regulatory framework to promote innovation in this space and support the use of AI‐based technologies’. He clearly recognises the need for including advanced digital health tools into the drug development pipeline. The Precertification Pilot Programme (Pre‐Cert) provides new regulatory guidelines for permitting the use of digital technologies that include machine learning.

Centralised image analysis

Central laboratory image analysis should be considered in study design due to advantages in standardisation of results. Although image analysis provides inherent reproducibility, inter‐platform variation between individual centres is a potential source of bias. Construction of SOPs with designated acceptable platforms, image analysis algorithms and validation steps ensuring inter‐site reproducibility would address this issue. Where companion diagnostic techniques reliant on specific staining platforms and accompanying image analysis platforms are required, adherence to manufacturer guidance would require clear documentation. Provision of unstained slides for immunohistochemical analysis and concurrent image analysis may be required as an additional quality assurance measure to account for inter‐laboratory staining variation. Ensuring both reproducible staining quality and image analysis between multiple laboratories could prevent centres participating in studies. Although centralised image analysis overcomes these issues, logistical constraints inherent to traditional studies undertaking centralised review remain. If many sites are conducting image analysis, repeatability analysis should be undertaken because of potential inter‐laboratory reproducibility concerns 50.

Regulation and guidelines

Slide scanners and image analysis algorithms when intended for medical use (including diagnosis) are classed as medical devices 51. The regulatory requirements for clinical performance studies of in vitro diagnostics (IVDs)/use of IVDs in clinical trials of medicinal products remains a subject of much debate and are fast evolving. The US FDA is testing a new Pre‐Cert model 52 with the intention of demonstrating by premarket review and excellence appraisal the same quality of information as a traditional approach to ensure safety and effectiveness standards are met. This novel model for the pre‐market review of digital health tools as medical devices includes implementing a new approach to the review of AI tools.

In the UK, the Medicines and Healthcare products Regulatory Agency (MHRA) regulates medicines, medical devices and blood components for transfusion in the UK. To ensure GCP compliance, the MHRA carries out inspections of trial sites (including laboratories) mostly based on a risk assessment score 53. How oversight of clinical trials utilising these technologies as medical devices will be regulated or inspected is a subject of debate. Constantly evolving AI applications pose additional challenges, but there are regulatory cleared AI algorithms in clinical practice.

To the best of our knowledge, there are no guidelines covering the use of digital pathology or image analysis in clinical trials. The low yield of clinically actionable biomarkers from a large volume of research studies with considerable resource outlay led to the construction of the REMARK recommendations for biomarker studies in 2005 54, 55. Studies that fail to meet these consensus requirements for a reputable biomarker study are increasingly excluded from systematic reviews of the evidence base for diagnostic approaches. A recent systematic review evaluating prognostic biomarker use in oesophageal adenocarcinoma demonstrated the effect of applying REMARK guidelines as inclusion criteria 56. Only 36 out of 214 eligible studies (17%) were included.

Constructing similar robust recommendations for digital pathology applications would minimise the extensive waste of resources encountered in prior biomarker studies. Failure of reproducibility driven by a lack of reporting of experimental detail has been described as a major factor in the inefficient development and adoption of clinically relevant biomarkers 57. Diligent reporting of the processes by which digital pathology applications were validated is essential to avoid repeating similar research practice failures. Direct adoption of existing biomarker guidelines could be considered: parameters assessed by digital pathology certainly meet the criteria of a biomarker as a ‘defined characteristic that is measured as an indicator of normal biological processes, pathogenic processes or responses to an exposure or intervention, including therapeutic interventions’ 58.

However, the application of guidelines intended for use in biospecimen‐derived biomarkers has been considered unsatisfactory when applied to imaging techniques, leading to the construction of separate guidelines for ‘imaging biomarkers’ 59. Synthesis of the most appropriate approaches from both biospecimen‐derived and imaging biomarkers would be logical for digital pathology applications.

The Health Research Authority (HRA) have recently updated their guidance on approval of new medical devices including software applications. The use of diagnostic platforms would require dedicated medical device clinical evaluation studies to obtain HRA approval, which would entail using systems in parallel with standard practice using light microscopy. Establishing the equivalence/non‐inferiority of a digital workstream for standard diagnostic practice is being undertaken in several UK centres with supporting guidance from the Royal College of Pathologists. The adoption of clinical studies using digital platforms would require the use of flagged research ethics committees to provide expertise in digital pathology. Lack of appropriate specialist availability to participate in the ethical approval process could hinder digital pathology development. Once standard diagnostic practice in clinical trials can be facilitated by digital platforms, approval of digital image analysis applications would be feasible. It is anticipated from joint HRA/MHRA guidance on software development that IVD performance evaluations would be undertaken in line with existing biochemical biomarker assays. Both industry and academic sponsors may find provision for clinical trial insurance to indemnify against diagnostic error induced by digital image analysis platforms challenging. It is uncertain whether actuarial assessment can be based on digital image use in a healthcare system where digital image analysis is more widely adopted, such as the USA. Combined assessment approaches of Pathologist plus machine may provide complex indemnity and regulatory challenges: appropriate mentoring from radiologists and industry specialists involved in digital radiology would be recommended to overcome these issues.

Conclusions

In this paper, we describe the use and potential future applications of digital pathology and image analysis technologies in clinical trials. The technologies can play a role in central review, training and image analysis and can be used to improve assessment of standard pathological features or extract novel insights. The current landscape sees the technologies most commonly being used where feasibility is already demonstrated such as central review or quality/efficiency benefits are obvious such as quantification of immune infiltrates in immune‐oncology trials.

In order to realise the potential of these technologies and dramatically improve the quality of the pathology input to clinical trials, the digital pathology community together with regulators and industry must establish practice standards for clinical trial use. Linking with international centres of excellence and involvement of other specialists such as software engineers, information network specialists is vital.

Table 1 lists the key issues that require consensus for the adoption of digital pathology and image analysis in clinical trials, and CM‐Path proposes to address these issues in a future multidisciplinary workshop.

Author contributions statement

RP, KO, MR, DS and CV conceived the outline of the paper and areas for discussion; additional oversight/commentary was provided by the CM‐Path QA Panel and HP, JR, NR. All authors were involved in writing the paper and had final approval of the submitted and published versions. This manuscript has been read and approved by all the authors, the requirements for authorship have been met and each author believes that the manuscript represents honest work.