Clinical requirements for extracellular vesicle assays.

The scientific and clinical interest in extracellular vesicles (EV) has grown exponentially during the past 15 years. As most research indicates that EVs can be utilised in diagnostics, prognostics and therapeutics, we may be on the brink of establishing the clinical utility of EV measurement, but how can we make this a reality? If we are to introduce EVs as biomarkers into clinical laboratories, it will be necessary to offer fully validated, International Organization for Standardization (ISO) standard 15189 assays. ISO 15189 defines the quality management system requirements particular to medical laboratories and is used internationally to determine accreditation. In order for a clinical laboratory to offer an accredited test for EVs, this assay must have been subjected to a thorough assay validation process. This process requires the generation of data related to defined performance characteristics, to ensure that an assay is performing in accordance with the needs of its clinical users. Each of the defined performance characteristics will be discussed in this review, along with the issues that specifically affect EV analysis. Accreditation is increasingly important for all clinical laboratories and the standards required to achieve this are becoming more and more stringent. Therefore, as companies seek to develop the best assays to detect EVs and their molecular contents for clinical utility, and as we move rapidly towards our goal of offering EV analysis in the diagnosis and monitoring of disease, it is timely to highlight the requirements for the clinical accreditation of such assays. It is essential to consider these parameters to ensure that we develop the highest quality assays possible and ultimately the best outcomes for patients.

Introduction

The scientific and clinical field of extracellular vesicles (EVs) has been rapidly expanding over the past 15 years. Publications focussing on exosomes and microvesicles have grown exponentially since 2000 [1]. As some of the first EV assays start to demonstrate a true clinical utility and gain FDA and international approvals [2], the process of introducing an assay into a clinical laboratory needs to be considered.

We have moved on from the initial “Wild West” of EV analysis, into a time where law and order starts to prevail, and this is being established with the promotion of standardisation studies and initiatives, such as EV Transparent Reporting and Centralising Knowledge (EV-TRACK) [3] and Minimal Information for Studies of EVs (MISEV) [4-6]. This review considers the next steps required as we move towards clinical implementation of such assays. It is noted that a wide range of techniques are currently available of EV analysis. These are listed in Table 1 along with their advantages and limitations in relation to their use clinical. Several recent reviews have discussed these techniques in more detail [7-12].

Table 1.

Technique comparison of clinical EV analysis.

Technique	Advantages	Limitations	References
Conventional Flow Cytometry	Widely available in clinical laboratoriesClinical validated machinesPossible to phenotypeHigh throughput and automationGood reproducibility	Limited sensitivityVariability known to occur between different machinesRequires expertise	[31,56]
High Sensitivity Flow Cytometry	Good resolutionPossible to phenotypeHigh throughputGood reproducibility	ExpensiveRequires expertise	[10,42,57,58]
Nanoparticle Tracking Analysis	High resolutionPossible to phenotypeGood reproducibility	ThroughputSpecificity – Designed for monodisperse populations	[10,18,59–62]
Dynamic Light Scattering	High resolutionGood reproducibility	ThroughputSpecificity – Designed for monodisperse populations	[60,62–64]
Resistive Pulse Sensing	High resolutionGood reproducibility	ThroughputSpecificity – Designed for monodisperse populations	[10,62,65]
Electron Microscopy	High resolutionPossible to phenotype	Not widely availableExpensiveThroughputPoor Reproducibility	[11]

The introduction of a new assay into a clinical laboratory requires several steps, which are laid out in Figure 1. The first step is the establishment of a clinical need, which is driven by the current literature and identification of gaps in the current diagnostic pathway being utilised. Suitable commercial or in-house assays are then identified and compared with existing diagnostic protocols, for example the current gold standard test for a disease. If the new assay demonstrates an advantage to the existing test, the assay will be subjected to further scrutiny prior to introduction, by either a Validation or Verification process (defined in Table 2).

Figure 1.

Introduction of clinical assays.

Table 2.

Definitions of terms.

Term	Definition
Clinical Assay	Any test being used to generate results that will be used diagnostically and may influence clinical decisions about a patient.
Clinical Laboratory Accreditation	A process to ensure the quality of results generated meet ISO 15189 standards
User Required Specifications (URS)	The required performance of a clinical assay, as defined by the clinical users either within an institution or nationally/internationally by a consensus publication
Validation	Analysis of performance characteristics to ensure that the assay meets the URS
Verification	Confirmation of performance characteristics of a commercial kit to ensure that the assay meets the URS

For in house methods, the laboratory will assess the performance characteristics to ensure that the assay performs at an adequate level within that clinical laboratory, a process known as Validation. For commercial assays, these performance characteristics should have been analysed by the company and published within the instruction manual. The clinical laboratory will then undertake confirmatory testing to ensure that the stated characteristics of the assay are reproducible within the clinical setting, a process known as Verification.

This review will focus on the steps of the Validation/Verification procedure and how all these steps relate specifically to EV analysis. However, it is important to note that other factors such as availability of equipment, maintenance, training of staff, information technology systems and clinical reporting all need to be considered in order for the clinical assay to be implemented. In addition, any new assay will be subjected to a continuous improvement process, via regular batch acceptance and audits of performance, to ensure that the test continues to perform at the highest level possible.

In order to fully validate an assay for clinical use and International Organization for Standardization (ISO) standard 15189 [13], which has been widely internationally accepted as the mandatory regulatory standard for clinical laboratories, it is essential to determine performance characteristics, including trueness, precision, clinical sensitivity and specificity, linearity, analytical sensitivity, analytical specificity, sample stability and diversity, and uncertainty of measurement (UoM) (defined in Table 3). These performance characteristics are compared to User Required Specification (URS), which are defined by the clinical users of the assay either within an institution or nationally/internationally by a consensus publication, and vary depending on that assays utility. Each of the performance characteristics are reviewed below along with specific issues affecting EV analysis.

Table 3.

Performance characteristics for assay validation.

Performance characteristic	Definition
Trueness	Qualitative assessment of the closeness of the measured value to the true value
Precision	Measurement of how close a group of measurements are to each another
Clinical Sensitivity	Ability to identify all individuals with a condition
Clinical Specificity	Ability to identify all individuals without a condition
Linearity	Ability to detect an analyte in a linear fashion across the reportable range
Analytical Sensitivity	Lowest level that can be reliably detected
Analytical Specificity	Effect of interfering substances on the assay
Sample Stability	Acceptable conditions and time that a sample can be stored prior to analysis
Sample Diversity	Range of sample types that can be used on the assay
Uncertainty of Measurement	Range of the values that could reasonably be attributed to the measured quantity

Assay performance characteristics

Although EVs occur in all body fluids, this review will focus on EVs in blood. Previous studies have found that EV measurement in plasma is sensitive to pre-analytical variables, such as phlebotomy technique, anticoagulant, sample transport and agitation, temperature, processing and storage [12,14-23]. In addition, analytical variables, such as the sensitivity of the flow cytometer used for detection, greatly impact on EV levels detected [22,24,25]. Therefore, differences in techniques need to be clearly defined to ensure that the levels of EVs reported can be reliably interpreted.

Trueness

Trueness is defined as a qualitative assessment of the closeness of the measured value to the true or conventionally true value. Trueness is essentially the same as the older and outdated term “accuracy”. Ideally trueness would be determined against an International Reference Preparation (IRP), which forms the basis for uniform reporting of results. Initially, it may appear difficult to verify the trueness of EVs, as there is currently no IRP available for EV measurement. However, this is also the case for many other clinical analytes [26]. Therefore, alternative strategies have been devised to determine the trueness of such analytes, including establishing an external quality assurance scheme [13].

This scheme would require a reference laboratory to produce a reference sample, which would be distributed to all participating laboratories. Such a process is currently being set-up by the Hendrix lab at University of Ghent [27]. The reference sample needs be homogenous, monodisperse, stable, safe (non-infectious), of sufficient volume to supply all participants and of a similar matrix to the samples being tested, i.e. plasma with EVs spiked in. The target values for each sample can be determined by an expert panel or by consensus of all participants. For this to be successful, we would need to establish the effects of storage and transportation on EV levels to ensure optimal reproducible conditions.

The participating laboratories will test the sample and return their results including details on the storage of the sample prior to analysis and the analysis platform used. The reference laboratory will then compile and analyse this data, generate a performance score for the laboratory based on the reported value versus the target value. If multiple platforms are used then this data can be displayed separately for different machines, to indicate any bias in a particular platform. The trueness of the EV measurement of any given techniques can then be determined using performance from the external quality assurance data.

Several studies have shown that different platforms, and even the same platforms in different laboratories, will give significantly different results for the same sample [28-30]. Differences in calibration and maintenance of equipment and training of operators may at least in part contribute to this. One way to reduce differences between laboratories and instruments would be to produce a universally agreed standardisation protocol. An inter-laboratory standardisation study has recently been conducted by van der Pol et al., which evaluated flow cytometry EV gating across 46 flow cytometers using software to correct for differences in refractive index and the optical set-up of the flow cytometers used [31].

External quality assurance schemes should strive to provide education to participants and to companies producing clinical assays in order to improve their performance, through commentaries, presentations and training days. Therefore, the commitment to become a reference laboratory for EV analysis would be a large undertaking and ideally would be shared between several expert groups in the field. Early collaborations are developing in specific areas of EV research, such as the Extracellular RNA Communication Consortium (ERCC) supported by National Institute of Health’s The Common Fund, who are focusing on the RNA extraction including standardised protocols (exrna.org).

Precision

Precision is a measurement of how close a group of measurements are to each another. It is reported as a Coefficient of Variation (CV%), with lower CV% indicating that the data replicates are close to one another. Thus, precision gives a measurement of confidence that an analyte measured once will give the same result if measured again.

For each assay it is essential to establish its precision and the level of imprecision should be acceptable for the URS. The acceptable levels of precision will vary dependent on techniques and the differences detected between patient groups. For example, a relatively high CV% may be acceptable if the difference in levels between patients and controls is large or binary, but if the difference is small the CV% needs to be much tighter.

Different techniques also generate different precision levels, for example flow cytometry produces greater imprecision than chemical analytes measured on automated machines, due to subjectivity and variability between operators [32]. Several methods have been utilised for EV analysis in blood, including flow cytometry, dynamic light scattering, nanoparticle tracking analysis (NTA), electron microscopy, atomic force microscopy, tunable resistance pulse sensing, fluorescence correlation spectroscopy and ELISA-based methods. Each of these methods have advantages and disadvantages, as reviewed previously [33,34], and each method is likely to have different levels of precision.

When establishing precision, it is necessary to define the intra-assay, inter-assay, inter-user and inter-batch to determine the likely variability when used in a clinical setting. This requires a panel of samples with varying levels of EVs to be run multiple times within one assay, across several different assays, performed by different scientists and over several lot numbers of a kit. In practice, the intra-assay, inter-assay and inter-user precision should be established before using the assay clinically, however establishing the inter-batch variability is usually carried out prospectively as this can take several months or even years.

In addition, clinical laboratories are required to batch test all new batches of diagnostics kits and there are limits of acceptability. If batches vary significantly from previous results, the kits may be rejected, wasting time and resources. Therefore, companies need to consider how reproducible their assays are between batches to prevent unnecessary wastage of products and to ensure a better clinical performance of the assay.

Clinical sensitivity and specificity

The definition of clinical sensitivity and clinical specificity are often confused. One way to define them is to consider the following two questions:

Clinical Sensitivity – Can the EV assay identify all patients with a condition (a test with poor clinical sensitivity may result in higher numbers of false negatives).

Clinical Specificity – Can the EV assay identify all individuals without disease (a test with poor clinical specificity may result in higher numbers of false positives).

In reality no clinical assays have 100% sensitivity and 100% specificity, and this is also very unlikely to be the case for EV-based assays. So it is important to optimise the cut-off for a clinical assay to achieve the best clinical sensitivity and clinical specificity, using Receiver Operating Curves (ROC). The area under the ROC curve can also be used to determine the overall performance of a test, a value of 1 indicating a perfect test and a value of 0.5 suggesting that the test has no clinical utility.

The trade-off between sensitivity and specificity will vary depending on intended clinical use of the assay, i.e. screening or diagnostic. Screening tests should have a high sensitivity, but may be less specific, whereas diagnostic tests need to be highly specific.

The definition of normal versus abnormal for EVs and hence where to place a clinical cut off may be clouded by the large spread in reported concentration of EVs detected in plasma of healthy humans [35-37]. However, in order for meaningful clinical analysis of EVs, it is essential to establish a “normal reference range”. The generation of the data to define reference ranges and thus to define cut-offs requires large numbers of samples that have been stringently processed. As previously stated there is great variability between analysis of EVs in different laboratories [10], including differences in EV extraction and analysis, making it impossible to combine this data from different studies. Therefore, we need standardisation of protocols across laboratories, before we can consider pooling data. This is an important area of validation that EV-specific Biobanks may be able to support, by providing well-defined samples from healthy individuals for assay development.

Although clinical sensitivity and specificity are stated as required ISO 15189 performance characteristics, in reality clinicians are often more concerned with a test’s Positive Predictive Value (PPV) and Negative Predictive Value (NPV). They can be calculated from the same data as clinical sensitivity and specificity and are defined as:

PPV – Percentage of people with a positive result who actually have the condition

NPV – Percentage of people with a negative result who do not have the condition

Therefore, when stating the clinical sensitivity and specificity for an assay with a defined reference range, it is useful to also provide the PPV and NPV to aid clinical utility of the assay.

Linearity

Linearity is the ability of a clinical assay to detect an analyte in a linear fashion throughout its reportable range and it is established by serially diluted samples with high concentrations of EVs. There is likely to be great variability in circulating EV levels between individuals, so clinical assays should be able to measure EVs over a wide range and be able to do this linearly, still a known issue in the field [10,34]. Linearity measurement can also be used to assist in the determination of the assays lower limit of detection, which must be clearly stated in the kit specifications.

Assays that utilise antibodies suffer from antigen-excess effects, particularly for analytes which have a very wide range of levels [30]. In addition to antigen-excess, EV analysis by flow cytometry faces the challenge of swarm detection. Standard flow cytometers are optimised to detect cells as an individual event, which are much larger than EVs. Therefore, depending on the flow cytometer set-up and flow rate, multiple vesicles can be detected as a single event [38,39]. Swarm detection can be identified by serial dilution of samples to check for linearity [40], and in plasma samples with particular high concentrations of EVs, it may be necessary to implement dilution factors in order to limit the impacts of swarm detection. Linearity experiments using NTA and tunable resistive pulse sensing (TRPS) of plasma EVs show a reasonable linearity in the range of 1.0–10.0 × 108 particles/mL for NTA and 1.0 × 108–1.8 × 109 particles/mL for TRPS [41], suggesting that these two methods are not affected by swarm detection.

Analytical sensitivity (lower limit of detection)

The analytical sensitivity for any clinical assay should be determined by the manufacturer and subsequently checked by the users. The lowest limit of detection will be the lowest concentration at which the assay can reproducibly report a result. This value can be determined by measuring the precision in samples with a known low concentration. The precision of the assay in these low samples should still meet the URS stated for the assay, to demonstrate consistent reproducibility of the assay at its limit of detection. Once the concentration reaches a point that the precision CV% is greater than the URS, then the sample is below the analytical sensitivity of the assay. Therefore, samples at this value and below should be reported as less than the limit of detection.

For EVs the analytical sensitivity could also refer to for example the smallest diameter of a vesicle that the assay can detect. It is important to note that the size distribution of EVs is log normal, meaning that there are many more small vesicles than large vesicles. Therefore, a small increase in the size sensitivity of an assay will result in a great increase in the number of vesicles it can detect. This has been demonstrated by high-sensitivity versus standard flow cytometry, and across other methods including NTA and resistive pulse sensing, where the increase in the resolution of the machine to detect smaller vesicles has led to an exponential increase in the numbers of vesicles detected [25,31,42,43].

Analytical specificity (interference)

Interference covers any other factors within a sample that affect the result other than the intended analyte. For EVs we are already aware of several such factors. It was noted that consumption of a high fat meal lead to a significant elevation of endothelial microvesicles in the circulation [44-46], although the impact of circulating chylomicrons may interfere with analysis.

Does this mean that EV analysis needs to be performed on fasting blood samples? Other analytes also require fasting bloods, such as lipid profiles and blood sugars. However, compliance is not high and it is not always possible, particularly if the patient presents to an emergency clinic, so there is a move to reduce the need for fasting samples [47]. The requirement for fasting samples may well be dependent on the nature and methodology of the assay, as some techniques may be more prone to detecting interfering lipids.

Another interfering factor is haemolysis, which is a common problem in clinic-obtained samples. By the very nature of haemolysis it will artificially induce EVs from RBCs, altering the quantification of EVs and potentially affecting the miRNA content [48]. So how do we control for haemolysis? Good phlebotomy procedures and protocols are essential, but in a clinical setting these may not always be adhered to, due to staff and time constraints. Therefore, laboratories will need to assess samples for evidence of haemolysis. A visual inspection of samples for haemolysis is insufficient and measurement of haemolysis can be done by spectrophotometry. Ultimately an assay that is not significantly affected by haemolysis or one that provides quality controls on samples that are haemolysed should be the aim.

The interfering factor in EV analysis that has troubled researchers for many years is platelets. No processing and centrifugation protocol can effectively remove all platelets whilst retaining all EVs, and to our knowledge, no assays have been described that sufficiently distinguish small platelets from large (platelet derived) EVs. Platelet-depleted-plasma is potentially the best samples we can hope to achieve. The freezing of this plasma can result in the post venepuncture generation of platelet-derived EVs and a skewing of the miRNA profile. It is essential that this is acknowledged within EV assay development, so the sample requirements for the assay are stringently defined whilst still being pragmatic for clinical practice. One suggestion would be to measure the platelet or RBC contamination in samples prior to storage.

Sample stability

Sample stability refers to how long and under which conditions a sample can be stored prior to analysis, without significantly impacting the results. It has been a key consideration of EV measurement in biological fluids for many years, whether it be the question of how quickly blood samples need to be centrifuged to produce platelet poor plasma [16-18,20], to how long urine samples can be stored in the fridge [49]. Stability data of EVs is an area where a reasonable literature base exists, although it should be noted that most of this literature was derived prior to the recommendation of standardisation protocols [47]. Therefore, novel stability data should be generated once standardisation protocols have been implemented.

The testing of sample stability should be applied to any clinical EV assay to determine the maximum time limits for sample storage. This is an essential performance characteristic for clinical samples, which often need to be taken in busy clinics and transported to laboratories prior to processing. EV assays will need to be robust enough to withstand such transport and logistical limitations. Several studies have suggested that miRNAs either free or within EVs are remarkably stable when isolated from both plasma and urine [50,51], suggesting that miRNAs may be a more robust and pragmatic biomarker of disease, than EVs themselves.

The stability of stored samples should also be considered. Many clinical laboratories will store samples prior to sample analysis. At what temperature and for how long can these samples be stored? Many EV assays for quantification, phenotyping and contents analysis have been shown to be relatively stable in samples stored at −80°C, but often clinical laboratories will not have the facilities to store large numbers of samples at this temperature. Therefore, the aim should be to generate assays that can be used for samples that have been stored under conditions that are widely available.

Sample diversity

Sample diversity refers to the range of acceptable sample types that can be used for the assay. Traditionally, for the measurement of circulating EVs, most protocols have advocated the use of tri-sodium citrate tubes. However, several recent studies have suggested that samples taken in EDTA tubes may provide better sample stability for the measurement of platelet and erythrocyte microvesicles [14,15], and therefore EDTA samples may be more applicable to a clinical setting. In addition, at the ISEV 2018 meeting, many clinical groups were using EDTA blood samples, potentially as these are more widely used stored for clinical studies. Other groups recommend the use of acid citrate dextrose (ACD) samples for EV analysis [52]. However, currently there is limited comparison data between the different coagulant blood tubes. In addition, appropriate anti-coagulants for downstream processing, such as miRNA analysis, should be considered. This area requires further investigation to determine the most appropriate samples, but could open up the potential of utilising samples that have already in stored, for example in clinical Biobanks.

Uncertainty of measurement

UoM is a parameter, associated with the result of a measurement (e.g. an assay) that defines the range of the values that could reasonably be attributed to the measured quantity [13]. UoM may well be the most important characteristic defined during assay validation. It gives a coefficient of variance which takes into account all sources of variation associated with the pre-analytical, analytical and post analytical variables. This measurement can be stated alongside a patients result in order to inform a clinician of the range of values in which that sample is likely to fall. This can impact decisions regarding diagnosis, prognosis and treatments. For certain clinical biochemical analyses the UoM may be very tight, and this is essential for analyses that even small changes have serious implications. For other functional assays, the UoM may be large and this needs to be clearly stated on results.

UoM is often considered to be the most difficult performance characteristic to calculate and is therefore often avoided. However, as stated above this value is often the most clinically relevant statistic to quote and therefore it should be addressed. It is also true that the statistics used to generate the UoM value are less applicable to qualitative than quantitative assays, but it can still be done [53].

For EVs there are some special considerations. We are already aware of many pre-analytical variables that may influence EV measurement, several of which are mentioned above. The UoM can be used to offset some of these concerns regarding pre-analytical variables, as it gives the opportunity to acknowledge and quantify their impact and to appropriately caveat clinical results.

Different diseases are likely to alter different subsets of EVs, due to differing cells becoming activated or dying; therefore, the measurement of multiple EV subtypes may be clinically useful. The UoM will be different depending on the sub-type of EVs being determined, with platelet-derived MVs likely to have the greatest UoM, due to the inherent problem with platelet contamination in plasma samples.

Discussion

The above performance characteristics have highlighted some major hurdles when it comes to the clinical application of EV diagnostics. The issues have been highlighted in Table 4 along with some of the potential solutions we may use to address them.

Table 4.

Challenges for clinical validation of EV diagnostics.

Challenges	Solutions
Lack of International Reference Preparation to determine Trueness	Setting up and use of EQA schemes
Overlap between patients and healthy controls	Setting an appropriate cut-off using ROC analysis
Interference from lipaemic, haemolytic and platelet contaminated samples	Applying strict, but achievable sample requirements for assays
Issues with sample stability	Determining appropriate collection and storage conditions that minimise EV generation and contaminating fractions of blood components
Issues with sample diversity	Stringent guidelines for sample acceptance criteria
Establishing uncertainty of measurement	Consideration of pre-analytical variables to ensure accurate reporting of EV levels

The lack of a defined IRP creates several difficulties in assay development. However, as has been highlighted here, there are several schemes being set-up by laboratories to address this issue, including sample exchanges. The aim of these schemes has been to identify and address issues with standardisation across multiple platforms. They are now likely to form the basis of future external quality assurance for clinical EV assays.

The large variability of EV levels detected even in healthy individuals and the likelihood that levels will overlap between health and disease makes it difficult to establish clinical cut-offs for normal versus abnormal. This can in part be rectified by using appropriate ROC analysis to define the most appropriate cut-off within a given population of patients. We must be aware that this cut-off is likely to be significantly altered depending on the technology used to detect EVs and the patient cohort being examined.

Interference from contaminating factors such as platelets and lipids in plasma samples remains an issue for EV analysis, both in research and clinical use. To minimise these effects, we impose stringent processing and storage requirements for plasma samples that we intend to use for EV analysis. In the clinical setting, these strict requirements may be impossible to achieve. Therefore, we need to be mindful during assay development to optimise assays for less-than-perfect samples and be pragmatic in our approach to sample requirements, as highlighted from the recent Clinical Wrap-Up session at ISEV 2018 [54].

Finally, the generation of UoM statistics for EV assays may be difficult to assess due to the pre-analytical variables known to impact detected EV levels. However, UoM is an opportunity to provide a range of results in which the true EV level falls. This is useful to both researchers and clinicians, as it can indicate whether a change in levels is significant or likely due to the variability of the assay. Therefore, UoM calculation should be encouraged for all EV analysis, even before it reaches the clinical stage.

Conclusions

Although assay validation might not be considered the most interesting or “sexy” side to EV research, it is right to acknowledge its importance. Accreditation is increasingly important for all clinical laboratories and the standards required to achieve this accreditation are becoming more and more stringent. Therefore, as we move rapidly towards our goal of offering EV analysis in diagnosis and monitoring of disease, we should consider these parameters to ensure that we develop the highest quality assays possible and ultimately the best outcomes for patients.

All this said, it is proving to be possible, as Exosome Diagnostics Inc, Waltham, USA who were recently the first to release exosome-based diagnostics test have achieved this for an EV Lung and prostate cancer diagnostic under its Clinical Laboratory Improvement Amendments (CLIA) act of 1988 as qualified to perform high complexity clinical testing and ISO 15189 certified locations in USA and Germany [55]. Therefore, clinically validated EV assays are achievable and this standard should be the goal for all new clinical techniques.