Validation of an electronic image reader training programme for interpretation of [18F]flutemetamol β-amyloid PET brain images.

OBJECTIVES: An electronic training programme (ETP) was developed for interpretation of images during routine clinical use of the PET amyloid imaging agent [F]flutemetamol injection (VIZAMYL). This study was carried out to validate the ETP.
MATERIALS AND METHODS: Five nuclear medicine technologists (NMTs) and five readers previously inexperienced in amyloid image interpretation were required to self-train using the ETP and pass a test to participate. A total of 305 [F]flutemetamol PET images were then tested as the validation set, following preassessment and reorientation (where required) by one of five NMTs. Next, a new set of readers blinded to clinical information independently assessed all 305 images. Images had been acquired in previous studies from patients representing the full spectrum of cognitive capacity. When available, a standard of truth determined by histopathology or clinical history was used to derive sensitivity and specificity for image interpretation from this validation set. Randomly selected images (n=29) were read in duplicate to measure intrareader reproducibility. Images were read first without, and subsequently with anatomic images, if available.
RESULTS: All NMTs and all readers scored 100% on the qualifying test. The interpretation of 135 cases without anatomic image support resulted in sensitivity ranging from 84% to 94% (majority 94%, median 92%) and specificity ranging from 77% to 96% (majority 92%, median 81%). Inter-reader agreement was very high, with most κ scores more than 0.8. Intrareader reproducibility ranged from 93 to 100%.
CONCLUSION: The self-guided ETP effectively trained new amyloid PET image readers to accurately and reproducibly interpret [F]flutemetamol PET images.

Introduction

Fibrillar β-amyloid plaques are a histological hallmark of Alzheimer’s disease (AD) 1. Noninvasive β-amyloid detection by the recently approved PET brain imaging agents may improve accuracy in diagnosing cognitive impairment 2–5. [18F]Flutemetamol injection is a PET agent approved in 2013 in the USA 6 and in 2014 in the EU 7 for imaging of brain β-amyloid plaque density in patients with cognitive impairment after it was studied in 761 patients in nine clinical trials 8–12. The active component, [18F]flutemetamol, is a neutral thioflavin derivative of [11C]PiB with a high affinity (Kd: 6.7 nmol/l) for amyloid 13,14. Its significant cortical uptake differentiates probable AD patients from aged healthy volunteers 11,12.

Amyloid PET images present unique challenges to readers compared with other nuclear medicine images. Differentiation of nonspecific uptake in white matter and brainstem from specific binding to amyloid plaques in grey matter requires an understanding of brain anatomy and experience in PET image interpretation. Atrophic thinning of cortical grey matter presents a further complication. Historically, clinical development of PET tracers included in-person training of image reader 15,16. However, as these agents obtained marketing approval, regulatory bodies promoted the development and validation of scalable training programmes for image readers, such as self-guided, electronic media-based approaches 17,18. These are intended to standardize training and encourage adoption of an objective image interpretation methodology by the end users.

GE Healthcare developed a self-guided electronic training programme (ETP) to train nuclear medicine personnel to accurately and reproducibly orientate [by both nuclear medicine technologists (NMTs) and readers] and interpret (readers only) [18F]flutemetamol PET images for detecting significant brain amyloid uptake. ETP effectiveness was assessed by evaluating the performance (accuracy and reproducibility) of five readers who interpreted PET images from young and elderly patients with absent, early, probable or late dementia. The ETP was initially developed for use in the USA, but more recently, has been approved for use in both Europe and Japan 19,20.

Materials and methods

Description of the electronic training programme

The ETP ( but preloaded on dedicated computers for the purpose of this study) familiarized nuclear medicine physicians, radiologists and technologists with the proper orientation and interpretation of [18F]flutemetamol PET images, as well as providing background information on brain anatomy and pathophysiology useful for image interpretation. The programme comprised four interactive modules including self-assessments covering the following:

Review of brain anatomy; introduction to brain distribution of [18F]flutemetamol using axial, sagittal and coronal views.

Image orientation and alignment; choice of colour scales for reading.

Strategies for classifying a scan as normal (negative) or abnormal (positive), evaluating five key regions using appropriate viewing planes, anatomic landmarks and recognizing atrophy (Fig. 1).

Fig. 1

The electronic training programme (ETP) teaches the persons being trained to make a positive scan classification upon identification of several features in the following regions. Frontal pole and lobe: the lack of a marked sulcal pattern (dotted lines) and/or sharp intensity gradient from grey matter to cerebrospinal fluid. Posterior cingulate and precuneus: presence of cortical uptake in the circled region. Lateral temporal lobe: heightened uptake throughout and loss of the gyral/sulcal pattern (dotted circles). Parietal lobe: high uptake and decreased sulcal pattern within the dotted circles. Striatum: >50% uptake in the dotted region between the thalamus and the frontal lobe (axial or sagittal view).

Independent sample images classified as positive (abnormal) or negative (normal) for the presence of amyloid.

To complete the testing process, the trainee was required to independently read and classify images and was provided expert feedback on a series of images. To be deemed ready to interpret [18F]flutemetamol images, users had then to correctly assign at least 14/15 images in the test at the end of the training.

The programme emphasized the importance and benefits of a colour (e.g. rainbow or Sokoloff) scale to facilitate identification of elevated levels of [18F]flutemetamol activity. By setting a known negative region, such as the cerebellar cortex, to 30%, and/or using pons activity to set near maximal intensity (90%), the colour gradient enabled differentiation between positive and negative regions of amyloid uptake in the cortex.

Electronic training programme validation

Training in this study consisted of image display and orientation (for NMTs and readers) and training nuclear medicine physicians or radiologists to interpret the images (Fig. 2).

Fig. 2

Diagrammatic representation of the training and reading process for Study GE067–021 (internal GE reference number for the validation study). 1. Images from 276 individual patients and an additional 29 repeat images interspersed were read only in the first phase of the study. 2. Those images that had a tissue-based pathological standard of truth were rerandomized and read again with the support of anatomic images (mainly CT and some MRI). CRF, case report form; CSR, clinical study report; CT, computed tomography.

Selection and training of nuclear medicine technologists and image readers

Eligible NMTs were certified, practicing in the USA and had predominantly routine clinical (nonresearch) experience in acquiring, processing and orienting nuclear brain images. Candidates were excluded if they were experienced in any type of amyloid imaging, had ever been debarred from clinical research or were unwilling to disclose potential conflicts of interest. Each candidate was required to watch the orientation module of the self-guided ETP, review the test images provided and orient them correctly in three different planes without assistance within three tries.

Each reader candidate was a board-certified nuclear medicine physician or a board-certified radiologist with clinical nuclear medicine training practicing in the USA, and extensive experience reading nuclear medicine images in a clinical, nonresearch setting. Exclusion criteria were the same as those for the NMTs. After completing the interactive content of the ETP, each reader candidate had to answer the review questions and read the test images without assistance. The reader candidate was required to pass the training programme to participate in the study.

All ETP training, orientation and reading was conducted at a site operated by a contract research organization, the American College of Radiology Image Metrix in Philadelphia, Pennsylvania, USA.

Imaging study participants and ethics

Images (n=276) collected from nine previous clinical studies of [18F]flutemetamol injection (Table 1) had been acquired from patients across the cognitive spectrum. These included young (age≤40 years) and elderly (age≥55 years) healthy volunteers; patients with amnesic mild cognitive impairment (aMCI) or probable AD; patients with known/suspected normal pressure hydrocephalus; and end-of-life patients (Table 1).

Table 1

Image selection for analyses of validity and reproducibility (GE067-021)

This study was carried out according to the Declaration of Helsinki 21, the International Conference on Harmonisation Good Clinical Practice guideline 22 and all applicable laws and regulations. Independent Ethics Committee (IEC) or Institutional Review Board (IRB) approval of the protocol was not sought because no patients received an investigational product. Patients whose brain images were used had participated in IRB/IEC-approved studies and they (or their legally authorized representative) had consented in writing to participate; data usage was considered to be covered by the previous consent. This study was registered on (NCT01672827) and was carried out between 9 July 2012 (start of NMT training) and 23 August 2012 (database lock).

Image acquisition and selection

Image acquisition started ∼90 min after the administration of [18F]flutemetamol injection (VIZAMYL™; GE Healthcare, Little Chalfont, UK) and comprised six 5 min frames after a 185 MBq dose (n=208) or five 2 min frames after a 370 MBq dose (n=68). To ensure similar image signals, only the first four imaging frames (20 min) were used from 185 MBq scans to approximate the image signal of a 10 min, 370 MBq scan.

Depending on the contributing study, image selection was complete (where histopathological SoT was available) or from a random sample (in larger studies without a SoT). No images selected for the blinded image evaluation portion of the validation study had been used previously in the reader training programme. To assess intrareader reproducibility (IRR), 29 (10.5%) of the 276 images were selected at random and duplicated, and the originals and duplicates were included in the image set, for a total of 305 images.

Conduct of the blinded read

The set of 305 images (Table 2; populations 4 and 5) was divided into five 61-image sets; each was distributed to one of five ETP-trained NMTs. Each NMT checked (and corrected as needed) the orientation of the images in his/her set. The recombined image sets were then randomized for reading without anatomic images. According to routine practice by many US clinical sites, if a reader wanted reorientation of an image during the blinded image read, the reader could reorient the image himself/herself or request that it be done by an on-call ETP-trained NMT. While blinded to all patient clinical information, each reader interpreted each patient’s images, classifying the patient as normal (negative) or abnormal (positive) for the presence of amyloid and concluded the interpretation before moving to the next image set; no images could be excluded or classified as unevaluable. After all images had been read without anatomic images, an identical second reading session was performed, for which a subset of the [18F]flutemetamol images (n=149) had been rerandomized for reading with associated anatomic images (MRI or CT). The readers had no contact with any GE Healthcare employees before or during the read and the ETP was their only source of training information.

Table 2

Analysis populations

Pathological standard of truth measures

Postmortem histopathological standards of truth were determined for 68 patients on the basis of two blocks per each of eight brain regions (midfrontal lobe, superior temporal, middle temporal, inferior parietal, anterior cingulate, posterior cingulate, precuneus and primary visual cortex) as outlined in Curtis et al. 16. Briefly, for each block, three slides were prepared, stained with Bielschowsky silver stain and read independently by two blinded neuropathologists who analysed five random fields of view and reached consensus on neuritic plaque density; these were later converted into a plaque abundance score (modified CERAD criteria 1,16) from 0 (none) to 3 (frequent). The regional scores were derived from the mean of all slide scores, which in turn were the mean of all field of view scores. Ultimately, each regional score was dichotomized to a ‘negative’ (≤1.5) or a ‘positive’ (>1.5) amyloid pathology outcome, and if at least one region was positive, the patient was declared positive for amyloid pathology.

Biopsy-based standards of truth were obtained from 36 patients suspected to have normal pressure hydrocephalus 8–10; five fields of view for each of up to three slides of a single fixed Bielschowsky silver stain-stained tissue section per patient were examined and scored as detailed above. Averaged scores were dichotomized as detailed above.

Statistical analysis plan

All statistical analyses were carried out using statistical analysis software (SAS Institute Inc., Cary, North Carolina, USA). Seven analysis populations were defined for this study (Table 2). Each PET image classification was compared with the corresponding SoT results, where available, and categorized as true positive (TP), true negative (TN), false positive (FP) or false negative (FN). These were counted (n) and used to calculate validity measures: sensitivity=nTP/(nTP+nFN); specificity=nTN/(nTN+nFP). Two-sided exact 95% binomial confidence intervals were calculated for sensitivity and specificity for each reader and the majority interpretation (the image interpretation made independently by at least three of the five readers). The null hypothesis stated that at least one of the lower bounds of the two-sided 95% confidence interval was 70% or more; the alternative hypothesis stated that neither was. A sample size of 39 abnormal and 50 normal patients was calculated to provide ∼90% power, assuming a true sensitivity of 92% or more and a true specificity of 90% or more. IRR was measured by percent agreement. Inter-reader agreement (IRA) was analysed using Fleiss’ κ statistics. κ values were classified as follows: 0.6–0.7=good; >0.7–0.8=very good; and >0.8=excellent 23,24.

Results

Training results

Of six NMT candidates screened, the first five were tested and passed the orientation module of the training programme. Two NMTs were on-call during the blinded visual image interpretation sessions in case a reader requested reorientation of an image. Of 18 reader candidates screened, the first five were trained with the ETP and tested. Each of the five trainees (three nuclear medicine physicians and two radiologists) correctly scored all 15 test images at the end of the ETP and subsequently participated in the study.

Validity analyses

The overall reader performance resulting from the primary study objective (sensitivity and specificity without anatomic images for population 1, Table 3) was high, with sensitivity ranging from 84 to 94% (median 92%, majority 94%) and specificity ranging from 77 to 96% (median 81%, majority 92%). Reading accuracy with anatomic images for population 1 was also strong, with sensitivity ranging from 86 to 98% (median 92%, majority 94%) and specificity ranging from 80 to 94% (median 85%, majority 94%). Similar performance was found in the reading accuracy for populations 2 and 3 (Table 3).

Table 3

Summary of accuracy measures using populations 1–3 characterized by different SoT measures

Reproducibility analyses

When evaluating images from analysis population 4, all readers except one (reader 4) achieved excellent (>0.8) κ scores. When the subset of patients with aMCI was evaluated (analysis population 6), IRA was excellent.

IRR for analysis population 5 was similarly excellent: 100% for reader 1, 97% for readers 2 and 4 and 93% for readers 3 and 5. For aMCI patients (n=8, analysis population 7), IRR was 100% for all five readers.

Challenging cases

In populations 1–3, readers interpreted up to 16% of images as FPs and up to 9% as FNs, with or without anatomic image support (see bold and italic values in Table 4 for population 1 data).

Table 4

Summary of individual reader performance for population 1 (GE067-021)

FPs are attributed to equivocal or low neuritic plaque levels with a substantial number of diffuse plaques; two autopsied cases had a low neuritic plaque score, but heavy diffuse plaque loads that may have elevated the [18F]flutemetamol PET signal to clearly abnormal levels. In a case from the biopsy cohort, underestimation of the global amount of amyloid may have occurred during the histopathological assessment because of the small fraction of brain tissue analysed. No FP interpretations were found by majority assessment in the healthy volunteers. FN interpretations are considered to be related to focal atrophy, a borderline neuritic plaque density or both; anatomic images did not resolve the FN. Finally, discordant interpretations may have derived from motion artefacts and a combination of the latter and borderline pathology.

Discussion

The need for training of nuclear medicine professionals in general molecular imaging was identified in a recent joint report by the European Association of Nuclear Medicine and the Society of Nuclear Medicine and Molecular Imaging 25, as well as in past reports by other nuclear medicine bodies 26,27. Although digital training resources have been created for radiology 28 and nuclear medicine 29–31, to date, only two such programmes for molecular imaging have been described 32,33. With the development of new molecular imaging agents, there is a parallel need for agent-specific training rapidly deployable globally, and web-based programmes may help fulfil this need. We therefore developed an ETP providing background anatomy and pathophysiology needed to interpret [18F]flutemetamol images. Originally DVD based, the ETP is now on the web 19.

This study showed that the self-guided ETP effectively trains readers inexperienced in assessing brain amyloid to accurately and reproducibly interpret [18F]flutemetamol PET images. Sensitivity and specificity were high for all readers. The availability of anatomic images (CT/MRI) did not significantly alter results and the results were similar across the analysis populations. IRA and IRR overall were very good or excellent.

The high image interpretation accuracy that we observed after ETP training is similar to that found after in-person training for the 68 autopsy patients 16; in both cases, majority reads were more than 86% sensitive and more than 84% specific, suggesting effective training in image interpretation. It remains important to maintain an inquisitive eye towards equivocal scans: poststudy discussions with readers suggest that stringent adherence to the ETP criteria is key to the accurate visual interpretation of [18F]flutemetamol images. Similar to when in-person reader training was used in an earlier study 16, with the ETP, the causes of false image interpretations appear to be cortical atrophy, head motion during scans and predominance of high-frequency diffuse amyloid plaque 25. FN reads because of cortical atrophy-related difficulty localizing grey matter can be reduced by examining the striatum or inferior parietal regions, which are less subject to atrophy 20. Motion artefacts can be reduced by using appropriate head supports and flexible head restraints. Finally, diffuse plaques may result in FP amyloid pathology if neuritic plaque density alone is used as the standard of truth 34. However, the most recent criteria for the histopathological diagnosis of AD have been updated to include the pathological detection of both neuritic and diffuse plaques 35, in contrast with past official guidelines 1.

The number of readers (five) was chosen on the basis of precedents set in previous clinical trials of [18F]flutemetamol 11 and other amyloid PET imaging agents 15,36,37, and on agreement with regulatory bodies, but nevertheless, may not represent all readers. Bias was reduced as much as possible by blinding readers to patient clinical information and having images read first without access to anatomic images. The patients in this study may not fully reflect those who will undergo amyloid PET scanning in clinical practice, for whom the decision to image will likely be made on the basis of criteria such as the Alzheimer’s Association appropriate use criteria 38). Nevertheless, the accuracy of image interpretation for the varied populations examined in this study shows that the methodological robustness derived from the ETP enables a reliable assessment of amyloid pathology load even in the absence of pre-existing clinical information. Moreover, the study comprised a limited set of patients (n=68) for whom autopsy confirmation was available. Larger numbers would make the results more robust; however, no considerable differences in sensitivity or specificity were observed across groups with varying standards of truth.

Conclusion

This study validated the self-guided ETP for accurately and reproducibly interpreting [18F]flutemetamol PET images. Images included in the validation had been acquired from patients representing a broad spectrum of cognitive function and amyloid burden. After receiving the training, readers previously inexperienced in amyloid PET image interpretation performed well and showed high sensitivity, specificity, IRA and IRR, which are all standard metrics for assessing a diagnostic radiopharmaceutical. The availability of this electronic media-based training programme obviates the need for costly, cumbersome, in-person training. It also prevents potential variations in emphasis or depth of training across sessions, depending on the trainer or the audience, and thus represents a consistent and scalable approach to reader training.