Interobserver Agreement Rates on Fibroblast Activation Protein Inhibitor-Directed Molecular Imaging and Therapy.

OBJECTIVES: Fibroblast activation protein (FAP) has emerged as a novel target for FAP inhibitor (FAPI)-directed molecular imaging and endoradiotherapy (ERT). We aimed to assess the interobserver agreement rates for interpretation of 68Ga-FAPI-4 PET/CT and decision for ERT. PATIENTS AND METHODS: A random order of 68Ga-FAPI-4 PET/CTs from 49 oncology patients were independently interpreted by 4 blinded readers. Per scan, visual assessment was performed, including overall scan impression, number of organ/lymph node (LN) metastases, and number of affected organs/LN regions. Moreover, a maximum of 3 target lesions, defined as largest in size and/or most intense, per organ compartment were identified, which allowed for an additional quantitative interobserver assessment of LN and organ lesions. To investigate potential reference tissues, quantification also included unaffected liver parenchyma and blood pool. Readers also had to indicate whether FAPI-directed ERT should be considered (based on intensity of uptake and widespread disease). Interobserver agreement rates were evaluated using intraclass correlation coefficients (ICCs) and interpreted according to Cicchetti (with 0.4-0.59 indicating fair, and 0.6-0.74 good, agreement).
RESULTS: On a visual basis, the agreement rate for an overall scan impression was fair (ICC, 0.42; 95% confidence interval [CI], 0.27-0.57). The concordance rate for number of affected LN areas was also fair (ICC, 0.59; 95% CI, 0.45-0.72), whereas the number of LN metastases, number of affected organs, and number of organ metastases achieved good agreement rates (ICC, ≥0.63). In a quantitative analysis, concordance rates for LN were good (ICC, 0.70; 0.48-0.88), but only fair for organ lesions (ICC, 0.43; 0.26-0.60). In regards to background tissues, ICCs were good for unaffected liver parenchyma (0.68; 0.54-0.79) and fair for blood pool (0.43; 0.29-0.58). When readers should decide on ERT, concordance rates were also fair (ICC, 0.59; 95% CI, 0.46-0.73).
CONCLUSIONS: For FAPI-directed molecular imaging and therapy, a fair to good interobserver agreement rate was achieved, supporting the adoption of this radiotracer for clinical routine and multicenter trials.

Overexpressed on cancer-associated fibroblasts, fibroblast activation protein (FAP) has been identified as a suitable target for molecular imaging,[1] providing high contrast PET images along with reduced background accumulation when compared with the current reference standard FDG.[2,3] Of note, FAP inhibitors (FAPIs) can also be labeled with β-emitters, and such theranostic approaches have demonstrated low toxicity profiles and increased radiotracer retention after administration of the therapeutic compound.[4,5] FAPI-directed PET is needed to identify patients eligible for FAPI-targeted endoradiotherapy (ERT), which further emphasizes the importance of accurate scan interpretation.[4-6] In this regard, a recent study reported on various diagnostic pitfalls, with false-positives due to scarring or degenerative lesions.[7] Thus, similar to somatostatin receptor (SSTR) or prostate-specific membrane antigen (PSMA)–directed molecular imaging, the reliability of scan interpretation for FAPI-directed PET must be proven, which may allow for more widespread adoption in trials or in the clinic.[8,9] In this study, we aimed to elucidate the interobserver agreement rates for interpretation of FAPI-directed PET scans on a visual and quantitative level for sites of disease, as well as unaffected organs for reference. In addition, we aimed to determine PET-based concordance rates for identifying patients eligible for ERT.

PATIENTS AND METHODS

Patient Population

In total, 49 patients afflicted with oncological disease were retrospectively evaluated. Table 1 displays patient’s characteristics. Parts of this cohort have been investigated in previous articles,[3,10,11] although those studies did not assess interobserver agreement rates. All patients had signed written informed consent for conducted procedures. The local ethics committee waived the need for further approval due to the retrospective nature of this investigation (number 20210415 02).

TABLE 1

Patients Characteristics

Female		18/49 (36.7)
Age, y		63.9 ± 11.6
Referred for	Staging	30/49 (61.2)
	Restaging	19/49 (38.8)
Diagnosis	Oral cavity tumor	15/49 (30.6)
	Head and neck cancer	10/49 (20.4)
	Pancreatic cancer	8/49 (16.3)
	Hepatocellular carcinoma	6/49 (12.2)
	Neuroendocrine neoplasia	3/49 (6.1)
	Lung carcinoma	2/49 (4.1)
	Sarcoma, adrenal carcinoma, colon carcinoma, GIST, SIFT	1/49 (2), each
Previous therapies	Surgery	16/49 (32.6)
	Chemotherapy	9/49 (18.4)
	Radiation therapy	2/49 (4.1)

For age, mean ± standard deviation is displayed. Percentages are indicated in parens.

GIST, gastrointestinal stromal tumor; SIFT, solitary fibrosis tumor.

FAPI-Directed PET/CT

PET/CTs were conducted using a Siemens Biograph mCT 64 or 128 (Siemens Healthineers, Erlangen, Germany). After administration of 136 ± 24 MBq 68Ga-FAPI-4 and an uptake time of 60 minutes, a full-body protocol (top of the skull to knees) was used. Helical CTs with or without intravenous contrast were acquired. PET emission data were then obtained using a 128 × 128 matrix in 3-dimensional (2–3 minutes emission time per bed position) and reconstructed iteratively as provided by the producer (Siemens Esoft; Siemens Healthineers, Erlangen, Germany).[11]

Scan Interpretation

PET/CTs were analyzed using syngo.via (VB50; Siemens Healthcare Erlangen, Germany). Four readers (minimum 3 years of experience; range, 3–4 years) analyzed all scans independently and were blinded to the clinical status except for diagnosis, sex, age, and previous therapies. All readers also underwent a training session to gain experience with the workstation before scan interpretation.

Visual Interpretation

As described in[12], overall scan impression was assessed in a binary fashion. Scans were rated as positive when uptakes in disease sites were above background level. In addition, the number of organs affected, the number of organ metastases, the number of affected lymph node (LN) areas, and the number of LN metastases were indicated on a 5-point scale (ranging from 1 to ≥5 for each parameter).[12] FAPI uptake density was evaluated using a 4-point scale (none, 0; low, 1; intermediate, 2; or high, 3).[12] In this regard, 0 was defined as having equivalent uptake when compared with unaffected background tissue (eg, in the muscle, unaffected liver, or blood pool), whereas 3 indicated most intense uptake relative to background. In addition, all readers also had to decide whether FAPI-directed ERT should be considered, either based on intensity of uptake or widespread disease (WD).

Quantitative Assessment

Readers selected a maximum of 5 target lesions (TLs; defined as having the highest uptake and/or the largest in size on PET), using the predefined maximum intensity threshold by the software. No more than 3 lesions per organ compartment could be chosen (primary, LN, lung, skeleton, liver, and soft tissue). Target lesions not allocated to LN were subsumed under organ lesions (OLs). Volumes of interest were placed over all TL to assess the SUVmax. Target lesions identified by all readers were then used for further analysis. In addition, to investigate their potential as reference tissue for background corrections, additional volumes of interest were placed on unaffected liver tissue and blood pool (left ventricle, carefully avoiding the ventricular wall), also providing SUVmax.

Statistical Analysis

For evaluation of agreement rates, we calculated intraclass correlation coefficients (ICCs; including 95% confidence intervals [CIs]), by applying a mean-rating, single-measure, consistency model, and average Maxwell’s interrater agreement.[12,13] The concordance rates were interpreted according to Cicchetti (poor agreement, ICC <0.4; fair, 0.4–0.59; good, 0.6–0.74; excellent, 0.75–1).[14] For statistical analysis, the following software tools were used: MedCalc statistical software (version 18.2.1; Med-Calc Software bvba) and R (version 3.6.1; R Core Team, 2019) with package irr (version 0.84.1) and boot (version 1.3–23).[15-17]

RESULTS

Interobserver Agreement for Visual Scan Interpretation

On a visual assessment, the ICC was fair for an overall scan impression (0.42 [0.27–0.57]). For the number of affected organs (ICC, 0.63 [0.49–0.75]), number of organ metastases (ICC, 0.74 [0.64–0.90]), and number of LN metastases (ICC, 0.74 [0.64–0.83]), good interobserver agreement rates were achieved, with the last 2 items just missing excellent agreement. For the number of affected LN areas, interrater agreement was fair (ICC, 0.59 [0.45–0.72]; Table 2; Figure 1A).

TABLE 2

Overview of ICC for Visual Assessment

Parameter	ICC
Overall scan impression	0.42 (0.27–0.57)
Uptake density	0.46 (0.31–0.61)
No. affected organs	0.63 (0.49–0.75)
No. organ metastases	0.74 (0.64–0.90)
No. affected LN areas	0.59 (0.45–0.72)
No. LN metastases	0.74 (0.64–0.83)

95% CIs are given in parens.

FIGURE 1

Forest plot showing ICCs (including 95% CIs) for (A) visual imaging interpretation of 68Ga-FAPI-4 PET, (B) quantitative assessment, and (C) decision for FAPI-directed ERT based on PET. On visual assessment (A), number of organ metastases and LN metastases almost reached excellent agreement rates. On quantitative assessment (B), LN lesions demonstrated better agreement rates when compared with OLs. For reference tissue, unaffected liver achieved higher ICCs relative to blood pool. Investigating patients eligible for ERT (C), fair agreement rates were recorded if the readers agreed on performing therapy based on intensity or WD alone, which was slightly better for WD. Agreement rate was also fair when both conditions were applicable, just missing good agreement. Dotted lines indicate ranges of fair (ICC, 0.4–0.59) and good (ICC, 0.6–0.74) agreement.

Interobserver Agreement for Quantitative Assessment

Among all readers, 511 TLs were recorded as follows: LN (149/511 [29.2%]), primary (123/511 [24.1%]), liver (103/511 [20.2%]), skeleton (79/511 [15.5%]), soft tissue (31/511 [6%]), and lung (26/511 [5.1%]). The identical TLs were recorded by all observers in 208/511 (40.7%) of the cases. For SUVmax derived from LN, the agreement rate was good (ICC, 0.70 [0.48–0.88]) and fair for the remaining OL (ICC, 0.43 [0.26–0.60]). Comparable results were recorded for the reference organs, with unaffected liver achieving good (ICC, 0.68 [0.54–0.79]) and blood pool fair concordance rates (ICC, 0.43 [0.29–0.58]; Table 3; Figure 1B).

TABLE 3

Overview of ICC for Investigated Quantitative Parameters

Compartment	ICC
Organ lesions*	0.43 (0.26–0.60)
LN lesions	0.70 (0.48–0.88)
Blood pool	0.43 (0.29–0.58)
Unaffected liver	0.68 (0.54–0.79)

Disease sites (organ or LN lesions) and reference tissues (blood pool, unaffected liver) were analyzed.

*Includes target lesions of the primary, lung, skeleton, liver, and soft tissue. 95% CIs are given in parens.

Decision for FAPI-Directed ERT

The agreement rate for ERT was fair for intensity (0.50 [0.37–0.65]) or WD alone (0.54 [0.41–0.69]) and demonstrated slightly higher ICC when both conditions were applicable (0.59 [0.46–0.73]; Table 4; Figure 1C), thereby just missing good concordance rates. These findings were further confirmed by a fair agreement rate for FAPI density on PET (ICC, 0.46 [0.31–0.61]; Table 2). Figure 2A shows a case where all readers agreed on conducting ERT based on intensity and WD, whereas Figure 2B displays a patient considered eligible for ERT only based on intensity.

TABLE 4

Overview of ICC for Deciding on ERT by Investigating Intensity on FAPI-PET, WD, or Both

Parameter	ICC
Intensity	0.50 (0.37–0.65)
WD	0.54 (0.41–0.69)
Intensity + WD	0.59 (0.46–0.73)

95% CIs are given in parens.

FIGURE 2

Patients imaged with 68Ga-FAPI-4 that were evaluated for ERT. A, Patient with diagnosis of a mixed neuroendocrine nonneuroendocrine neoplasm after resection of the primary in the duodenum. MIP, transaxial PET, CT, and PET/CT revealed intensive 68Ga-FAPI-4 uptake in multiple liver lesions. All readers considered this patient eligible for ERT, based on both intensity and WD. B, Patient with a carcinoma of the oropharynx. MP, transaxial PET, CT, and PET/CT revealed a FAPI-avid LN metastasis in the left cervical region. All readers agreed that FAPI-directed ERT would be feasible based on intensity of uptake, but not on WD.

DISCUSSION

Analyzing a broad spectrum of tumor entities imaged with 68Ga-FAPI-4, we observed a fair to good interobserver agreement rate for visual and quantitative assessments of visceral and LN lesions. In addition, unaffected liver uptake also had a higher ICC than blood pool, supporting the notion that liver parenchyma is suitable for reliable background corrections among multiple observers. Last, based on intensity of uptake and advanced metastatic disease, decision for FAPI-targeted ERT also demonstrated a fair interobserver agreement. Given those substantial concordance rates for imaging and therapy, 68Ga-FAPI-4 may be used in multicenter settings for assessing sites of disease and identifying potential treatment candidates.

In recent years, FAPI-directed molecular imaging has been applied in multiple types of cancer, demonstrating that this imaging agent provides high contrast images,[18] which can even outperform the current reference radiotracer, 18F-FDG.[2,3] In addition, FAPI PET has also provided clinical value, for example, by triggering a TNM-based upstaging in more than half of cases or assisting in volumetric segmentation for radiotherapy planning.[19,20] Nonetheless, interpretation of 68Ga-FAPI-4 PET/CT has multiple pitfalls, leading to false-positives in degenerative lesions, scarring, or in the uterus.[7] As such, before an increased use of this PET agent in daily routine, a high interobserver rate should be proven. We observed an ICC of 0.74 for both LN and visceral lesions (Fig. 1A), thereby just missing excellent agreement rates.[14] This is in line with previous investigations on 68Ga-labeled radiotracers for SSTR- or PSMA-directed imaging, which had comparable or slightly higher ICCs for LN or distant organ metastases (SSTR, ≥0.77; PSMA, ≥0.74).[8,21] As a possible explanation for those discrepant ICC values, such previous studies focused on more homogeneous cohorts of enrolled patients, for example, gastrointestinal neuroendocrine neoplasms or prostate cancer, respectively.[8,21] This is in contrast to our study, which included randomly chosen patients with different tumor entities. As such, the presented findings emphasize the potential of FAPI as a pantumor radiotracer, as multiple observers still have a fair to good agreement rate, even in difficult scenarios with a large variety of tumor types. Nonetheless, interpretation of 68Ga-FAPI-4 still seems to substantially differ among multiple readers, for example, for the number of affected LN areas (ICC, 0.59). Thus, similar to SSTR- or PSMA-directed PET/CT, standardized frameworks providing reliable methods for scan interpretation may further increase ICCs,[22-24] preferably established in a consensus setting among a panel of experts in the field.[23]

In addition to a visual assessment, we also investigated the interobserver repeatability for quantification of sites of disease. Again, visceral and LN lesions demonstrated fair to good interobserver agreement rates, in particular with LN-derived SUV. The observed variation among readers may be explained by intratumor lesion variability of FAP expression, as such a heterogeneity of fibroblast activation within a metastasis has also been observed in immunohistochemical analyses.[25] Nonetheless, the acceptable concordance rate may be of importance for future studies using FAPI-directed PET/CT for identifying high-risk patients prone to early progressive disease under anti–tumor-specific therapies or to use the changes in uptake between baseline and follow-up scans to assess efficacy of various therapeutics.[26] In addition to its potential use as a response assessment tool, quantification may be also of importance for treatment planning, for example, in radiation oncology or to assess the target retention for antifibrotic treatments.[20,27] As such, given the ICCs we observed, the reader may have certainty that SUVs will not substantially vary among multiple readers for LN or distant organ metastases (Table 3). In addition, recent reports also demonstrated that tumor-to-background ratios (TBRs) on FAPI-PETs are significantly higher relative to TBR derived from 18F-FDG, for example, by using the SUVmax of the liver or blood pool.[2,3] As such, SUVmax from those reference tissues were also investigated in the present analysis, yielding a higher ICC for unaffected liver parenchyma when compared with blood pool derived from the left ventricle. As a possible explanation, a recent study revealed a tight link between radiotracer accumulation in the myocardium and risk factors for cardiovascular disease in a large cohort of patients with cancer.[28] Similar results of a large variety of uptake in oncology patients scanned with FAPI-PET have also been reported for the vasculature.[29] As such, given the lower ICC for the blood pool along with previous reports showing a potential interaction in cardio-oncology settings, unaffected liver parenchyma may be better suited for TBR calculations.

In theranostic scenarios, recent reports demonstrated relatively long tumor retention on posttherapeutic scans, which may lead to an increased use of FAPI-targeted ERT.[4,5,30,31] Of note, regardless if the β-emitter 90Y, 177Lu, or 153Sm was used, patient selection was primarily conducted with 68Ga-labeled FAPI PET/CT.[4,5,30,31] In our analysis, the ICCs for FAPI-directed ERT were fair among all readers. Nonetheless, relative to concordance rates for visual and quantitative assessments, ICCs were slightly lower for deciding on FAPI-directed treatment (Fig. 1C). Fendler and coworkers also investigated the observer agreement for SSTR-directed imaging and decision for ERT in neuroendocrine neoplasms. Similar to results of our study, a larger variety for deciding on therapy was noted when compared with ICC for imaging.[8] When a standardized framework for SSTR-PET/CT interpretation was applied, interobserver rates for appropriateness of SSTR ERT improved.[12] Given the fact that FAPI-targeted therapies are increasingly used as salvage approaches,[4,5,30,31] suggested standardized frameworks for FAPI-PET should then not only include recommendations on image interpretation, but also for selecting candidates for treatment.

Several limitations have to be considered. Our findings should be confirmed in larger, prospective trial, preferably including more readers with different levels of experience for PET/CT interpretation. Moreover, interobserver agreement rates for FAPI-directed imaging and decision on ERT may have been biased by patients with low FAPI-avid tumor findings, but TL-based assessment still provided a high number of lesions. Nonetheless, the present study aimed to mimic a real-world scenario of whom not every patient is eligible for radiolabeled therapy or does per se have a high uptake in every site of disease. Last, clinical applications of FAPI-directed molecular imaging have been recently expanded, for example, to image systemic sclerosis,[32] and thus, future studies may also investigate the interobserver agreement if FAPI-directed PET is used to image nonmalignant diseases.

CONCLUSIONS

Evaluating the interobserver rate for FAPI-directed imaging and therapy in a cohort with various tumor entities, a fair to good concordance rate was achieved on both visual and quantitative levels, including sites of disease and reference tissues. As such, this radiotracer may be appropriate for larger multicenter trials, for example, to select treatment candidates for FAPI-directed ERT.