Diagnostic performance of VEGF-D for lymphangioleiomyomatosis: a meta-analysis.

OBJECTIVE: VEGF-D is a potential biomarker for lymphangioleiomyomatosis (LAM); however, its diagnostic performance has yet to be systematically studied.
METHODS: We searched PubMed, EMBASE, Scopus, Web of Science, and Cochrane Library to identify primary studies on VEGF-D in relation to the diagnosis of LAM. The quality of the studies was evaluated using the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2). Summary estimates of diagnostic accuracy were pooled using a bivariate random effects model. Subgroup and sensitivity analyses were performed to explore possible heterogeneity. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) was applied to rate the quality of evidence and indicate the strength of recommendations.
RESULTS: Ten studies involving 945 patients were of high risk in quality, as assessed using the QUADAS-2. The pooled diagnostic parameters were indicated as follows: sensitivity = 0.82 (95% CI, 0.71-0.90); specificity = 0.98 (95% CI, 0.94-0.99); and diagnostic OR = 197 (95% CI, 66-587). The AUC of summary ROC analysis was 0.98. The subgroup and sensitivity analyses revealed that the overall performance was not substantially affected by the composition of the control group, prespecified cutoff value, the country of origin, or different cutoff values (p > 0.05 for all). A strong recommendation for serum VEGF-D determination to aid in the diagnosis of LAM was made according to the GRADE.
CONCLUSIONS: VEGF-D seems to have great potential implications for the diagnosis of LAM in clinical practice due to its excellent specificity and suboptimal sensitivity.

INTRODUCTION

Lymphangioleiomyomatosis (LAM) is a rare systemic neoplastic disease that primarily affects women of reproductive age. It is characterized by progressive and multiple cystic destruction of the lung and abnormalities of lymphatic system. , Histologically, LAM lesions are characterized by the proliferation of neoplastic smooth muscle-like cells (LAM cells) in small clusters located on the edges of lung cysts and along blood vessels and lymphatics. LAM cell infiltration causes airway obstruction, vascular wall thickening, lymphatic vessel disruption, venous occlusion, and hemorrhage, leading to the deterioration of lung function and eventually respiratory failure. -

Given the deadly harm of LAM and the advances in disease-specific treatments, such as sirolimus, it is increasingly important to make an early correct diagnosis. , However, the diagnosis of LAM remains challenging. According to the European Respiratory Society (ERS), a definite diagnosis of LAM depends on the typical changes in the findings of HRCT plus lung biopsy in the absence of compatible clinical presentations, such as renal angiomyolipoma, chylothorax, or tuberous sclerosis complex. In fact, several patients do not have these typical clinical features, and biopsy is often needed to achieve a definite diagnosis. However, biopsy methods, either transbronchial lung biopsy or video-assisted thoracic surgery, are invasive, technically difficult, and operator-dependent. Furthermore, invasive diagnostic procedures are unsuitable for screening for LAM in high-risk groups. Thus, searching for noninvasive and effective methods to aid in the diagnosis of LAM has become imperative, among which disease-specific biomarkers, has become imperative.

VEGF-D, a glycoprotein produced by LAM cells, has been found to promote the lymphangiogenesis via VEGF receptor 3. , In 2006, Seyama et al. first found increased serum VEGF-D levels in patients with LAM. Consequently, an increasing number of studies have investigated the diagnostic value of VEGF-D levels for LAM; however, the results have not entirely been consistent. For example, in the study by Xu et al., VEGF-D levels demonstrated an excellent diagnostic performance with an AUC of 0.99; in another study, VEGF-D levels for diagnosing LAM had an AUC of 0.75. Although a threshold of 800 pg/mL is indicated by the American Thoracic Society (ATS), serum VEGF-D levels appear to vary at different dosages or cutoff values. The optimal cutoff value for VEGF-D levels differed across studies, ranging from 440 pg/mL to 1,239 pg/mL. Identification of an appropriate threshold for VEGF-D levels in patients with LAM, especially those with characteristic lung cysts on HRCT but without additional clinical findings, is essential. To date, no studies have systematically pooled these data to estimate the overall diagnostic performance of VEGF-D levels for LAM or further explore potential issues that impact on its diagnostic accuracy. Therefore, we performed a meta-analysis based on the currently available evidence to establish summary estimates for the diagnostic accuracy of VEGF-D levels for LAM, which would yield relevant implications in clinical practice.

METHODS

This study was a meta-analysis in design whose protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO, Protocol no. CRD42020164137), an international database of prospectively registered systematic reviews. Our study was conducted and the findings were reported in accordance with the methods recommended by the Cochrane Diagnostic Test Accuracy Working Group and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Chart S1). - Ethical approval was not required owing to the retrospective nature of the meta-analysis.

Search strategy and selection criteria

Two reviewers independently searched the following databases: PubMed, EMBASE, Scopus, Web of Science, and Cochrane Library (until June 26, 2021) to identify related studies. The combinations of the following search strings were used: (“lymphangioleiomyomatosis” or “LAM”) AND (“endothelial growth factor-D” or “VEGF-D”). References listed in the considered articles or review articles were also manually checked to obtain additional relevant articles. LAM was described and defined in accordance with the ATS/Japanese Respiratory Society (JRS) guidelines (Chart S2) and the ERS guidelines (Chart S3). Studies were incorporated into the meta-analysis when they met all of the following criteria: 1) original studies that examined the diagnostic ability of VEGF-D for LAM; 2) sufficient data for constructing a two-by-two table of true positive, true negative, false positive, and false negative outcomes; and 3) studies that included at least 20 subjects (studies with smaller sample sizes may be vulnerable to selection bias). There were no limitations on languages, and the lowest date limit was restricted to 1997, when VEGF-D was first identified. Only the data associated with the best diagnostic performance were included from the studies in which several different cutoff values were used.

Data extraction and quality assessment

Two reviewers independently assessed the included studies. The following information was extracted: first author, publication year, country of origin, test method, cutoff value, diagnostic standard, and two-by-two tables for true positive, false positive, true negative, and false negative outcomes. We contacted the corresponding authors to identify additional information if necessary.

Quality Assessment of Diagnostic Accuracy Studies, version 2 (QUADAS-2) was applied to evaluate the methodology of the included studies. It covers four main domains to evaluate the risk of bias: patient selection, index test, reference standard, and flow and timing. Concerns regarding applicability were also assessed in patient selection, index text, and reference standard domains. The tool provides signaling questions to help rate different studies in the abovementioned domains. The questions are answered as “yes,” “no” or “unclear.” If the answers to all signaling questions for a domain are “yes,” then an overall judgment of a “low risk of bias” can be made. If any signaling question is answered as “no,” potential bias exists. If data provided by the primary study are insufficient in one or more domains to permit a judgment, the “unclear” category should be used. We further assessed the following information as a supplement for methodology evaluation: study registration, real-world design, institutional review board, sample size calculation, prospective data collection, consecutive data collection, inclusion/exclusion criteria (stringent or lenient), multicenter study, and conflicts of interest.

The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) was applied to evaluate the certainty of evidence. It classifies the quality of evidence into four levels (high, moderate, low, and very low). The GRADE system starts with a “high quality” rating and then it is downgraded by one level for each of the five factors considered: risk of bias, indirectness, inconsistency, imprecision, and publication bias. The risk of bias was evaluated using the QUADAS-2. To evaluate indirectness, we considered our evaluation of applicability concerns and the directness of VEGF-D test results on patient-important outcomes. Inconsistency was evaluated according to interstudy heterogeneity (assessed using the chi-squared-based Q-test and inconsistency index). The width of 95% CI was considered when evaluating imprecision: 95% CI < 10% was considered as “not serious”; 10% ≥ 95% CI < 20% was considered as “serious”; and 95% CI ≥ 20% was considered as “very serious.”

The GRADE system offers two grades of recommendations: “strong” and “weak.” When the effects of an intervention clearly outweigh the undesirable effects, or clearly do not, it offers strong recommendations. When the trade-offs are less certain, weak recommendations become mandatory. Any discrepancies in study selection, data extraction, and quality assessment were resolved by a third reviewer.

Statistical analysis

The meta-analysis was performed using a bivariate model. , We analyzed the overall diagnostic test accuracy by calculating the following indices: sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and diagnostic OR. We also calculated the AUC, which suggests the degree of accuracy of a diagnostic tool (poor, 0.50-0.75; good, 0.75-0.92; very good, 0.93-0.96; and excellent, > 0.97). Furthermore, we combined the hierarchical summary ROC (HSROC) model to conduct the meta-analysis, because we expected that the studies would use different thresholds to dichotomize test results measured on a continuous scale. The method can denote whether a threshold effect is present when the cutoff value is non-prespecified. It is better to summarize the results than to use a single joint summary estimate of sensitivity and specificity. , We further applied Fagan’s nomogram to estimate the extent to which the results of the diagnostic test change the possibility that a patient has a certain disease, which is based on the following equivalent formula:

To detect the inter-study heterogeneity, we applied the chi-squared-based Q-test and inconsistency index (I2). A p value of < 0.10 was used to suggest the presence of heterogeneity beyond what would be expected by chance, and an I2 value of > 50% was used to quantify the degree of heterogeneity. We further performed subgroup and sensitivity analyses to explore potential issues that may affect the diagnostic performance of VEGF-D for LAM. Publication bias was assessed by Deeks’ funnel plots.

Stata, version 15.0 (Stata Corp LP, College Station, TX, USA), Review Manager 5.2 (The Cochrane Collaboration, Copenhagen, Denmark), and Meta-Disc (XI Cochrane Colloquium, Barcelona, Spain) were used, and a two-sided p < 0.05 was considered statistically significant.

RESULTS

Study characteristics

After title and abstract review of the 115 records identified in the initial screening, we selected 33 articles for full-text review, of which 3 studies were excluded because of duplicate data. In addition, 14 of these studies were excluded because they did not include a control group, and another 6 studies were further excluded because two-by-two tables could not be constructed, although the authors of those studies had been contacted. Finally, 10 studies were included in the meta-analysis (Figure 1).

Figure 1

Flow diagram of study selection.

The included studies were conducted between 2009 and 2019. The average sample size was 95 (range, 33-173), yielding a total population of 945 patients. Two studies were conducted in North America, , 4 in Europe, - 3 in Asia, , , and 1 in South America. The mean age of the patients in the included studies ranged from 30 to 55 years.

All included studies were conducted in tertiary hospitals, and serum was used as the sample to test VEGF-D levels; the human VEGF-D enzyme-linked immune sorbent assay kit (R&D System Inc., Minneapolis, MN, USA) was used in 8 studies. , , - , - The diagnostic standard of LAM met the diagnostic criteria of definite LAM according to the ERS guidelines in 8 studies, , , - , , while part of the included patients met the diagnostic standard for probable LAM in 2 studies. , All included studies described the composition of the control group in detail. The control groups (Table 1) consisted of patients with other polycystic lung diseases such as pulmonary Langerhans cell histiocytosis, Birt-Hogg-Dubé syndrome, and Sjögren’s syndrome in 3 studies, , , healthy individuals in 4 studies, , , , and patients with other polycystic lung diseases as well as healthy participants in 3 studies. -

Table 1

Characteristics of the included studies on VEGF-D levels in relation to the diagnosis of lymphangioleiomyomatosis.a

Author	Country	P/C, n	Mean age, years (P/C)	Smoking, n (%)	Assay method	VEGF-D test manufacturer	Prespecified cutoff value	Cutoff value, pg/mL	LAM+ TSC, n (%)	mTOR inhibitor therapy, n (%)	Lymphatic involvement, n (%)	Diagnostic standard	Controls	TP	FP	FN	TN
Glasgow et al. 16	USA	106/40	51.0/49.6	UN	ELISA	R&D Systems	No	1,239	0 (0)	UN	77 (73)	Definite LAM	Healthy	61	1	45	39
Young et al. 29	USA	15/18	UN	UN	ELISA	R&D Systems	Yes	800	0 (0)	UN	UN	Definite LAM	OPLD	12	0	3	18
Cottin et al. 32	France	45/42	48/?	UN	UN	UN	Yes	800	UC	6(13%)	UN	Definite LAM	OPLD+ Healthy	34	1	11	41
Chang et al. 30	USA	45/32	46/36	1 (2.2%)	ELISA	R&D Systems	No	440	2 (4.4)	UN	UN	Definite LAM	Healthy	41	1	4	31
Radzikowska et al. 31	Poland	29/46	41.0/37.6	UN	ELISA	R&D Systems	No	468	7 (24)	UN	UN	Definite LAM	OPLD	25	5	4	41
Xu et al. 15	China	50/40	40.8/41.4	UN	ELISA	R&D Systems	No	850.7	2 (4%)	7 (14%)	UN	Definite LAM	Healthy	48	0	2	40
Daccord et al. 33	Switzerland	15/21	UN	UN	ELISA	UN	No	520	UC	3 (20%)	UN	Definite+ Probable LAM	OPLD+ Healthy	12	1	3	20
Amaral et al. 36	Brazil	104/40	43/43	UN	ELISA	R&D Systems	Yes	800	21 (20)	0 (0%)	20 (24)	Definite LAM	Healthy	52	0	52	40
Hirose et al. 34	Japan	108/65	UN	UN	ELISA	R&D Systems	No	645	16 (14.8)	35 (32%)	UN	Definite+ Probable LAM	OPLD+ Healthy	90	2	18	63
Mou et al. 35	China	50/34	39.3/46.9	UN	ELISA	R&D Systems	No	901	3 (6)	UN	UN	Definite LAM	OPLD	47	0	3	34

P/C: patient/control; LAM: lymphangioleiomyomatosis; TSC: tuberous sclerosis complex; mTOR: mammalian target of rapamycin; TP: true positive; FP: false positive; FN: false negative; TN: true negative; UN: unknown; OPLD: other polycystic lung disease; and UC: unclear. aAll of the studies were conducted at tertiary hospitals, and VEGF-D levels were determined from serum samples.

Methodological quality assessment

We used the QUADAS-2 tool to evaluate the methodological quality of the included studies. A high risk may exist in the patient selection and index test domains. The studies had a high or an unclear risk of bias because of the following: 1) 90% of the included studies did not describe in detail whether participants were enrolled consecutively or whether they used random sampling , , - , - ; 2) 4 studies did not avoid a case-control design , , , ; 3) 4 studies did not avoid inappropriate exclusion, in which healthy subjects, rather than patients with a disease that needs to be differentiated from LAM, were recruited as controls , , , ; and 4) the cutoff value of VEGF-D in serum was prespecified in 33.3% of the included studies. , , In the reference standard domain, all included studies were at a low risk. In the flow and timing domain, 1 study showed a high risk of bias because not all of the patients in the studies were included in the analysis. With regard to applicability concerns, 5 studies were of high concern owing to deficiencies in patient selection, , , , , ) and 1 study was of high concern in index test. ) (Figure S1).

In addition to the information obtained using the QUADAS-2, additional information related to the quality of the study design was also extracted. No studies were registered in the WHO International Clinical Trials Registry Platform, European Union Clinical Trials Register, or ClinicalTrials.gov. In the sample as a whole, 7 studies were approved by an institutional review board. , , , , - Two studies had a prospective design. , Inclusion/exclusion criteria were stringent in 2 studies. , Potential conflicts of interest may exist in 2 studies. , More detailed information is provided in Table S1.

Diagnostic accuracy

Figure 2 shows that the pooled sensitivity and specificity were 0.82 (95% CI, 0.71-0.90) and 0.98 (95% CI, 0.94-0.99), respectively. The positive likelihood ratio was 35.5 (95% CI, 14.4-87.6), and the negative likelihood ratio was 0.18 (95% CI, 0.11-0.30). The overall diagnostic OR was 197 (95% CI, 66-587), and the AUC was 0.98 (95% CI, 0.97-0.99).

Figure 2

Forest plots of the sensitivity and specificity of VEGF-D levels for the diagnosis of lymphangioleiomyomatosis. The sensitivity and specificity point estimates from each study are shown using solid squares. Error bars indicate the 95% CIs. df: degrees of freedom.

The HSROC model showed that the β-value was −0.31 (p = 0.678) and the λ-value was 5.69, suggesting that the HSROC curve was symmetric and the overall diagnostic value was excellent (Figure 3).

Figure 3

Hierarchical summary ROC (HSROC) curve plot of sensitivity vs. specificity of VEGF-D levels for the diagnosis of lymphangioleiomyomatosis.

We used Fagan’s nomogram to support decision making and evaluate the clinical utility (Figure S2). For instance, in an average-risk population with a pretest probability of 20%, the VEGF-D test would increase the probability of diagnosing LAM to 90% when the test result is positive and would decrease that probability to 4% when the test result is negative.

Subgroup analysis

The I2 values for the pooled sensitivity and specificity were 92.70% and 57.81%, respectively, indicating significant heterogeneity among the included studies. We first searched for the presence of a threshold effect, a source of heterogeneity unique to the diagnostic meta-analysis. No threshold effect was detected (Spearman’s correlation coefficient = 0.097; p = 0.789). We further investigated the potential factors that may affect the overall results of the meta-analysis. As shown in Table 2, the AUC was not significantly affected by the composition of the control group (p = 0.115), prespecified cutoff value (p = 0.839), country of origin, or different cutoff values; this indicated that these covariates did not substantially affect the diagnostic accuracy of VEGF-D for LAM in our study (p = 0.889).

Table 2

Subgroup analysis of the included studies on the VEGF-D in relation to the diagnosis of lymphangioleiomyomatosis.

Covariate	Study, n	AUC	SE (AUC)	Z	p
Composition of the control group
Not healthy	6	0.96	0.02	1.576	0.115
Healthy	4	0.99	0.01
Prespecified cutoff value
Yes	3	0.98	0.04	0.203	0.839
No	7	0.97	0.02
Country of origin
Asian countries	3	0.99	0.00	0.139	0.889
Non-Asian countries	7	0.96	0.02
Cutoff values
≤ 600 pg/mL	3	0.94	0.04	1.558a	0.119a
> 800 pg/mL	3	0.99	0.00	0.739a	0.460a
600-800 pg/mL	4	0.98	0.02	Ref	Ref

Compared with the cutoff values (> 600 to ≤ 800 pg/mL).

Sensitivity analysis

Not all the patients with LAM met the diagnostic standard for definite LAM in the 2 primary studies, , ) and 2 studies were abstracts. , We performed a sensitivity analysis to explore the effects of these studies on the overall results of the meta-analysis. The results indicated that the overall AUC was not materially altered after excluding the 2 primary studies that included the patients with probable LAM and the 2 abstracts, suggesting the robustness of our study (Table 3).

Table 3

Summary of overall analysis and sensitivity analysis.

Variable	Study, n	AUC	SE (AUC)	Z	p
Overall results of the meta-analysis	10	0.98	0.005
Excluding abstracts	8	0.97	0.015	0.632a	0.527a
Excluding two studies with probable LAM patients	8	0.9813	0.01116	0.106a	0.915a

Compared with the overall results of the meta-analysis.

Publication bias

Publication bias was explored using Deeks’ funnel asymmetry plot test. The results showed a non-significant value (p = 0.92). The shape of the funnel plots did not reveal any evidence of asymmetry (Figure S3), indicating a low likelihood of publication bias.

Quality of evidence according to the GRADE system

The GRADE system was applied to rate the quality of evidence and to indicate the strength of the recommendations. In our study, the quality of the pooled results was downgraded for the risk of bias (based on the results of the QUADAS-2), inconsistency (I2 > 0.5 and p < 0.1), and imprecision (95% CI width > 10%; Table S2). As a result, the quality of evidence was classified as low based on the GRADE summaries.

DISCUSSION

To the best of our knowledge, this is the first study to systematically explore the diagnostic performance of VEGF-D for LAM. The analysis showed that the overall diagnostic accuracy of VEGF-D determination for LAM was excellent based on the 10 studies with a high risk of bias assessed using the QUADAS-2; the VEGF-D level also exhibited suboptimal sensitivity and high specificity. Subgroup and sensitivity analyses revealed that the overall performance was not substantially affected by the composition of the control group, prespecified cutoff value, country of origin, or different cutoff values. A strong recommendation for serum VEGF-D testing to aid in the diagnosis of LAM in clinical practice was made according to the GRADE system.

In accordance with the ATS/JRS guidelines, the VEGF-D test is recommended before diagnostic lung biopsy for patients with suspected LAM who present with cystic abnormalities on CT but without confirmatory clinical or extrapulmonary radiological features. However, the guidelines only quoted 7 studies. The studies were mainly conducted in the USA and Europe. - , - , Data from the rest of the world are limited. In addition, the overall sensitivity or specificity of VEGF-D levels was not provided. After the publication of the guidelines, more clinical studies have been reported, and we performed the meta-analysis by including more studies conducted in other parts of the world, which reinforces the use of the VEGF-D test in a broader population.

In our study, the specificity of VEGF-D was excellent; but the sensitivity was moderate, which indicates that a positive result can be considered as LAM, whereas a negative result does not exclude LAM. In fact, a high specificity seems to be more acceptable in clinical practice, because an incorrect diagnosis of LAM may lead to missed opportunities to treat the real disease, avoiding adverse effects and unnecessary expenses due to inappropriate treatment. However, if a patient with LAM has a false negative result, the likely consequence is that the patient will proceed to the next diagnostic test for obtaining the final diagnosis-pulmonary biopsy, which reduces the undesirable effects of false negative to a large extent. Therefore, the determination of VEGF-D levels has a potential clinical impact on the LAM diagnosis algorithm. For patients who presented with pulmonary cystic abnormalities but lack confirmatory features of LAM, VEGF-D levels should be considered before proceeding to lung biopsy. It was estimated that 90% of probable LAM cases, in accordance with the ERS guidelines, could be upgraded to definite LAM if the VEGF-D test result is added to the diagnostic criteria. Therefore, many invasive diagnostic procedures can be avoided, which will substantially reduce patient risk and medical burden.

In our study, we found 7 studies that identified their optimal cutoff values using the Youden index. , , - , , Data-driven selection of optimal cutoff values for a test of a continuous variable may lead to overoptimistic estimates of diagnostic accuracy, especially in studies with small sample sizes. Using a prespecified cutoff value is an efficient approach to reduce the problem, but it is often difficult to achieve. In the early phases of a test evaluation, there is usually limited information available on the likely value of the optimal cutoff value, especially for a rare disease. In our study, we found that the test performance reported in the studies with data-driven selection of cutoff values was not better than that reported in the studies with prespecified cutoff values.

Although the ATS recommends a VEGF-D threshold value of 800 pg/mL, the cutoff values in the included studies were varied, ranging from 440 pg/mL to 1,239 pg/mL. This is reasonable because VEGF-D levels can be affected by many factors. Several studies have reported that VEGF-D levels were higher in LAM patients with lymphatic involvement than in those without lymphatic involvement. In addition, the severity of the disease, sirolimus treatment status, and whether there is accompanying tuberous sclerosis complex may also affect VEGF-D levels. , Additionally, differences in sample handling, standard preparation, or other aspects of assay conduct may lead to discrepancies in VEGF-D levels across different studies. , Therefore, the cutoff value is likely to vary with the severity of disease, equipment, and treatment history. On the one hand, a cutoff value of 800 pg/mL as a diagnostically specific threshold for LAM can provide a good diagnostic value for overall patients with LAM, and a uniform standard for the VEGF-D level is needed if its use is to become a standard practice for diagnosing LAM. On the other hand, investigators should also be open to the possibility that different cutoff values may be needed for patients with different disease statuses.

Our study has some limitations that need to be addressed. First, the number of studies on LAM, since it is a rare disease, was relatively small, although we performed a comprehensive literature search. Second, not all of the LAM patients in this meta-analysis met the diagnostic criteria of definite LAM. , Nevertheless, our sensitivity analysis indicated that the studies involved did not materially alter the pooled results. Lastly, the possible selection bias should not be ignored, given that all of the included studies were conducted in tertiary hospitals. Multicenter studies conducted in hospitals at different levels are needed to further validate our findings.

In conclusion, this study demonstrated that VEGF-D determination is highly specific and moderately sensitive for the diagnosis of LAM. Given that the effects of VEGF-D for LAM outweigh the undesirable effects, a strong recommendation for serum VEGF-D determination to aid in the diagnosis of LAM in clinical practice was made according to the GRADE system; however, multicenter studies conducted in hospitals at different levels are needed to further validate these findings.