Quality of Radiomics Research on Brain Metastasis: A Roadmap to Promote Clinical Translation.

OBJECTIVE: Our study aimed to evaluate the quality of radiomics studies on brain metastases based on the radiomics quality score (RQS), Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist, and the Image Biomarker Standardization Initiative (IBSI) guidelines.
MATERIALS AND METHODS: PubMed MEDLINE, and EMBASE were searched for articles on radiomics for evaluating brain metastases, published until February 2021. Of the 572 articles, 29 relevant original research articles were included and evaluated according to the RQS, TRIPOD checklist, and IBSI guidelines.
RESULTS: External validation was performed in only three studies (10.3%). The median RQS was 3.0 (range, -6 to 12), with a low basic adherence rate of 50.0%. The adherence rate was low in comparison to the "gold standard" (10.3%), stating the potential clinical utility (10.3%), performing the cut-off analysis (3.4%), reporting calibration statistics (6.9%), and providing open science and data (3.4%). None of the studies involved test-retest or phantom studies, prospective studies, or cost-effectiveness analyses. The overall rate of adherence to the TRIPOD checklist was 60.3% and low for reporting title (3.4%), blind assessment of outcome (0%), description of the handling of missing data (0%), and presentation of the full prediction model (0%). The majority of studies lacked pre-processing steps, with bias-field correction, isovoxel resampling, skull stripping, and gray-level discretization performed in only six (20.7%), nine (31.0%), four (3.8%), and four (13.8%) studies, respectively.
CONCLUSION: The overall scientific and reporting quality of radiomics studies on brain metastases published during the study period was insufficient. Radiomics studies should adhere to the RQS, TRIPOD, and IBSI guidelines to facilitate the translation of radiomics into the clinical field.

INTRODUCTION

Brain metastases are the most common type of intracranial tumors in adults [1], and these substantially affect the overall prognosis of patients with underlying cancers [2]. Although the clinical outcomes of patients with brain metastases have largely improved due to advances in neuro-oncology treatment, there is still room for improvement, because the management of brain metastases is often complex and controversial [3]. As MRI is the imaging modality of choice for brain metastasis [4], many studies have attempted to use MRI to address many questions in the field of brain metastasis. Radiomics is a recently introduced method that enables data mining from MRI; therefore, many studies have used MRI with radiomics in patients with brain metastases. As radiomics is more likely to be used with MRI in the field of brain metastases in the near future, it is highly desirable to assess the quality of published studies to evaluate the current status.

Radiomics involves the exploitation of MRI data to extract high-dimensional quantitative imaging features, which can be used to support clinical decision-making [56]. Although previous radiomics studies in the field of neuro-oncology have mainly focused on gliomas [789101112], there has been an increase in the number of studies on brain metastases. Indeed, radiomics studies have demonstrated promising results in the discrimination of brain metastasis from other tumors [1314151617181920], identification of primary tumor types in patients with brain metastases [21222324], prediction of specific genetic mutations [2526272829], prediction of survival [3031], differentiation between radiation necrosis and brain metastasis [32333435], and prediction of outcome after radiosurgery [363738394041]. However, such studies on brain metastases are confined within the limits of experimental settings, without translation into real-world clinical settings [42]. To lessen this translational gap, scientific and reporting quality must be high, which will enable a standardized evaluation of performance and increase the reproducibility and clinical utility of radiomics.

The radiomics quality score (RQS) is determined using a scoring system that incorporates the crucial aspects of radiomics studies and assesses their quality [4344]. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist is also a well-known tool for assessing the quality of prediction models [45]. Additionally, the Image Biomarkers Standardization Initiative (IBSI) provides a comprehensive and detailed review of each mandatory step for radiomics analyses, including nomenclature of radiomics features, general schemes, and datasets for calibration [46]. To the best of our knowledge, no study has evaluated radiomics studies on brain metastases according to the criteria of these three quality assessment tools. Assessing the quality of current radiomics studies on brain metastases may further promote the use of radiomics as a clinical tool. Therefore, in light of the RQS, TRIPOD checklist, and IBSI guidelines, this study aimed to evaluate the quality of radiomics studies that focused on brain metastases.

MATERIALS AND METHODS

Systematic Search Strategy and Study Selection

PubMed MEDLINE (n = 516) and EMBASE (n = 56) databases were searched to collect all original research papers on radiomics analysis published until February 2021, and the following terms were used for the search: (“brain metastasis” OR “brain metastases”) AND (“radiomic” OR “radiogenomic” OR “texture”). From a total of 572 papers retrieved using the specified search terms, 39 duplicate articles were removed. Of the remaining articles, 504 were excluded because they were non-radiomics studies (n = 100), conference abstracts (n = 25), not in the field of interest (n = 255), non-human (n = 6), non-brain (n = 20), review articles (n = 60), technical notes (n = 18), editorial articles (n = 1), erratum (n = 1), comments (n = 5), or case reports (n = 11), or included < 50 brain metastases (n = 2). Finally, 29 articles were included in the analysis (Fig. 1).

Fig. 1

Flow chart of the study selection process.

Analysis of Method Quality Based on RQS

The RQS was determined using 16 items classified into six domains, as reported in previous studies (Supplementary Table 1) [474849]. Two reviewers (with 8 and 10 years of experiences in neuroradiology) independently evaluated the domain scores of all included articles (Supplement, Radiomics Quality Scoring) after first achieving consensus on the evaluation criteria through discussion. If disagreements occurred between the two reviewers, a final decision was made after a consensus was reached.

Additional discussions were required to reach consensus on the following topics: issues of “validation” (domain 2), “comparison with the gold standard” (domain 3), and “detection and discussion of biologic correlation” (domain 3) (Supplement, Consensus Reached for RQS Scoring) due to the distinct nature of studies on brain metastasis.

Analysis of Reporting Completeness Based on the TRIPOD Checklist

The TRIPOD checklist, consisting of 37 items in 22 criteria, was applied to each article to determine the completeness of the report [45]. The analysis type of the prediction model was determined as follows: development only, type 1a; development and validation using resampling, type 1b; random split-sample development and validation, type 2a; nonrandom split-sample development and validation, type 2b; development and validation using separate data, type 3; or validation only, type 4. Details of the TRIPOD checklist and data extraction are shown in Supplement (Reporting Completeness Based on TRIPOD Statement).

Reporting of Image Processing and Radiomics Feature Extraction according to the IBSI Guidelines

The IBSI guidelines provide a total of 76 items for complete reports on image processing and image biomarker extraction (https://ibsi.readthedocs.io/en/latest/04_Radiomics_reporting_guidelines_and_nomenclature.html#reporting-guidelines) [50]. Among these items, the two reviewers evaluated all the journals according to the six items related to the pre-processing: 1) bias-field correction algorithm, 2) isovoxel resampling, 3) skull stripping, 4) gray-level discretization, 5) signal intensity normalization, and 6) use of IBSI guideline-compliant radiomics feature extraction software. These six items were chosen because they are the most commonly performed pre-processing methods in the field of radiomics. In addition, the segmentation method (i.e., manual, semi-automatic, or automatic segmentation) was retrieved from each journal. Evaluations were discussed to reach a consensus. The detailed assessment criteria for each item are described in Supplement (Assessment of pre-processing steps according to the IBSI guidelines).

Statistical Analysis

All statistical analyses were performed using R software (version 4.0.2; R Foundation for Statistical Computing).

In total, 29 articles were reviewed. In the cases where a score of one point per item was obtained, the study was considered to have basic adherence to each item of the RQS, TRIPOD checklist, or IBSI guideline. The basic adherence rate (%) was calculated as the proportion of the number of articles with basic adherence to the total number of articles. Sixteen items of the RQS were scored. The RQS for each item was reported as the median and range values. The percentage of the ideal score (%) was calculated as the ratio of the mean score to the ideal score for each item, and the total RQS (-8 to 36) was calculated for all articles. A total of 35 items on the TRIPOD checklist was scored. When calculating the overall adherence rate, validation items (10c, 10e, 12, 13, 17, and 19a) and “if done” items (5c and 11) were excluded from both the denominator and numerator. Seven items from the IBSI guidelines are scored.

RESULTS

Characteristics of Radiomics Studies on Brain Metastasis

The characteristics of the 29 included radiomics studies [13151617181922242526282930313233353738394051] are documented in Table 1. In the included studies, the median number of patients included was 77 (range, 24–439). The types of journals were clinical journals (7, 24.1%), imaging journals (18, 62.1%), and computer/neuroscience journals (4, 13.8%). Radiomics studies were either diagnostic (21, 72.4%) or prognostic (8, 27.6%). Except for eight prognostic studies covering prediction of survival and prediction of outcome after radiosurgery, all other studies were diagnostic. Detailed characteristics of the included studies are presented in Supplementary Table 2.

Table 1

Characteristics of the 29 Included Radiomics Studies

Article Characteristics		Data
No. of patients, median (range)		77 (24–439)
Journal type
	Clinical journal	7 (24.1)
	Imaging journal	18 (62.1)
	Computer science/neuroscience journal	4 (13.8)
Biomarker
	Diagnostic	21 (72.4)
	Predictive	0 (0.0)
	Prognostic	8 (27.6)
Topics in brain metastasis
	Differentiation of brain metastasis and other brain tumors	8 (27.6)
	Differentiation of brain metastasis and radiation necrosis after radiosurgery	4 (13.8)
	Prediction of primary tumor type in patients with brain metastasis	4 (13.8)
	Prediction of specific gene mutations in patients with brain metastasis	5 (17.2)
	Prediction of outcome of radiosurgery in patients with brain metastasis	6 (20.7)
	Prediction of survival in patients with brain metastasis	2 (6.9)
Sequence used for feature extraction*
	Conventional images	23 (82.1)
	Advanced images (diffusion-weighted image or diffusion tensor image)	5 (17.9)
Segmentation method
	Automatic	1 (3.4)
	Semi-automatic	9 (31.0)
	Manual	18 (62.1)
	Both automatic and manual	1 (3.4)
External validation
	Performed	3 (10.3)
	Not performed	26 (89.7)
Magnetic field strength (Tesla)*
	1.5T	10 (35.7)
	3T	11 (39.3)
	1.5T & 3T	6 (21.4)
	Not illustrated	1 (3.6)

Data are the number of studies with a percentage in parentheses unless specified otherwise. *Relevant for 28 radiomics studies that used MRI except for one study that used PET.

Except for one study using positron emission tomography, 28 studies used MRI. Twenty-three (79.3%) studies used conventional images for feature extraction, and five (17.2%) used additional advanced images. Most studies performed manual segmentation (62.1%). Only three studies performed an external validation. Multiple brain metastases (> 1 lesion per patient) were included in the dataset of 20 studies (69.0%). Among them, seven studies demonstrated the statistical analysis that was relevant to the handling of cluster-correlated data. In three studies, patients with multiple metastases were assigned to the validation set or the training set to prevent bias from cluster correlation [182231]. Correction for false discovery rate [30], correlation tests between the features [2427], and marginal proportional-hazards models were used for multiple lesions within the same patient [40]. In the remaining nine studies, only one lesion per patient constituted the dataset.

Rate of Basic Adherence to the Reporting Quality according to the Six Key Domains

The rate of basic adherence to reporting quality for the 16 items (six domains) of the RQS is documented in Table 2. In terms of domain 1, 20 studies (69.0%) included well-documented image protocols. Nine studies (31.0%) involved multiple segmentations performed by different readers to evaluate segmentation reliability [141516172326293540]. No study has used a test-retest approach or a phantom study.

Table 2

Radiomics Quality Score according to the Six Key Domains

		Basic Adherence Rate (%)*	Median (Range)	Percentage of the Ideal Score (%)†
Total 16 items (ideal score 36)		50.0	3 (-6–12)	9.6 (3.4)
Domain 1: protocol quality and stability in image and segmentation (0 to 5 points)		69.0 (20)	1 (0–2)	20.0 (1.0)
	Protocol quality (2 points)	76.0 (19)	1 (0–1)	34.5 (0.7)
	Multiple segmentations (1 point)	16.0 (4)	0 (0–1)	31.0 (0.3)
	Test-retest (1 point)	0 (0)	0 (0–0)	0 (0)
	Phantom study (1 point)	0 (0)	0 (0–0)	0 (0)
Domain 2: feature selection and validation (-8 to 8 points)		82.8 (24)	-2 (-8–6)	-4.3 (-0.3)
	Feature reduction or adjustment of multiple testing (-3 or 3 points)	82.8 (24)	3 (-3–3)	65.5 (2.0)
	Validation (-5, 2, 3, 4, or 5 points)	37.9 (11)	-5 (-5–3)	-46.2 (-2.3)
Domain 3: biologic/clinical validation and utility (0 to 6 points)		55.2 (16)	1 (0–4)	16.7 (1.0)
	Non-radiomics features (1 point)	44.8 (13)	0 (0–1)	44.8 (0.4)
	Biologic correlations (1 point)	13.8 (4)	0 (0–1)	13.8 (0.1)
	Comparison to “gold standard” (2 points)	10.3 (3)	0 (0–2)	8.6 (0.2)
	Potential clinical utility (2 points)	10.3 (3)	0 (0–2)	10.3 (0.2)
Domain 4: model performance index (0 to 5 points)		89.7 (26)	2 (0–4)	35.2 (1.8)
	Cut-off analysis (1 point)	3.4 (1)	0 (0–1)	3.4 (0.0)
	Discrimination statistics (2 points)	89.7 (26)	2 (0–2)	81.0 (1.6)
	Calibration statistics (2 points)	6.9 (2)	0 (0–2)	5.2 (0.1)
Domain 5: high level of evidence (0 to 8 points)		0 (0)	0 (0–0)	0 (0)
	Prospective study (7 points)	0 (0)	0 (0–0)	0 (0)
	Cost-effectiveness analysis (1 point)	0 (0)	0 (0–0)	0 (0)
Domain 6: open science and data (0 to 4 points)		3.4 (1)	0 (0–1)	0.9 (0.0)

*Numbers in parentheses are the number of studies, †Numbers in parentheses are the mean values.

In terms of domain 2, 24 studies (82.8%) performed feature reduction or adjustment for multiple tests. Ten studies (34.5%) performed validation using a dataset obtained from the same institution, gaining two points [13151722272930313238]. One study (3.4%) performed validation using a dataset from another institute, earning three points [14]. In the remaining 18 studies (36%), validation was missing.

For domain 3, 13 studies (44.8%) performed multivariate analyses with non-radiomics features. Four studies (13.8%) earned 1 point each for using biological correlation components [23293341]. Only three studies (10.3%) compared radiomics-based methods with the “gold standard” method (e.g., neuroradiologists’ decision to distinguish brain metastasis from other tumors) [141720]. Three studies (10.3%) provided potential applications of radiomics for the prediction of mutation status or progression-free survival through decision curve analysis or nomograms [263141].

In terms of domain 4, only one study (3.4%) performed a cut-off analysis to stratify patients with significantly different progression-free survival [31]. Among the 26 studies (90.0%) that used discrimination statistics or a resampling method, 21 studies (80.8%) used both discrimination statistics and a resampling method [131415161722232425262728293132333537383940]. Only two studies (6.9%) used calibration statistics with statistical significance [3141], with only one study also applying a resampling method [41].

In terms of domain 5, none of the studies were prospective or executed a cost-effectiveness analysis. For domain 6, only one study (3.4%) made the code publicly available, which defined image pre-processing and feature extraction.

Assessment of the RQS

The median RQS for the 25 radiomics studies was 3.0 (range, -6 to 12). The mean score was 3.4, which was 9.6% of the ideal score (Table 2, Fig. 2). When considering each domain, the mean score and percentage of the ideal score were the lowest in domain 2 (feature selection and validation) and highest in domain 4 (model performance). Among the six domains, domains 5 (high level of evidence) and 6 (open science and data) had mean scores of 0 and 0.03, respectively. In five studies (17.2%), neither feature selection nor validation was performed [1920213441].

Fig. 2

RQS assessment results according to the six key domains.

RQS = radiomics quality score

Completeness of Reporting a Radiomics-Based Multivariable Prediction Model Using the TRIPOD Checklist

Of the 35 items in the TRIPOD checklist, the mean number of items reported ± standard deviation was 17.2 ± 3.8 (range, 6–26). When we excluded “if relevant” and “if done” (item 5c) items from both the numerator and denominator, the adherence rate to this checklist was 60.3%. The adherence rates to each TRIPOD checklist are shown in Table 3.

Table 3

Adherence of Radiomics Studies to Individual Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis Checklist Items

		Total (35 Items)	Data (%)
Title and abstract (n = 29)
	Title	1. Identify developing/validating a model, target population, and the outcome	1 (3.4)
	Abstract	2. Provide a summary of objectives, study design, setting, participants, sample size, predictors, outcome, statistical analysis, results, and conclusions	23 (79.3)
Introduction (n = 29)
	Background and objectives	3a. Explain the medical context and rationale for developing/validating the model	26 (89.7)
		3b. Specify the objectives, including whether the study describes the development/validation of the model or both	24 (82.8)
Methods (n = 29 except for item 5c, for which n = 12)
	Source of data	4a. Describe the study design or source of data (randomized trial, cohort, or registry data)	26 (89.7)
		4b. Specify the key dates	25 (86.2)
	Participants	5a. Specify key elements of the study setting including number and location of centers	4 (13.8)
		5b. Describe eligibility criteria for participants (inclusion and exclusion criteria)	22 (75.9)
		5c. Give details of treatment received, if relevant (n = 12)	9 out of 12
	Outcome	6a. Clearly define the outcome, including how and when assessed	20 (69.0)
		6b. Report any actions to blind assessment of the outcome	0 (0)
	Predictors	7a. Clearly define all predictors, including how and when assessed	28 (96.6)
		7b. Report any actions to blind assessment of predictors for the outcome and other predictors	7 (24.1)
	Sample size	8. Explain how the study size was arrived at	11 (37.9)
	Missing data	9. Describe how missing data were handled with details of any imputation method	0 (0)
	Statistical analysis methods	10a. Describe how predictors were handled	29 (100)
		10b. Specify type of model, all model-building procedures (any predictor selection), and method for internal validation	27 (93.1)
		10d. Specify all measures used to assess model performance and if relevant, to compare multiple models (discrimination and calibration)	26 (89.7)
	Risk groups	11. Provide details on how risk groups were created, if done	3 (10.3)
Results (n = 29 except for item 14b, for which n = 20)
	Participants	13a. Describe the flow of participants, including the number of participants with and without the outcome. A diagram may be helpful	10 (34.5)
		13b. Describe the characteristics of the participants, including the number of participants with missing data for predictors and outcome	21 (72.4)
	Model development	14a. Specify the number of participants and outcome events in each analysis	26 (89.7)
		14b. Report the unadjusted association between each candidate predictor and outcome, if done (yes or no, n = 20)	13 (44.8)
	Model specification	15a. Present the full prediction model to allow predictions for individuals (regression coefficients, intercept)	0 (0)
		15b. Explain how to the use the prediction model (nomogram, calculator, etc.)	3 (10.3)
	Model performance	16. Report performance measures (with confidence intervals) for the prediction model	22 (75.9)
Discussion
	Limitations	18. Discuss any limitations of the study	26 (89.7)
	Interpretation	19b. Give an overall interpretation of the results	28 (96.6)
	Implications	20. Discuss the potential clinical use of the model and implications for future research	24 (82.8)
For validation (types 2a, 2b, 3, and 4) (n = 7 except for items 10e and17, for which n = 0)
	Methods-statistical analysis methods	10c. Methods-Statistical analysis methods: describe how the predictions were calculated	6 (85.7)
		10e. Describe any model updating (recalibration), if done (n = 0)	0 (0)
	Methods	12. Identify any differences from the development data in setting, eligibility criteria, outcome, and predictors	4 (57.1)
	Results	13c. Show a comparison with the development data of the distribution of important variables	4 (57.1)
	Results-model updating	17. Report the results from any model updating, if done (n = 0)	0 (0)
	Discussion-interpretation	19a. Discuss the results with reference to performance in the development data and any other validation data	2 (28.6)

Data are the number of studies.

Reporting of Image Pre-Processing and Radiomics Feature Extraction according to the IBSI Guidelines

Among the 29 studies, 20 (69.0%) performed signal intensity normalization. Only six studies (20.7%) performed bias-field correction (five performed N4 bias correction while one performed N3 bias correction), and nine (31.0%) performed isovoxel resampling, four (13.8%) performed skull stripping, and four (13.8%) performed gray-level discretization (Table 4, Fig. 3). The software used for the extraction of radiomics features included Pyradiomics (27.6%), Matlab (27.6%), LIFEx (13.8%), IBEX (6.9%), and others (13.8%). Two studies (6.9%) did not mention the software used. Among these, Pyradiomics and LIFEx adhered to the IBSI guidelines (Table 4, Fig. 3).

Table 4

Quality of Image Processing and Radiomics Feature Extraction according to the Image Biomarker Standardization Initiative Guidelines

Pre-Processing Performed		Number of Studies (%)
Bias-field correction		6 (20.7)*
Isovoxel resampling		9 (31.0)
Skull stripping		4 (13.8)
Gray-level discretization		4 (13.8)
Signal intensity normalization		20 (69.0)
Software for feature extraction
	Pyradiomics	9 (31.0)
	Matlab	8 (27.6)
	LIFEx	4 (13.8)
	IBEX	2 (6.9)
	Others	4 (13.8)
	N/A	2 (6.9)

*One study performed N3 bias correction, and the remaining five studies performed N4 bias correction. N/A = not available

Fig. 3

Reporting of image pre-processing and radiomics feature extraction according to Image Biomarker Standardization Initiative.

N/A = not available

DISCUSSION

In this study, radiomics studies on brain metastasis were evaluated with respect to scientific and reporting quality. Overall, the scientific and reporting quality of radiomics studies on brain metastases was suboptimal, with a low overall adherence rate for the RQS (50.0%) and TRIPOD checklist (60.3%) items. Only three studies performed external validation, indicating that most of the studies lacked generalizability. According to the RQS, none of the studies involved test-retest or phantom studies, prospective studies, or cost-effectiveness analyses. The majority of the studies did not address biologic correlations, compare radiomics-based methods to the gold standard method, state the potential clinical utility, perform cut-off analysis or calibration statistics, and provide open science and data. According to the TRIPOD checklist, the majority of studies did not report the title adequately. No study has described the blind assessment of outcomes and handling of missing data or presented a full prediction model. The detailed pre-processing steps were missing in most of the studies.

The basic adherence rate to the RQS items was low (50.0%). Specifically, the evaluation of feature robustness was insufficient, with only a few studies performing multiple segmentations, while no studies performed phantom studies or collected images at multiple time points. A considerable proportion of the studies executed feature extraction, which is essential to avoid overfitting [52]. However, only three studies performed external validation, whereas 18 studies did not perform validation at all. As validation is the key process to allow radiomics to be integrated as a reliable tool for clinical practice, future radiomics studies should include a validation step. Most studies did not describe biological correlations or comparisons with the gold standard. However, there are no established gold standards for the prediction of outcomes following radiosurgery or prediction of primary tumor types, except for pathologic confirmation, which inevitably decreases the adherence rate. Few studies have provided the potential clinical utility of radiomics and made the code publicly available. None of the studies were prospective and performed cut-off analysis, calibration statistics, and cost-effectiveness analysis, resulting in low adherence rates. Thus, the quality of radiomics studies on brain metastases has the potential to improve several categories of RQS.

In the reporting of radiomics studies according to the TRIPOD checklist, there were several highly problematic items. Only one study included the terms “development” and “validation” with the target population and outcome mentioned in the title. Similarly, most of the studies lacked the description of “development” and “validation” in the abstract and did not describe the specific study design. None of the studies reported handling of missing data. Furthermore, only a few studies reported their actions for blind assessment of predictors. These results of suboptimal reporting are in line with those of previous studies that investigated multivariable prediction model studies and oncologic studies according to TRIPOD [4753]. Meanwhile, since the current TRIPOD checklists focus on regression-based prediction model approaches, several studies that were irrelevant to the regression analyses were not applicable to the assessment of TRIPOD. Specifically, the checklists associated with model development and specification are based on regression analyses. The use of TRIPOD in artificial intelligence and machine learning (ML) studies is limited, as these studies frequently did not entail regression analyses [54]. Therefore, the development of a new version of the TRIPOD statement specific to ML (TRIPOD-ML) is currently in progress, focusing on ML prediction algorithms and building an established methodology for prediction research [54].

Adherence to the nomenclature of standard radiomics features and calculations described by the IBSI are important for improving the reproducibility of studies [5255]. However, only a few studies have reported the quality of radiomics studies that assessed adherence to the IBSI of each journal [4456]. In our study, we focused on the pre-processing steps in IBSI and reported the number of published radiomics studies on brain metastases following the guidelines. Bias-field correction aims to reduce low-frequency intensity nonuniformity [57], while isotropic resampling reduces directional biases [58] and gray-level discretization cluster pixels according to intensity values to enhance feature reproducibility [59]. Image intensity normalization is also a necessary step in increasing repeatability [60]. However, the majority of the studies did not execute all pre-processing steps and only performed a few. Radiomics studies on brain metastases should focus on pre-processing, which will increase the reproducibility of radiomics features.

Among the 20 studies that enrolled multiple brain metastases of more than one lesion per patient to constitute the dataset, only seven studies demonstrated the statistical analysis relevant to the handling of cluster-correlated data. Assignment of the respective metastases either to the validation set or to the training set for patients with multiple metastases was most frequently used to avoid bias from cluster correlation. As brain metastases tend to be multiple and multiple lesions from one patient may correlate with each other, specific considerations in the statistical analysis should be considered in future studies.

Our study had several limitations. First, the sample size was relatively small. Second, radiomics is still a developing imaging biomarker, and the suggested RQS and TRIPOD criteria may be too idealistic and strict. Phantom studies and multiple imaging acquisitions have rarely been applied to real-world practice. Additionally, it may be too extensive to provide the whole unadjusted association between each candidate predictor and outcome and the full prediction model. However, considering these details is necessary to improve the scientific and reporting quality of radiomics to enable the clinical translation of radiomics in the future.

In conclusion, the overall scientific and reporting quality of radiomics studies on brain metastases published during the study period was insufficient, and this low quality may hamper the use of radiomics in the clinical field. The RQS, TRIPOD checklist, and IBSI guidelines should be adhered to make radiomics a more robust decision-making tool in clinical practice.