Evaluation of the predictive ability of ultrasound-based assessment of breast cancer using BI-RADS natural language reporting against commercial transcriptome-based tests.

PURPOSE: The objective of this study was to assess the classification capability of Breast Imaging Reporting and Data System (BI-RADS) ultrasound feature descriptors targeting established commercial transcriptomic gene signatures that guide management of breast cancer.
MATERIALS AND METHODS: This retrospective, single-institution analysis of 219 patients involved two cohorts using one of two FDA approved transcriptome-based tests that were performed as part of the clinical care of breast cancer patients at Harbor-UCLA Medical Center between April 2008 and January 2013. BI-RADS descriptive terminology was collected from the corresponding ultrasound reports for each patient in conjunction with transcriptomic test results. Recursive partitioning and regression trees were used to test and validate classification of the two cohorts.
RESULTS: The area under the curve (AUC) of the receiver operator curves (ROC) for the regression classifier between the two FDA approved tests and ultrasound features were 0.77 and 0.65, respectively; they employed the 'margins', 'retrotumoral', and 'internal echoes' feature descriptors. Notably, the 'retrotumoral' and mass 'margins' features were used in both classification trees. The identification of sonographic correlates of gene tests provides added value to the ultrasound exam without incurring additional procedures or testing.
CONCLUSIONS: The predictive capability using structured language from diagnostic ultrasound reports (BI-RADS) was moderate for the two tests, and provides added value from ultrasound imaging without incurring any additional costs. Incorporation of additional measures, such as ultrasound contrast enhancement, with validation in larger, prospective studies may further substantiate these results and potentially demonstrate even greater predictive utility.

Introduction

Breast cancer continues to be a significant problem around the world, accounting for 29% of newly diagnosed cancers in women, with women in the United States having a 12.3% chance of developing breast cancer over their lifetimes [1]. Despite increased prevalence, breast cancer death rates have decreased by 36% from 1989 to 2012 due to screening and improved treatment regimens [2]. The use of ultrasound in screening, particularly in women younger than 45 years of age with fibroglandular, dense tissue, has improved cancer detection sensitivity in comparison to mammography alone [3]. The ability of ultrasound to detect small (< 1 cm), mammographically occult lesions [4] as well as potentially downgrading breast masses [5] provide examples of how ultrasound can further guide clinical decision making in cancer screening and detection.

The clinical utility of diagnostic modalities that provide more than just an assessment of malignant versus benign, but rather prognostic information has driven the development of transcriptome-based tests. The OncotypeDX® (Genomic Health Inc, Redwood City, CA) test calculates a recurrence score on a scale from 0–100 (higher scores reflect higher risks of recurrence) for early-stage hormone receptor-positive breast cancer, in order to assess the potential benefit from chemotherapy after breast cancer surgery (7). This test is currently included in the National Comprehensive Cancer Network (NCCN) and American Society of Clinical Oncology (ASCO) guidelines. The MammaPrint® (Agendia Inc, Irvine, CA) test calculates a binary (high versus low) recurrence risk score based upon a 70-gene expression profile, and also guides chemotherapy treatment decisions [6, 7]. In patients with an OncotypeDX® score less than 11 or a MammaPrint® low-risk score, chemotherapy can be omitted, and anti-estrogen therapy can be administered alone.

The development of the Breast Imaging-Reporting and Data System (BI-RADS) [8] by the American College of Radiology (ACR) to include standardized language and descriptors in the reporting of breast imaging has proven very successful in standardizing communication among radiologists, oncologists, and surgical oncologists in the decision-making and management of breast lesions [9]. Other fields have successfully developed similar approaches in prostate, lung, and liver lesions (PI-RADS, Lung-RADS, LI-RADS) [10-13]). Although BI-RADS has an established role in breast imaging and interdisciplinary communication (e.g. medical oncologists, surgical oncologists, and internists), with the ongoing development of new treatments and and genomic-based tests, we need to be able to derive as much information as possible from clinical imaging measurements [14, 15]. The BI-RADS categorization of sonographic findings has sufficient positive predictive value to be used as a predictor of malignancy [16]. Along a parallel track, correlations between cross-sectional imaging and molecular profiling have identified potential surrogate roles as imaging biomarkers in a variety of diseases, including breast cancer [17-20].

The clinically established use of ultrasound BI-RADS reporting affords an opportunity to assess how much information can be derived from imaging alone, and whether various descriptors may supplement prognostic gene panels. We sought to assess the potential of breast ultrasound feature descriptors to identify cohorts that would or would not benefit from chemotherapy using structured natural language processing (NLP) of BI-RADS terminology targeting established transcriptomic assays (OncotypeDX® and MammaPrint®).

Materials and methods

This retrospective study received approval with waiver of written patient consent, from the Institutional Review Board at the Los Angeles Biomedical Research Institute at Harbor-UCLA, and is Health Insurance Portability and Accountability Act (HIPAA)-compliant. All data were fully anonymized for subsequent study analyses.

Patient demographics, histological classification of biopsied samples, receptor status (when available), and transcriptomic-test were extracted from the electronic medical record system. Breast cancer patients were included in this study if they had undergone testing with either of the two genomic assays (OncotypeDX® and MammaPrint®) between April 2008 and January 2013, and were divided into two cohorts depending on which assay was performed. Ultrasound reports for each patient at the time of initial cancer diagnosis were stored in a collection of Microsoft Word® documents (.docx).

The diagnostic ultrasound studies, interpreted and reported by fellowship trained, board-certified breast radiologists (each with at least 4 years experience), with strict adherence to BI-RADS standards [8], were parsed with custom scripts focusing on BI-RADS descriptive terminology. Four classes of ultrasound BI-RADS field descriptors were consistently reported across all reports, accounting for the features that were most consistently described: margins, echogenicity, internal echo pattern, and retrotumoral phenomenon [8]. The cumulative collection of these descriptive terms from all reports defined the dictionary of terms.

The BI-RADS ultrasound report files were parsed as regular expressions focusing on the FINDINGS and IMPRESSION sections of the diagnostic reports using Python version 2.7.10 (https://www.python.org/). The FINDINGS were parsed according to size (‘Longitudinal’, ‘Transverse’, ‘Anteroposterior’), ‘’, ‘’, ‘’, ‘’, and ‘’. The reported BI-RADS score was extracted from the IMPRESSION section of the reports. ‘’ values were, ‘ill-defined’, ‘irregular’, ‘smooth’, ‘lobulated’, or ‘N/A’. ‘’ values included, ‘hypoechoic’, ‘isoechoic’, ‘anechoic’. ‘’ values included, ‘homogeneous’ and ‘heterogeneous’. ‘’ values included, ‘small ca++’, ‘large ca++’, and ‘none’. ‘’ values included, ‘irregular posterior shadowing’ and ‘posterior shadowing’. Since the sizes of the masses were not consistently measured in all three dimensions for the majority of the cases, tumor size measurement was discarded from subsequent analyses (S1 and S2 Tables).

The ability of the BI-RADS ultrasound features to predict risk score classification by the OncotypeDX® and MammaPrint® transcriptome-based tests was assessed using recursive partitioning and regression trees (CART) using analysis of variance (ANOVA). Training to testing validation sets were split in 3:1 ratios, randomly split with the Mersenne-Twister random number generator (seed = 202). Twenty-fold cross-validation was performed with a minimum of 10 observations per splitting-node and a minimum of 6 observations per terminal node. We defined an area under the curve (AUC) of the receiver operator characteristic (ROC) curve to be at least 0.9 in order to be considered as a candidate classifier to potentially compete with molecular tissue markers and an AUC of 0.6 to be a reportable but not competitive as an radiogenomic surrogate. Statistical significance for all portions of the study were defined as p < 0.05. The analyses were performed using R (https://cran.r-project.org/).

Results

Cohort characteristics

In the two cohorts of patients, 149 had undergone testing with the OncotypeDX® assay and 70 had the MammaPrint® assay. Both of the genetic test cohorts showed a significant association with tumor grade (Tables 1 and 2) as determined by ANOVA (p <0.05), as expected. There was no significant association with age, race or tumor histology with OncotypeDX® or MammaPrint® classifications (Tables 1 and 2). There were significant negative correlations between ER and PR for OncotypeDX® and MammaPrint® with weaker correlation coefficients for the latter (S3 and S4 Tables), grossly consistent with other published reports [21, 22].

Table 1

Summary statistics for the OncotypeDX® cohort by age (mean +/- sd years), grade (mean +/- sd), race (A: Asian, AA: African American, C: Caucasian, F: Philipina, H: Hispanic, ME: Middle Eastern, -: not documented), and histology (IDC: invasive ductal carcinoma, ILC: invasive lobular carcinoma, IDC/ILC: invasive ductal carcinoma with lobular features).

The bottom row highlights ANOVA p-values. * indicates statistical significance (p <0.05).

	Age (years)	Grade*	Race	Histology
	54.2+/-9.4	1.8+/-0.70	A: 11	IDC: 56
			AA: 13	ILC: 8
			C: 12	mucinous: 5
			F: 3	other: 1
			H: 30
			ME: 1
p-value	0.6	0.0000002	0.55	0.64

Table 2

Summary statistics for the MammaPrint® cohort by age (mean +/- sd years), grade (mean +/- sd), race (A: Asian, AA: African American, C: Caucasian, F: Philipina, H: Hispanic, ME: Middle Eastern, -: not documented), and histology (IDC: invasive ductal carcinoma, ILC: invasive lobular carcinoma, IDC/ILC: invasive ductal carcinoma with lobular features).

The bottom row highlights ANOVA p-values. * indicates statistical significance (p <0.05).

	Age (years)	Grade*	Race	Histology
	51.2+/-10.3	2.22+/00.68	A: 15	IDC: 132
			AA: 38	IDC/ILC: 3
			C: 21	ILC: 11
			F: 5	mucinous: 1
			H: 69	other: 2
			-: 1
p-value	0.37	0.00023	0.16	0.23

Ultrasound imaging features

Collectively the 219 sonographically detectable masses characterized according to five semantic features (see Materials and Methods) and were assessed for a possible 144 different classifications of the masses. All BI-RADS scores were 3 or greater, as would be expected, based upon interpretation and biopsy recommendations of BI-RADS [8, 23]. Since the ‘echogenicity’ of the masses was described as ‘hypoechoic’ in 217 out of 219 masses (with one described as ‘anechoic’ and the other as ‘isoechoic’), the descriptor was removed from subsequent analyses. Following removal of the echogenicity feature there remained 48 possible unique sonographic classifications of the masses from 4 different features (‘margins’, ‘internal echo pattern’, ‘internal shadowing’, and ‘retrotumoral phenomenon’).

Ultrasound BI-RADS classifiers

The CART classification trees alongside their corresponding ROC curves are presented in Figs 1 and 2, with AUCs of 0.77 (OncotypeDX®, Fig 1) and 0.65 (MammaPrint®, Fig 2). Incorporation of tumor grade information into the regression analysis did not improve the predictive value of the classification trees. Mass margins and retrotumoral phenomena appear at the top of the classification tree for both tests. Additionally, for these cohorts, although there were four different possible values for the tumor margin feature, the classification separation boundaries occurred along binary lines (smooth versus non-smooth and smooth/lobulated versus irregular), which is concordant with “benign versus malignant” suspicion in the BI-RADS based assessment [8].

Fig 1

OncotypeDX® classification based upon BI-RADS ultrasound feature descriptors for hypoechoic breast masses.

A) The classification tree involves two features, the margins of the tumor and the type of shadowing phenomenon for the tumor. B) The area under the ROC curve was 0.77 with 52 subjects in the training set and 18 in the testing set.

Fig 2

MammaPrint® classification based upon BI-RADS ultrasound feature descriptors for hypoechoic breast masses.

A) The classification tree involves three BI-RADS ultrasound features, the type of shadowing phenomenon for the tumor, the margins of the tumor, and the internal echo pattern. B) The area under the ROC curve was 0.65 with 111 subjects in the training set and 38 in the testing.

Discussion

In this study we analyzed the potential for BI-RADS ultrasound descriptors to track the OncotypeDX® or MammaPrint® classifications using NLP in conjunction with classification and regression trees, resulting the identification of three sonographic features (‘margins’, ‘retrotumoral’ and ‘internal echoes’) that may provide non-invasive correlates of the transcriptome profiles. Through the use of specific terminology and a well-defined vocabulary with systematic report recommendations, ultrasound BI-RADS has been an effective mechanism to provide consistent, transparent, and unambiguous recommendations to referring physicians and their patients to interpret the results of breast imaging studies (14). The use of a structured language and well-defined vocabulary is particularly useful since one challenge with respect to quantitative imaging of breast ultrasound is the non-tomographic, operator dependent nature of image acquisition, resulting in variation in acquisition with respect to anatomic planes as well as ultrasound parameters (e.g. different transducer probes, use of harmonics, differences in gain and time gain compensation, focal zones, etc). These sources of variation limit the application of automated or semi-automated quantitative imaging approaches to ultrasound. However, the BI-RADS descriptions of sonographically detectable masses provide an opportunity to use NLP based methods in order to identify features with prognostic and therapeutic implications and correlates with other diagnostic tests, such as the transcriptomic tests evaluated in this study.

The assessments of ultrasound imaging correlates using standardized language and descriptors compared to their relationship to the FDA approved tissue-based transcriptomic tests (OncotypeDX® and MammaPrint®) provide a biological context to interpret the transcriptomic measurements. For example, hypoechoic masses are concerning for malignancy, and it is such a common observation, that there is no further prognostic information to be derived from it, thus although it is an important BI-RADS feature, it is not an important prognostic predictor (since the ROC curves in Figs 1 and 2 implicitly assume a priori that the masses are hypoechoic). Conversely the CARTs enable evaluation of multiple features that portend higher or lower risk in which one feature may be more suspicious for malignancy but another feature is not (e.g. irregular margins but no retrotumoral phenomena, Figs 1 and 2).

Added value without added costs

The search for imaging correlates of transcriptomic tests can be classified to serve as non-invasive 1) alternatives, 2) complementary, or 3) supplementary roles to more invasive, biopsy-dependent tests. Many radiogenomic applications focus on the first point, which is beneficial when an imaging test is less expensive or cheaper than a tissue-based test. For example, recently MRI has been explored as a radiogenomic surrogate for some of these tests. Unfortunately classification of the breast MRI features do not achieve an accuracy that can reasonably compete or provide surrogacy for the established transcriptomic tests [24, 25]. Additionally the cost of MRI scans are non-trivial and rival the MammaPrint® and OncotypeDX® test costs (doubling the cost without providing substantive additional information). However, the ultrasound-based assessments used in this study was focused on the latter the second and third classifications (complementary or supplementary to tissue-based tests).

Although the role of OncotypeDX® and MammaPrint® in management of breast cancer have been promising, there is a non-negligible cost for these tests, in the $3000-$4000 range. Recent evidence suggests that this may not be cost effective [26], thus it would be beneficial to have a low- or no-cost non-invasive screening test, to determine whether there would be added value from these tests. In a similar vein, the cost of bilateral breast MRI is on average $3000, nearly ten-fold the cost of breast ultrasound [27, 28]. In contrast, part of the established diagnostic evaluation of breast masses involves the breast ultrasound, so there is no additional cost burden. Given the invasive nature of tissue-based tests and the costs associated with tissue biopsies, processing and analysis, in addition to the costs of commercial tests [29], the use of ultrasound imaging information to help identify cases in which transcriptomic tests may alter patient management, provides a potential means to make the transcriptomic tests more cost effective.

The principle limitations of this study include the sample size, the number of available features, and the lack of quantitative measurements. The difference in the sample size between the two cohorts may in part explain why the performance of the OncotypeDX® predictor exceeded the MammaPrint® predictor and highlights the point that, with larger sample sizes and prospective evaluation at different hospitals, classification performance may improve. Although both tests provide guidance for treatment (i.e. low scores for both tests can justify sole anti-estrogen treatment), the MammaPrint® test is applicable to estrogen receptor positive and negative women, whereas the Oncotype® test has been applied demonstrated to estrogen positive cohorts. The difference in the clinical applications may also provide an explanation for the difference in the performance of the ultrasound feature descriptors. For example, ‘internal echoes’ may not have any predictive significance in estrogen positive women, although testing this in an independent cohort is warranted before drawing such a conclusion.

Despite the aforementioned limitations, the AUC of the ROC curves for the regression decision trees suggest that there is a role for the use of ultrasound BI-RADS descriptors beyond just a probability assessment for malignancy versus benignity. Incorporation of additional features such as color Doppler flow and ultrasound contrast agents may also further improve the molecular predictive value of breast mass sonography. Furthermore, new contrast agents [30] may provide further improvements in the specificity of the classifier, providing more precise diagnostic and prognostic value. Future studies may also evaluate other genomic tests, such as the Breast Cancer Index, EndoPredict, Mammostrat, and Prosigna Breast Cancer Prognostic Gene Signature Assay, for any potential correlation with imaging studies as well [31-34].

Conclusions

Although BI-RADS was developed to guide decision making in breast imaging studies and to assess the probability of malignancy, the use of a standardized lexicon and descriptive features for ultrasound masses provided the opportunity to use NLP to construct regression trees classifiers for prognostic FDA approved transcriptome-based tissue tests. Using the structured language of ultrasound BI-RADS, we assessed the ability of ultrasound feature characteristics to predict OncotypeDX® and MammaPrint® transcriptome-based classifications across 219 patients. Interestingly, NLP classifications of the BI-RADS reports were able to generate classification trees that were concordant with the transcriptomic tests. Ultrasound findings, notably the ‘retrotumoral’ and ‘margins’ features, if abnormal, may help provide justification to obtain one of the transcriptomic tests; future multi-institutional prospective studies will be important in determining if these observations persist in larger cohorts.

Table of imaging features, IHC, race, age, and OncotypeDX® scores.

(XLSX)

Click here for additional data file.

Table of imaging features, IHC, race, age, and MammaPrint® scores.

(XLSX)

Click here for additional data file.

Summary of correlation between OncotypeDX® and IHC markers.

(XLSX)

Click here for additional data file.

Summary of correlation between MammaPrint® and IHC markers.

(XLSX)

Click here for additional data file.

21 Jun 2019

PONE-D-19-15654

Evaluation of the predictive ability of ultrasound-based assessment of breast cancer using BI-RADS natural language reporting against commercial transcriptome-based tests

PLOS ONE

Dear Dr. Jamshidi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Aug 05 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Azra Alizad, MD

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2.  In your ethics statement in the manuscript and in the online submission form, please provide additional information about the patient records used in your retrospective study. Specifically, please ensure that you have discussed whether all data were fully anonymized before you accessed them and/or whether the IRB or ethics committee waived the requirement for informed consent. If patients provided informed written consent to have data from their medical records used in research, please include this information.

3.

We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

Additional Editor Comments (if provided):

Thank you for sending your great work to PLOS ONE. Please respond to the reviewers’ comments and submit your revisions to PLOSONE.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Summary:

This is an interesting paper that explores an aspect of radiogenomics. I think it certainly adds to the current literature on this subject.

Introduction:

1)Consider changing the first paragraph to focus more on the value of US in breast cancer. It is too generic for the focus of this paper.

Methods:

2) It would be important to know what reporting system is used for US …is it powerscribe where the radiologist can describe any terminology they want (compliant with BIRADS or not) or is it a program like MagView where the radiologist is somewhat forced to use BIRADS terms?

3) Need to mention how many radiologists reported on US and what was their experience in terms of years.

4) 4th paragraph of methods: capitalize where appropriate

Results:

Easy to follow

Discussion

5) I am not sure the entire discussion on costs is applicable. It is ok to mention it in a few sentences but such a long section seems out of scope for this paper.

Reviewer #2: The premise of this study is very intriguing and I would recommend publication after consideration of minor revisions. It is clear that while this is an interesting study with potentially very interesting implications, substantially more data from a very large, multicenter, prospective trial, will be needed to be confident that an AUC of 0.6 truly provides sufficient accuracy for ultrasound features to be considered a supportive adjunct to guide the decision of whether molecular based tissue testing is warranted. The transcriptome tests guide critical clinical decisions regarding the benefit of administering vs. not administering chemotherapy and stratify high risk patients from lower risk patients. A significantly greater level of confidence from a much larger study(ies) will be needed before imaging characteristics of tumors can influence clinical decisions about whether to use a tissue-based tests to clarify the need for chemotherapy. While the authors do not purport to draw a definite conclusion, more definitive language from the authors about the limitations of this study, which in essence provides preliminary data, and the need for additional robust data is strongly recommended.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

5 Aug 2019

We thank the reviewers for their evaluation of the manuscript and constructive comments. The manuscript has been revised to reflect the changes that are described below.

Reviewer #1: Summary:

This is an interesting paper that explores an aspect of radiogenomics. I think it certainly adds to the current literature on this subject.

Thank you, we too believe that this initial evaluation may prompt further studies to extract further information from ultrasound studies, with the potential to serve as useful adjuncts in motivating the need for tissue based transcriptome studies, in the appropriate cohorts.

Introduction:

1) Consider changing the first paragraph to focus more on the value of US in breast cancer. It is too generic for the focus of this paper.

This is a great point, since the focus of the manuscript is on ultrasound. We have added some additional references and statements highlighting the role and relevance of ultrasound.

Methods:

2) It would be important to know what reporting system is used for US …is it powerscribe where the radiologist can describe any terminology they want (compliant with BIRADS or not) or is it a program like MagView where the radiologist is somewhat forced to use BIRADS terms?

There were template reports, however a dictation system was used (Powerscribe, Nuance). All of the reports used BI-RADS terminology (as provided in the Supporting Information).

3) Need to mention how many radiologists reported on US and what was their experience in terms of years.

The studies were read by three sub-specialty trained radiologists (all with at least 4 years of experience). Methods were updated to reflect this.

4) 4th paragraph of methods: capitalize where appropriate

Done.

Discussion

5) I am not sure the entire discussion on costs is applicable. It is ok to mention it in a few sentences but such a long section seems out of scope for this paper.

We have abbreviated the paragraph regarding the cost benefits and trade-offs.

Reviewer #2: The premise of this study is very intriguing and I would recommend publication after consideration of minor revisions. It is clear that while this is an interesting study with potentially very interesting implications, substantially more data from a very large, multicenter, prospective trial, will be needed to be confident that an AUC of 0.6 truly provides sufficient accuracy for ultrasound features to be considered a supportive adjunct to guide the decision of whether molecular based tissue testing is warranted. The transcriptome tests guide critical clinical decisions regarding the benefit of administering vs. not administering chemotherapy and stratify high risk patients from lower risk patients. A significantly greater level of confidence from a much larger study(ies) will be needed before imaging characteristics of tumors can influence clinical decisions about whether to use a tissue-based tests to clarify the need for chemotherapy. While the authors do not purport to draw a definite conclusion, more definitive language from the authors about the limitations of this study, which in essence provides preliminary data, and the need for additional robust data is strongly recommended.

Thank you for your comments and review of the manuscript.

We have added additional text to highlight the fact that additional studies with larger cohorts will be needed to further test the potential utility of these features (particularly multi-institutional, given the potential for imager-dependent variability in ultrasound exams).

We feel it is worth mentioning however, given that the transcriptomic tests are dependent on insurance company approval and the ability of a patient to have the test is driven in large part by whether they have “good” or “bad” insurance, we feel that any additional data (especially if there is no additional cost associate with it) that can help indicate potential benefit from the transcriptomic test provides some value (especially considering some published studies promoting MR based “radiogenomic” predictors compared to transcriptomic tests have argued for the role of surrogacy with AUCs < 0.8 – to that end, we feel that an AUC ~ 0.7 for an adjunctive test is reasonable).

Submitted filename: responses.1.docx

Click here for additional data file.

21 Oct 2019

PONE-D-19-15654R1

Evaluation of the predictive ability of ultrasound-based assessment of breast cancer using BI-RADS natural language reporting against commercial transcriptome-based tests

PLOS ONE

Dear Dr. Jamshidi,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Dec 05 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Azra Alizad, MD

Academic Editor

PLOS ONE

Additional Editor Comments (if provided):

Thank you very much for your submission and your response to the reviewers' comments.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #3: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thanks for addressing all the comments. No further comments. I think this will be a valuable manuscript.

Reviewer #3: This is an interesting look at the breast US features, as they correlate with breast cancer prognostic and predictive markers. It is nicely presented. However, I do take issue with the determination, under the method section, that " AUC of 0.6 to be considered as a supportive adjunct to help guide the decision of whether a molecular based tissue test is warranted"(page 37). I am not sure how this conclusion was reached, since this is not a study assessing the need for employing a prognostic marker, but rather a study looking at correlating a prognostic marker with imaging features. The decision on when to use a prognostic marker resides with the oncologist and is influenced by many patient and tumor characteristics and I do not believe it can be made on imaging features alone, at this point in time.

The conclusion that the US features employed here can be used to predict when to use a prognostic marker is, therefore, far reaching and should be revised.

What this study demonstrates is a correlation between the US features and Oncotype DX and, to a lesser degree, to the MammaPrint, but the correlation is moderate at best. Therefore, it can not be used in place of these tests to predict who needs chemotherapy.

The study is promising in that it shows a correlation and I agree with the conclusion that adding other imaging features to the ones evaluated in this study may improve the AUC and hopefully achieve the >0.9 AUC.

The Discussion is very lengthy, as mentioned by another reviewer. I would consider moving some of the discussion under the introduction part, for example the costs of MRI and prognostic markers and focus the discussion on the findings of this study and how it compares to other studies, like you did under the Conclusion section.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

6 Nov 2019

Thank you for the comments. We have enclosed a pdf with our point by point responses.

Submitted filename: comments.1.pdf

Click here for additional data file.

4 Dec 2019

Evaluation of the predictive ability of ultrasound-based assessment of breast cancer using BI-RADS natural language reporting against commercial transcriptome-based tests

PONE-D-19-15654R2

Dear Dr. Jamshidi,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Azra Alizad, MD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Thank you for submitting your interesting work to PLos One. I gladly recommend your paper for publication.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: I have no additional comments or concerns, and I am satisfied with the response to my previous comments.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

12 Dec 2019

PONE-D-19-15654R2

Evaluation of the predictive ability of ultrasound-based assessment of breast cancer using BI-RADS natural language reporting against commercial transcriptome-based tests

Dear Dr. Jamshidi:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Azra Alizad

Academic Editor

PLOS ONE