A semi-automated machine-learning based workflow for ellipsoid zone analysis in eyes with macular edema: SCORE2 pilot study.

BACKGROUND AND
OBJECTIVE: To develop a semi-automated, machine-learning based workflow to evaluate the ellipsoid zone (EZ) assessed by spectral domain optical coherence tomography (SD-OCT) in eyes with macular edema secondary to central retinal or hemi-retinal vein occlusion in SCORE2 treated with anti-vascular endothelial growth factor agents.
METHODS: SD-OCT macular volume scans of a randomly selected subset of 75 SCORE2 study eyes were converted to the Digital Imaging and Communications in Medicine (DICOM) format, and the EZ layer was segmented using nonproprietary software. Segmented layer coordinates were exported and used to generate en face EZ thickness maps. Within the central subfield, the area of EZ defect was measured using manual and semi-automated approaches via a customized workflow in the open-source data analytics platform, Konstanz Information Miner (KNIME).
RESULTS: A total of 184 volume scans from 74 study eyes were analyzed. The mean±SD area of EZ defect was similar between manual (0.19±0.22 mm2) and semi-automated measurements (0.19±0.21 mm2, p = 0.93; intra-class correlation coefficient = 0.90; average bias = 0.01, 95% confidence interval of limits of agreement -0.18-0.20).
CONCLUSIONS: A customized workflow generated via an open-source data analytics platform that applied machine-learning methods demonstrated reliable measurements of EZ area defect from en face thickness maps. The result of our semi-automated approach were comparable to manual measurements.

RCT Entities:   Population Interventions Outcomes

Introduction

Optical coherence tomography (OCT) scans are routinely used in clinical practice for monitoring therapeutic efficacy in patients with macular edema. Central retinal thickness measurements obtained from OCT are a key outcome measure for clinical trials; however, central retinal thickness is not considered a biomarker or a predictor for visual recovery because of its weak correlation with visual acuity (VA).[1] The ellipsoid zone (EZ), previously named the photoreceptor inner segment-outer segment junction, is visualized as a hyperreflective layer in the outer retina on spectral domain optical coherence tomography (SD-OCT) images.[2] SD-OCT enables in vivo high-resolution, cross-sectional imaging of the retina, permitting qualitative and quantitative assessment of EZ integrity, which has been correlated with VA in multiple retinal diseases,[3, 4] including retinal vein occlusion (RVO).[5-7]

Many studies examining the EZ layer are qualitative, describing abnormalities of the layer based on signal intensity.[8, 9] Current methods for quantitative assessment of the EZ involve segmentation of the retinal layers using manual or semi-automated techniques.[10, 11] This generates EZ layer thickness measurements with an Early Treatment Diabetic Retinopathy Study (ETDRS) grid. An alternative and more intuitive method would be to assess the area of EZ defect or loss using en face thickness maps created from the segmented layers.[5] The en face maps are two dimensional and present areas of EZ absence, i.e. very low thickness areas are dark regions against a brighter background of normal EZ thickness. Manual segmentation of the EZ is laborious and time intensive.[12] Therefore, we sought to utilize accessible, machine-learning methods to facilitate the identification of these regions. We trained a classifier model using the Trainable Weka Segmentation plugin[13] of Fiji [14, 15], an open-source image processing software package specifically designed for scientific image analysis, to detect the areas of EZ defect. Using this trained classifier, we then applied it within a novel, customized workflow using the open-source software platform Konstanz Information Miner (KNIME)[16] to evaluate the EZ assessed by SD-OCT generated en face thickness maps in eyes with macular edema secondary to central retinal vein occlusion (CRVO) or hemi-retinal vein occlusion (HRVO).

Methods

Participants

Study data were obtained from the tudy of mparative Treatments for tinal Vein Occlusion (SCORE2), a multicenter, prospective, randomized non-inferiority trial of eyes with macular edema secondary to CRVO or HRVO comparing intravitreal anti-vascular endothelial growth factor agents (anti-VEGF) bevacizumab vs. aflibercept (Clinicaltrials.gov identifier NCT01969708).[17] The study was approved by institutional review boards (IRB) associated with each center (University of Wisconsin Madison IRB Number 2014-0256-CR006) and adhered to the tenets of the Declaration of Helsinki. All participants provided written informed consent. The SCORE2 design and methods have been previously described in detail.[18] In summary, 362 participants were randomized to receive either intravitreal bevacizumab or aflibercept. The study visits were conducted per protocol with treatment provided per protocol from baseline through month 12, and then at the discretion of the investigator thereafter. Inclusion criteria were center-involved macular edema defined as central subfield thickness of ≥300 μm (or ≥320 μm if measured with Heidelberg Spectralis Machine).

Seventy-five participants were randomly selected from the SCORE2 baseline dataset, which represented approximately 20% of the total number of trial subjects. Participants were stratified based on baseline VA into three groups, including good (73–59 letters: 20/40-20/63), moderate (58–49 letters: 20/80-20/100), and poor (48–19 letters: 20/125-20/400). Twenty-five subjects were randomly selected from each stratum.

SD-OCT image acquisition

All SD-OCT images were acquired by certified technicians using the SCORE2 reading center (Fundus Photograph Reading Center, University of Wisconsin) approved protocol with either Carl Zeiss Meditec Cirrus (Carl Zeiss Meditec, Dublin, CA) or Heidelberg Spectralis (Spectralis Heidelberg Engineering, Heidelberg, Germany) OCT machine.[18] The Zeiss macular volume scans were 6 mm and comprised of 512 A-scans and 128 B-scans, and the Heidelberg scans were 20 x 20 degrees and comprised of 512 A-scans and 97 B-scans. SD-OCT images were evaluated at baseline, month 1, month 6, and month 12 for all participants.

Segmentation of EZ layer

The SD-OCT macular volume scans were received in proprietary formats at the central reading center and converted to Digital Imaging and Communications in Medicine (DICOM) format.[19] The EZ layer was segmented in the central subfield (CSF) using custom segmentation software developed using MATLAB (The Mathworks Inc, Natick, Massachusetts, USA.).[20] The CSF consisted of 17 (Spectralis) or 23 (Cirrus) B-scans. The EZ layer is typically visible as a hyperreflective line between the external limiting membrane and the retinal pigment epithelium (RPE). The inner border of the second outer hyperreflective band (EZ layer) and the inner border of the third outer hyperreflective band (RPE) were selected as the EZ layer boundaries ().[2]

Semi-automated analysis of ellipsoid zone (EZ) defect using en face approach.

A. B-scan with segmentation lines (green) from the top of the EZ layer to the top of the retinal pigment epithelium highlighting EZ defect (vertical red lines). B. Distances between segmented lines were linearly interpolated to form en face thickness map showing intact (white/grey pixels ^) and defective (black/dark pixels *) EZ. C. Application of Trainable Weka Segmentation (TWS) classifier to entire thickness map identifies regions of intact (green) and absent (red) EZ. D. The area of absent EZ (red) within the central subfield (green circle) was measured on the en face thickness map via the KNIME workflow.

Generation of En face thickness maps

EZ layer xy coordinates were exported as Extensible Markup Language (XML) format, and those files were used to generate en face thickness maps for selected layers via linear interpolation within a customized workflow in the open-source data analytics platform KNIME, version 3.7.2 (). Areas of EZ defect appear as dark areas on the thickness maps compared to bright areas with normal EZ.

Ellipsoid zone area analysis

A machine-learning tool, the Trainable Weka Segmentation (TWS) plugin, was used to generate a classifier to segment regions of EZ defect within generated thickness maps automatically (. This tool is a Fiji plugin that combines a collection of machine-learning algorithms with a set of selected image features to produce pixel-based segmentations.[13] Default segmentation settings in TWS were applied, though the maximum sigma was set to 32. The training features included were Gaussian blur, Hessian, Membrane projections (directional filtering), Sobel filter, Difference of gaussians, and Variance. The classifier applied was Fast Random Forest, a multi-threaded version of random forest by Fran Supek.[21] An estimated subset (~10%) representing the heterogeneity of the thickness maps of the entire dataset were selected to train the classifier to detect areas of EZ defect. Once trained, where the appropriate regions were reliably and reproducibly segmented, the classifier was then applied to the larger dataset for automatic segmentation and area measurements via the customized KNIME workflow (). All scans were obtained from the SCORE2 dataset and were therefore from eyes affected by either CRVO or HRVO. The KNIME workflow was applied using a standard desktop computer (Processor: Intel® Core™ i5-45900 CPU @ 3.30 GHz; Memory: 8.00 GB; System Type: 64-bit Operating System).

KNIME workflow.

There are 5 distinct sections (metanodes) of the customized KNIME workflow including: (A) selection of input parameters, which includes a source folder containing .xml layer files, an .xls file including xy coordinates for central subfield (CSF) positions, the selection of two layers for thickness map generation, a classifier .model file generated via the Trainable Weka Segmentation (TWS) plugin of Fiji, the path to local Fiji installation, the selection of the segmentation method (either ‘automatic’ or ‘manual’), and the output folder for all results files; (B) generation of the en face thickness maps and masks of CSF regions; (C) either ‘automatic’, using the TWS classifier, or ‘manual’ segmentations; (D) segmentation area measurements, which are calculated only within the CSF region; and finally (E) an interactive segmentation view for a visual check of all segmentations made via the workflow.

For manual measurements, the EZ boundary was manually traced within the KNIME workflow, and only the areas of EZ defect within the CSF were quantified. CSF xy coordinates were exported from the custom MATLAB software and used to overlay the CSF regions unique to each image (). All thickness maps and area measurements were generated via the same methods for both automatic and manual approaches. The only difference between the semi-automated and manual workflows was the segmentation method applied (). The KNIME workflow allowed visual inspection of all segmented regions with image overlays of the area of EZ defect on the en face thickness map (). All images were analyzed and reviewed by two masked graders and study authors (T.E. and C.P.). Graders reviewed images independently and were masked to segmentation results. Inter-rater reliability and agreement were assessed for manual EZ defect measurements.

The minimum area of EZ defect was defined as 0.004 mm2 and the maximum area of EZ defect was 0.78 mm2 (based on the area of the CSF). The minimum area was selected based on the lowest limit of area measurability used within reading center grading protocols (e.g. drusen circle C0 established by the Age-Related Eye Disease Study (AREDS) Research Group).[22]

Statistical analysis

We investigated the reliability and agreement of the manual and semi-automated approaches for determining the area of EZ defect within the CSF. Reliability was determined by calculating the intra-class correlation coefficient (ICC).[23] Agreement was determined by calculating the average bias between the manual and semi-automated measurements using the Bland-Altman method.[24] If the average bias does not exceed the 95% confidence interval (CI) of the limits of agreement (LOA), then the methods do not disagree and can be used interchangeably.[25] ICC and average bias were calculated to assess inter-rater reliability and agreement for manual EZ defect measurements. Area measurements were not normally distributed. Therefore, the non-parametric Wilcoxon Signed-Ranks Test was used to compare the area measurements between semi-automated and manual methods. A two-tailed p-value less than 0.05 was considered significant for all hypothesis testing. Statistical analysis was performed using R v3.6.1 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Of the 75 randomly selected study eyes, one did not have follow-up images after baseline, resulting in 74 study eyes available for analysis. Assessment of the EZ layer was performed at baseline, month 1, month 6, and month 12; however, EZ assessment was not possible on baseline scans due to a >90% rate of ungradable images resulting from signal blockage by hemorrhage or fluid (). Therefore, only SD-OCT images at months 1, 6, and 12 were analyzed. Of the selected study visits, SD-OCT images were missing from 7 study visits and the retinal layers were not visible for segmentation in 31 eyes due to signal blockage from hemorrhage or fluid (21 at month 1, 6 at month 6, and 4 at month 12), resulting in a total of 184 gradable volume scans for analysis (53 at month 1, 65 at month 6, and 66 at month 12).

(A) Representative center point SD-OCT scan deemed ungradable at baseline for ellipsoid zone (EZ) assessment. EZ was considered ungradable when the retinal pigment epithelium was not visible due to signal blockage by hemorrhage and edema. (B) Month 1 follow-up SD-OCT scan after single intravitreal anti-VEGF injection. EZ considered gradable.

Using semi-automated measurements, EZ defect was seen in 36 of 53 thickness maps (67.9%) at month 1, 27 of 65 (41.5%) at month 6, and 29 of 66 (43.9%) at month 12. Combining all time points for a total of 184 images, 92 (50.0%) had an EZ defect area. The mean±SD area of EZ defect as measured by the semi-automated approach was 0.23±0.25 mm2 at month 1 (range 0.005–0.76 mm2), 0.21±0.21 mm2 at month 6 (range 0.007–0.76 mm2), and 0.10±0.15 mm2 at month 12 (range 0.005–0.66 mm2) (). The mean±SD area of EZ defect for all time points combined was 0.19±0.21 mm2 (range 0.005–0.76 mm2), respectively.

The manual approach revealed a mean±SD EZ area defect measurement of 0.24±0.24 mm2 at month 1 (range 0.005–0.78 mm2), 0.19±0.22 mm2 at month 6 (range 0.006–0.78 mm2), and 0.12±0.18 mm2 at month 12 (range 0.005–0.78 mm2). The mean±SD area measurement for all time points combined was 0.19±0.23 mm2 (range 0.005–0.78 mm2). The area of EZ defect combining all time points was indistinguishable between the semi-automated and manual measurements (p = 0.93) ().

Analysis of reliability and agreement between manual and automated area measurements of ellipsoid zone (EZ) defect.

A. Mean±SD area of EZ defect was similar between manual (0.19±0.23 mm2) and automated (0.19±0.21 mm2) measurements (p = 0.93). B. Intra-Class Correlation was 0.90 comparing area measurements of EZ defect. C. Bland-Altman Plot displaying differences against the averages between manual and automated area measurements of EZ defect. The average bias was 0.01 with a 95% confidence interval of the limits of agreement from -0.18–0.20. Abbreviation: ns not significant, SD standard deviation.

The ICC between semi-automated and manual measurements for the area of EZ defect across all time points was 0.90, indicating excellent reliability (). The average bias between measurements was 0.01 mm2 (95% CI -0.18–0.20) (). Inter-rater reliability (ICC = 0.81) and agreement (average bias 0.07, 95% CI -0.17–0.31) for manual measurements between graders was good.

Area measurements did not differ between semi-automated and manual approaches at month 1 (p = 0.89), month 6 (p = 0.27), and month 12 (p = 0.16) (). The ICC at month 1 (0.89), month 6 (0.94), and month 12 (0.88) were good to excellent. The average bias between measurements was 0.00 (95% CI LOA: -0.24–0.24) at month 1, 0.01 (95% CI LOA: -0.14–0,16) at month 6, and 0.03 (95% CI LOA: -0.15–0.18) at month 12.

After segmentation of the EZ layer outside of the KNIME workflow, the time required to generate en face thickness maps from 184 XML files with EZ layer segmentation coordinates was 6 minutes 26 seconds with an average of 2.1 seconds per file. Using the semi-automated approach, the time required to identify and measure the EZ defect area within the CSF was 15 minutes 12 seconds with an average of 4.9 seconds per thickness map. Using the manual approach, the time required to identify and manually trace the EZ defect boundaries, and automatically quantify the area within the CSF was 2 hours 36 minutes with an average of 51 seconds per thickness map. The semi-automated approach reduced the time required to identify and quantify the EZ defect area within the CSF of 184 thickness maps by 2 hours with an average of 46.1 seconds per thickness map.

Discussion

In this study, we developed a customized workflow that began with a semi-automated segmentation method for defining the EZ layer using nonproprietary software. These layer coordinates were used to then automatically generate en face thickness maps in eyes with macular edema secondary to CRVO or HRVO in a randomly selected subgroup of SCORE2 subjects for both manual and semi-automatic segmentation of areas of EZ defect using the open-source software platform, KNIME. This semi-automated approach took advantage of machine-learning methods via the open-source Fiji[14] plugin Trainable Weka Segmentation (TWS)[13] and demonstrated excellent reliability and agreement when compared to manual measurements. These data suggest that our semi-automatic and open-source approach can be efficiently and reliably applied to similar workflows quantitatively assessing SD-OCT derived retinal morphology.

Quantitative assessment of the EZ commonly use manual measurements via the en face [26, 27] method, which consists of creating conforming segmentation lines along the upper and lower boundaries of the EZ band. The B-scan intensities through the segmented boundaries are projected to form an en face image with enhanced contrast between intact and absent EZ.[28] The EZ boundary is then traced to create a geographic representation. Manual measurement of the EZ is time intensive,[12] particularly in clinical trials with high volume data. Therefore, we sought to utilize accessible machine-learning tools to facilitate the identification of these regions.

The ability to measure the EZ is dependent on the visibility of the EZ layer. In the acute stages of RVO, evaluation of the EZ is impaired by the presence of hemorrhage and edema that reduces the outer retinal signal intensity.[27] At baseline, >90% of images were ungradable due to the presence of confounding hemorrhage and edema. Initial qualitative grading of the EZ was performed by expert graders at the SCORE2 central reading center (Fundus Photograph Reading Center, University of Wisconsin). The EZ was qualitatively deemed ungradable when the RPE was not visible due to signal blockage by hemorrhage and edema (). After intravitreal anti-VEGF treatment, 72% of available study eyes were gradable at month 1 for the evaluation of EZ integrity (). Similar approaches to evaluating EZ integrity exclude patients with severe edema, which has been defined as a CSF thickness greater than 600 μm.[5] Although we did not use this threshold, we did exclude EZ defect area measurements less than or equal to 0.004 mm2. Area measurements less than this value were considered absent. We reviewed all data points outside the 95% CI of the LOA comparing the manual versus semi-automated measurements. Discrepancies between area measurements in all cases were due to poor image contrast as a result of impaired signal intensity from multiple factors, including media opacity, hemorrhage, and edema. These data suggest the semi-automated measurements did not disagree with those obtained by the manual approach, and semi-automated measures performed as well as manual outlines by expert graders ().

This study has several limitations. We compared the manual to semi-automated area measurements of EZ defect using the mean area measurement and not by colocalizing the EZ defect areas generated from the two methods. This comparison does not capture the differences in position, orientation, and shape between the EZ boundary measurements. This measurement is impaired by the fact that similar EZ areas may have dissimilar EZ boundaries. Despite this limitation, the mean area measurement is similar to the difference in widths that have been used in reliability studies of EZ defect from individual B-scans in other disease processes, namely retinitis pigmentosa.[29] Additionally, because our study is limited to a smaller region of the CSF, we expect this difference to be minimal (). The selection of scans for analysis of the EZ was performed by expert graders and was based on the SCORE2 SD-OCT grading protocol. However, all scans deemed of sufficient quality for EZ assessment (i.e. the RPE was visible) were evaluated by both the semi-automated and manual approaches. Future studies may examine the application of machine-learning for image selection. This analysis did not examine the correlation between the area of EZ defect with VA and other clinical data. Validation of our approach and association of EZ defect area measurements with clinical data will be performed on the entire SCORE2 cohort upon study completion. Studies have examined the association of EZ integrity at baseline as a predictor of future visual acuity.[30-32] However, these studies included eyes with branch retinal vein occlusion, which presents with significantly less hemorrhage and fluid compared to central and hemi-retinal vein occlusion, especially within the CSF.[1] These studies also utilized high resolution SD-OCT scans, whereas the SCORE2 trial did not to improve the generalizability to the study methods and results. Although post-image processing to remove shadowing cast by hemorrhage and fluid has been performed,[27, 33] these studies have also utilized high resolution SD-OCT scans from a single SD-OCT device. Our method combined multiple SD-OCT machines. Finally, we examined EZ disruption in macular edema secondary to CRVO or HRVO. Future studies applying our methods to diabetic macular edema and other diseases may be beneficial.

There are many strengths of our approach. Our workflow combines multiple steps in the generation of EZ absence area measurements, such as EZ segmentation, export of files containing segmentation coordinates, generation of en face thickness maps that highlight the contrast between intact and absent EZ, and automatic delineation and quantification of EZ defects. The analysis utilized a classifier that acts as a computer-generated grader that can, once fully trained, measure regions of EZ defect, adding precision and high-throughput speed beyond that of manual tracing. Manual identification of EZ defect is time-consuming and error prone. In a study comparing methods of EZ boundary identification in retinitis pigmentosa, the average time required to delineate and trace the EZ boundary, not including segmentation of the EZ layer, was 4.1 minutes for the en face method.[12] Our semi-automated process is rapid, permitting the analysis of SD-OCT scan in seconds in a reliable and reproducible manner. Differences between manual EZ defect tracing times between the previously reported study and ours were likely due to area measurements only within the CSF.

Conclusions

We developed a customized workflow using open-source software that applied manual and semi-automated methods for quantifying the area of EZ defect assessed by SD-OCT derived en face thickness maps in eyes with macular edema secondary to CRVO or HRVO. Semi-automated EZ defect area measurements were obtained through machine-learning and demonstrated excellent agreement and reliability when compared to manual measurements, suggesting that our workflow may be applied to other quantitative assessments of SD-OCT derived retinal morphology. Our semi-automated approach will be validated in the entire SCORE2 cohort upon study completion.

20 Jan 2020

PONE-D-19-34185

A Semi-Automated Machine-Learning Based Workflow for Ellipsoid Zone Analysis in Eyes with Macular Edema: SCORE 2 Pilot Study

PLOS ONE

Dear Dr Domalpally,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The manuscript as reviewed by two exert reviewers and was found to be of interest, but they suggest some minor revisions. I agree and we look forward to the revised version

We would appreciate receiving your revised manuscript by Mar 05 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Demetrios G. Vavvas

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please amend your Data availability statement to describe how other researchers may obtain the datasets from the SCORE 2 study used in this study.

3. Please amend your Ethics statement to provide the approval numbers.

4. Please  amend your Competing interests statement to declare author commercial affiliations.

5. Thank you for including your ethics statement:  "The study was approved by the institutional review boards associated with each center and adhered to the tenets of the Declaration of Helsinki. All participants provided written informed consent."

Please amend your current ethics statement to include the full name of the ethics committee/institutional review board(s) that approved your specific study.

Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research.

6. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

7. Thank you for stating the following in the Financial Disclosure section:

"The Standard Care vs Corticosteroid for Retinal Vein Occlusion (SCORE) Study was supported by National Eye Institute (National Institutes of Health, Department of Health and Human Services) grants 5U10EY014351, 5U10EY014352, and 5U10EY014404; and Allergan, Inc."

We note that you received funding from a commercial source: Allergan, Inc

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

Please provide an amended Competing Interests Statement that explicitly states this commercial funder, along with any other relevant declarations relating to employment, consultancy, patents, products in development, marketed products, etc.

Within this Competing Interests Statement, please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

8. Thank you for stating the following in the Financial Disclosure section:

"The Standard Care vs Corticosteroid for Retinal Vein Occlusion (SCORE) Study was supported by National Eye Institute (National Institutes of Health, Department of Health and Human Services) grants 5U10EY014351, 5U10EY014352, and 5U10EY014404; and Allergan, Inc."

We note that one or more of the authors have an affiliation to the commercial funders of this research study : KNIME GmbH.

1. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

2. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: 1. A very well structured manuscript, technically sound, with data that support the conclusions.

2. The statistical analysis described is thorough.

3. The lack of EZ assessment at baseline scans defeats the purpose of using EZ integrity as a prognosticator of final visual outcome:

As described in the introduction, one of the reasons why EZ integrity is often assessed is its potential use as a predictive factor of the final visual outcome. In the present study, by converting cross sectional images into en face, hemorrhage or edema-associated signal blockage prevented the assessment of EZ integrity at baseline. (>90% of baseline images were ‘ungradable’)

Hence, as described, EZ integrity of only SD-OCT images at months 1, 6, and 12 was assessed, analysed and quantified in this study. Consequently, if a semi-automated machine learning system is used for the assessment of EZ integrity in en face images, the only prognostic information one can derive comes from the EZ integrity after intravitreal anti-VEGF treatment (at 1st month)

Reviewer #2: The paper by Tyler Etheridge et al. entitled 'A Semi-Automated Machine-Learning Based Workflow for Ellipsoid Zone Analysis in Eyes with Macular Edema: SCORE 2 Pilot Study' is an interesting study on the evaluation of the combination of a customized workflow with KNIME and machine learning tool for Fiji plugin: Trainable Weka Segmentation. The authors developed the customized workflow with semi-automated optical coherence tomography measurements and examined its agreement and reliability. And then, they found their new approach was reliable and time-saving. The study is well constructed. The obtained results will be of interest to researchers in the field. Their handy method using KNIME and Fiji plugin could be applied to other retinal diseases at a low cost. And I think the study has great potential not only in academic research but also in daily clinical practice.

I have few concerns,

(1) Page 7, line 170-172; what kind of dataset did the authors use for the learning process? From entire SCORE2 data? Or other normal subject data? And how much data did the authors use for training? Did the authors check if the performance reached the plateau or not?

(2) Page 11, line 247-255; how long did it take for the training process?

(3) Page 9, line 210-218; how did the authors judge whether each image could go through the analysis or not. Manually by graders or semi-automated with a program? Did 'gradable' mean manually measured? Did only images that could be manually measured go through semi-automated measurements? If the judgment is done by a human, not by a semi-automated program, then I suggest discussing this essential process in terms of the semi-automated approach limitation.

(4) Page 10, line 231-233 (Figure 3A); the authors evaluated the reliability with Bland-Altman plot in addition to intraclass correlation coefficient. But, I guess intraclass correlation coefficient is enough to evaluate both the reliability and agreement (ref. Terry K. Koo, Mae Y. Li. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016 Jun; 15(2): 155–163. Published online 2016 Mar 31. doi: 10.1016/j.jcm.2016.02.012 PMCID: PMC4913118. Correction in: J Chiropr Med. 2017 Dec; 16(4): 346. Pubmed Central PMCID: PMC5731844).

(5) For the researchers and clinicians who want to use the workflow and semi-automated methods in their settings, is it possible to show the link of the programs or training dataset? If it's possible, it must be helpful for our fields.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

26 Mar 2020

Review Comments to the Author

Reviewer #1:

1. A very well structured manuscript, technically sound, with data that support the conclusions

2. The statistical analysis described is thorough.

3. The lack of EZ assessment at baseline scans defeats the purpose of using EZ integrity as a prognosticator of final visual outcome:

As described in the introduction, one of the reasons why EZ integrity is often assessed is its potential use as a predictive factor of the final visual outcome. In the present study, by converting cross sectional images into en face, hemorrhage or edema-associated signal blockage prevented the assessment of EZ integrity at baseline. (>90% of baseline images were ‘ungradable’)

Hence, as described, EZ integrity of only SD-OCT images at months 1, 6, and 12 was assessed, analysed and quantified in this study. Consequently, if a semi-automated machine learning system is used for the assessment of EZ integrity in en face images, the only prognostic information one can derive comes from the EZ integrity after intravitreal anti-VEGF treatment (at 1st month)

Reply: No changes were made. Please see the discussion section (page 13, line 282-298). The inability to accurately assess the EZ on en face thickness maps due to reduced outer retinal signal intensity from blockage by hemorrhage and fluid is an inherent challenge in retinal vein occlusion. Without histopathological correlates it is difficult to know whether EZ defects on SD-OCT are artifacts resulting from signal blockage or loss of photoreceptors secondary to disease. However, our approach of assessing month 1 EZ integrity with subsequent visual acuity matches other studies referenced in the manuscript:

Chan EW, Eldeeb M, Sun V, et al. Disorganization of Retinal Inner Layers and Ellipsoid Zone Disruption Predict Visual Outcomes in Central Retinal Vein Occlusion. Ophthalmol Retina 2019;3:83-92.

Tang F, Qin X, Lu J, Song P, Li M, Ma X. Optical Coherence Tomography Predictors of Short-Term Visual Acuity in Eyes with Macular Edema Secondary to Retinal Vein Occlusion Treated with Intravitreal Conbercept. Retina 2019.

Fujihara-Mino A, Mitamura Y, Inomoto N, Sano H, Akaiwa K, Semba K. Optical coherence tomography parameters predictive of visual outcome after anti-VEGF therapy for retinal vein occlusion. Clin Ophthalmol 2016;10:1305-13.

Reviewer #2:

The paper by Tyler Etheridge et al. entitled 'A Semi-Automated Machine-Learning Based Workflow for Ellipsoid Zone Analysis in Eyes with Macular Edema: SCORE 2 Pilot Study' is an interesting study on the evaluation of the combination of a customized workflow with KNIME and machine learning tool for Fiji plugin: Trainable Weka Segmentation. The authors developed the customized workflow with semi-automated optical coherence tomography measurements and examined its agreement and reliability. And then, they found their new approach was reliable and time-saving. The study is well constructed. The obtained results will be of interest to researchers in the field. Their handy method using KNIME and Fiji plugin could be applied to other retinal diseases at a low cost. And I think the study has great potential not only in academic research but also in daily clinical practice.

I have few concerns,

(1) Page 7, line 170-172; what kind of dataset did the authors use for the learning process? From entire SCORE2 data? Or other normal subject data? And how much data did the authors use for training? Did the authors check if the performance reached the plateau or not?

Reply: No changes were made. Please see the methods section (page 8, line 175-179). The authors selected en face thickness maps that represented the inherent variability of the total dataset analyzed (184 scans). We chose approximately 10% of the dataset (12 en face thickness maps) for classifier training. Areas of normal EZ and EZ defects were manually traced using the Trainable Weka Segmentation plugin of Fiji as described by Arganda-Carreras et al.*. The authors sought to maximize the classifiers identification of regions of interest and minimize non-region of interest (false-positive) identification. A reliable and accurate classifier was trained with the initial 12 thickness maps, more would have been used if necessary, as is customary in such training protocols.

* Arganda-Carreras I, Kaynig V, Rueden C, Eliceiri KW, Schindelin J, Cardona A, et al. Trainable Weka Segmentation: a machine learning tool for microscopy pixel classification. Bioinformatics. 2017;33(15):2424-6. doi: 10.1093/bioinformatics/btx180. PubMed PMID: 28369169

(2) Page 11, line 247-255; how long did it take for the training process?

Reply: No changes were made. The authors did not time the classifier training. However, after generation of the en face thickness maps and selection of images representing the heterogeneity of the image dataset, the training of the classifier required minimal time compared to manual annotations (20-30 minutes to obtain the first, most-reliable classifier with appropriate settings).

(3) Page 9, line 210-218; how did the authors judge whether each image could go through the analysis or not. Manually by graders or semi-automated with a program? Did 'gradable' mean manually measured? Did only images that could be manually measured go through semi-automated measurements? If the judgment is done by a human, not by a semi-automated program, then I suggest discussing this essential process in terms of the semi-automated approach limitation.

Reply: The following changes were made (page 14, line 308-312). “The selection of scans for analysis of the EZ was performed by expert graders and was based on the SCORE2 SD-OCT grading protocol. However, all scans deemed of sufficient quality for EZ assessment (i.e. the RPE was visible) were evaluated by both the semi-automated and manual approaches. Future studies may examine the application of machine-learning for image selection.”

(4) Page 10, line 231-233 (Figure 3A); the authors evaluated the reliability with Bland-Altman plot in addition to intraclass correlation coefficient. But, I guess intraclass correlation coefficient is enough to evaluate both the reliability and agreement (ref. Terry K. Koo, Mae Y. Li. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016 Jun; 15(2): 155–163. Published online 2016 Mar 31. doi: 10.1016/j.jcm.2016.02.012 PMCID: PMC4913118. Correction in: J Chiropr Med. 2017 Dec; 16(4): 346. Pubmed Central PMCID: PMC5731844).

Reply: No changes were made. The authors sought to perform a rigorous comparison of the two approaches and subtle differences exist between reliability and agreement, which is described by Kottner et al.*.

*Kottner J, Streiner DL. The difference between reliability and agreement. J Clin Epidemiol. 2011 Jun;64(6):701-2; author reply 702. doi: 10.1016/j.jclinepi.2010.12.001. Epub 2011 Mar 16. PMID: 21411278.

(5) For the researchers and clinicians who want to use the workflow and semi-automated methods in their settings, is it possible to show the link of the programs or training dataset? If it's possible, it must be helpful for our fields.

Reply: The following changes were made to the Data Availability Statement. “The KNIME workflow is available at https://hub.knime.com/tetheridge/spaces/Public/latest/Raw_Thickness_Maps. The KNIME workflow protocol and training dataset is available at https://doi.org/10.6084/m9.figshare.11774577.”

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

13 Apr 2020

PONE-D-19-34185R1

A Semi-Automated Machine-Learning Based Workflow for Ellipsoid Zone Analysis in Eyes with Macular Edema: SCORE2 Pilot Study

PLOS ONE

Dear Dr Domalpally,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The manuscript has improved and there are some minor points remaining to be addressed. We look forward to the revised version

We would appreciate receiving your revised manuscript by May 28 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Demetrios G. Vavvas

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors opted to make no changes on comment #3. Regarding their answer:

1. In their article entitled Spectral-domain optical coherence tomography (SD-OCT) patterns and response to intravitreal bevacizumab therapy in macular edema associated with branch retinal vein occlusion, Kang et al found that the integrity of the EZ before treatment could not be evaluated in cross sectional OCT scans in 8 of 67 patients with BRVO-ME.

Herein the authors by converting cross sectional images into en face report >90% of baseline images were ‘ungradable’. This is vastly different from the ungradable percentages in cross sectional OCT. Thus it seems the inability to accurately assess the EZ is not an inherent challenge in RVO (as the authors response to the comment suggests) rather an inherent limitation of en face images, which should be clearly stated in the limitations.

2. A prognosticator available at 1 month is not as useful to clinicians and patients as baseline prognosticators for long term functional outcomes in cases those exist. In BRVO the EZ integrity at baseline is a well established prognosticator that the en face approach seems to fail to offer.

All 3 articles that the authors cite in their response refer to cross sectional OCT studies and 2/3 actually did evaluate EZ disruption at baseline with no mention on the potential effect of macular edema. Hence, it seems that the en face transformation of cross sectional OCT images limits the ability to assess the EZ integrity due to macular edema and/or haemorrhage.

3. Further, the authors reply to the comment suggesting that histopathological studies would be needed to determine whether EZ defects on SD-OCT are artifacts resulting from signal blockage or loss of photoreceptors secondary to disease. However the article by Kanakis et al they are citing in the relevant part of their discussion (lines 282-298) describe the development of ‘a method of en face representation of the ellipsoid zone, along with the removal of shadows, to evaluate the ellipsoid layer disruption’ using ‘a MATLAB (Mathworks, Inc, Natick, MA) implementation of the algorithms for shadows removal’ concluding that ‘It could be assumed that the defects at the level of photoreceptors could be the result of the coexisting macular edema. However, the concordance of the ischemic area as shown in FA with the area of ellipsoid disruption at the en face reconstruction of the OCT makes this hypothesis unlikely.’

Good quality images are crucial in Machine Learning. The authors herein employed Machine Learning and transformed the cross sectional images to en face ending up >90% of baseline images were ‘ungradable’in terms of the main parameter the authors seek to study. To make their study stronger, and given their machine learning expertise, I would suggest the authors should embrace a similar approach as the en face OCT article by Kanakis et al they are citing and supplement their methods by developing an algorithm for artifact removal so that the quality control of the images they are feeding the Machine Learning is improved.

Reviewer #2: Almost all my concerns are answered clearly.

Just to be sure, please let me confirm it.

(1) Methods section (page 8, line 175-179). "Total dataset" comes from SCORE2 data. So, it didn't have normal subject data. Is it correct?

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

14 Apr 2020

Response to editor : Our protocol is located with the KNIME workflow uploaded to the KNIME Hub and is completely open access. The link has provided in the data sharing component of the submission: https://hub.knime.com/tetheridge/spaces/Public/latest/Raw_Thickness_Maps

Review Comments to the Author

Reviewer #1: The authors opted to make no changes on comment #3. Regarding their answer:

1. In their article entitled Spectral-domain optical coherence tomography (SD-OCT) patterns and response to intravitreal bevacizumab therapy in macular edema associated with branch retinal vein occlusion, Kang et al found that the integrity of the EZ before treatment could not be evaluated in cross sectional OCT scans in 8 of 67 patients with BRVO-ME.

Herein the authors by converting cross sectional images into en face report >90% of baseline images were ‘ungradable’. This is vastly different from the ungradable percentages in cross sectional OCT. Thus it seems the inability to accurately assess the EZ is not an inherent challenge in RVO (as the authors response to the comment suggests) rather an inherent limitation of en face images, which should be clearly stated in the limitations.

Reply: Thank you for the comment. Both central and hemi-retinal vein occlusion have significantly more hemorrhage and fluid than branch retinal vein occlusion, making a comparison between the disease processes problematic. The SCORE2 study data were obtained from eyes with macular edema secondary to central or hemi-retinal vein occlusion. To further emphasize the challenge inherent in evaluating the ellipsoid zone on SD-OCT scans in central and hemi-retinal vein occlusion, regardless of cross sectional or en face evaluation, the authors have include a new figure (Figure 3 lines 217, 286, 288, and 478-481) depicting a representative cross sectional B-scan at baseline with significant hemorrhage and fluid blocking the outer retinal signal intensity to the point of being unable to identify the retinal pigment epithelium. The figure also includes a follow-up month 1 cross sectional image after resolution of a majority of hemorrhage and fluid with intravitreal anti-VEGF therapy and gradable ellipsoid zone.

Change made to manuscript: Figure 3. (A) Representative center point SD-OCT scan deemed ungradable at baseline for ellipsoid zone (EZ) assessment. EZ was considered ungradable when the retinal pigment epithelium was not visible due to signal blockage by hemorrhage and edema. (B) Month 1 follow-up SD-OCT scan after single intravitreal anti-VEGF injection. EZ considered gradable.

2. A prognosticator available at 1 month is not as useful to clinicians and patients as baseline prognosticators for long term functional outcomes in cases those exist. In BRVO the EZ integrity at baseline is a well established prognosticator that the en face approach seems to fail to offer.

All 3 articles that the authors cite in their response refer to cross sectional OCT studies and 2/3 actually did evaluate EZ disruption at baseline with no mention on the potential effect of macular edema. Hence, it seems that the en face transformation of cross sectional OCT images limits the ability to assess the EZ integrity due to macular edema and/or haemorrhage.

Reply: Thank you for the comment. The studies by Chan et al, Tang et al, and Fujihara-Mino et al included eyes with central retinal vein occlusion and branch retinal vein occlusion. As stated above, comparing the two distinct disease processes is problematic owing to the significant difference in hemorrhage and fluid in central retinal vein occlusion compared to branch retinal vein occlusion. In addition, these studies used high resolution SD-OCT scans, whereas the SCORE2 clinical trial did not to add generalizability to the study methods and results. This has been included in the discussion of our studies limitations.

Change made to manuscript ( line 324): Studies have examined the association of EZ integrity at baseline as a predictor of future visual acuity.However, these studies included eyes with branch retinal vein occlusion, which presents with significantly less hemorrhage and fluid compared to central and hemi-retinal vein occlusion, especially within the CSF.

3. Further, the authors reply to the comment suggesting that histopathological studies would be needed to determine whether EZ defects on SD-OCT are artifacts resulting from signal blockage or loss of photoreceptors secondary to disease. However the article by Kanakis et al they are citing in the relevant part of their discussion (lines 282-298) describe the development of ‘a method of en face representation of the ellipsoid zone, along with the removal of shadows, to evaluate the ellipsoid layer disruption’ using ‘a MATLAB (Mathworks, Inc, Natick, MA) implementation of the algorithms for shadows removal’ concluding that ‘It could be assumed that the defects at the level of photoreceptors could be the result of the coexisting macular edema. However, the concordance of the ischemic area as shown in FA with the area of ellipsoid disruption at the en face reconstruction of the OCT makes this hypothesis unlikely.’

Good quality images are crucial in Machine Learning. The authors herein employed Machine Learning and transformed the cross sectional images to en face ending up >90% of baseline images were ‘ungradable’in terms of the main parameter the authors seek to study. To make their study stronger, and given their machine learning expertise, I would suggest the authors should embrace a similar approach as the en face OCT article by Kanakis et al they are citing and supplement their methods by developing an algorithm for artifact removal so that the quality control of the images they are feeding the Machine Learning is improved.

Reply: Thank you for the comment. Similar to the studies by Chan et al, Tang et al, and Fuijihara-Mino et al, the aforementioned study by Kanakis et al was performed using high resolution SD-OCT scans not obtained for use by clinical trials or clinical practice. The SCORE2 SD-OCT scans were not high-resolution. In addition, the study by Kanakis et al only utilized Spectralis SD-OCT machine. The SCORE2 trial utilized Spectralis and Cirrus SD-OCT machines. Therefore, our methods permitted the use of either imaging device. This limitation is included in the discussion (lines 320-321).

Change made to manuscript: These studies also utilized high resolution SD-OCT scans, whereas the SCORE2 trial did not to improve the generalizability to the study methods and results. Although post-image processing to remove shadowing cast by hemorrhage and fluid has been performed,these studies have also utilized high resolution SD-OCT scans from a single SD-OCT device. Our method combined multiple SD-OCT machines.

Reviewer #2: Almost all my concerns are answered clearly.

Just to be sure, please let me confirm it.

(1) Methods section (page 8, line 175-179). "Total dataset" comes from SCORE2 data. So, it didn't have normal subject data. Is it correct?

Reply: Thank you for the clarification. The total dataset comes from SCORE2 data and therefore does not include data from “normal” or “healthy” subjects. This clarification was made in the methods (lines 177-179).

Change made to manuscript: All scans were obtained from the SCORE2 dataset and were therefore from eyes affected by either CRVO or HRVO.

_______________________________________

Submitted filename: Response to Reviewers041420.docx

Click here for additional data file.

16 Apr 2020

A Semi-Automated Machine-Learning Based Workflow for Ellipsoid Zone Analysis in Eyes with Macular Edema: SCORE2 Pilot Study

PONE-D-19-34185R2

Dear Dr. Domalpally,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Demetrios G. Vavvas

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

21 Apr 2020

PONE-D-19-34185R2

A Semi-Automated Machine-Learning Based Workflow for Ellipsoid Zone Analysis in Eyes with Macular Edema: SCORE2 Pilot Study

Dear Dr. Domalpally:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Demetrios G. Vavvas

Academic Editor

PLOS ONE

Table 1

Comparison of ellipsoid zone defect area measurements.

	Manual			Automated
Time Points	Minimum	Maximum	Mean ± SD	Minimum	Maximum	Mean ± SD	p-value	ICC	Average Bias (95% CI LOA)
All (mm2)	0.005	0.78	0.19 ± 0.23	0.005	0.76	0.19 ± 0.21	0.76	0.90	0.01 (-0.18–0.20)
M01 (mm2)	0.005	0.78	0.24 ± 0.24	0.005	0.76	0.23 ± 0.25	0.89	0.89	0.00 (-0.24–0.24
M06 (mm2)	0.006	0.78	0.19 ± 0.22	0.007	0.76	0.21 ± 0.21	0.27	0.94	0.01 (-0.14–0.16)
M12 (mm2)	0.005	0.78	0.12 ± 0.18	0.005	0.66	0.10 ± 0.15	0.16	0.88	0.03 (-0.12–0.18)

Abbreviations: M01 Month 1, M06 Month 6, M12 Month 12; ICC, intra-class correlation coefficient, CI confidence interval, LOA limits of agreement