The role of structured reporting and structured operation planning in functional endoscopic sinus surgery.

Computed tomography (CT) scans represent the gold standard in the planning of functional endoscopic sinus surgeries (FESS). Yet, radiologists and otolaryngologists have different perspectives on these scans. In general, residents often struggle with aspects involved in both reporting and operation planning. The aim of this study was to compare the completeness of structured reports (SR) of preoperative CT images and structured operation planning (SOP) to conventional reports (CR) and conventional operation planning (COP) to potentially improve future treatment decisions on an individual level. In total, 30 preoperative CT scans obtained for surgical planning of patients scheduled for FESS were evaluated using SR and CR by radiology residents. Subsequently, otolaryngology residents performed a COP using free texts and a SOP using a specific template. All radiology reports and operation plannings were evaluated by two experienced FESS surgeons regarding their completeness for surgical planning. User satisfaction of otolaryngology residents was assessed by using visual analogue scales. Overall radiology report completeness was significantly higher using SRs regarding surgically important structures compared to CRs (84.4 vs. 22.0%, p<0.001). SOPs produced significantly higher completeness ratings (97% vs. 39.4%, p<0.001) regarding pathologies and anatomical variances. Moreover, time efficiency was not significantly impaired by implementation of SR (148 s vs. 160 s, p = 0.61) and user satisfaction was significantly higher for SOP (VAS 8.1 vs. 4.1, p<0.001). Implementation of SR and SOP results in a significantly increased completeness of radiology reports and operation planning for FESS. Consequently, the combination of both facilitates surgical planning and may decrease potential risks during FESS.

Introduction

Functional endoscopic sinus surgery (FESS) represents the gold standard in the surgical management of paranasal sinus disease [1, 2]. Three-dimensional high-resolution computed tomography (CT) scans are required for any surgical approach to treat both chronic and acute rhinosinusitis, particularly with potentially occurring complications [3, 4]. Prior to surgery, CT scans are indispensable to determine the extent of FESS as well as to identify anatomical structures with an increased risk regarding catastrophic complications. Such complications may include injuries of the anterior skull base, the orbit, optic nerve or internal carotid artery with potential lethal bleedings, ischemic stroke or need for blood transfusions [5-9]. Especially in the era of powered instruments such as microdebriders, a profound understanding of the anatomy is of utmost importance, as these particular instruments can cause severe complications when handled without sufficient care [10, 11]. Therefore, a solid preoperative knowledge of the anatomy using radiological and surgical knowledge is key for any successful interdisciplinary surgical treatment planning which may prevent premature revision surgery [12]. Additionally, the accessibility of CT scans sets the foundation towards performing image-guided surgery by using intraoperative navigation [13].

Despite major improvements in the quality of CT scans making it possible to address a wide variety of diseases and anatomical variances in great detail, radiologists and otolaryngologists have grossly different perspectives on these scans [4, 14]. Whereas radiologists mainly focus on the extent and the effects of the disease, otolaryngologists are also very much interested in anatomical variances to plan surgical approaches [15]. In addition, residents in training, both in radiology and otolaryngology, often struggle which anatomical structures which have to be considered in reporting and operation planning and how to grade them [16, 17]. Conventional CT checklists have been implemented in many otolaryngology departments to give residents some sort of guidance. Nevertheless, these checklists usually exist in analogous form. Therefore, they are frequently abandoned after initial implementation as they are not directly integrated into the clinical workflow [18].

Structured reporting (SR) has been advocated for various diagnostic modalities in radiology as well as in otolaryngology and other specialties with impact on treatment decisions and interdisciplinary communication [19-25]. The main benefit, especially for residents in training, is the standardization of the content and terminology which is known to improve report quality and time efficiency when compared to conventional reporting (CR) [19-21]. Additionally, the implementation of structured operation planning potentially reduces the risk of missing key structures during the planning process. This may be due to the fact that structured reporting templates can highlight important features as well as pertinent negative findings within predefined checklists [26, 27]. Consequently, SR and SOP may also promote the learning curve of younger radiologists and otolaryngologists in training [28].

Overall, the aim of this study was to perform a comparative analysis of the completeness of CT reports and surgical plannings using SR and SOP, respectively, to the conventional approaches (e.g. CR and COP) which may have the potential to improve treatment decisions and patient outcome in the long term.

Materials and methods

Study design and study sample

This study was designed to evaluate the impact of structured reporting as well as structured operation planning in the process of the surgical management of chronic rhinosinusitis. Therefore, n = 30 consecutive preoperative CT scans of adult patients (m = 17, w = 13, mean age at surgery: 38.2 years) scheduled for FESS due to chronic rhinosinusitis with (n = 10) and without polyposis (n = 20) were included in this study (see Table 1). Existing conventional reports of sinus CTs were identified. These CT scans were assessed once again using a specifically designed online-based SR template (n = 30, Smart Reporting GmbH, Munich, Germany, https://smart-reporting.com) by one board-certified radiologist. The template was designed in consideration of the latest radiological and otolaryngological recommendations for preoperative CT analysis before FESS procedures, anatomical structures and terminology. The templates guide the user through clickable decision-trees specific for a wide range of benign and malignant paranasal sinus diseases. By completing the decision-trees, the software creates full semantic sentences from previously defined text elements, thus creating complete and consistent reports. In order to reduce the likelihood to consult additional medical literature or classification tables during reporting and planning, info boxes can be added to the template displaying classifications and medical guidelines which have been shown to improve time efficiency [19].

Table 1

Demographics and characteristics of the study sample.

Characteristics	Value
Number of patients included	30
Age at surgery (mean ± SD)	41.4 ± 12.8 years (range: 23–58 years)
Gender	Male: n = 17 Female: n = 13
Indication for functional endoscopic sinus surgery	Chronic rhinosinusitis with polyposis: n = 20
	Chronic rhinosinusitis without polyposis: n = 10
Number of participating otolaryngology residents	6
Years of residency	4.5 ± 0.9 (range: 4–6 years)

Subsequently, residents in training (n = 6, 4.5 ± 0.9 years of residency) who were assigned to the particular FESS procedure were to plan the operation by conventional operation planning (COP, n = 30) using the CT scans on file. To do so, they completed the department’s standard FESS planning form in which the categories nasal septum, middle nasal meatus, ethmoid infundibulum, maxillary sinus, ethmoid sinus, sphenoid sinus and frontal sinus have to reported upon for each side in writing by hand. In a second step, residents planned the same procedures using a specifically designed online-based template for SOP (n = 30) of FESS procedures (Smart Reporting GmbH, Munich, Germany, https://smart-reporting.com). Time needed for each type of planning was recorded.

Finally, participating residents completed a user satisfaction questionnaire regarding COP and SOP, respectively.

Ethics approval and consent to participate

Ethics approval was obtained by the Institutional Review Board (Ethik-Kommission der Landesärztekammer Rheinland-Pfalz. Reference number: 2018–13225). All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Oral and written patient information was given by the examining physician. Written informed consent was obtained prior to the examination.

Sample size calculation

As previously described in the literature, the number of patients needed was calculated based on the anticipated effect size when comparing the percentage of CRs/COPs with 80% completeness or higher to SRs/SOPs. We assumed that 45% of CRs/COPs would have very high completeness ratings (i.e. of 80% or higher), considering the report completeness of other head and neck ultrasound imaging studies as shown in the literature [19-21]. Additionally, we estimated that completeness ratings would go up to 80% by using SRs/SOPs. The power was set to 80% with a significance level of α = 0.05. Using these parameters, the minimum number of reports required for the study was calculated to be n = 58 (29 reports in each group).

Evaluation of radiological reports and operation plannings

Collectively, 60 anonymized reports (30 SRs and CRs each) were independently analyzed for completeness (i.e. reporting of critical anatomy and pathology) by a board-certified radiologist and an otolaryngologist. Additionally, an evaluation template was implemented in order to standardize the analysis.

Following this, 60 anonymized operation plannings (30 SOPs and COPs each) were also evaluated for completeness (i.e. identification of critical anatomical structures and disease as well as determination of the extent of the procedure) by two board-certified otolaryngologists with a high expertise in FESS based on an evaluation checklist. Due to the free text structure of COPs which contain handwritten plannings, readability was subjectively assessed using a five-point Likert scale. Time expenditure for each type of planning was also recorded and compared.

Otolaryngology residents were asked for their opinions regarding practicability (question 1), usefulness in everyday practice (question 2), improvement in report-quality (question 3), time efficiency (question 4), justification of additional time needed (question 5) as well as benefits for inexperienced physicians being trained in FESS (question 6) of both types of planning by applying a ten-point visual analogue scale.

Statistical analysis

Data are reported as the mean percentage of the greatest possible outcome (i.e. percentage of maximum completeness, etc.), mean time needed to complete the report (seconds) and mean VAS values ± SD. Wilcoxon signed-rank test for paired nominal data was used to compare operation planning evaluations and questionnaire findings while Mann-Whitney test for unpaired data was used to compare radiology reports. A p-value of less than 0.05 was considered statistically significant. All statistical analyses were performed using Prism 8 (GraphPad Software, Inc., San Diego, CA, USA).

Results

Analysis of radiological reports

Radiological reports, both CR and SR, were completed by radiology residents. Compared to CRs, SRs showed a significantly higher overall completeness level for all categories (84.4 vs. 22.0%, p<0.001, Fig 1a).

Fig 1

Comparison of overall completeness between structured reports (SR) and conventional reports (CR) of paranasal sinus computed tomographies (CT).

Analysis reveals significantly superior completeness ratings of SR (a). Detailed completeness assessment of relevant items reveals superior results of SR in all items when compared to CR (b). *** p < 0.001.

In particular, SRs were more complete in regard to description of nasal septum (100 vs. 0%, p<0.001), middle nasal meatus (85.8 vs. 3.3%, p<0.001), maxillary sinus (95.6 vs. 62.2%, p<0.001), ethmoid sinus (82.7 vs. 24.3%, p<0.001), and frontal sinus (75 vs. 33.3%, p<0.001). Additionally, the anterior skull base (98.3 vs. 1.7%, p<0.001) and the absence of potential masses (70 vs. 13.3%, p<0.001) were reported more frequently. Concerning classifications, SRs had a higher level of completeness in regard to the Keros classification (83.3 vs. 0%, p<0.001) and the Lund-Mackay Score (73.3 vs 0%, p<0.001). These results are summarized in Fig 1b.

Analysis of operation plannings

Surgical planning of FESS, by both COP and SOP, was performed by otolaryngology residents who were assigned to assist or to perform (under supervision of a board-certified otolaryngologist) each FESS procedure by using the preoperative CT scans for surgical planning. When compared to mostly handwritten COPs, SOPs showed a significantly better readability (100 vs. 62%, p<0.001). Analysis revealed significantly higher overall completeness ratings for SOP when compared to COP (97 vs. 39.4%, p<0.001). In detail, SOPs received significantly superior ratings in respect to the nasal septum (100 vs. 56.7%, p<0.001), the middle nasal meatus (96.7 vs. 54.2%, p<0.001), the ethmoid infundibulum (92.5 vs. 50%, p<0.001) and the maxillary sinus (93.3 vs. 44.7%, p<0.001). In addition, the ethmoid sinus (99.2 vs. 33.3%, the sphenoid sinus (93.9 vs. 30%, p<0.001) and the frontal sinus (98.6 vs. 31.9%, p<0.001) were documented more completely using SOP. Potential masses in the paranasal sinuses (100 vs. 15%, p<0.001) and particularly the Keros classification of the anterior skull base (100 vs. 40%, p<0.001) were considered significantly more reliably using SOP as well. Evaluation of time needed to plan the operation shows a tendency towards a better time efficiency for SOPs without reaching significance level (148 s vs. 160 s, p = 0.61). A comprehensive presentation of operation planning analysis is shown in Fig 2.

Fig 2

Analysis of readability, time efficiency and completeness of structured and conventional operation plannings.

Structured operation plannings (SOP) reveal a significantly superior readability and overall completeness when compared to conventional operation planning (COP, a). Evaluation of time needed to plan the operation shows a tendency towards a better time efficiency for SOPs without reaching significance level (b). Analysis of detailed completeness levels of relevant anatomical features for FESS plannings outlines significantly higher completeness ratings for all analyzed items when compared to COP (c). n.s. = not significant, *** p < 0.001.

User satisfaction

In total, the questionnaire showed a significant preference for SOPs by all participating otolaryngology residents (8.1 vs. 4.1, p<0.001). In detail, the use of SOP was rated to be usable in everyday practice (7.8 vs. 3.7, p = 0.035), to improve the quality of preoperative planning (8.8 vs. 4.0, p = 0.038) and to have a favorable time efficiency (7.2 vs. 3.5, p = 0.04). Additionally, residents thought that any extra time spent on SOP was justified compared to COP (8.2 vs. 3.3, p = 0.038). All other questions revealed a positive tendency towards a preference of SOP without reaching the level of significance. A detailed analysis of questionnaires is shown in Fig 3.

Fig 3

Analysis of user satisfaction.

User satisfaction analysis reveals a significantly superior overall satisfaction when using structured operation planning (SOP) when compared to conventional operation planning (COP). In detail, SOP received significantly higher ratings concerning everyday practice (Q2), quality improvement (Q3) and time efficiency (Q4). In case of additional time needed for SOP, users thought that this additional time was likely to be well spent (Q5). The items practicability (Q1) and training effect (Q6) showed a tendency toward better ratings for SOP without reaching significance level. * p < 0.05.

Discussion

FESS has been proven as an effective surgical treatment of refractory chronic conditions of paranasal sinuses and, therefore, represents the surgical gold standard in these cases [29]. While technical developments that lead to modern FESS procedures have improved the surgical efficacy even further [30-32], the influence of structured approaches in radiology reports and pre-treatment surgical plannings have not been studied comprehensively. Yet, precise imaging and accurate reporting of findings are generally found to be relevant for therapy planning. Furthermore, detailed planning of surgery can increase confidence, boost the learning curves of residents in training and reduce complications during and after surgery [33, 34].

The main finding of our study outlines an added value of a structured approach to radiology reporting in combination with structured surgical planning to improve treatment decisions. In summary, structured data acquisition did indeed lead to superior completeness, both for information collected by otolaryngology and radiology residents. This may very well be due to the fact that radiologists and otolaryngologists often focus on quite different aspects when evaluating CT scans of the paranasal sinuses [15, 35, 36]. Whereas radiologists mainly evaluate CT scans for potential pathologies and their consecutive extent (e.g. signs and extent of chronic rhinosinusitis), otolaryngologists usually reflect CT scans in terms of a surgical approach considering the pathological findings [37]. Interestingly, implementation of SOP caused otolaryngology residents to consider potentially dangerous structures such as the lamina papyracea and particularly the anterior skull base and its asymmetries as described by the Keros classification (100 vs. 31.3%, p = 0.0007) significantly more often. While these results clearly demonstrate superior completeness ratings of SOPs, the insufficient completeness of COPs may be due to the fact that the SOP template queries all structures for every operation planning while unremarkable findings may not be reported using COPs due to a lack of necessity. Consequently, while important structures are actually considered by otolaryngology residents for each and every surgery, this may be underrepresented in COP documentation. Nevertheless, without proper documentation, the chance to miss an atypical critical structure in the surgical planning process, which may lead to intraoperative complications, is increased. Additionally, the surgical approach to the frontal sinus was described significantly more frequent which may very well reduce the rate of revision surgery due to insufficient drainage of the frontal sinus [38, 39]. As described by Stammberger et al., the uncapping the egg technique to the frontal recess and sinuses requires thorough preparation in order to identify ethmoidal cells protruding into the frontal sinus [40]. If planned insufficiently, dissection of only the lower portion of such cells will not provide sufficient drainage of the frontal sinus and consequently will fail to resolve the frontal sinus disease.

Structure and content of radiology reports of paranasal sinuses in terms of surgical therapies may be unclear to radiologists, and in particular for inexperienced residents, even though these scans are performed as part of the surgical planning [15, 35, 36]. Therefore, the creation of specific structured reporting templates may also enhance interdisciplinary discussion and may provide common grounds for exchange transmission of information [4]. The template used in this study was carefully designed in consideration of the current guidelines by a team of board-certified otolaryngologists and radiologists with a special expertise in FESS and head and neck radiology, respectively. In this process, anatomical structures that are of central importance for radiological reporting and FESS were identified and their interdisciplinary importance was discussed to implement all necessary information for detailed and accurate surgical planning. In this course, an interdisciplinary approach to preoperative reporting and planning of FESS was established. This may be highly desirable not only from a surgical perspective, but also from radiologists’ point of view in order to maximize the clinical value of their reports leading to higher satisfaction of referring physicians. Additionally, it decreases the risk that atypical anatomical structures, which may be concealed by extensive paranasal sinus disease, are missed, both by radiologists and otolaryngologists, and therefore compromised during surgery [41-43].

These results are in line with previously reported findings on structured reporting for imaging: In radiology, structured reporting has shown superior results in terms of report completeness, quality and time efficiency when compared to conventional free-text reporting [22, 44]. Therefore, structured reporting is recommended by leading societies [45, 46]. Additionally, the utilization of structured reporting for pathology reports did lead to better patient management and outcomes [47] and showed beneficial effects in the setting of ENT sonography reporting [19-21]. While several studies report higher completeness, better readability, advantages in information extraction and educational benefits [19–24, 44, 48–50], sometimes a too rigid structure is claimed to be an obstacle for experienced readers in diagnostic imaging [51]. However, the results presented in this study show a comparable time investment towards surgical planning regarding both approaches. Due to the fact that the pre-existing CRs that had been created during the daily clinical practice, no information was available regarding the time needed to finish the report. Such being the case, no statement can be made concerning differences in time efficiency between CR/SR of paranasal sinus CTs. Also, structured reporting templates are now offering more flexible solutions and allowing the utilization of pre-defined text elements that can help to reduce orthographic mistakes as they are based on expert-reviewed text components [45]. This may also enhance report quality in the era of telemedical approaches which often suffer from non-native speaking reporting physicians [52-54]. Additionally, most of the recent publications have dismissed the problem of impaired radiology reports due to too rigorous structures which may be attributed to enhanced information technologies incorporated into SR templates [19-21].

The results presented in this study have to be interpreted in the context of its design: First, the results of this study are based on treatment planning for only one surgical procedure, namely FESS. As the surgical approach varies noticeably depending on the pathological pattern and the department’s internal standard and expertise, structured therapy planning may be of a very high importance in order to increase standardization. This may secondly lead to an increased scientific comparability of radiological findings as well as surgical planning, especially in the context of multicenter studies [55, 56]. Therefore, generalization of the results to other surgical procedures should only be done with great care. Additionally, the operation planning was carried out by residents in training, which does not represent the standard of care of board-certification. Second, imaging reporting of paranasal sinuses is a very suitable application for structured reporting in comparison to other diagnostic procedures with a higher degree of variability of pathological findings, like for example an MRI scan for unclear vertigo. In addition, structures to be reported may be valued significantly different by radiologists and otolaryngologists. In consequence, SR may be a valuable tool to ensure standardization while enabling the necessary variability. Third, while implementation of SR/SOP resulted in a significant increase of completeness, this study did not evaluate correctness of the reports. Therefore, no statement concerning the impact of SR/SOP on report content accuracy can be made. This has to be evaluated in future studies. Fourth, since participating Otolaryngology residents used the same CT scans for corresponding SOPs and COPs, potential bias due to testing or learning effects cannot be ruled out. A sequence of creating COPs before SOPs was chosen in order to reduce bias since, unlike SOPs, COPs do not offer any feedback to the user. Consequently, potential training effects are minimized to greatest possible extent. Additionally, residents prepared SOPs and COPs before the operation to reduce potential bias that may arise from additional knowledge acquired from intraoperative findings. Since there were only n = 6 residents with a sufficient experience in FESS procedures available at the University Medical Center Mainz, each resident prepared 5 corresponding SOPs/COPs within this study. By assigning residents one after the other, potential bias due to learning effects cannot be ruled out but are minimized since a greatest possible time interval between assignments may blur details used within SOPs. Finally, further analyses in other patient collectives, other healthcare systems and other treatment modalities may be needed to validate the findings presented in this study.

Conclusions

Implementation of SR and SOP results in a significantly increased completeness of radiologic reports and operation planning in the surgical management of paranasal sinus disease. Consequently, the combination of both may enhance the learning curve, both in radiology and otolaryngology, and decrease potential risks during endoscopic sinus surgery in rhinology centers.

Flowchart of the structured radiology reporting and structured operation planning structures.

(TIF)

Click here for additional data file.

Screenshot of structured reporting template.

For full decision tree, refer to S1 Table.

(TIF)

Click here for additional data file.

Screenshot of structured operation planning template.

For full decision tree, refer to S1 Table.

(TIF)

Click here for additional data file.

Key features of the structured reporting template.

(DOCX)

Click here for additional data file.

Key features of the structured operation planning template.

(DOCX)

Click here for additional data file.

Supporting raw data evaluated in this study.

(XLSX)

Click here for additional data file.

6 Oct 2020

PONE-D-20-22030

The role of structured reporting and structured operation planning in functional endoscopic sinus surgery

PLOS ONE

Dear Dr. Ernst,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Nov 20 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Giannicola Iannella, M.D

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for providing the full details of your ethics board approval and informed patient consent in the Ethics Statement. We ask that you additionally include this information in the Methods section.

3.Thank you for stating the following in the Competing Interests section:

[Wieland H Sommer is the founder and CEO of the company Smart Reporting GmbH, which hosts an online platform for structured reporting. Dominik Nörenberg and Thomas Huber are part-time employees of Smart Reporting GmbH. Matthias F Froelich is a medical consultant of Smart Reporting GmbH. The other authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.  This manuscript is part of a medical doctoral thesis presented by Manuel R Reissig at the University Mainz Medical School.].

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In this original article, the authors compared the completeness of structured reports of preoperative CT images and structured operation planning to conventional reports and conventional operation planning to improve treatment decisions on an individual level in 30 patients undergoing FESS surgery. The paper is well written and the topic interesting. References are adequate. I recommend acceptance in the present form.

Reviewer #2: Potential for Conflict of Interest:

Because of the relationship of some authors with SmartReporting GmbH, provide more information about the statement “no relationships with any companies, whose products or services may be related…”. Even if there were no actual conflicts of interest, is there any management plan (e.g., by the University of Mainz) for any potential, possible, future conflict of interest?

Methods:

- Submit a flow-chart of the SR and SOP structures and decision-trees for review (as supporting information).

- How many residents participated in preparing report? Were they at the same stage in their training? Did a given resident do an SR and a CR for the same patient? If so, would this introduce a bias for which ever report was prepared secondly? There are similarly questions if a given resident prepared reports on more than one patient.

Results:

(1) Is there any data concerning an association or correlation between “completeness” (an outcome measure) and “treatment decisions on an individual level” (part of the overall aim of the study)?

Discussion:

(a) How could it be determined whether the use of SR (for pre-operative CT images) and SOP “may decrease potential risks during FESS?

(b) What about having a structured outcomes report (“SOR” for post-operative outcomes) to go along with the SR and SOP?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: John H Anderson, MD, PhD

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

11 Oct 2020

Dear Ladies and Gentlemen,

we would like to thank the reviewers and editor for giving our manuscript so much consideration. We carefully revised our manuscript PONE-D-20-22030 according to the reviewer’s suggestions and feel confident that the manuscript meets your expectations by now.

Editor:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf

and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

The manuscript and the associated files meet the PLOS ONE’s style requirements.

2. Thank you for providing the full details of your ethics board approval and informed patient consent in the Ethics Statement. We ask that you additionally include this information in the Methods section.

The Ethics Statement was included in the Methods sections of our revised manuscript.

3. Thank you for stating the following in the Competing Interests section:

[Wieland H Sommer is the founder and CEO of the company Smart Reporting GmbH, which hosts an online platform for structured reporting. Dominik Nörenberg and Thomas Huber are part-time employees of Smart Reporting GmbH. Matthias F Froelich is a medical consultant of Smart Reporting GmbH. The other authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article. This manuscript is part of a medical doctoral thesis presented by Manuel R Reissig at the University Mainz Medical School.].

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Since there are no restrictions, the statement "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” Was added to our Competing Interest statement in our cover letter.

4. Have the authors made all data underlying the findings in their manuscript fully available? (Reviewer #2: No)

As part of the PLOS ONE policy on data sharing, we prepared a supporting information table that provides the raw data that includes:

• The values behind the means, standard deviations and other measures reported;

• The values used to build graphs;

• The points extracted from images for analysis.

When revisiting our entire raw data collection to prepare the supporting information table (S3 Table), we noticed a calculation mistake concerning the mean VAS values of our COP user satisfaction questionnaire and have corrected it accordingly. Consequently, mean VAS values for COP as well as p-values for questions 1 (Q1) through question 6 (Q6) slightly differ. However, these changes did not result in any changes regarding the level of significance of inter-group-varability. Mean values for SOP as well as overall user satisfaction that represents the sum of all six questions was calculated correctly, both for SOP and COP. Changes can be tracked in the “tracked changes” version of our revised manuscript. The corrected values can be objectively reproduced through the raw data provided in the S3 Table. Figure 3 was revised accordingly.

Old values: New values:

User satisfaction Q1: 8.0 vs. 5.5, p>0.05 8.0 vs. 4.0, p=0.058

User satisfaction Q2: 7.8 vs. 5.3, p=0.041 7.8 vs. 3.7, p=0.035

User satisfaction Q3: 8.8 vs. 6.0, p=0.022 8.8 vs. 4.0, p=0.038

User satisfaction Q4: 7.2 vs. 5.3, p=0.028 7.2 vs. 3.5, p=0.04

User satisfaction Q5: 8.2 vs. 5.4, p=0.018 8.2 vs. 3.3, p=0.038

User satisfaction Q6: 8.3 vs. 6.7, p>0.05 8.3 vs. 5.8, p=0.34

Time efficiency: 148 vs. 160s, p=0.71 148 vs. 160s, p=0.61

We apologize sincerely for these miscalculations that were honest mistakes that did not happen in bad faith. We would like to take an entirely open approach in addressing this topic by disclosing it here. As mentioned above the level of significance of differences between groups was not changed by these recalculations and therefore there was no impact on the conclusions of this study.

Reviewer #1:

In this original article, the authors compared the completeness of structured reports of preoperative CT images and structured operation planning to conventional reports and conventional operation planning to improve treatment decisions on an individual level in 30 patients undergoing FESS surgery. The paper is well written and the topic interesting. References are adequate. I recommend acceptance in the present form.

Thank you very much for your review and your recommendation for acceptance of our manuscript.

Reviewer #2:

1. Because of the relationship of some authors with Smart Reporting GmbH, provide more information about the statement “no relationships with any companies, whose products or services may be related…”. Even if there were no actual conflicts of interest, is there any management plan (e.g., by the University of Mainz) for any potential, possible, future conflict of interest?

Thank you very much for your comment. As the reviewer pointed out correctly, Wieland H Sommer is the founder and CEO of Smart Reporting GmbH. Dominik Nörenberg and Thomas Huber are part-time employees of Smart Reporting GmbH. Matthias F Froelich is a medical consultant of Smart Reporting GmbH. The human resources department of University Medical Centre Mannheim has approved all part-time and consulting contracts. None of the other authors does currently have a relationship with Smart Reporting GmbH.

In addition, University Medical Center Mainz has a management plan in place for handling of potential conflicts of interests for the institution and its employees including several standardized procedures and thorough approval processes by the management in cooperation with both the legal department and the human resources department:

• Every employment contract of employees of University Medical Center Mainz are subject to approval by the personnel department. The corresponding form can be found here: https://www.verwaltung.personal.uni-mainz.de/files/2020/06/Antrag-NT-Beamte-neu.pdf

Furthermore, a corresponding employment contract would have to be carried out without opposing the activities for University Medical Center Mainz. This is stated specifically in corresponding legislation, the “Landesgesetz über die Errichtung der Universitätsmedizin der Johannes Gutenberg-Universität Mainz“ in §10, §35 and §36: http://landesrecht.rlp.de/jportal/portal/t/y9j/page/bsrlpprod.psml?doc.hl=1&doc.id=jlr-JohGutUniMedGRPrahmen:juris-lr00&documentnumber=1&numberofresults=44&showdoccase=1&doc.part=X¶mfromHL=true

• There is no agreement or contract regarding the publication or other topics between Smart Reporting GmbH and the ENT department of University Medical Center Mainz. The publication process was initiated, maintained and supervised by the corresponding author at the lead site, University Medical Center Mainz. Every future contract would be subject of approval by the legal department of University Medical Center Mainz. Additionally, the institutional ethics committee audited potential conflicts of interest before giving approval for this study.

• Additionally, while the founder and employees of Smart Reporting GmbH were involved in the planning phase of this study and contributed by providing the structured reporting templates within the Smart Reporting Software Solution free of charge. Data were analyzed by employees of the University Medical Center Mainz with no relationship to Smart Reporting GmbH. Data tables and corresponding analyses were not shared with Smart Reporting GmbH.

2. Methods:

2.1 Submit a flow-chart of the SR and SOP structures and decision-trees for review (as supporting information).

A flow-chart of the SR and SOP structures and decision-trees was created and submitted for review as supporting information.

2.2 How many residents participated in preparing report? Were they at the same stage in their training? Did a given resident do an SR and a CR for the same patient? If so, would this introduce a bias for which ever report was prepared secondly? There are similarly questions if a given resident prepared reports on more than one patient.

In total, n=6 Otolaryngology residents with a mean work experience of 4.5 ± 0.9 years participated in preparing operation plannings. With respect to the low standard deviation, it can be assumed that they were at the same stage in their training. All structured radiology reports were created by one board-certified radiologist. These SRs were compared to pre-existing CRs which were created during clinical routine.

Since participating residents used the same CT scans for corresponding SOPs and COPs, potential bias due to testing or learning effects cannot be ruled out. In a first step, COPs of the CT scans were created analogously to the clinical routine. In a second step, participating residents used the same images to generate corresponding SOPs. This sequence was chosen in order to reduce bias since, unlike SOPs, COPs do not offer any feedback to the user. Consequently, potential training effects are minimized to greatest possible extent. Additionally, residents prepared SOPs and COPs before the operation to reduce potential bias that may arise from additional knowledge acquired from intraoperative findings.

Within this study, participating residents assigned to carry out FESS procedures under the direct supervision of the corresponding author one after another. Since there were only n=6 residents with a sufficient experience in FESS procedures available at the University Medical Center Mainz, each resident prepared 5 corresponding SOPs/COPs within this study. By assigning residents one after the other, potential bias due to learning effects cannot be ruled out but are minimized since the time interval between assignments is as long as possible. A long time interval may blur details used within SOPs.

These points raised by the reviewer were added and critically discussed in the discussion section of the revised manuscript.

3. Results:

Is there any data concerning an association or correlation between “completeness” (an outcome measure) and “treatment decisions on an individual level” (part of the overall aim of the study)?

Thank you for this important comment. We have to apologize that our statement concerning the overall aim of this study was potentially imprecise with regard to “treatment decisions on an individual level”. The principle aim of this study was to perform a comparative analysis of the completeness of CT reports and surgical plannings using SR and SOP, respectively, to the conventional approaches (e.g. CR and COP). Additionally, we hypothesized that an increase in completeness of radiology reports and operation plannings may have the potential to improve treatment decisions and patient outcome in the long term. This hypothesis has to be evaluated by future studies.

4. Discussion:

4.1 How could it be determined whether the use of SR (for pre-operative CT images) and SOP “may decrease potential risks during FESS?

To evaluate the impact of structured reporting approaches on diagnosing and treating paranasal sinus disease, we plan to use three different approaches. In a first step, which is the topic of the present study, we evaluated structured reporting templates for radiology reporting and operation planning and their impact on report completeness. This study represents the preoperative part of the project. In a second step, we plan to evaluate the impact of a structured reporting template, which creates the actual operation report with respect to completeness, time efficiency and extent of the operation. Due to the structured and standardized format of the radiology report, the operation planning and the operation report, similarities and differences can be analyzed semi-automatically. Consequently, combining the preoperative and intraoperative approach could help determine whether the use of SR for pre-operative CT images and SOP may decrease potential risks during FESS by pointing out potential hazards before the operation.

4.2 What about having a structured outcomes report (“SOR” for post-operative outcomes) to go along with the SR and SOP?

Thank you for this thoughtful comment that represents the final step of our project. By creating a structured reporting template for the documentation of postoperative care and postoperative outcome, the impact of the first two steps (as described in 4.1) can be evaluated with respect to the actual incidence of potential risks. Due to the pleasingly low rate of severe complications in FESS, a dedicated study will have to involve multiple centers in order to recruit a sufficient amount of patients thus generating the appropriate power.

Thank you very much in advance for re-reviewing our manuscript!

Best regards,

Dr. Benjamin Ernst

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

10 Nov 2020

The role of structured reporting and structured operation planning in functional endoscopic sinus surgery

PONE-D-20-22030R1

Dear Dr. Ernst,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Giannicola Iannella, M.D

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

the authors have well revised the manuscript in accordance with the comments of revision 2. in my opinion this is an interesting paper that is suitable to be published on PLOS-ONE. Congratulations on your efforts.

Best regards

Dr Giannicola Iannella

Reviewers' comments:

Reviewer #2: The authors provided appropriate and complete responses to all issues.

The following were noted:

(1) The detailed response to my review issue #1, regarding the potential for a conflict of interest, was comprehensive and fully adequate. Note an important point made by the authors, “Data were analyzed by employees of the University Medical Center Mainz with no relationship to Smart Reporting GmbH. Data tables and corresponding analyses were not shared with Smart Reporting GmbH.”

(2.1) Flow-charts of the SR and SOP structures were submitted as supporting information.

(2.2) The detailed response to my review issue #2.2 is fully adequate. Note that “In a first step, COPs of the CT scans were created analogously to the clinical routine. In a second step, participating residents used the same images to generate corresponding SOPs. This sequence was chosen in order to reduce bias since, unlike SOPs, COPs do not offer any feedback to the user.” Although there remains the possibility for bias, reasonable efforts have been made to reduce the possibility.

(4.2) In regard to the future possibility of a structured outcomes report, the authors noted the importance of this and “a dedicated study (for this) will have to involve multiple centers in order to recruit a sufficient amount of patients thus generating the appropriate power.”

12 Nov 2020

PONE-D-20-22030R1

The role of structured reporting and structured operation planning in functional endoscopic sinus surgery

Dear Dr. Ernst:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Giannicola Iannella

Academic Editor

PLOS ONE