Changing antibiotic prescribing practices in outpatient primary care settings in China: Study protocol for a health information system-based cluster-randomised crossover controlled trial.

BACKGROUND: The overuse and abuse of antibiotics is a major risk factor for antibiotic resistance in primary care settings of China. In this study, the effectiveness of an automatically-presented, privacy-protecting, computer information technology (IT)-based antibiotic feedback intervention will be evaluated to determine whether it can reduce antibiotic prescribing rates and unreasonable prescribing behaviours.
METHODS: We will pilot and develop a cluster-randomised, open controlled, crossover, superiority trial. A total of 320 outpatient physicians in 6 counties of Guizhou province who met the standard will be randomly divided into intervention group and control group with a primary care hospital being the unit of cluster allocation. In the intervention group, the three components of the feedback intervention included: 1. Artificial intelligence (AI)-based real-time warnings of improper antibiotic use; 2. Pop-up windows of antibiotic prescription rate ranking; 3. Distribution of educational manuals. In the control group, no form of intervention will be provided. The trial will last for 6 months and will be divided into two phases of three months each. The two groups will crossover after 3 months. The primary outcome is the 10-day antibiotic prescription rate of physicians. The secondary outcome is the rational use of antibiotic prescriptions. The acceptability and feasibility of this feedback intervention study will be evaluated using both qualitative and quantitative assessment methods. DISCUSSION: This study will overcome limitations of our previous study, which only focused on reducing antibiotic prescription rates. AI techniques and an educational intervention will be used in this study to effectively reduce antibiotic prescription rates and antibiotic irregularities. This study will also provide new ideas and approaches for further research in this area. TRIAL REGISTRATION: ISRCTN, ID: ISRCTN13817256. Registered on 11 January 2020.

Background

Antibiotic resistance is a widespread concern around the world [1]. In the past decade, 50% of antibiotic prescriptions worldwide have been used to treat colds, and, according to a report of World Health Organization (WHO), many of these cases have no indication of antibiotic use [2]. The main forms of antibiotic overuse and abuse are non-indication, overdose, and multi-drug combined use of antibiotics which violate the principles of antibiotic use [3, 4]. Overuse and abuse of antibiotics are major risk factors for antibiotic resistance [5-9]. The total consumption of antibiotics in 71 countries (including China) increased by more than 50% from 2000 to 2010 [10]. The antibiotic prescription rate in China is twice as high as that recommended by WHO, and higher than that in developed countries and most developing countries [9-11]. Guizhou province, located in southwest China, has the largest number of poor people and is the largest poverty-stricken area in China. Our retrospective study of Guizhou’s 39 primary care facilities showed that most of the medical staff have less than a university education, and the unreasonable rate of antibiotic prescriptions was over 90%. The incidence of bacterial resistance in Guizhou province is on the rise [12-14].

Previous studies have shown that there are a variety of methods for antibiotics prescription control, including educational intervention [15], communication training [16], nursing point testing [13], electronic decision support system [17], and delayed prescription [18], but reports of a feedback intervention are rare. Existing feedback interventions mainly focused on email or poster information [18-26], regular or irregular assessment/audit of antibiotic prescriptions [21, 22, 25–29], or prescription recommendations from experts and peers delivered at a meeting or online [18, 19, 24, 25, 30–32]. Some studies even report prescribing information publicly [20, 33]. However, these methods are somewhat mandatory and censored, which can cause negative emotions to the physician, and needs long-term intervention by professionals [34-36]. As a result, some studies have shown negative results [37, 38].

We previously conducted a cluster randomised crossover-controlled trial to reduce antibiotic prescription rates based on existing health information system (HIS) in primary care institutions in Guizhou [39]. In this study, the antibiotic prescription rates of the two groups decreased by 15% over the 6-month study period. However, a limitation was that prescription of unreasonable antibiotics were not considered.

In view of this, this new study will analyze and process the digital information in electronic medical records with big data technology, and use the depth map neural network (DMNN) in artificial intelligence (AI) technology to provide physicians with the best diagnosis and treatment suggestions in real time.

This is a cluster-randomised, open controlled, crossover, superiority trial. We aim to describe an automatically-presented, privacy-protecting, DMNN technology-based feedback intervention model. The feedback intervention model can not only effectively remind physicians of the deviation of their prescribing behavior, but also humanely give reasonable suggestions, which can greatly improve the enthusiasm of physicians to participate. In addition, an educational handbook developed by us for primary outpatient institutions will be distributed to primary physicians. A pilot study will be conducted to test the physician motivation and intervention effectiveness. The comparators are usual care i.e., primary care hospitals within Guizhou which did not receive any intervention.

Methods design

Trial setting

The trial will be carried out in primary care institutions in four geographical regions of Guizhou province: the east, west, north, and south. We have identified the primary care institutions as township public hospitals and community health service centers in a previous study, which provide primary health care services to the majority of rural residents in China [39]. Guizhou province has a population of approximately 39 million and is one of the most impoverished provinces in China. A township public hospital or community health service center is a comprehensive institution for health administration and medical prevention work established by a county or township. Up to 2019, there were 1,329 township hospitals in Guizhou province, with only 7,211 practicing physicians, most of whom have only received vocational education equivalent to a junior college or technical secondary school level [39-41]. According to our 2018 study of 16 primary hospitals in rural areas of Guizhou province, most (63%) of the antibiotic prescriptions were made by resident physicians with a below college level of education and most antibiotic prescriptions were deemed to be inappropriate [13].

The Health Information System (HIS) involved in this study was designed and developed by Guizhou Lianke Weixin Technology Co., Ltd. (LWTC) under the authorization of Information Center Guizhou Provincial Health Commission (ICGPHC). By accessing the port of ICGPHC platform, relevant data can be obtained. All of the research team’s preliminary research data will come from the platform. One of the interventions included in this study, the antibiotic prescription rate ranking feedback intervention early warning system, was jointly developed by the company’s technical staff based on the platform and the applicant’s study requirements. It has been successfully implemented for 6 months from February to August 2018 in 31 primary care settings in Guizhou province [39].

Graph neural network technology

Graph neural network technology (GNN) is an advanced form of AI technology [42]. A GNN model can realize the formulation and recommendation of an ideal treatment plan according to the optimized causal reasoning function of the model, develop an AI real-time warning system for unreasonable antibiotic prescription, and conduct intelligent and reasonable interventions on the antibiotic prescription patterns of physicians in primary care institutions. In this process, the network system will involve repeated self-learning and correction to improve the early warning system for the unreasonable use of antibiotics [43-45]. Finally, through the evaluation of the intervention effect of the multi-level model, an intervention model of antibiotic prescription can be obtained to provide an economic, feasible, and effective reference plan for solving the overuse and abuse of antibiotic prescription in primary care institutions, thus reducing the drug resistance rate and burden of public finances in rural areas of China.

Recruitment

Clusters

According to the inclusion criteria of the previous study [39], a cluster is defined as one in which: 1) the primary care institutions are in Guizhou province and have the same HIS system; 2) each primary care institution has at least 3 outpatient general physicians (GPs), each of whom has a history of seeing at least 100 patients, on average, every 10 days; 3) all physicians have worked in the hospital in their current position for more than 1 year. In 2020, 252 primary care institutions in Guizhou used the LWTC HIS system of which 132 met the eligibility criteria. These institutions were randomly located in 6 cities of Guizhou province—Bijie, Zunyi, Tongren, Anshun, Qiannan and Liupanshui.

Patients

The study subjects were primary care institutions that provided health care services for township residents in Guizhou province. For this purpose, identified eligible patients as all patients who received an initial diagnosis and were prescribed by an outpatient physician at the participating care institutions. The main diagnostic categories for all diseases were based on the International Classification of Diseases, 10th Edition (ICD-10) codes [46].

Process

A complete trial process is shown in Fig 1. To avoid selection bias [47] we will use a crossover design in which each group will receive different treatments at different times [48]. We will divide outpatient physicians in each selected primary care institution equally into two sequences according to the principle of randomization.

Fig 1

Overview of enrollment, intervention, and assessments of the cross-over design trial.

The trial will be divided into two stages and will last for 6 months. The first stage will last for 3 months, with the Group 1 enrolled in the intervention group and the Group 2 in the control group. The second stage will also last for 3 months, with the two groups crossing over. The cross-over design is shown in Fig 2.

Fig 2

Cross-over trial diagram.

Control

In the control group, no form of intervention will be provided. The control group of physicians participating in the trial will continue to provide their usual treatment methods and experience to diagnose and treat patients.

Intervention

In the intervention group, based on previous trials [39], an antibiotic feedback intervention composed of three parts will be developed, including a real-time warning of improper antibiotic use and a 10-day summary of antibiotic prescription rate ranking and related information. The distribution of homemade educational manuals will also be made (details to be given in section 3).

(1) AI-based real-time warning pop-up windows of improper antibiotic use. Based on the HIS system of primary care institutions, the warning plug-in uses graph neural network technology to automatically access the prescription data in the background. It will compare each prescription with the big data and DMNN modeling results, determine whether the antibiotic prescription (including type, dosage, and course of treatment) is reasonable to be used in the consultation service and will provide a real-time automatic warning alert for unreasonable antibiotic prescription. Once a physician prescribes an unreasonable antibiotic, a pop-up window will automatically appear in the lower right corner of the screen to alert the physician that the prescription is unreasonable and indicate the type of unreasonable use of antibiotics. The form of pop-up window is shown in Fig 3. The pop-up window will disappear if the physician clicks on it. It will also automatically disappear after 5 minutes. The duration of the pop-up window will be recorded automatically by system. Extreme durations will be noticed (i.e., 1 second or 5 minutes). According to previous research [13], we define unreasonable prescription of antibiotics with the following indicators: 1. Incorrect or unnecessary use: for example, a physician gives antibiotics for which there is no clear indication; 2. Incorrect antibacterial spectrum: for example, prescribing aminoglycoside drugs for gram-positive bacteria; 3. Combined antibiotic use: administration of more than one injectable or oral system antibiotic at a time without any indication, for example, amoxicillin capsule and ceftazidime injection in combination.

Fig 3

Example of unreasonable use of antibiotic warning pop-ups.

(2) Pop-up windows of antibiotic prescription rate ranking. This reminder system is a plug-in developed in a previous study [39]. We will implement pop-up windows of antibiotic prescription use in the HIS system. The system will appear on the physician’s screen in the form of an automatic pop-up window every 10 days, informing them of their ranking in terms of their antibiotics prescription rate within the same outpatient department, actual antibiotic prescription rate and related information. The information seen by each physician will be confidential. The physicians have the freedom to read this feedback massage or not. When the physician logs into the HIS, a pop-up window or link will appear on the computer screen, prompting him or her to view the message. If a physician presses the ESC button, it will disappear. All the on-screen procedures, including click rate and the time of the message, will be recorded automatically.

Based on a previous study involving 16 hospitals in the early stage [13], we will invite 48 medical experts to conduct two rounds of demonstration using the Delphi method. Two guidance proposals with expert consensus will be formed as detailed in the section below.

(3) Distribution of educational manuals. The educational manuals include 2 parts: "Instruction and Recommendations for Outpatient Clinical Use of Antibiotics in Primary Care Institutions" and "Instruction and Recommendations for Diagnosis of Common Infectious Diseases in Outpatients of Primary Care Institutions". We have consolidated them into a manual for distribution to outpatient physicians in primary care institutions.

The first part is the recommendation for the rational use of antibiotics, and the second part is the diagnostic guidance of the symptoms, signs and auxiliary examinations for common infectious diseases such as digestive system, respiratory system, and urinary system.

In the first part, we divided the criteria for rationality of antibiotics into four categories: 1. Suitable: preferred antibiotic; 2. Optional: the antibiotic can be used or substituted; 3. Wrong-spectrum: the antibacterial spectrum is not used correctly; 4. No use: In the second part, based on the proportion of different diagnostic criteria and the weight, we set the most valuable diagnostic criteria as“4”, and the standard for low diagnostic value was set to "1".

Pilot study

Prior to the formal trial, we will conduct a pilot study to test the feasibility of this intervention trial. Specifically, we will work with HIS engineers to test the sensitivity and reliability of our newly developed AI-based early warning system, which is based on the existing pop-ups of antibiotic prescription information. We will also distribute the educational materials to physicians. Our study group members will be trained to explain the intervention process to the relevant manager and physicians.

The pilot study will be conducted in the outpatient department of a hospital. We will interview the director of the hospital and all qualified outpatient physicians. They will be informed about the precautions, processes, risks and benefits of the pilot study and the method of data collection in the informed consent form. We will include all physicians who signed the informed consent form in accordance with the intention-to-treat (ITT) principle [49]. And we will follow the same outline as in the formal trial.

The pilot study will last 3 months. At the end of the pilot study, the attitude, opinions, and suggestions of physicians on the feedback intervention will be obtained through a questionnaire survey. According to the feedback results of the questionnaires and the field work conducted during the pilot study, we will determine the following research points before the formal intervention trial: 1. Whether the newly developed AI-based plug-in can realize long-term, large-scale and high-precision real-time warning of prescriptions; 2. Whether all the pop-up windows and links of the warning system can work normally; 3. Whether physicians can understand and grasp all the antibiotic prescription intervention information reasonably quickly; 4. Whether the feasibility of the feedback intervention, specifically, the majority of primary care institutions and their outpatient physicians think that our research work is feasible. In other words, most of the outpatient physicians who received the intervention will feel that our study is helpful to their work by the end of the trial and will continue to use our feedback intervention.

Data collection and management

Approved by ICGPHC, with the help of engineers from LWTC, we will retrieve antibiotic prescriptions and total prescriptions of the HIS in primary care institutions for statistical analysis through the downloaded program written by engineers. Data collection will proceed from April to October 2021. All data will be collected real-time from ICGPHC’s data center, thus there will be no interference to hospitals and physicians during the data collection process. All researchers participating in the data collection will sign a confidentiality agreement.

Due to the large amount of data, the downloading process and management of prescription data will be the responsibility of the two main authors (YYF and CY). The collected data will be entered into a standard format database to store all the outcome and covariate data. At the same time, we will generate codes that connect physicians and patients to facilitate the analysis and processing of individual-level data. In addition, the demographic and professional information of physicians will be obtained from the personnel management department of the primary hospital.

Randomization and blinding

After recruitment, the information technology staff of LWTC will be invited to randomly assign (computer-generated random numbers) all recruited primary care institutions to an intervention group and a control group in a ratio of 1:1.

Since this is a behavioural intervention trial, the physicians will have a clear idea of the intervention when they sign the informed consent form and will be able to determine whether they are in the intervention or control group at the start of the trial. Therefore, the design of this study makes it impossible to use a blinded approach to participants and researchers [39].

Outcome measures

Primary outcome

The primary outcome is the 10-day antibiotic prescription rate of physicians defined as the number of antibiotic prescriptions divided by the total number of prescriptions in each 10-day time period (the term “prescription” as used here refers to one drug) [39]. Indicators of related covariates include: 1. Baseline characteristics of the physician, such as age, gender, job title, education, and working years; 2. Patient characteristics, such as gender, age, ethnicity, and disease; 3. Antibiotic information, such as name, dosage form, route, and amount in grams.

Secondary outcome

The secondary outcome is the rational rate of antibiotic prescription defined as the reasonable amount of antibiotics prescribed during the study period divided by the total amount of antibiotics prescribed though the study period.

Sample size calculation

Since the outcome variable (antibiotic prescription rate per physician) is a continuous variable, we used the two independent means formula (two-tailed test) to calculate the required number of physicians to recruit into the study as given by the following formula.

In the above formula, the parameters are as follows: α = 0.05, Z(0.975) = 1.96, β = 0.2, Z(0.8) = 0.84. Based on the data from our previous study, the pre-intervention mean (μ1) = 35.0, and the pre-intervention variance (σ1) = 15.0; the post-intervention mean (μ2) = 30.0, and the post-intervention variance (σ2) = 15.0 [39]. Since a 1:1 ratio was adopted in the experimental design, the sample size ratio (r) of the two groups was 1.0. From this we can calculate the sample size as n1 (group 1) = n2 (group 2) = 142 physicians per group.

To allow for a 10% non-response rate, the sample size was increased to 160 physicians per group for a total of 320 physicians. Since most of these primary hospitals have 3–4 outpatient physicians who meet the inclusion criteria, we will randomly select hospitals using a computer-generated number from the list of 132 hospitals that met the inclusion criteria. The total number of hospitals to be included in the study will be determined by whether the hospitals have 320 outpatient physicians who meet the inclusion criteria. We will visit the selected hospitals and ask the physicians to sign the informed consent form. Fig 4 shows a flow chart of the trial where group 1 represents the group of physicians who will receive the intervention in the first stage.

Fig 4

Flow chart of the crossover trial.

Statistical analyses

We will follow the statistical method of the previous study [39]. Descriptive statistics will be presented for the outcome variables and related covariates. We will compare the antibiotic prescription rate and the antibiotic prescription rationality rate between the two groups (Group 1 and Group 2) at baseline, crossover point and at the end of the trial. After testing of the data for normality, Student’s t-test, paired t-test, Wilcoxon signed-rank test or Rank-sum test will be used to compare antibiotic prescription rate and antibiotic rationality rate between the two groups for horizontal (between groups) comparison or vertical (at different time points within the same group) comparison, to observe the difference and trend of change. Secondly, because it will take some time before feedback interventions have any impact on antibiotic prescription rates, a transition model suitable for studies that includes regular follow-up intervals and changing exposure and outcome states will be used to predict the impact of the intervention on changes in antibiotic prescription rates and rationalization rates within the same physician over time. The transition model ensures that the "carry-over" impact is adjusted and that correlations over time are addressed.

Following ITT principles [49], outpatient physicians from all participating primary care facilities will be included in the analysis. We will report all results according to the CONSORT guidelines [50].

All data analyses will be done using R version 4.0.4

Process evaluation

Based on the Medical Research Council’s 2008 framework [51] and Grant’s framework for process evaluations of cluster randomised trials of complex interventions [52], we will carry out a mix of qualitative and quantitative process evaluation method.

The purpose of the process evaluation is to determine if our complex interventions are effective. Therefore, we will focus on the following key research questions that need to be addressed: "What part of the feedback intervention worked?", "Is the intervention effective for the target population?" and “Why are feedback interventions effective?” to set up the process evaluation plan. The specific evaluation content of the evaluation will include the following aspects: 1. To evaluate the feasibility, universality and acceptability of the AI-based real-time warning system of inappropriate use of antibiotics, which is based on GNN technology, and to assess the warning system for a high proportion of antibiotic prescription; 2. To assess the feasibility and reliability of the educational materials; 3. To evaluate the sampling and recruitment process at the cluster level (primary care hospitals) and individual level (outpatient physicians); and 4. To assess the response of the study to the intervention trial at the cluster level and the individual level.

We will use document review (recruitment standards, informed consent, and education manual), telephone return visit of outpatient physicians in primary care institutions and questionnaire survey of intervention trials as the evaluation methods of the process evaluation. The survey and interview guidelines used in the process evaluation are based on the Theoretical Domains Framework guidance [53]. This will give us a deeper understanding of how feedback interventions work. We will use the same method for data collection in both the control group and the intervention group. The prescription data will be aggregated every 10 days.

For the qualitative study, the sample size will be determined based on the feedback intervention test results. The sample size will be adjusted midway during the study according to the personnel turnover and the situation of withdrawal from the study. The qualitative research method will adopt the explanatory description method.

The results of the process evaluation will provide useful information for future feedback intervention trials.

Trial management

Prof. Yue Chang and Mr. Yuanfan Yao from the Guizhou Medical University and Dr. Zhezhe Cui from the Guangxi Zhuang Autonomous Region Center for Disease Control and Prevention will be the co-guarantees of the trial and will have full access to the trial dataset. We have signed a data confidentiality agreement with the Guizhou Provincial Health Commission to protect the safety and privacy of the physicians, patients concerned and to ensure that all data is collected in accordance with accepted ethical guidelines, properly stored and used for research purposes only. We will also invite experts in relevant fields to set up a trial steering committee. Conference calls will be held every month until the completion of the study. Committees may also meet on an ad-hoc basis as required. Any modification of the agreement will be decided by the meeting.

Trial registration

The trial was registered at Current Controlled Trials: ISTRCTN13817256 on 11 January 2020.

Trial status

We expect to formally launch the trial in April 2021. The pilot study has started under the consent of the local health department and will be finished by the time of this proposal submission. We aim to publish the results in international medical journals, and promote our research results in Guizhou province.

Ethical approval and consent

The trial has been approved by Human Trial Ethics Committee of Guizhou Medical University (Certificate No.: 2019 (148)) in Dec. 27, 2019. All the physicians at the primary care institutions participating in the trial will sign the informed consent. When collecting and analyzing data, we will remove patient confidential information, such as name and national ID. Physicians’ prescription data will also be subject to strict confidentiality measures.

Discussion

This study builds on a previous feedback intervention trial conducted by our research team in primary care institutions in Guizhou province [39], where we conducted the trial based on HIS and successfully reduced antibiotic prescription rates among outpatient physicians. Our current study has some advantages over this and other previous studies.

The purpose of this intervention is to establish a highly compliant, economic, and feasible artificial intelligence early warning system for antibiotic prescription control in primary care institutions of Guizhou province. The system adds an AI-based real-time warning of inappropriate antibiotic use and self-compiled education manuals to a previous study [39] that included only a 10-day pop-up alert message of antibiotic prescribing rate ranking and related information. Based on the results of our preliminary study, this intervention is suitable for primary care institutions in developing countries with paperless office conditions.

To overcome the limitations of our previous study, which only focused on reducing antibiotic prescription rates, new techniques and evaluation indexes will be used to expand our study. GNN technology will be adopted in this study to develop an AI real-time warning system for unreasonable antibiotic prescriptions, to make intelligent judgments and interventions on antibiotics prescribed by the primary care physicians. This new heterogeneous and composite network structure model and iterative optimization method will make the GNN not only have the performance of traditional high network expression ability, but also avoid the problems of traditional deep learning technology, such as complex iteration, poor understanding, and poor interpretation [44, 45]. Once the research results are applied into practice, they will provide more effective, convenient, quick, and economical intervention measures for preventing the overuse and abuse of antibiotics in primary care institutions.

In addition, we will invite 48 domestic experts to construct a recommended manual for rational use of antibiotics and diagnosis of common infectious diseases for primary care institutions using the Delphi method, which we expect to be highly praised by physicians and hospital managers in the pilot study and will have a good guiding effect on primary care institutions [54, 55].

Despite these advances, our study will inevitably follow the limitations of previous studies. Firstly, infectious diseases in the southwest of China show a seasonal pattern. Our intervention trial will begin in April 2021 and continue for six months. Therefore, seasonal effects may not be completely eliminated during the trial period. Secondly, in the field of epidemiological trials and infection prevention, changes in subjects’ behavior due to the Hawthorne effect will have a certain impact on the results. Finally, when the trial moves from the first stage to the second stage, the experimental conditions of the subjects are changed, which may affect the results due to the carry-over effect. Since the feedback intervention trial in this study is a behavioral intervention trial, it is difficult to eliminate the behavior once it is developed, so the effect cannot be eliminated by setting a washout period as commonly implemented in drug intervention trials.

SPIRIT + checklist.

(DOCX)

Click here for additional data file.

AI system in details: Deep learning and training of antibiotic prescription data in deep graph neural network technology.

(DOCX)

Click here for additional data file.

14 Jul 2021

PONE-D-21-16838

Changing antibiotic prescribing practices in outpatient primary care settings in China: study protocol for a health information system-based cluster-randomised crossover controlled trial

PLOS ONE

Dear Dr. Chang,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Aug 28 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Vijayaprakash Suppiah, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

3. Thank you for stating the following financial disclosure:

“The funders had and will not have a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

At this time, please address the following queries:

a) Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution.

b) State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

c) If any authors received a salary from any of your funders, please state which authors and which funders.

d) If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“The study was funded by the National Natural Science Foundation of China Grant on “Research on feedback intervention mode of antibiotic prescription control in primary medical institutions based on the depth graph neural network technology” (71964009) and the Science and Technology Fund Project of Guizhou Provincial Health Commission Grant on “Application Research of Deep Learning Technology in Rational Evaluation and Intervention of Antibiotic Prescription” (gzwjkj2019-1-218).”

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“The funders had and will not have a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

5. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript.

6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions?

The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field.

Reviewer #1: Partly

Reviewer #2: Partly

Reviewer #3: Yes

**********

2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses?

The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory.

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Yes

**********

3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable?

Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

4. Have the authors described where all data underlying the findings will be made available when the study is complete?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics.

You may also provide optional suggestions and comments to authors that they might find helpful in planning their study.

(Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an interesting protocol. The protocol used artificial intelligence - based real-time warnings, pop-up windows and educational manuals to evaluate whether those interventions could reduce antibiotic prescription rates. However, as a protocol, the article did not answer the questions whether these three can reduce the use of antibiotics. Moreover, I have some other concerns, listed as follows:

1. The introduction part needs to be more streamlined.

2. Can the results from Guizhou reflect China or other places of the world?

3. The reason for choosing Guizhou is because of the higher rate of bacterial resistance in this area? Or is the rate of overuse of antibiotics higher?

4. The study is designed to analysis whether interventions can reduce the antibiotic use rate of physicians. But it will be more interesting if the study can simultaneously analyze whether the prognosis of these doctors' patients has changed, or whether the bacterial resistance rate has changed.

5. One intervention of this protocol is "pop-up windows of antibiotic prescription rate ranking". I am curious whether this ranking will have a negative effect on medical decision-making. I mean, if in some departments where the use of antibiotics always high, such as department of infectious diseases or respiratory, will the physician change the correct medical decision because of this ranking?

6. The authors tried to analyze whether their intervention could reduce the use of antibiotics. Does the author have a theoretically reasonable interval of antibiotic usage in each department and how are they calculated? I believe that in different departments, this should be different.

7. As I mentioned before, this is a protocol. Although interesting, it still needs further expected.

Reviewer #2: In this paper, relating to the author's previous research results, doctors from township public hospitals and community health service centers in Guizhou Province were designated as participants. Two independent average formulas were used to calculate the number of doctors recruited into the study, and they were randomly divided into control groups and intervention groups. The study was divided into two stages, three months as a stage. After the completion of one stage, the two groups were crosschanged. The final evaluation indicators were a 10 day antibiotic prescription rate and an unreasonable prescription utilization rate.

The intervention model examined in this paper mainly includes the intervention early alert system of antibiotic prescription rate monitoring feedback and the artificial intelligence real-time early warning system of irrational antibiotic prescription development based on graphical neural network technology (GNN). Both of them are indicated on the doctor's computer screen in the form of an adaptive pop-up window to release early warning information. Before the preliminary study, the authors underwent a three-month pre study, and the participating institutions gave a positive answer to the intervention, believing that the intervention was beneficial to their work.

The main advantage of this paper is that the author wants to construct an artificial intelligence early detection system for antibiotic prescription control in primary medical institutions in Guizhou Province, which is technically consistent and economically feasible. Combine it with the opening of a doctor's antibiotic prescription, and reduce the antibiotic prescription rate through this intervention mode. If the intervention measures are effectively established, it will provide a good reference program for the opening of antibiotic prescriptions in the majority of hospitals, and has objective significance.

But at the moment I'm confused

1. Why do you take the way of cross cohort study?

2. What is the relationship between cross cohort and primary endpoint?

3. Because the trial scheme is open, doctors participating in the study may subjectively reduce the number of antibiotic prescriptions on the basis of understanding the content of the study. How to control the interference of this factor in this study?

Reviewer #3: This is a meaningful study. I am interested in the artificial intelligence system. Is it regional, or national? can you show me more detailed information?

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

1 Aug 2021

REBUTTAL LETTER

Dear academic editor and reviewers,

Thank you very much for your review. We are grateful to the editor and 3 reviewers for giving us many good suggestions. Below are our responses.

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

Response: Thank you for the reminder. We have reviewed and revised the manuscript for formatting problems as requested by PLOS ONE.

2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

Response: Thank you for the reminder. We have checked all the funding information to ensure that they now match.

3. Thank you for stating the following financial disclosure:

“The funders had and will not have a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

At this time, please address the following queries:

a) Please clarify the sources of funding (financial or material support) for your study. List the grants or organizations that supported your study, including funding received from your institution.

Response: The study was funded by the National Natural Science Foundation of China Grant on “Research on feedback intervention mode of antibiotic prescription control in primary medical institutions based on the depth graph neural network technology” (71964009) and the Science and Technology Fund Project of Guizhou Provincial Health Commission Grant on “Application Research of Deep Learning Technology in Rational Evaluation and Intervention of Antibiotic Prescription” (gzwjkj2019-1-218).

b) State what role the funders took in the study. If the funders had no role in your study, please state: “The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

Response: Our apologies. The funders had a role in the study which we should acknowledge. Specifically, all funders provided travel expenses during the data collection process, as well as the expert's expenses for providing guidance on the study design, technological guidance, and data analysis.

c) If any authors received a salary from any of your funders, please state which authors and which funders.

Response: No author received any salary from the funders.

d) If you did not receive any funding for this study, please state: “The authors received no specific funding for this work.”

Response: The corresponding author, Yue Chang, was financially supported by the National Natural Science Foundation of China Grant on “Research on feedback intervention mode of antibiotic prescription control in primary medical institutions based on the depth graph neural network technology” (71964009) and the Science and Technology Fund Project of Guizhou Provincial Health Commission Grant on “Application Research of Deep Learning Technology in Rational Evaluation and Intervention of Antibiotic Prescription” (gzwjkj2019-1-218).

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

Response: Yes, we have added the following to our cover letter:

“The study was funded by the National Natural Science Foundation of China Grant on “Research on feedback intervention mode of antibiotic prescription control in primary medical institutions based on the depth graph neural network technology” (71964009) and the Science and Technology Fund Project of Guizhou Provincial Health Commission Grant on “Application Research of Deep Learning Technology in Rational Evaluation and Intervention of Antibiotic Prescription” (gzwjkj2019-1-218).

The funders provided expert advice on study design, travel expenses for data collection and guidance on data analysis.

None of the authors received any salary from the funders.”

4. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“The study was funded by the National Natural Science Foundation of China Grant on “Research on feedback intervention mode of antibiotic prescription control in primary medical institutions based on the depth graph neural network technology” (71964009) and the Science and Technology Fund Project of Guizhou Provincial Health Commission Grant on “Application Research of Deep Learning Technology in Rational Evaluation and Intervention of Antibiotic Prescription” (gzwjkj2019-1-218).”

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“The funders had and will not have a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

Response: Thank you. We have removed all funding-related text from the manuscript and moved it to the Funding Statement section.

The study was funded by the National Natural Science Foundation of China Grant on “Research on feedback intervention mode of antibiotic prescription control in primary medical institutions based on the depth graph neural network technology” (71964009) and the Science and Technology Fund Project of Guizhou Provincial Health Commission Grant on “Application Research of Deep Learning Technology in Rational Evaluation and Intervention of Antibiotic Prescription” (gzwjkj2019-1-218).

We have included our amended statements within the cover letter.

5. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered the online submission form will not be published alongside your manuscript.

Response: Thank you. Our ethics statement now appears in the Methods section of the manuscript on page 23 line 389-394:

“Ethical approval and consent

The trial has been approved by the Human Trial Ethics Committee of Guizhou Medical University (Certificate No.: 2019 (148)) in Dec. 27, 2019. All the physicians at the primary care institutions participating in the trial will sign the informed consent. When collecting and analyzing data, we will remove patient confidential information, such as name and national ID. Physicians' prescription data will also be subject to strict confidentiality measures.”

6. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Response: Thank you. We have reviewed the reference list to ensure that it is complete and correct. We did not cite any retracted article.

According to the requirements of Reviewer #1, we simplified the content of the introduction part, so the following references were deleted:

[12] Xiaoyuan Q, Yin C, Sun X, et al. Consumption of antibiotics in Chinese public general tertiary hospitals (2011-2014): Trends, pattern changes and regional differences. Plos One. 2018;13(5).

[14] Wang J, Wang P, Wang X, et al. Use and prescription of antibiotics in primary health care settings in China. Jama Intern Med. 2014;174(12):1914-1920.

[15] Radon K, Saathoff E, Pritsch M, et al. Protocol of a population-based prospective COVID-19 cohort study Munich, Germany (KoCo19). BMC Public Health. 2020;20(1):1036.

[23] Davey P, Marwick CA, Scott CL, et al. Interventions to improve antibiotic prescribing practices for hospital inpatients. Cochrane Database Syst Rev. 2017;2:CD003543.

[45] Benke K, Benke G. Artificial Intelligence and Big Data in Public Health. Int J Environ Res Public Health. 2018;15(12).

[46] Zhang X, Perez-Stable EJ, Bourne PE, et al. Big Data Science: Opportunities and Challenges to Address Minority Health and Health Disparities in the 21st Century. Ethn Dis. 2017; 27(2):95-106.

[48] Dong L, Yan H, Wang D. Antibiotic prescribing patterns in village health clinics across 10 provinces of Western China. J Antimicrob Chemother. 2008;62(2):410-415.

In addition, we updated the following 10 old and inappropriate references with references in bold:

[7] Froom J, Culpepper L, Jacobs M, et al. Antimicrobials for acute otitis media? A review from the International Primary Care Network. BMJ. 1997;315(7100):98-102.

[7] Christaki E, Marcou M, Tofarides A. Antimicrobial Resistance in Bacteria: Mechanisms, Evolution, and Persistence. Journal of Molecular Evolution. 2020 ;88(1):26-40.

[8] Neuhauser M, Weinstein A, Rydman R, et al. Antibiotic resistance among gram-negative bacilli in US intensive care units: implications for fluoroquinolone use. JAMA. 2003; 289(7):885-888.

[8] Septimus EJ. Antimicrobial Resistance: An Antimicrobial/Diagnostic Stewardship and Infection Prevention Approach. Medical Clinics of North America. 2018 ;102(5):819-829.

[19] De Santis G, Harvey J, Howard D, et al. Improving the quality of antibiotic prescription patterns in general practice. The role of educational intervention. Med J Aust. 1994;160(8):502-505.

[15] Dekker ARJ, Verheij TJM, Broekhuizen BDL, Butler CC, Cals JWL, Francis NA, et al. Effectiveness of general practitioner online training and an information booklet for parents on antibiotic prescribing for children with respiratory tract infection in primary care: a cluster randomized controlled trial. Journal of Antimicrobial Chemotherapy. 2018 ;73(5):1416-1422.

[20] Briel M, Langewitz W, Tschudi P, et al. Communication training and antibiotic use in acute respiratory tract infections. A cluster-randomised controlled trial in general practice. Swiss Med Wkly. 2006;136(15-16):241-247.

[16] Little P, Stuart B, Francis N, Douglas E, Tonkin-Crine S, Anthierens S, et al. Antibiotic Prescribing for Acute Respiratory Tract Infections 12 Months After Communication and CRP Training: A Randomized Trial. Annals of Family Medicine. 2019;17(2):125-132.

[31] Welschen I, Kuyvenhoven MM, Hoes AW, et al. Effectiveness of a multifaceted intervention to reduce antibiotic prescribing for respiratory tract symptoms in primary care: randomised controlled trial. BMJ. 2004;329(7463):431.

[26] Pettersson E, Vernby A, Mölstad S, Lundborg CS. Can a multifaceted educational intervention targeting both nurses and physicians change the prescribing of antibiotics to nursing home residents? A cluster randomized controlled trial. Journal of Antimicrobial Chemotherapy. 2011;66(11):2659-2666.

[34] Bjerrum L, Cots JM, Llor C, et al. Effect of intervention promoting a reduction in antibiotic prescribing by improvement of diagnostic procedures: a prospective, before and after study in general practice. European Journal of Clinical Pharmacology. 2006;62(11):913.

[29] Gerber JS, Prasad PA, Fiks AG, Localio AR, Grundmeier RW, Bell LM, et al. Effect of an outpatient antimicrobial stewardship intervention on broad-spectrum antibiotic prescribing by primary care pediatricians: a randomized trial. JAMA. 2013;309(22):2345-2352.

[35] Altiner A, Brockmann S, Sielk M, et al. Reducing antibiotic prescriptions for acute cough by motivating GPs to change their attitudes to communication and empowering patients: a cluster-randomized intervention study. J Antimicrob Chemother. 2007;60(3):638-644.

[30] Butler CC, Simpson SA, Dunstan F, Rollnick S, Cohen D, Gillespie D, et al. Effectiveness of multifaceted educational programme to reduce antibiotic dispensing in primary care: practice based randomised controlled trial. BMJ. 2012;(344):d8173.

[56] LaValley, Michael P. Intent-to-treat Analysis of Randomized Clinical Trials. ACR/ARHP Annual Scientific Meeting 2003.

[49] McCoy CE. Understanding the Intention-to-treat Principle in Randomized Controlled Trials. Western Journal of Emergency Medicine. 2017;18(6):1075-1078.

[61] Hongzhou S, Qinjian Y, Qianjin Z. Review of Development and Application of Delphi Method in China — One of Series Papers of Nanjing University Knowledge Mapping Research Group. Journal of Modern Information. 2011;31(5).

[54] Araújo V, Teixeira PM, Yaphe J, Correia de Sousa J. The respiratory research agenda in primary care in Portugal: a Delphi study. BMC Family Practice. 2016 ;17(1):124.

[62] Yingyao C, Ming N, Yaozhi H, et al. Selection of indicators to measure public interest of public medical institutions: Based on the Delphi method. Chinese Journal of Health Policy. 2012;5(1):6-10.

[55] Banno M, Tsujimoto Y, Kataoka Y. The majority of reporting guidelines are not developed with the Delphi method: a systematic review of reporting guidelines. Journal of Clinical Epidemiology. 2020; 124:50-57.

Reviewer #1

This is an interesting protocol. The protocol used artificial intelligence - based real-time warnings, pop-up windows and educational manuals to evaluate whether those interventions could reduce antibiotic prescription rates. However, as a protocol, the article did not answer the questions whether these three can reduce the use of antibiotics. Moreover, I have some other concerns, listed as follows:

(1) The introduction part needs to be more streamlined.

Response: According to your suggestion, we have shortened the introduction on page 4-7, line 51-125:

“Antibiotic resistance is a widespread concern around the world. In the past decade, 50% of antibiotic prescriptions have been used to treat colds, and, according to a report of World Health Organization (WHO), many of these cases have no indication of antibiotic use. The main forms of antibiotic overuse and abuse are non-indication, overdose, and multi-drug combined use of antibiotics, which violate the principles of antibiotic use. Overuse and abuse of antibiotics are major risk factors for antibiotic resistance. The total consumption of antibiotics in 71 countries (including China) increased by more than 50% from 2000 to 2010. The antibiotic prescription rate in China is twice as high as that recommended by WHO, and higher than that in developed countries and most developing countries. Guizhou province, located in southwest China, has the largest number of poor people and is the largest poverty-stricken area in China. Our retrospective study of Guizhou's 39 primary care facilities showed that most of the medical staff do not hold a university degree, and the rate of inappropriate antibiotic prescriptions was over 90%. The incidence of bacterial resistance in Guizhou province is on the rise.

Previous studies have shown that there are a variety of methods for antibiotics prescription control, including educational intervention, communication training, nursing point testing, electronic decision support system, and delayed prescription, but reports of a feedback intervention are rare. Existing feedback interventions mainly focused on email or poster information, regular or irregular assessment/audit of antibiotic prescriptions, or prescription recommendations from experts and peers delivered at a meeting or online. Some studies even report prescribing information publicly. However, these methods are somewhat mandatory and censored, which can cause negative emotions to the physician, and needs long-term intervention by professionals. As a result, some studies have shown negative results.

We previously conducted a cluster randomised crossover-controlled trial to reduce antibiotic prescription rates based on an existing health information system (HIS) in primary care institutions in Guizhou. In this study, the antibiotic prescription rates decreased by 15% over the 6-month study period. However, a limitation was that prescription of unreasonable antibiotics were not take into account. In view of this, this new study will analyze and process the digital information in electronic medical records with big data using depth map neural network (DMNN) in artificial intelligence (AI) technology to provide physicians with the best diagnosis and treatment suggestions in real time.

This is a cluster-randomised, open controlled, crossover, superiority trial. We aim to describe an automatically-presented, privacy-protecting, DMNN technology-based feedback intervention model. The feedback intervention model can not only effectively remind physicians of the deviation of their prescribing behavior from their peers, but also humanely give reasonable suggestions, which can greatly improve the enthusiasm of physicians to participate. In addition, an educational handbook developed by us for primary outpatient institutions will be distributed to all primary physicians. A pilot study will be conducted to test the motivation of the physicians and effectiveness of the intervention. The comparators are usual care, i.e. primary care hospitals within Guizhou which did not receive any intervention.”

(2) Can the results from Guizhou reflect China or other places of the world?

Response: Antibiotic abuse varies widely from region to region in China. The economic level of Guizhou province is low and the health resources are scarce. Due to the limited medical capacity of physicians in primary care institutions, antibiotic abuse is relatively serious in Guizhou (Chang Y, 2019). Therefore, the results of this study are intended to be used in areas with high levels of antibiotic abuse, rather than nationwide. At the same time, we expanded the sample size based on the original study (Chang Y, 2020), and also increased the reliability of this research results.

We have added a sentence in the discussion section reflecting these ideas. Page 23, line 404-405:

“Based on the results of our preliminary study, this intervention is suitable for primary care institutions in developing countries with paperless office conditions.”

(3) The reason for choosing Guizhou is because of the higher rate of bacterial resistance in this area? Or is the rate of overuse of antibiotics higher?

Response: Guizhou is one of China's poorest provinces. The level of medical care is relatively low. Our previous study (Chang Y, 2019) showed that the rate of overuse and misuse of antibiotic prescriptions in primary care institutions in Guizhou was over 90%. Therefore, Guizhou is very representative. In addition, the incidence of bacterial resistance in Guizhou province is on the rise.

In the background section on lines 77-83 on page 5, we focused on explaining why Guizhou Province was chosen as the research site:

“Guizhou province, located in southwest China, has the largest number of poor people and is the largest poverty-stricken area in China. Our retrospective study of Guizhou’s 39 primary care facilities showed that most of the medical staff have less than a university education, and the unreasonable rate of antibiotic prescriptions was over 90%. The incidence of bacterial resistance in Guizhou province is on the rise.”

(4) The study is designed to analysis whether interventions can reduce the antibiotic use rate of physicians. But it will be more interesting if the study can simultaneously analyze whether the prognosis of these doctors' patients has changed, or whether the bacterial resistance rate has changed.

Response: Thank you for your advice, which is exactly the direction of our future research. In addtion, as we mentioned in the outcome measurement section on page 17-18, our goal is not only to reduce the prescribing rate of antibiotics (primary outcome), but also to reduce the irrational prescribing rate of antibiotics (secondary outcome). We have also added a sentence on page 3, line 40:

“…to effectively reduce antibiotic prescription rates and antibiotic irregularities.”

(5) One intervention of this protocol is "pop-up windows of antibiotic prescription rate ranking". I am curious whether this ranking will have a negative effect on medical decision-making. I mean, if in some departments where the use of antibiotics always high, such as department of infectious diseases or respiratory, will the physician change the correct medical decision because of this ranking?

Response: Thank you. This comment is thought-provoking. In our previous study (Yue Chang, 2019/2020), we found that the primary care institutions in Guizhou Province were mainly general clinics, and a few large hospitals had independent clinics of traditional Chinese medicine, orthopedics and gynecology. Considering the feasibility of our outpatient ranking intervention, this study only included GPs in the general practice room. In addition, we also highlighted the situation of GPs in the "recruitment section" of our methodology on page 9 line 165:

“ …2) each primary care institution has at least 3 outpatient general physicians (GPs), …”

Furthermore, we also pointed out in the "Intervention Section" :

“The information seen by each physician will be confidential. The physicians have the freedom to read this feedback message or not.”

We think this is the mildest and most compliant type of feedback intervention. In conclusion, we therefore believe that, in principle, the ranking system will not interfere with the physician’s decision to prescibe the correct medication, rather, it will prompt those who have a high rate of prescribing, relative to their peers, to reconsider their behaviour carefully.

(6) The authors tried to analyze whether their intervention could reduce the use of antibiotics. Does the author have a theoretically reasonable interval of antibiotic usage in each department and how are they calculated? I believe that in different departments, this should be different.

Response: As mentioned above, we are dealing with primary care physicians who are treating patients with common diseases at the primary care level. According to our previous study (Chang Y, 2020), 10 days was the most reasonable interval. The inclusion criteria for physicians was a minimum of 100 prescriptions in 10 days. This facilitated our calculation of antibiotic prescription rates. The transition model we use is also well suited for this kind of crossover design study with the same interval. Our ranking feedback intervention is updated every 10 days, three times a month. Our previous Chinese article collected the opinions of physicians, who also thought that three times a month was a reasonable rate. So, we didn't design different time intervals for the general outpatients.

(7) As I mentioned before, this is a protocol. Although interesting, it still needs further expected.

Response: Thank you very much for your comments.

Reviewer #2

(1) Why do you take the way of cross cohort study?

Response: A crossover design study is a repeated measurement study in which each unit receives different treatments at different times. A crossover design study is commonly considered in RCT because it is more effective than a parallel design. The crossover design not only allows all participants to experience all interventions, but can also use inter-group comparisons to reduce confounding factors (Berggren L, 2021; Senn S, 2002; David H DS, 2016). We also mentioned this on page 9 line 177-178：

“To avoid selection bias, we will use a crossover design in which each group will receive different treatments at different times.”

(2) What is the relationship between cross cohort and primary endpoint?

Response: Firstly, as mentioned on page 11, lines 183-186,

“The trial will be divided into two stages and will last for 6 months. The first stage will last for 3 months, with Group 1 enrolled in the intervention group and Group 2 in the control group. The second stage will also last for 3 months, with the two groups crossing over.”

We will compare the antibiotic prescription rates from baseline to the cross-over point (after 3 months) and from the cross-over point to the end point (after 6 months) in Group 1 and Group 2, respectively. The antibiotic prescription rates in Group 1 and Group 2 were also compared at baseline (0 month), cross-over point (3 months), and end point (6 months).

We also mentioned this on page 19 line 329-331:

“We will compare the antibiotic prescription rate and the antibiotic prescription rationality rate between the two groups (Group 1 and Group 2) at baseline, cross-over point and at the end of the trial.”

(3) Because the trial scheme is open, doctors participating in the study may subjectively reduce the number of antibiotic prescriptions on the basis of understanding the content of the study. How to control the interference of this factor in this study?

Response: Unlike drug intervention trials, our feedback interventions were designed to improve the prescribing behavior of the doctors. This type of behavioral intervention is inherently difficult to define when to start and when to end, which is why our cross-over design has no washout period. At the same time, due to the good effect achieved in the early intervention studies (Chang Y, 2020), we have also received the support of Guizhou Provincial Health Commission and will carry out long-term prescription intervention in primary health hospitals in the future.

So, we expect that doctors will automatically reduce their prescription of antibiotics and increase the rationality of antibiotics because of our intervention, which is our long-term goal.

Reviewer #3

(1) This is a meaningful study. I am interested in the artificial intelligence system. Is it regional, or national? can you show me more detailed information?

Response: Thank you for your query. This was a regional study. In the future, we also want to scale it up to the whole country and even to other developing countries. We will provide more details about the AI system in the Appendix (Appendix AI system in details).

Kind Regards.

Yuanfan Yao

07-31-2021

Reference

[1] Chang Y, Sangthong R, McNeil EB, et al. Effect of a computer network-based feedback program on antibiotic prescription rates of primary care physicians: A cluster randomized crossover-controlled trial. J Infect Public Health. 2020.

[2] Chang Y, Chusri S, Sangthong R, et al. Clinical pattern of antibiotic overuse and misuse in primary healthcare hospitals in the southwest of China. PLoS One. 2019;14(6):e0214779.

[3] Berggren L. Analysing a cross-over study. Statistical work and challenges related to planning, conducting and analysing a clinical trial with cross-over design. Sweden: Stockholm University; 2012.

[4] Senn S. Cross-over trials in clinical research (second edition). Department of statistical scientice and department of epidemiology and public health university college London, UK: John Wiley & sons, Ltd.; 2002.

[5] David H DS, Eduardo B. Which Treatment Is Better? Ascertaining Patient Preferences with Crossover Randomized Controlled Trials. J Pain Symptom Manage. 2016;49(3): 625–631. doi:10.1016/j.jpainsymman.2014.11.294.

Submitted filename: 20210801Rebuttal letter.CY.docx

Click here for additional data file.

12 Oct 2021

Changing antibiotic prescribing practices in outpatient primary care settings in China: study protocol for a health information system-based cluster-randomised crossover controlled trial

PONE-D-21-16838R1

Dear Dr. Chang,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Vijayaprakash Suppiah, PhD

Academic Editor

PLOS ONE

Reviewer's Responses to Questions

Comments to the Author

1. Does the manuscript provide a valid rationale for the proposed study, with clearly identified and justified research questions?

The research question outlined is expected to address a valid academic problem or topic and contribute to the base of knowledge in the field.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Is the protocol technically sound and planned in a manner that will lead to a meaningful outcome and allow testing the stated hypotheses?

The manuscript should describe the methods in sufficient detail to prevent undisclosed flexibility in the experimental procedure or analysis pipeline, including sufficient outcome-neutral conditions (e.g. necessary controls, absence of floor or ceiling effects) to test the proposed hypotheses and a statistical power analysis where applicable. As there may be aspects of the methodology and analysis which can only be refined once the work is undertaken, authors should outline potential assumptions and explicitly describe what aspects of the proposed analyses, if any, are exploratory.

Reviewer #1: Partly

Reviewer #2: Yes

**********

3. Is the methodology feasible and described in sufficient detail to allow the work to be replicable?

Descriptions of methods and materials in the protocol should be reported in sufficient detail for another researcher to reproduce all experiments and analyses. The protocol should describe the appropriate controls, sample size calculations, and replication needed to ensure that the data are robust and reproducible.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors described where all data underlying the findings will be made available when the study is complete?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception, at the time of publication. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above and, if applicable, provide comments about issues authors must address before this protocol can be accepted for publication. You may also include additional comments for the author, including concerns about research or publication ethics.

You may also provide optional suggestions and comments to authors that they might find helpful in planning their study.

(Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have answered well and revised the manuscript. This manuscript is ready to publish and I have no other concerns.

Reviewer #2: The author kindly answered our questions, clearly explained why the cross cohort study was adopted, and explained the relationship between cohort design and expected endpoint. However, how to reduce the subjective reduction of antibiotic prescriptions when participating doctors know the research content still needs to further design better research methods to reduce this interference factor.At the same time, the real data have not been obtained, so it is difficult to evaluate the specific effect of this method.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

15 Oct 2021

PONE-D-21-16838R1

Changing antibiotic prescribing practices in outpatient primary care settings in China: study protocol for a health information system-based cluster-randomised crossover controlled trial

Dear Dr. Chang:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Vijayaprakash Suppiah

Academic Editor

PLOS ONE