Analysis of low resource setting referral pathways to improve coordination and evidence-based services for maternal and child health in Ethiopia.

BACKGROUND: In low-resource settings, patient referral to a hospital is an essential part of the primary health care system. However, there is a paucity of study to explore the challenges and quality of referral coordination and communication.
OBJECTIVE: The purpose of this research was to analyze the existing paper-based referral registration logbook for maternal and child health in general and women of reproductive age in particular, to improve referral coordination and evidence-based services in Low-Resource Settings.
METHODS: This study analyzed the existing paper-based referral registration logbook (RRL) and card-sheet to explore the documentation of the referral management process, and the mechanism and quality of referrals between the health center (Jimma Health Center-case, Ethiopia) and the Hospital. A sample of 459 paper-based records from the referral registration logbook were digitized as part of a retrospective observational study. For data preprocessing, visualization, and analysis, we developed a python-based interactive referral clinical pathway tool. The data collection was conducted from August to October 2019. Jimma Health Center's RRL was used to examine how the referral decision was made and what cases were referred to the next level of care. However, the RRL was incomplete and did not contain the expected referral feedback from the hospital. Hence, we defined a new protocol to investigate the quality of referral. We compared the information in the health center's RRL with the medical records in the hospital to which the patients were referred. A total of 201 medical records of referred patients were examined.
RESULTS: A total of 459 and 201 RRL records from the health center and the referred hospital, respectively, were analyzed in the study. Out of 459, 86.5% referred cases were between the age of 20 to 30 years. We found that "better patient management", "further patient management", and "further investigation" were the main health-center referral reasons and decisions. It accounted for 40.08%, 39.22%, and 16.34% of all 459 referrals, respectively. The leading and most common referral cases in the health center were long labor, prolonged first and second stage labor, labor or delivery complicated by fetal heart rate anomaly, preterm newborn, maternal care with breech presentation, premature rupture of membranes, malposition of the uterus, and antepartum hemorrhage. In the hospital RRL and card-sheet, the main referral-in reasons were technical examination, expert advice, further management, and evaluation. We found it overall impossible to match records from the referral logbook in the health center with the patient files in the hospital. Out of 201, only 13.9% of records were perfect matching entries between health center and referred hospital RRL. We found 84%, 14.4%, and 1.6% were appropriate, unnecessary and unknown referrals respectively.
CONCLUSION: The paper illustrates the bottlenecks encountered in the quality assessment of the referrals. We analyzed the current status of the referral pathway, existing communications, guidelines and data quality, as a first step towards an end-to-end effective referral coordination and evidence-based referral service. Accessing, monitoring, and tracking the history of referred patients and referral feedback is challenging with the present paper-based referral coordination and communication system. Overall, the referral services were inadequate, and referral feedback was not automatically delivered, causing unnecessary delays.

1. Introduction

The World Health Organization (WHO) defines referral as “a process in which a health worker at one level of the healthcare system, having insufficient resources to manage a clinical condition, seeks the assistance of a better or differently resourced facility at the same or higher level to assist in, or take over the management of, the client’s case” [1]. An effective referral system can play an important role in delivering cost-effective health-care services and ensuring access to better quality care. There is a growing body of literature that recognizes the automation of the paper-based referral system to reduce unnecessary waiting time and delay [2,3], improve the communication gap [4-7], and provide a fast response rate and accurate data [8]. However, in Ethiopia, not all health centers have the required early detection facilities to make sound referral decisions [9,10]. Literature evidence suggests that assisting the referral decision process through decision support systems enables to: (I). aid the process of filling out a referral template [11], (II). provide an alternative recommendation using historical records [12], and (III). prioritize illness severity instead of first-come first-serve rules to ensure on-time treatment of emergency patients [13]. Therefore, there is a need to introduce referral quality improvement techniques beyond looking at the statistical patterns and trends [14]. Clinical pathways (CP) have been introduced for point of care patient management and defined as a concise and evidence-based summary of the care process, including an algorithm, the informed guidelines or the best evidence, coordinating roles, sequencing activities, documenting and evaluating variance [15,16]. Data from several studies suggest that CP has been successful for minimizing cost, reducing delay, or improving patient outcomes [17-20]. CP helps to bring the whole treatment of a special disease in one concrete setting [21-23]. CP also implemented by tracing historical record and has effective for improving outcomes, reducing delay and cost, as described in [24-28]. In all, clinical decision support in CPs has largely been implemented to manage the quality of the care process (Chawla et al. 2016) [29] but are often out of reach for developing countries. Therefore, it makes sense to develop an automated plan of care helping to practice and enable evidence-based decision-making considering low clinical competence and shortage of resources. It will also empower local and (unexperienced) care providers and enable them to make better referral decisions.

The health network model with three-tiered structure (primary, secondary and tertiary levels) is adopted to structure the Ethiopian healthcare system [30]. Each level is expected to serve thousands of people in the health center to millions of people in specialized hospitals. The referral system is organized based on geographically defined catchment areas. The catchment network links primary hospitals to health centers and the health centers are linked with the health posts (the lowest level of health care service under the health center). The primary hospitals connect with general hospitals and the highest level of network connects general hospitals to specialized hospitals [9,30]. The primary referral service is delivered if a specific patient has met the required referral criteria, prerequisites, and the liaison office’s approval. presents the overall summary of the Ethiopian primary healthcare referral flow.

Upon arrival, the health professional will capture the idea (chief compliant), the information, and the feeling of the patient (signs and symptoms) for delivering the required treatment or referral service. A patient can be referred to the next level when a patient needs more inpatient care, expert advice, technical examination, or a technical intervention that is beyond the capabilities of the facility. A referral is also made when the referring facility cannot accept more patients due to a shortage of beds or the unavailability of professionals. In addition, referral from the primary hospital to the health center is also made based on the specific condition of the patient and the prescribed service [30]. Once the referral reason is fulfilled, the liaison office is responsible for: (I). matching, and validating the referral-out,(II). checking that the referral form is filled-in and signed by the physician and ensuring that feedback is sent to the referring health facility, and (II). compiling, analyzing and interpreting data to improve the referral service. The referral side (usually the referred hospital) should confirm the availability of the service, the availability of a bed, and professional readiness. Then, the admitting physicians in the referred hospital are responsible for delivering appropriate diagnosis and evaluation, adhering to the guideline, indicating the level of urgency, ensuring the required procedures and completing all the required documentation. An effective evidence-based service and point-of-care instrument is key for the referral quality and facilitation of seamless referral coordination between the health center (HC) and the referred hospital. Though the referral linkage was documented well, far too little attention has been paid to examine the applicability and implementation challenges in low resource settings that arose from the current paper-based referral registration logbook coordination and communication. Since unnecessary referral will create bottlenecks and heavy patient load in the referred hospital and missed referral will create a delay and complications in the management of the individual patient, a case study on the Ethiopian primary healthcare system (specifically at Jimma Health Center) was conducted to investigate the referral mechanism between the health center and the hospitals, the reasons for the referral, and the “quality” of these referrals. Assessing the quality and completeness of the paper-based records related to the referral system proved to be extremely difficult with respect to completeness, availability of referral feedback and referral reason matching [31]. The aim of this investigation was to examine the existing paper-based referral registration logbook(RRL) and card-sheet to explore the documentation of the referral management process and the mechanism and quality of referrals between the health center (Jimma Health Center-case, Ethiopia) and the Hospital. To address this objective, sample records from the referral registration logbook at the Jimma Health Center and referred hospital were digitized to explore: (i) the reason for referral decision (how the referral decision is made), (ii) what cases are referred to the next level, and (iii) what are the missing feedbacks and conclusions to distinguish appropriate referral from unnecessary referral.

2. Methods

2.1. Ethical considerations

Ethical clearance was obtained from the Jimma University, Institute of Health, Institutional Review Board (IRB). IHRPGI/467/2019 is the reference number. Permission was granted by the Jimma health center, and the data was gathered and analyzed anonymously after written consent was obtained.

2.2. Study setting and design

The research was limited to the exploration of the existing paper-based referral registration logbook for maternal and child health in general and women of reproductive age in particular, to improve referral coordination and evidence-based services in Low-Resource Settings (LRS). A retrospective observational study in the Ethiopian primary healthcare service, namely at Jimma Health Center (which is in East and Sub-Saharan Africa) was conducted after securing the required ethical clearance and administrative procedure. We collected n = 459 cases from the RRL to investigate our research questions. The number of women of reproductive age who received care at the health center is estimated to be N = 15646. The minimum required sample size was calculated to be n(minimum) = 375. The following formulae were applied: in which Zα/2 is the critical value of the Normal Distribution at α/2 (confidence level is set to 95% and hence the critical value is 1.96, α is 0.05), the margin of error (MOE) is 0.05, p is the sample proportion and N is the population size. The theory behind these calculations is explained in [32,33]. We follow the recommendations of WHO stepwise approach to surveillance for the remaining values [34]. These value settings allow us to simplify the calculations as formulated in Eqs 1 and 2: p = 0,5 maximizes the nominator in Eq 2 and produces a worst case (i.e. maximal) value for n(minimum), and p = 0,5 is recommended in cases where no a priori results can be used from previous studies; since we only consider women of reproductive age (age 15–48), the number of “age-gender” categories is equal to 1; the response rate is ~100% since we are performing a retrospective analysis based on RRL; the design effect is set equal to 1 which is recommended for random samples, following the WHO guidelines no finite population correction is applied. The representativeness of the data in our sample is guaranteed, since we digitalized RRL records of n = 459 referral consultations, which is larger than n(minimum).

2.3. Data source

Clinical guidelines, card-sheet, and referral registration logbook were the main data sources. In health center, the paper-based referral registration logbook (RRL) used to capture documentation about how the referral process was managed, manage the quality of referral using referral feedback to ensure whether the referral was appropriate for the patient or not, manage referral coordination and communication. The RRL used as a standard referral-in and referral-out data entry protocol for managing the referral cases and quality [30]. On average, the health-center paper-based RRL contains 22 records per page and 20–25 pages in total. More information on the structure and layout of the health center paper based RRL is presented in

2.4. Data collection process

Two data collectors were recruited from the Jimma Health Center to collect the data from the patient referral-out registration logbook and register in the pre-prepared electronic data-sheet template. The data collection was conducted from August to October 2019 and maintained a single data entry process for the duration of the study. We examined the RRL records referred in the year 2010/2011 E.C. (i.e. 2018/19 G.C). Sample records from the referral registration logbook at the Jimma Health Center and referred hospital were digitized. presents the summary of the dataset.

To assure the quality of data, the data collectors were recruited based on their familiarity and experience with the existing workflow process, patient card-sheet management, professional expertise, and exposure to handle clinical and health information. The data collectors were already working as full-timers (health information system, data monitoring and recorder professional) and they agreed to cooperate on the data collection process during the weekend and after working hours.

However, the health center referral logbook did not have the expected referral feedback from the hospital. To get the referral feedback, we defined a new strategy and recruited two additional data collectors to collect from the referred hospital logbook and card-sheet. The referred hospital RRL data was also collected using a pre-prepared electronic data-sheet template. The overall data collection process was random and obtaining the required information was time-consuming and tougher than expected. Then, we tried to compare the information in the referral logbook of the health center with the medical records in the hospital to which the patients were referred. Though the health-center cases were referred to Jimma University Specialized Hospital and Shanan Gibe General Hospital, this study was limited to extract referral feedback from Jimma University Specialized Hospital.

Finally, the alignment of the CP dataset classification based on International Classification of Diseases 10th Revision (ICD-10) standards was performed [35].

2.5. Data preprocessing and analysis

Data pre-processing, visual inspection and analysis of the CP dataset were conducted using a python based interactive referral CP visualization tool. An interactive and RRL data-driven tool was designed for: (I). Analyzing and visualizing RRL records, (II). Pre-processing and visual inspection of RRL records such as handling missing, noisy and outlier values, and (III). Conducting automated referral feedback and reasoning analysis.

To answer our research questions, all the records were collected from the Jimma Health Center RRL and the referred hospital (i.e. Jimma University Specialized Hospital) registration logbook and card-sheet. Since all the records were presented in hard-copy format this activity was time-consuming, and it was difficult to extract the right and perfect matching records.

WHO data-quality review and metrics were adopted to assess the quality of RRL [36]. The referral quality was assessed with respect to: (I). The completeness and timeliness of RRL data, (II). The internal consistency of RRL such as the presence of anomalies, the RRL format and type, and consistency over time (history of recording), (III). The accuracy, validity, and uniqueness of RRL data to describe the RRL definition, RRL column type, RRL column value, and level of duplication, and (IV). The external consistency such as the matching of health-center referral reasoning and referred hospital referral feedback.

The RRL analysis for the health-center was then conducted. The RRL contains information on patient referrals such as "disease name (cases)," "referred to," "referral reason," "referred date," and "collected signs and symptoms" that were used in primary care prior to making a referral decision. We examined the reasons for referral, disease names (cases), and conclusion (outcome) to see the reasons for the referral decision and which cases were referred to the next level. The referral justification was registered in the RRL “referral-out reasoning” column and documented based on the national guidelines (or referral templates). In order to assess referral cases, it is crucial to examine this referral-out justification (reasons). Fig 1 gives further information about the reason for the referral (the decision to refer or not to refer a patient) based on the national guidelines, which should be medically sound, objective, and in the best interests of the patient.

Fig 1

The primary health care referral flow (FDRE MoH, 2010).

The referred hospital RRL, on the other hand, was analyzed to identify the missing feedbacks and conclusions to distinguish appropriate referral from unnecessary referral. Examining the RRL column "the final remark and feedback," which was filled by the referring hospital and forwarded to the health center for future referral reference and quality improvement, helped determine the appropriateness of the referral. The health-center referral reason, hospital referral feedback, and conclusion were all analyzed to examine which referrals were appropriate and which were not. We also compared the RRLs from the health center and the referring hospital to ensure that the datasets (or individual records) were matched in general and that the referral reasoning was accurate in particular.

The final CP dataset contains medical record number, disease name (cases), age, sex, referring organization, referred to, referral reason, referred date, date of arrival at the hospital, collected signs and symptoms, referral feedback and conclusion (outcome), International Classification of Diseases 10th Revision (ICD-10) Classification, ICD-10 parent category, matching terms and descriptions attributes. Overall, we analyzed the health center RRL for investigating the referral case and reasons, and the referred hospital RRL for exploring the missed referral feedback. Comparison between the health center and referred hospital RRL were also conducted to verify the matching of the datasets and referral reasoning.

3. Results

A total of 459 cases were collected from the Jimma Health Center’s RRL. The Jimma Health Center refers patients to the Jimma University Specialized Hospital and Shanan Gibe General Hospital. Out of 459, 86.5% referred cases were between the age of 20 to 30 years. is depicted to illustrate the summary of referral age.

The finding regarding RRL data quality metrics, indicate that all RRL columns were documented except “referral feedback” column. However, the “referral feedback” column is consistently empty. A detailed analysis of RRL data quality metrics is presented in .

The referral decision (the reason for the referral) is made based on the signs and symptoms collected at the RRL health center in accordance with the diseases categories (or cases). The collected signs and symptoms were used to assess the risk of diseases (cases), and the referring health professional documented the referral-out justification. We found that meeting any of the six referral criteria (the reason for the referral) influenced the decision to refer based on the referral guidelines, which include “When a patient needs an expert advice as determined by the attending health professional”, “When technical examination is required that is not available at the referring facility”, “When a technical intervention that is beyond the capabilities of the facility is required”,“When patients require inpatient care that cannot be given at the referring facility”, “When the referring facility cannot no more accept patients due to shortage of beds and unavailability of professionals”, and “Referrals are also made to the lower level health facilities and community based organizations in the best interest of the patient depending on the condition of the patient and the capacity of the lower level health facility /community based organization” [30].

Using health-center RRL categories such as disease name (or cases), signs and symptoms, conclusion, and referral-out reason, we examined the referral cases and the reasons for the referral decisions. These findings suggest that long labour, prolonged first and second stage labour, labour or delivery complicated by fetal heart rate anomaly, preterm newborn, maternal care with breech presentation, premature rupture of membranes, malposition of the uterus and antepartum hemorrhage were the leading and most frequent referral cases in the health center. We found that “better patient management”, “further patient management”, and “further investigation” were the main health center referral reasons. It accounted for 40.08%, 39.22%, and 16.34% of all 459 referrals, respectively. Further patient management indicates that there are some findings in the health center, but more experienced physicians should do extra examinations and investigations whereas in the case of better patient management there are not enough results in the health care center to confirm the patients’ diagnosis. In addition, normal delivery was referred to Shanan Gibe General Hospital with a referral reason of lack of light (electricity breakdown), prolonged 2nd stage labor, and high blood pressure. The detailed summary of the health center referral reason is presented in

We found that there was no evidence in the health-center RRL to distinguish appropriate referrals from unnecessary referrals. The health center’s referral logbook includes the required column for “referral feedback” but it is consistently empty. The next section of the study was concerned with investigating the quality of referral (or referral feedback) from the referred hospital and a total of 201 cases were examined. We found it overall impossible to match referral reasoning from the referral logbook in the health center with the referral feedback in the hospital. Out of 201, 13.9,18.9, and 67.2% of records were perfect matching, a high probability of matching with educated guesses, and no clue who-is-who respectively. We observed that the referral reasons in the hospital record were explicit and more explanatory in comparison with the HC’s RRL referral reason. Out of 201 cases, 84.0,14.4, and 1.6% were appropriate referrals (the referral was right, and this is encouraging to refer similar cases in the future), unnecessary referrals and not-known respectively. The criterion for an appropriate referral was extracted from the hospital RRL (feedback and conclusion section filled-in by the physician). As mentioned before, further management and evaluation, investigation and better management were the leading referral reasons next to technical examination and expert advice. There was unnecessary referral in cases of a referral reason for technical examination and expert advice, further management and evaluation, better medical prescriptions, investigation, and Mx. (medication management). The summary of the result is presented in

Further analysis showed that “further investigation”, “further management” and “better management” were the leading referral-out reasons, whereas technical examination, expert advice, further management and evaluation were the main referral-in reasons in the hospital RRL, as mentioned in the hospital card-sheets. presents the detailed aggregated summary of referral reason mentioned in the heath-center RRL and the hospital RRL. We also noticed that the registration of health-center RRL referral reasons did not exactly match and follow the national guidelines.

Our finding revealed that it is challenging (almost impossible) to satisfy and deliver the expected referral feedback with the current paper-based referral coordination. In summary, these results show that: (I). All referrals were made from the health-care center to the hospital, (II). The health-center “referral feedback” RRL column was incomplete and empty, (III). The leading referral reasons were “better patient management”, “further patient management”, and “further investigation” respectively, and (IV). Getting referral feedback from the referred hospital was very challenging and difficult to match health-center referral reasoning and referred hospital referral feedback.

4. Discussion

This study was set up to assess the referral decision and quality of referrals between the health center (Jimma Health Center) and the hospital (Jimma University Specialized Hospital). In accordance with the present results, 11/12 (91.7%) columns were filled which met the national guidelines expected target of 2019/20 (i.e. 90% of data completeness). With respect to the national health sector transformation plan, the utilization of evidence-based information for decision-making was not known and only 29% of the health facilities met the minimum information standard (criteria) whereas report completeness and timeliness was 72% and 84% respectively [9].

The referring health professional makes the decision and fills out the RRL refer-out reason based on the referral criteria. Prior to making a referral decision, the referring health professional is responsible for filling out the RRL form with the essential information and attaching any relevant documents. The RRL contains information on patient referrals such as "disease name (cases)," "referred to”, "referral reason”, and "collected signs and symptoms" that were used in primary care. RRL categories "signs and symptoms" and "diseases (or cases)" were used to make the referral decision, as well as the conclusion (or outcome). The signs and symptoms were used to assess the risk of diseases (cases), and the referring health professional documented the referral-out justification. Referral decisions are made when the criteria for referral are medically appropriate and in the best interests of the patient or client. Furthermore, the national guidelines are also used as a reference guide to address the most common presenting symptoms and referral criteria, which aids in the prioritization and documentation of the referral condition [30,37]. It is somewhat surprising that the registration of health-center RRL referral reason did not exactly match and follow the national guidelines. For instance, the guideline set referral reason as “A patient needs an expert advice as determined by the attending heath professional”, “Technical examination is required that is not available at the referring facility”, “Technical intervention that is beyond the capabilities of the facility is required” and so on whereas the referral reason written in the RRL looks like “better patient management”, “further patient management”, “further investigation” and so on. A possible explanation for this might be that the benefit of time savings due to the unnecessity of data transcription and the layout of the paper-based RRL not enough to accommodate the national guideline referral reasoning.

In the health-center RRL, the “referral feedback” column that helps to mitigate unnecessary referral, missed referral (or referral delay), and complications in the management of the individual patient were missed. The referral feedback is crucial to verify whether the referral case is correct or not. The final remark and feedback were sent from the referred hospital to the health center for future referral reference and quality improvement. Missing timely referral feedback and evidence will affect the quality of the healthcare service in the health center and the staff cannot know whether their referral policy is of good quality. When the patient who was referred to hospital show up for subsequent follow-up in the health center, surprisingly the health center has no information on what happened in the hospital. Hence, they often consider this as a new case for the patient. With the referral logbook being incomplete it is impossible to answer the initial research questions on the quality of referral. Therefore, as mentioned earlier, we defined a new approach to investigate the quality of referral and tried to compare the information in the referral logbook of the health center with the medical records in the hospital to which the patients were referred. However, the observed referral reasoning difference between the referral logbook in the health center with the referral feedback in the hospital was significant. The difference of referral reason between health center RRL and hospital RRL may arise from:(i) the limitation of the dataset (for instance, the proportion of matching entries on referral between health center and referred hospital RRL was 13.9% only), (ii) lack of proper written documentation and digitization, and (iii) the level of expertise, availability of resources and level of investigation.

This finding suggested that it is very challenging to access, monitor, and track the history of the referral patients and referral feedback using current paper-based referral coordination and communication. There are several possible explanations for this result: (I). The lack of adequate communication between the primary care and the referred hospital, (II). The lack of adequate infrastructure and integrated platform with the existing workflow to enable automated referral system, (III). The layout and format of the handwritten information presented in the patient card-sheet and RRL were inconsistent from one record to another, (IV). The referring provider did not automatically notice the missed referral feedback. This observation may support the hypothesis, enabling an automated referral system may improve the communication gap, and reduce waiting time between the primary health center and the referred hospitals [6]. The evidence-based point-of-care instrument can thus be suggested that assist the process of filling out a referral template to reduce time and error [ and helps to assist the referral follow-up [38]. However, further investigation is required to explore the barriers and facilitators for enhancing the quality of referral service and to deliver a fully functional and effective referral system.

Overall, these findings raise intriguing questions regarding the nature and extent of the missed referral feedback, the quality of referrals, and referral decisions. Furthermore, the RRL analysis showed that all referrals are made from the health-care center to the hospital. Unfortunately, we did not find a referral made to lower-level health facilities.

Our study has several limitations. We sampled from a Jimma Health-center and focused on women of reproductive age and limited to extract referral feedback from Jimma Specialized Hospital. Our findings thus may not easily generalize to other health-centers. Moreover, this study examined the bottlenecks encountered in the quality assessment of the referrals with current paper-based documentation, coordination, communication, and patient files.

5. Conclusions

The study addressed the question of how the referral decision (the reason for referral decision), coordination, and communication are made based on the collected dataset from the health center RRL and referred hospital RRL. It uncovers the missed referral feedback using the referred hospital RRL. With the current paper-based referral coordination and communication, it is very challenging to access, monitor and track the history of the referred patients and referral feedback. We also observed that the health center RRL reason was not exactly followed by the hospital and does not match the national guidelines. The most crucial problem was in the identification of referral records (no unique, consistent and easy searchable identifier was present in the RRLs and card-warehouse). The layout and format of the handwritten information presented in the patient card-sheet and RRL were inconsistent from one record to another, and hence it was difficult to audit the records. In addition, if changes were made, it was not easy to track where the changes were made. Because of all these, the quality of the referral services in primary healthcare settings was compromised, referral feedback was not delivered automatically, and this caused unnecessary delay. Therefore, when the necessary infrastructure and human resources are in place, implementing an automated patient management system may help to correct and impose a standardized referral service workflow. It creates the opportunity to track and monitor the referred patients, access their medical records, retrieve referral feedbacks, and standardize the interaction between the patients and health professionals. As mentioned before, implementing a patient data (or smart) card seems to be the fastest solution to improve the referral service quality in the absence of distributed digital infrastructure in such low-resource setting. In summary, implementing digital referral services compatible with the existing clinical guidelines is urgently needed for improving and assessing the quality of referral services in low-resource settings. This is the subject of our further research. Furthermore, even before viable infrastructure is implemented and competent human resources are attracted, a minimal infrastructure (a low-cost alternative) and trained personnel can facilitate referral coordination and offer evidence-based services to ensure a buy-in and acceptance by all stakeholders. This transition needs careful planning and openness to adopt best practices in order to reduce resource waste, to ensure training, maintenance, and avoid heavy service expenses.

1 Dec 2021

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Natasha McDonald, PhD

Associate Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. As per PLOS ONE's data sharing policy, PLOS ONE does not allow for censored information. As such please revise your manuscript accordingly and provide the names of the participating sites of your study.

3. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well.

4. Please include a separate caption for each figure in your manuscript.

5. We suggest you thoroughly copyedit your manuscript for language usage, spelling, and grammar. If you do not know anyone who can help you do this, you may wish to consider employing a professional scientific editing service. Whilst you ma We suggest you thoroughly copyedit your manuscript for language usage, spelling, and grammar. If you do not know anyone who can help you do this, you may wish to consider employing a professional scientific editing service.

Whilst you may use any professional scientific editing service of your choice, PLOS has partnered with both American Journal Experts (AJE) and Editage to provide discounted services to PLOS authors. Both organizations have experience helping authors meet PLOS guidelines and can provide language editing, translation, manuscript formatting, and figure formatting to ensure your manuscript meets our submission guidelines. To take advantage of our partnership with AJE, visit the AJE website (http://learn.aje.com/plos/) for a 15% discount off AJE services. To take advantage of our partnership with Editage, visit the Editage website (www.editage.com) and enter referral code PLOSEDIT for a 15% discount off Editage services.  If the PLOS editorial team finds any language issues in text that either AJE or Editage has edited, the service provider will re-edit the text for free.

Upon resubmission, please provide the following:

The name of the colleague or the details of the professional service that edited your manuscript

A copy of your manuscript showing your changes by either highlighting them or using track changes (uploaded as a *supporting information* file)

A clean copy of the edited manuscript (uploaded as the new *manuscript* file)y use any professional scientific editing service of your choice, PLOS has partnered with both American Journal Experts (AJE) and Editage to provide discounted services to PLOS authors. Both organizations have experience helping authors meet PLOS guidelines and can provide language editing, translation, manuscript formatting, and figure formatting to ensure your manuscript meets our submission guidelines. To take advantage of our partnership with AJE, visit the AJE website (http://learn.aje.com/plos/) for a 15% discount off AJE services. To take advantage of our partnership with Editage, visit the Editage website (www.editage.com) and enter referral code PLOSEDIT for a 15% discount off Editage services. If the PLOS editorial team finds any language issues in text that either AJE or Editage has edited, the service provider will re-edit the text for free. Upon resubmission, please provide the following: The name of the colleague or the details of the professional service that edited your manuscript A copy of your manuscript showing your changes by either highlighting them or using track changes (uploaded as a *supporting information* file) A clean copy of the edited manuscript (uploaded as the new *manuscript* file)

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: REVIEWERS COMMENTS

ABSTRACT

In the Abstract section the results indicate that out of 201. It is not clear what the N(number of matching samples are that lead to the 13%). Kindly indicate this. The results of the study are not compatible with the objectives. The objectives indicate that the study intended to explore the practice whereas the results indicate the main reasons for referral. Kindly align the two. The conclusion section should indicate the overall implication of the study within the health system. It currently reads like an objective. Kindly revise to indicate the implication.

DATA COLLECTION SECTION

In page 11 you mention that 459 cases were used. It is not clear how the sampling was done. As much as the authors indicate the stepwise approach the calculation that is present in the paragraph below does not match the sample size given. Kindly clarify.

DISCUSSION SECTION

In Page 16 you indicate that the proportion of “matching”. The sentence seems incomplete. Is it matching entries on referral?

In the second paragraph in the discussion section you mention that the reasons for referral do not match the guidelines. In my view – the intention is not to exactly match the guidelines word for word. Whereas the reasons for referral in the RRL book indicate reasons that you mention such as “better patient management” they convey the nuance of the needs of the healthcare worker and should hence not be seen as a radical departure. The guidelines should be seen as guidelines and not an absolute way of filling in a referral form. I suggest that you discuss the evidence on the practical use of the form as a way of describing the departure from the guidelines.

In the fourth paragraph pg. 16 revise spelling for found. The sentence on the limitations also needs to start in its own paragraph.

Reviewer #2: Justification must be made to how data was generated and analysed. The various statistical levels the data was subjected to must be explicitly stated. Referencing in the entire write-up should be worked on. I came across Harvard referencing style at some points, APA and MLA especially with the endnote referencing.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Jackline Oluoch-Aridi

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: REVIEWERS COMMENTS_Plos One.docx

Click here for additional data file.

Submitted filename: Reviewer_Report_Plos One_24.11.21.docx

Click here for additional data file.

1 Feb 2022

Rebuttal letter

Dear Editor,

PLOS ONE Journal

Website: https://journals.plos.org/plosone/s/journal-information

Dec 27, 2021.

Dear Sir/Madam

Thank you for allowing us the opportunity to submit a revised draft of the manuscript. We thank you and the reviewers for a thorough reading and constructive criticism of our manuscript and for the opportunity to revise and resubmit. We are pleased to submit the improved research article, including a proposed slight title change, “Analysis of referral pathways between low resource health settings to improve coordination and evidence-based services for maternal and child health in Ethiopia” for your consideration in the PLOS ONE. On the following pages, you will find our response to the editor and reviewer comments.

On behalf of my co-authors, I thank you for your consideration of this resubmission. We appreciate your time and look forward to your response.

Please see below, in blue, for a point-by-point response to the reviewers. All page numbers refer to the revised manuscript file with tracked changes. All modifications in the manuscript have been highlighted in red.

Sincerely,

Geletaw Sahle Tegenaw

Title:

The title should include person and time.

The title of the article does not fully cover the objective of the research. For instance, there is no point in time in the research findings where the care of patients was analysed using the information on Referral Registration Logbook (RRL) as suggested by the title as an evidence-based point-of-care instrument.
We appreciate the efforts by you and the reviewers on the manuscript. To better frame the main thesis of our paper, we updated the title to “ Analysis of referral pathways between low resource health settings to improve coordination and evidence-based services for maternal and child health in Ethiopia”

Abstract:

Reviewer 1: From page 8 under the heading Objectives line 1-2. The objective of the study suggests that the analysis is to enable data-driven clinical pathway but the discussion and conclusion of the research largely focused on the completeness and feedback mechanism of the RRL and card-sheet.
Reviewer 2: In the Abstract section the results indicate that out of 201. It is not clear what the N(number of matching samples are that lead to the 13%). Kindly indicate this. The results of the study are not compatible with the objectives. The objectives indicate that the study intended to explore the practice whereas the results indicate the main reasons for referral. Kindly align the two. The conclusion section should indicate the overall implication of the study within the health system. It currently reads like an objective. Kindly revise to indicate the implication.

Thanks for the reviewers’ comment and we appreciate your suggestion. We updated the Methods, Results and Conclusion of the abstract for clarification. 
Added and updated as:

Methods
This study analyzed the existing paper-based referral registration logbook (RRL) and card-sheet to explore the documentation of the referral management process, and the mechanism and quality of referrals between the health center (X Health Center-case, Ethiopia) and the Hospital. (X anonymized for blind review). A sample of 459 paper-based records from the referral registration logbook were digitized as part of a retrospective observational study. For data preprocessing, visualization, and analysis, we developed a python-based interactive referral clinical pathway tool. The data collection was conducted from August to October 2019. X Health Center’s RRL was used to examine how the referral decision was made and what cases were referred to the next level of care. However, the RRL was incomplete and did not contain the expected referral feedback from the hospital. Hence, we defined a new protocol to investigate the quality of referral. We compared the information in the health center’s RRL with the medical records in the hospital to which the patients were referred. A total of 201 medical records of referred patients were examined. 

Results
A total of 459 and 201 RRL records from the health center and the referred hospital, respectively, were analyzed in the study. Out of 459, 86.5% referred cases were between the age of 20 to 30 years. We found that “better patient management”, “further patient management”, and “further investigation” were the main health-center referral reasons and decisions. It accounted for 40.08 %, 39.22 %, and 16.34 % of all 459 referrals, respectively. The leading and most common referral cases in the health center were long labor, prolonged first and second stage labor, labor or delivery complicated by fetal heart rate anomaly, preterm newborn, maternal care with breech presentation, premature rupture of membranes, malposition of the uterus, and antepartum hemorrhage. In the hospital RRL and card-sheet, the main referral-in reasons were technical examination, expert advice, further management, and evaluation. We found it overall impossible to match records from the referral logbook in the health center with the patient files in the hospital. Out of 201, only 13.9% of records were perfect matching entries between health center and referred hospital RRL. We found 84%, 14.4%, and 1.6% were appropriate, unnecessary and unknown referrals respectively.

Conclusion
The paper illustrates the bottlenecks encountered in the quality assessment of the referrals. We analyzed the current status of the referral pathway, existing communications, guidelines and data quality, as a first step towards an end-to-end effective referral coordination and evidence-based referral service. Accessing, monitoring, and tracking the history of referred patients and referral feedback is challenging with the present paper-based referral coordination and communication system. Overall, the referral services were inadequate, and referral feedback was not automatically delivered, causing unnecessary delays.

Paragraph 3 of page 8 under the methods portion does not indicate the study type used for the research

.
We updated the methods and added the study types such as a retrospective observational study. We used this terminology consistently in the text (e.g. also in Section 2.1)

Methods:

In page 11 you mention that 459 cases were used. It is not clear how the sampling was done. As much as the authors indicate the stepwise approach the calculation that is present in the paragraph below does not match the sample size given. Kindly clarify.

We appreciate your suggestion and n(minimum)=375 was computed as the minimal required sample size. The data in our sample is guaranteed to be representative since we digitalized RRL records from n = 459 referral consultations.

We added the following explanation based on the suggestion, in Section 2.1. We collected n=459 cases from the RRL to investigate our research questions. The number of women of reproductive age who received care at the health center is estimated to be N=15646. The minimum required sample size was calculated to be n(minimum)=375. The following formulae were applied:

n(minimum) = N*X / (X + N – 1) equation 1 X = Zα/22 *p*(1-p) / MOE2 equation 2

in which Zα/2 is the critical value of the Normal Distribution at α/2 (confidence level is set to 95% and hence the critical value is 1.96, α is 0.05), the margin of error (MOE) is 0.05, p is the sample proportion and N is the population size. The theory behind these calculations is explained in (Israel 1992; Daniel and Cross 2018). We follow the recommendations of WHO stepwise approach to surveillance for the remaining values (WHO 2017b). These value settings allow us to simplify the calculations as formulated in equations 1 and 2: p = 0,5 maximizes the nominator in equation 2 and produces a worst case (i.e. maximal) value for n(minimum), and p = 0,5 is recommended in cases where no a priori results can be used from previous studies; since we only consider women of reproductive age (age 15-48), the number of “age-gender” categories is equal to 1; the response rate is ~100 % since we are performing a retrospective analysis based on RRL; the design effect is set equal to 1 which is recommended for random samples, following the WHO guidelines no finite population correction is applied. The representativeness of the data in our sample is guaranteed, since we digitalized RRL records of n = 459 referral consultations, which is larger than n(minimum).

Page 12 paragraph 1 line 3, the study design used for research is quoted as “a case study” but every evidence about how the data was obtained and analyzed indicated that this was a retrospective observational study design.
We appreciate that the reviewer asked for more accurate wording regarding the study types. We have substituted “a case study” to “a retrospective observational study”, which was employed in our study (e.g. in the modified text of Section 2.1)

Also, in page 11 paragraph 2, there was no mention of the tool that was used to carry out the study. 

Added in Section 2.2: “The referred hospital RRL data was also collected using a pre-prepared electronic data-sheet template.”

In page 12 paragraph 3, line 2 – 5 the analysis was overly concentrated on the completeness and feedback of the referrals but did not cover the other objectives of the research. 
Thank you for the comment. We have added the following relevant information for clarification, in Section 2.4. 

The RRL analysis for the health-center was then conducted. The RRL contains information on patient referrals such as "disease name (cases)," "referred to," "referral reason," "referred date," and "collected signs and symptoms" that were used in primary care prior to making a referral decision. We examined the reasons for referral, disease names (cases), and conclusion (outcome) to see the reasons for the referral decision and which cases were referred to the next level. The referral justification was registered in the RRL “referral-out reasoning” column and documented based on the national guidelines (or referral templates). In order to assess referral cases, it is crucial to examine this referral-out justification (reasons). Figure 1 gives further information about the reason for the referral based on the national guidelines, which should be medically sound, objective, and in the best interests of the patient. 

The referred hospital RRL, on the other hand, was analyzed to identify the missing feedbacks and conclusions to distinguish appropriate referral from unnecessary referral. Examining the RRL column "the final remark and feedback," which was filled by the referring hospital and forwarded to the health center for future referral reference and quality improvement, helped determine the appropriateness of the referral. The health-center referral reason, hospital referral feedback, and conclusion were all analyzed to examine which referrals were appropriate and which were not. We also compared the RRLs from the health center and the referring hospital to ensure that the datasets (or individual records) were matched in general and that the referral reasoning was accurate in particular.

Results:

In page 13 paragraph 3, the study results are well-represented but this does not look at the issues raised in the objectives which include the decision to refer or not to refer a client (to determine unnecessary referrals).

We appreciate the motivation for these comments. We have clarified, and added the following further clarification. 

Added: The referral decision (the reason for the referral) is made based on the signs and symptoms collected at the RRL health center in accordance with the diseases categories (or cases). The collected signs and symptoms were used to assess the risk of diseases (cases), and the referring health professional documented the referral-out justification. We found that meeting any of the six referral criteria (the reason for the referral) influenced the decision to refer based on the referral guidelines, which include “When a patient needs an expert advice as determined by the attending health professional”, “When technical examination is required that is not available at the referring facility”, “When a technical intervention that is beyond the capabilities of the facility is required”, “When patients require inpatient care that cannot be given at the referring facility”, “When the referring facility cannot no more accept patients due to shortage of beds and unavailability of professionals”, and “Referrals are also made to the lower level health facilities and community based organizations in the best interest of the patient depending on the condition of the patient and the capacity of the lower level health facility /community based organization” (FDRE-MoH 2010). 

Added. Using health-center RRL categories such as disease name (or cases), signs and symptoms, conclusion, and referral-out reason, we examined the referral cases and the reasons for the referral decisions. 

Added. It accounted for 40.08 %, 39.22 %, and 16.34 % of all 459 referrals, respectively.

In paragraph 3 under results, the issue of timeliness was not duly covered in the table 2: Analysis of RRL data quality metrics: 
Thank you for your comment. We have now separated timelines from completeness. Added Timeliness Row on Table 2. 
The RRL column records were found to be up to date and reported as such. Referral feedback columns, on the other hand, were usually missing and were not used for real-time reporting.

Can the results also show information outcome of referrals as a proxy appropriateness of referrals based on the condition for treatment?
Thanks for raising these question. The hospital will assess the health center RRL-out information and register as well as update the hospital RRL-in. Examining the RRL column "the final remark and feedback," which was filled by the referring hospital and forwarded to the health center for future referral reference and quality improvement, helped establish the appropriateness of the referral. The referral feedback is crucial in establishing whether or not the referral case is correct. Unfortunately, this feedback and remark was lost in the health center's RRL. We can only verify the appropriateness of referrals based on hospital RRL.

Discussion:

Reviewer 1:

In page 15, line 3, there should be a space between results, and 11/12.
Thank you for the comment. We added Space

In page 15 line 7, the % sign was omitted from the 72.
Thank you for the comment. Added: %

In page 15 under the discussion, again this does not cover the entirety of the study objectives tough it clearly highlights the bottleneck of referrals as the non-provision of feedbacks from the referral point to the referring facility.
We appreciate that the reviewer brought this up. We started our discussion with an overview of data quality evaluation. Then, we describe how the referring healthcare practitioner makes the referral choice (evaluating the "signs and symptoms" and "diseases (or cases") for referral "the conclusion or outcome") and complete the RRL form documentation process. Following that, referral feedback and matching of referral entries between the health center RRL-out and the referring hospital RRL-in are performed. Finally, utilizing existing paper-based referral coordination and communication, we examined the overall history of the referred patients as well as referral feedback.

However, on paragraph two, we added in Section 4 the following for further explanation: 
The referring health professional makes the decision and fills out the RRL refer-out reason based on the referral criteria. Prior to making a referral decision, the referring health professional is responsible for filling out the RRL form with the essential information and attaching any relevant documents. The RRL contains information on patient referrals such as "disease name (cases)," "referred to”, "referral reason”, and "collected signs and symptoms" that were used in primary care. RRL categories "signs and symptoms" and "diseases (or cases)" were used to make the referral decision, as well as the conclusion (or outcome). The signs and symptoms were used to assess the risk of diseases (cases), and the referring health professional documented the referral-out justification. Referral decisions are made when the criteria for referral are medically appropriate and in the best interests of the patient or client. Furthermore, the national guidelines are also used as a reference guide to address the most common presenting symptoms and referral criteria, which aids in the prioritization and documentation of the referral condition (FDRE MoH, 2010, FDRE-MoH 2017).

Reviewer 2:
In Page 16 you indicate that the proportion of “matching”. The sentence seems incomplete. Is it matching entries on referral?

Thanks for the comment and update as …. the proportion of matching entries on referral between health center and referred hospital RRL was 13.9% only)

In the second paragraph in the discussion section you mention that the reasons for referral do not match the guidelines. In my view – the intention is not to exactly match the guidelines word for word. Whereas the reasons for referral in the RRL book indicate reasons that you mention such as “better patient management” they convey the nuance of the needs of the healthcare worker and should hence not be seen as a radical departure. The guidelines should be seen as guidelines and not an absolute way of filling in a referral form. I suggest that you discuss the evidence on the practical use of the form as a way of describing the departure from the guidelines.

We appreciate that the reviewer asked the intention is not to exactly match the guidelines word for word. Despite the fact that the guidelines are not an absolute means of filling out a referral form, it is anticipated that the RRL documentation is based on the national guidelines. This minimizes the variances between health-center referral-out reasoning and referring hospital referral-in reasoning. We added an explanation of the referral practice including the reason and criteria for the referral. The updated version is listed below.

We added:

The referring health professional makes the decision and fills out the RRL refer-out reason based on the referral criteria. Prior to making a referral decision, the referring health professional is responsible for filling out the RRL form with the essential information and attaching any relevant documents. The RRL contains information on patient referrals such as "disease name (cases)," "referred to”, "referral reason”, and "collected signs and symptoms" that were used in primary care. RRL categories "signs and symptoms" and "diseases (or cases)" were used to make the referral decision, as well as the conclusion (or outcome). The signs and symptoms were used to assess the risk of diseases (cases), and the referring health professional documented the referral-out justification. Referral decisions are made when the criteria for referral are medically appropriate and in the best interests of the patient or client. Furthermore, the national guidelines are also used as a reference guide to address the most common presenting symptoms and referral criteria, which aids in the prioritization and documentation of the referral condition (FDRE MoH, 2010, FDRE-MoH 2017).

In the fourth paragraph pg. 16 revise spelling for found. The sentence on the limitations also needs to start in its own paragraph.

Thanks for the comment. The spelling “found” updated and the updated looks like. Unfortunately, we did not find a referral made to lower-level health facilities. The limitations now starts in a new paragraph.

Conclusions:

In page 17 paragraph 2 the conclusion indicates that the study addressed the issues of how the referrals decisions were made, the coordination and communication of the referrals but these were not presented in the results and further discussions. 
We appreciate the motivation for these comments. We have clarified and added the clarification in the Results and Discussion to answer how the referral decisions (the reason for the referral) were made .

In paragraph 2 line 12, the recommendation did not target a specific stakeholder.
We appreciate the motivation for these comments. We added “in primary care settings”

References:

We appreciate the positive comments from Reviewers. In response to [18-25], we rectified all references and citations. The citation is created automatically using the Harvard reference style in Latex/Overleaf. The citation is now consistent in terms of formatting, capitalization, space, punctuation, usage of colon and semicolon, and so on. For instance, we use the format of "FDRE-MoH 2016”, "Teklu et al. 2020”, and "Bettencourt-Silva, Clark, et al. 2015; Zhang, Padman, and Patel 2015; Bettencourt-Silva, Mannu, and Iglesia 2016; Funkner, Yakovlev, and Kovalchuk 2017; Zhang and Padman 2017” for “single author citation”, “multiple authors citations”, and ”multiple sources in one reference” respectively. Also, organized chronologically from oldest to newest.

In page 9, line 5, amia2011: Authors should consider capitalizing A and provide a space between Amia and 2011
The citation is corrected and updated

In page 9, line 8, Teklu, et al 2020: I suggest the punctuation should come after the et al.
Updated as: (FDRE-MoH 2016; Teklu et al. 2020)

In page 9, line 10, Heimly2011: The authors should provide a space between the author’s name and year of publication and a semi-colon after the year of publication.
Updated as: (Heimly and Nytrø 2011)

In page 9, line 18, Fehlings & ODonnel: should be stated Fehlings and ODonnel.
Updated as: (De Bleser et al. 2006; Vargus-Adams Glader Fehlings and ODonnel 2017). Also, organized chronologically from oldest to newest.

In page 9, line 20, (Chu, 2001a, Chu, 2001b, DiJerome, 1992 and Donald et al., 2016)): consider the arrangement of the year of publications. I suggest it should start from the oldest to the current year of publication. Also, please be consistent with the typing of the et al, it's either you are using a full stop together with a comma or choosing one of the afore-mentioned punctuation signs throughout. Again, consider closing the in-text reference with just a bracket.
Updated as: (DiJerome 1992; Chu 2001a; Chu 2001b; Donald et al. 2016). Also, organized chronologically from oldest to newest.

In page 9, line 22, Gonz´alez-Ferrer et al. ,2013: Be consistent with formatting, the et al issue again.
Updated as: Gonz ´alez-Ferrer et al. 2013

In page 9, line 24 and 25, (Bettencourt-Silva, Mannu, and Iglesia, 2016; Zhang and Padman, 2017; Bettencourt-Silva, Clark, et al., 2015; Funkner, Yakovlev, and Kovalchuk, 2017; Zhang, Padman, and Patel,2015): Bettencourt-Silva has no year of publication and I suggest the years are arranged from the oldest to the newest.
Updated as: (Bettencourt-Silva, Clark, et al. 2015; Zhang, Padman, and Patel 2015; Bettencourt-Silva, Mannu, and Iglesia 2016; Funkner, Yakovlev, and Kovalchuk 2017; Zhang and Padman 2017). Also, organized chronologically from oldest to newest.

In page 11, line 9 (under Study Setting and Design): (Daniel, W. W., and Cross, C. L., 2018; Israel, G. D., 1992): In-text referencing should be stated appropriately.
Updated as: (Israel 1992; Daniel and Cross 2018). Also, organized chronologically from oldest to newest.

Submitted filename: Response to Reviewers PLOS ONE.docx

Click here for additional data file.

16 May 2022

21 Jun 2022

Reviewer #3: There is a problem with the title. The use of the word' between' suggests that more than one low resource setting was studied, whereas the study was conducted in Ethiopia only.

We appreciate the efforts by you and the reviewers on the manuscript. To better frame the main thesis of our paper, we updated the title to “Analysis of low resource setting referral pathways to improve coordination and evidence-based services for maternal and child health in Ethiopia”

Please either use 'low resource setting' or 'poor resource setting’

Thanks for the reviewers comment and we appreciate your suggestion. Now, we updated the 'poor resource setting' to 'low resource setting’.

In the conclusion, you suggest use of automated patient management as a solution- is the infrastructure available? is the human resource trained? what would be the 'cost benefit' implication? Whereas you mention that this is inline with the clinical guidelines, context is very important.

We appreciate the motivation for these comments. We updated and provided context. We added “when the necessary infrastructure and human resources are in place”. The whole sentences are as follows:

Therefore, when the necessary infrastructure and human resources are in place, implementing an automated patient management system may help to correct and impose a standardized referral service workflow.

Finally, to provide additional explanation and context, we added:

Furthermore, even before viable infrastructure is implemented and competent human resources are attracted, a minimal infrastructure (a low-cost alternative) and trained personnel can facilitate referral coordination and offer evidence-based services to ensure a buy-in and acceptance by all stakeholders. This transition needs careful planning and openness to adopt best practices in order to reduce resource waste, to ensure training, maintenance, and avoid heavy service expenses.

Submitted filename: Response to Reviewers_v2.docx

Click here for additional data file.

9 Aug 2022

Analysis of low resource setting referral pathways to improve coordination and evidence-based services for maternal and child health in Ethiopia

PONE-D-21-20872R2

Dear Dr.  Geletaw Sahle Tegenaw

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Jackline Oluoch-Aridi, Ph.D.

Guest Editor

PLOS ONE

Additional Editor Comments (optional):

To the authors,

The article has undergone significant improvements with the authors acknowledging the feedback of our reviewers.

Reviewers' comments:

17 Aug 2022

PONE-D-21-20872R2

Analysis of low resource setting referral pathways to improve coordination and evidence-based services for maternal and child health in Ethiopia

Dear Dr. Tegenaw:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Jackline Oluoch-Aridi

Guest Editor

PLOS ONE

Table 1

Summary of the health center and referred hospital RRL dataset.

No. of Cases	Data Sources	Status
459	Jimma Health Center RRL	Without referral feedback
201	Jimma Referred Hospital Card-sheet and RRL	With proper and written feedback

Table 2

Analysis of the RRL data quality metrics.

Data Quality Metrics		Result Found
RRL column format and type		• The overall RRL was recorded according to the RRL column format and type. There were three columns with a value and format of date and time, numeric, and categorical features. Whereas, the remaining RRL columns were string value and format.
Accuracy and uniqueness		• All the RRL column attribute values were valid and it conforms to the syntax (format and type) of the RRL definition.• We observed that the RRL information reflects how the referral process coordination and communication were managed• None of the RRL values were duplicated.
Completeness		• Of 459, 100% of all RRL columns were documented except the “referral feedback” columns. The “referral feedback” column was consistently empty (0%).• Among the RRL “referral date” column value, 98% were non-missing values. Only 2% of the referral date were not recorded.
Timeliness		• The RRL column records were found to be up to date and reported as such. Referral feedback columns, on the other hand, were usually missing and were not used for real-time reporting.
Consistency and validity	Internal consistency of RRL	• Anomalies were found. For example, two age values (i.e. 88 and 2423) replaced by the average value of the age because the two values were considered as a typing error.• The layout and format of the handwritten information presented in the patient card-sheet and RRL were inconsistent from one record to another• Health center RRL reasons did not match the national guidelines.
	External consistency between Health-center RRL-out and Hospital RRL-in	• We found it overall impossible to match referral reasoning from the referral logbook in the health center with the referral feedback in the hospital.• The health center RRL reason was not exactly followed by the hospital.