Performance of a new symptom checker in patient triage: Canadian cohort study.

BACKGROUND: Computerized algorithms known as symptom checkers aim to help patients decide what to do should they have a new medical concern. However, despite widespread implementation, most studies on symptom checkers have involved simulated patients. Only limited evidence currently exists about symptom checker safety or accuracy when used by real patients. We developed a new prototype symptom checker and assessed its safety and accuracy in a prospective cohort of patients presenting to primary care and emergency departments with new medical concerns.
METHOD: A prospective cohort study was done to assess the prototype's performance. The cohort consisted of adult patients (≥16 years old) who presented to hospital emergency departments and family physician clinics. Primary outcomes were safety and accuracy of triage recommendations to seek hospital care, seek primary care, or manage symptoms at home.
RESULTS: Data from 281 hospital patients and 300 clinic patients were collected and analyzed. Sensitivity to emergencies was 100% (10/10 encounters). Sensitivity to urgencies was 90% (73/81) and 97% (34/35) for hospital and primary care patients, respectively. The prototype was significantly more accurate than patients at triage (73% versus 58%, p<0.01). Compliance with triage recommendations in this cohort using this iteration of the symptom checker would have reduced hospital visits by 55% but cause potential harm in 2-3% from delay in care.
INTERPRETATION: The prototype symptom checker was superior to patients in deciding the most appropriate treatment setting for medical issues. This symptom checker could reduce a significant number of unnecessary hospital visits, with accuracy and safety outcomes comparable to existing data on telephone triage.

Background

Worldwide, 70% of internet users go online to access health information [1-3]. Most inquiries begin with search engines such as Google, but results are often incomplete, inaccurate, or inaccessible to lay persons [3-5]. Potential consequences of unreliable or inappropriate health information include delay of care, inappropriate hospital visits, and cyberchondria [6, 7]. These online health-seeking behaviors may be driven in part by a knowledge gap with respect to what people should do if they become ill, given that public health education focuses primarily on preventive health [8, 9]. This knowledge is also difficult to teach, given that even trained health professionals may struggle to identify patients who would most benefit from emergency department care [9-13].

During the COVID-19 pandemic, computerized algorithms known as “symptom checkers” were widely implemented to limit unnecessary contact between patients and healthcare providers, alleviate pressures on telehealth systems, and empower patients to decide where best to access care for their symptoms [14-18]. Symptom checkers are not new, and many already existed prior to the pandemic [19]. Unfortunately, few studies have been published about the safety or accuracy of symptom checkers and, of existing studies, all suffer from at least one of the following limitations: the literature was not peer-reviewed; the results were based solely on simulated data, the selection of patients covered a limited range of conditions; the data was not generated by patients entering in their own symptoms; or the studies did not report safety outcomes [20]. Development of a safe and effective symptom checker could lower barriers to accessing care, encourage patients to go to the hospital for potentially life-threatening problems, discourage unnecessary healthcare visits, and reduce wait times by encouraging appropriate healthcare utilization [19, 21–25].

We developed our own prototype symptom checker [26] and conducted this study to address the methodological limitations found in existing studies [27, 28]. In particular, we detail the performance of the symptom checker when used by adults seeking care from family physicians and emergency departments.

Methods

Development of the symptom checker

The prototype symptom checker was designed by author FC and coded by computer science students at Western University [26]. The version of the algorithm used for this study was comprised of a total of 247 questions or computational steps manually compiled into a decision tree with recursive elements. Relevant questions are presented sequentially to the user based upon previous responses.

To create the questions, common medical diagnoses and concerns were first collected through medical textbooks (e.g. Toronto Notes, 2018; Edmonton Manual, 2018; Harrison’s Internal Medicine 20e) and online physician resources (e.g. Uptodate, Dynamed, College of Family Physicians of Canada). Literature review was then performed for each diagnosis to identify questions and factors that would impact escalation of treatment. Diagnostic questions were specifically gathered for conditions considered emergency and conditions that can be managed by patients at home. Diagnostic questions that did not change triage recommendations were discarded. No patient vignettes were used for the development of this symptom checker.

Study design

A prospective cohort study was done with patients who self-presented to emergency departments and primary care clinics. Ethics approvals were obtained from the research ethics boards of Western University, the University of British Columbia, and Whitehorse General Hospital, respectively. The prospective sample included Canadian patients presenting to 2 emergency departments (Whitehorse General Hospital in Whitehorse, YT; Royal Inland Hospital in Kamloops, BC) and 13 full-time family physician practices based out of 3 centres in London, Ontario (Victoria Family Medical Centre, St. Joseph’s Family Medical Centre, and Byron Family Medical Centre). The research ethics committees approved the lack of parent or guardian consent for minors because all intended participants are 16 years of age or older. The sample was representative of adult Canadian patients who seek professional medical attention.

In the waiting rooms of each location, an unattended kiosk and information poster invited adult patients to use a tablet computer programmed with the prototype symptom checker. Inclusion criteria were adults ≥16 years reporting that they were seeking care for a new medical issue. The information poster contained details typically found in a patient letter of information. Patients were incentivized to participate with opportunities to win $50 gift cards. On the tablets, patients signed consent and entered their name, age, gender, and symptom checker responses. Patient recruitment and data collection occurred from June 2019 to February 2020. Patients were blinded to the symptom checker’s triage recommendation after completion and treating physicians were blinded to patient participation.

Data management

Identifying information and responses to the prototype symptom checker were stored in separate files on the tablets to ensure blinding during analysis. Identifying information was used to link medical records. If the corresponding patient records could not be found, the symptom checker responses were excluded from analysis. A list of valid medical records was collected and authors (MP, LR, AS, JP, AT, CP, TR) extracted the impression and plan written by the treating physician.

The diagnoses and treatments provided by the treating physician were reviewed by physician authors FC and SL and, by consensus, assigned an appropriate categorization for comparison with triage recommendations provided by the symptom checker (Table 1). The process of dividing patient encounters into broad categories of immediate/emergent, urgent, primary/routine care, and home/self-care is a standard process in similar studies that examine the performance of telemedicine triage and emergency triage systems [29, 30]. Categorization was done with blinding to triage recommendations made by the symptom checker. The treating physicians’ diagnoses and management plans were used the gold standard for comparison with recommendations provided by the symptom checker; patients’ presenting symptoms were not considered when categorizing the true severity of the patient’s medical issue.

Table 1

Criteria used for categorizing the acuity of patient medical encounters with a healthcare professional.

Information from patient medical records	Subsequent categorization of the patient	Most appropriate triage location for this categorization
Patient was admitted to hospital, sent to hospital, or required immediate treatment usually only available in hospital.	Emergency	Hospital
Patient’s diagnosis was not life or limb threatening, but required timely assessment by an emergency physician or primary care provider for treatment or referral for treatment	Urgency	Hospital or primary care
Initial treatment provided was wholly within scope of an outpatient family physician practice. Minimal risk for significant harm would result from delay in providing the treatment administered by the physician.	Routine	Primary care
Management advice was given to the patient for an issue that did not require a prescription, referral, or further diagnostic testing.	Home- Appropriate	Home

Some physicians documented multiple diagnoses or issues for the healthcare visit. In these cases, each diagnosis was separately categorized as an emergency, urgency, routine, or home appropriate. For example, a patient encounter with dual diagnoses of “1) vaginal discharge not yet diagnosed and 2) lower back pain” were categorized as routine and home appropriate, respectively, because investigations were ordered by the physician for the vaginal discharge and home-treatment solutions were recommended for the lower back pain. Symptom checker responses were then assessed to determine the most appropriate physician diagnosis. In this specific encounter, the patient selected on the symptom checker that the concern was not related to pain; thus the diagnosis “vaginal discharge not yet diagnosed” was used for the purposes of assessing symptom checker performance.

Patients were excluded from final analysis if the treating physician commented that the patient was being seen for follow-up, or if there was a very significant mismatch between patient and physician concerns for visit (e.g. the patient selected “I want to seek help related to violence against women” and the visit diagnosis was “otitis media”).

Statistical methods

Based on data gathered in the first month, study recruitment to allow for 80% power was guided by a predicted 20% prevalence and a conservative 75% sensitivity for urgencies. 234 participants were required from each of hospital and clinic settings. To adjust for patient records that could not be identified, minimum recruitment was set at 292 participants.

Outcomes measures were pre-specified. Triage was considered accurate if the triage categorization, as determined by the treating physician’s management plan, matched the triage recommendation provided by the symptom checker. Data were calculated with 95% confidence intervals about the mean. Sensitivity and positive predictive value (PPV) were calculated with 95% confidence intervals (Wilson-score method). A two-tailed McNemur’s Test was used to compare the accuracy of patient decision making with the prototype. Calculations were done with Microsoft Excel (2016) and SPSS (version 21).

Results

Included for final analysis were 281 and 300 patients who presented to emergency departments and primary care clinics, respectively (Fig 1). Mean patient age was 38±16 years (range 16 to 91 years) for emergency department patients and 48±18 years (range 16 to 91 years) for patients seen by primary care family physician. 366 of 581 patients (63%) were female. There was a diverse collection of patient concerns captured including trauma, mental health, pregnancy, immunizations, infections, cardiovascular, respiratory, gastrointestinal, and dermatologic concerns, among others. Of patients with mismatches between patient and physician responses, mean age was 40±15 years (range 19 to 94 years) for emergency patients and 50±20 years (range 19 to 94 years) for patients seen by primary care. The algorithm’s triage “accuracy” for these mismatched cases was 90%, but was not included for analysis because the patient used the symptom checker for an entirely different reason than what was discussed with the physician.

Fig 1

Flow of study participants.

For emergency department patients (Table 2; n = 281), the accuracy of the prototype symptom checker (172/281 encounters or 61%; 95% CI of 55% to 67%) was significantly better than that of patients (90/281 encounters, 32%; CI of 27% to 38%; p<0.01). No emergencies were missed by the symptom checker (n = 8); the diagnoses of these emergencies were: ingestion of death camas; incarcerated inguinal hernia; biceps tendon tear or rupture; alcohol intoxication with coffee ground emesis; upper gastrointestinal bleed requiring transfusion; intoxication, chest pain, and possible Brugada; self-inflicted wrist laceration; hand laceration from a drill through the hand; and antepartum hemorrhage. Sensitivity to urgencies was 90% (73/81 encounters; CI of 82% to 95%). Under-triage for urgencies occurred in 3% (8 encounters, CI of 1% to 6%); diagnoses for these cases were: abscess of thigh, phalanx fracture, boxer’s fracture, post lipoma excision hematoma, hand laceration, metacarpal fracture, toe fracture, and metallic foreign body in shin. Over-triage to hospital occurred in 76 encounters (27%, CI 22% to 33%). Patient compliance with the symptom checker’s triage advice would have reduced total hospital visits by 55% (155/281 encounters, CI of 49% to 61%).

Table 2

Triage recommendations made by the prototype symptom checker for patients self-presenting to hospital.

		Encounter categorization (n = 281)
		Emergency	Urgency	Routine	Home-Appropriate	PPV
Symptom checker’s management recommendation	Hospital	9	41	54	22	40% (32–48%)
	Primary Care	0	32	78	8	93% (87–97%)
	Home	0	8	17	12	32% (0–14%)
Event rate		3%	29%	53%	15%
Sensitivity		100%	90%	52%	29%
		(70–100%)	(82–95%)	(44–60%)	(17–44%)

Green colored cells represent accurate triage recommendations made by the symptom checker. Red colored and uncolored cells represent under-triage and over-triage, respectively. Sensitivity and positive predictive value (PPV) are reported with 95% confidence intervals).

For patients who presented to primary care (Table 3; n = 300), the accuracy of the symptom checker (85% or 255/300 encounters; CI of 81%-89%) was similar to that of patients’ (83% or 249/300; CI of 78% to 87%; p = 0.11). One patient with an emergency diagnosis of “possible deep vein thrombosis” presented to primary care, but the symptom checker would have correctly advised the patient to go to the hospital. Sensitivity to urgencies was 97% (34/35, CI 85% to 99%). Under-triage occurred in one urgency encounter wherein the diagnosis was “possible cellulitis surrounding tracheostomy”. Over-triage to hospital occurred in 20 encounters (7%, CI 4% to 10%) in which hospital triage was suggested for issues that could be managed either at home or by primary care.

Table 3

Triage recommendations made by the prototype symptom checker for patients self-presenting to family physician clinics.

		Encounter categorization (n = 300)
		Emergency	Urgency	Routine	Home-Appropriate	PPV
Symptom checker’s management recommendation	Hospital	1	26	15	5	57% (43–70%)
	Primary Care	0	8	193	18	88% (83–92%)
	Home	0	1	6	27	79% (63–90%)
Event rate		0.3%	12%	71%	17%
Sensitivity		100%	97%	90%	54%
		(21–100%)	(85–99%)	(85–93%)	(40–67%)

Green colored cells represent accurate triage recommendations made by the symptom checker. Red colored and uncolored cells represent under-triage and over-triage, respectively. Sensitivity and positive predictive value (PPV) are reported with 95% confidence intervals).

The overall accuracy of the symptom checker was 73% (427/581, 95% CI 70% to 77%), which was significantly better than the 58% accuracy of patients (339/581, CI 54% to 62%; p<0.01). Under-triage causing potential harm from delay in care occurred in a total of 9 encounters (2%, CI 1% to 3%), in which the symptom checker recommended home management was for an urgency.

Discussion

This is the first study to report a direct comparison between the triage accuracy of a symptom checker against decisions made by patients. The overall accuracy of our prototype symptom checker was 73% (95% CI 70% to 77%), with potential to harm in 2–3% of encounters. These results are promising given that, in context, telephone triage is reported to have a median accuracy 75% and a rate of harm from under-triage of 1.3–3.2% [29]. The results of this study may be most directly applied to a scenario wherein presenting patients at healthcare facilities have an opportunity to obtain a rapid computer generated opinion about their medical concern, with the possibility of redirecting care to a nearby hospital or clinic providing primary care for walk-in patients.

The symptom checker was less accurate among patients who presented to hospital compared to those who presented to primary care. Over-triage to hospitals occurred more often for patients who self-presented to the emergency department (76/281, 27%) compared to those who sought primary care (20/300, 7%). This may have occurred because patients who go to the hospital reported having more severe symptoms when using the symptom checker, which resulted in the symptom checker recommending a higher acuity response. Under-triage for urgencies occurred in several instances specifically related to distal extremity injuries and improvements will need to be made for these types of presenting concerns. The physician visit diagnosis and patient’s symptom checker responses were significantly mismatched for 135 patients and excluded from analysis (S1 Table); this may have been because patients wanted to utilize the symptom checker out of personal interest, but did not wish to reveal details about their personal health information.

This is the first study to fulfill three key criteria in the assessment of a symptom checker’s triage performance which was found to be lacking in previous studies: triage recommendations are compared to care provided by a clinician as the reference standard; testing was done in a general population of patients who continue to receive standard care; and an unrestricted range of symptoms was assessed [27, 28]. All peer-reviewed studies involving patients published thus far do not meet one or more of the above criteria (Table 4) [25, 31–39]. Three non-peer reviewed reports were reviewed; two had significant risks for bias and the one government report did not provide sufficient information to interpret outcome measures [40-42].

Table 4

Summary of peer-reviewed papers on symptom checker usage by patients.

All studies were limited by at least one of three factors (in grey).

Paper	Assessment of Triage Accuracy	Population	Symptoms assessed
Meyer et al.[23]	Did not assess accuracy	General population of registered users of the symptom checker	Unrestricted
Berry et al.[31]	Assessed accuracy	Patients presenting to an outpatient internal medicine clinic with abdominal pain symptoms	Abdominal pain only
Nijland et al.[32]	Did not assess accuracy	General population of web users	Unrestricted
Poote et al.[33]	Risks of bias in assessing accuracy (Treating physicians were not blinded to triage outcomes, potentially subjective triage categorization schema)	University students only	Unrestricted
Verzantvoort et al.[34]	Accuracy was measured using nursing triage as the reference standard. Triage does not assess potential emergencies in determination of accuracy	General population of primary care patients	Unrestricted
Sole et al.[35]	Only 5% (n = 59) of symptom checker users were able to be assessed by a physician for accuracy	College students only	Unrestricted
Price et al.[36]	Assessed accuracy	Children only	Influenza-like illness only
Winn et al.[37]	Did not assess accuracy	Users of an online chat bot	Unrestricted
Cowie et al.[38]	Did not assess accuracy	General population in Scotland	Unrestricted
Miller et al.[39]	Did not assess accuracy	General population in England	Unrestricted

Limitations

Across emergency department and primary care settings, the prototype appropriately triaged 10 of 10 patients diagnosed with an emergency but the study was not powered for emergencies. The sample size of emergencies was small because of the relative infrequency of emergencies compared to other medical presentations. Emergencies were also more likely to arrive by ambulance which bypasses the waiting room. Patients who felt very unwell were also less likely to use the symptom checker, based on the study design. Further studies will be needed to ensure sensitivity to emergencies.

Reproducibility of the study’s results in other countries may be challenging given that local guidelines, cultural factors, and access to resources may differ. The results of these studies cannot be generalized to other symptom checkers given the diversity of triage approaches [19].

Some questions remain unanswered by this study. It is unclear if patients will respond to recommendations by the symptom checker. It is also uncertain how the symptom checker would perform among a population of patients who have chosen to stay at home, instead of seeking care at a hospital or outpatient clinic. Implementation of a symptom checker on a community level would be necessary to demonstrate if implementation results in changes in healthcare utilization and changes in population morbidity.

Conclusions

The prototype symptom checker was superior to patients in deciding the most appropriate treatment setting for medical issues. Use of the symptom checker by patients seeking medical care could reduce a significant number of unnecessary hospital visits, with accuracy and safety outcomes comparable to existing data on telephone triage.

Raw data: Comparison of symptom checker recommendations to physician diagnosis and treatment.

De-identified data containing information ported directly from patient charts may be available upon approval from the Western Research Ethics Board for authorized individuals.

(XLSM)

Click here for additional data file.

26 May 2021

PONE-D-20-39920

Performance of a New Symptom Checker in Patient Triage: Canadian Cohort Study

PLOS ONE

Dear Dr. Chan,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The reviewers have identified several aspects of your study design that require further clarification. Please ensure that you address these points thoroughly in your revisions.

Please submit your revised manuscript by Jul 09 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Jamie Males

Staff Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please state in your methods section whether you obtained consent from parents or guardians of the minors included in the study or whether the research ethics committee or IRB approved the lack of parent or guardian consent.

3. In your Methods section, please provide additional information about the participant recruitment method and the demographic details of your participants. Please ensure you have provided sufficient details to replicate the analyses such as: a) the recruitment date range (month and year), b) a description of any inclusion/exclusion criteria that were applied to participant recruitment, c) a table of relevant demographic details, d) a statement as to whether your sample can be considered representative of a larger population, e) a description of how participants were recruited, and f) descriptions of where participants were recruited and where the research took place.

4.We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: A good small proof of concept study to assess a computerized symptom checker with actual care provided in 2 emergency departments (Whitehorse and Kamloops, BC) and 13 full-time family physicians in London.

Nice concept but lack details and has the several weaknesses

1. Very small sample size evident from large confidence intervals of the reported results

2. Inter-rater reliability of physician reviewer for categorization

3. Very scant details regarding the algorithms created

4. While the checker performs in a reasonable way for identifying emergencies (with wide confidence intervals due to small sample size), its overall accuracy of around 60% is a concern.

Reviewer #2: This study addresses a major gap in the evaluation of symptom checkers (SC), the use by real patients at the time of presentation with a medical complaints in the ED or urgent primary care with results compared to actual clinical decisions. The authors developed their own symptom checker to advise on triage and appropriate level of care. This was tested in two hospital emergency departments and in 13 primary care sites in Canada. Patients and clinicians were blinded to the symptom checkers responses. Records were linked from patients name, age and gender. Urgency of triage was assessed with a 4 point scale (immediate/emergency, urgent, primary/routine, home care).

Results show that the new SC performed equivalently in sensitivity and PPV to telephone triage (based on a systematic review from 2012). No emergency cases were missed but 8 patients in hospital with the need for urgent care would have been recommended to stay home. One patient in primary care with an urgent problem would have similarly been told to stay home. In the hospital context the SC was significantly more accurate at triage than the patients themselves, but in primary care accuracy was very similar. The only significant weakness noted is a lack of information on the patient’s experience in using the SC.

Overall this is a very good study that is much larger and higher quality that nearly all existing studies. It has a low risk of bias, based on accepting all ED or primary care patients, having patients enter data on their own symptoms and comparing the SC performance to the actual physicians’ decisions. This study makes a major contribution to the field.

Minor revisions:

Clarification is required in the handling of cases where there was more than 1 diagnosis. It is stated that “Symptom checker responses were then assessed to determine the most appropriate physician diagnosis.” What were the criteria for determining the appropriate one – whether the data collected by the symptom checker was more supportive of one of the physician’s diagnoses? What if the data included questions relevant to both? What number/percentage of cases had more than one physician diagnosis?

Patient were excluded “if there was a very significant mismatch between patient and physician concerns for visit”. What number/percentage of cases were excluded by this criteria?

In listing the 3 key criteria for quality of studies the authors may wish to add a 4th, data entered by patients on their own symptoms, which is missing from most existing studies of SCs, that use existing or made up cases.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Dr Hamish Fraser

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

13 Oct 2021

Reviewer #1: A good small proof of concept study to assess a computerized symptom checker with actual care provided in 2 emergency departments (Whitehorse and Kamloops, BC) and 13 full-time family physicians in London.

Nice concept but lack details and has the several weaknesses

1. Very small sample size evident from large confidence intervals of the reported results

2. Inter-rater reliability of physician reviewer for categorization

3. Very scant details regarding the algorithms created

4. While the checker performs in a reasonable way for identifying emergencies (with wide confidence intervals due to small sample size), its overall accuracy of around 60% is a concern.

Response to Reviewer #1:

1) Contextually in this field of study, the sample size in our study is actually quite large (n=581). Of existing studies on symptom checkers that assessed for triage accuracy when used patients (Table 4), the sample sizes were 49 (Berry et al.) and 294 (Price et al.) individuals. As a result of the large sample size, the confidence intervals were quite narrow except for encounters for medical emergencies, which is likely what the reviewer has issue with. The sample size for emergencies was small, as expected (n=10). This is also addressed under the limitations section of the manuscript, which is quoted here for convenience: “Across emergency department and primary care settings, the prototype appropriately triaged 10 of 10 patients diagnosed with an emergency but the study was not powered for emergencies. The sample size of emergencies was small because of the relative infrequency of emergencies compared to other medical presentations. Emergencies were also more likely to arrive by ambulance which bypasses the waiting room. Patients who felt very unwell were also less likely to use the symptom checker, based on the study design. Further studies will be needed to ensure sensitivity to emergencies.”

2) Inter-rater reliability data is not available because it was done by consensus between FC and SL. This detail is now make more clear in the Methods section.

3) The purpose of this paper is not to detail how the algorithm functions, but how well it functions. Contextually, in most studies assessing the accuracy of patient telephone triage, the information provided is even more limited regarding the protocols in place to triage patients and compliance with operational standards. In addition, the algorithm has been made available to be used for free online at symptomcheck.ca, as cited in the manuscript.

4) The overall accuracy of the symptom checker was 73%. Contextually, telephone triage has a similar level of accuracy, as stated in the Discussion section. In the sub-population of patients from emergency departments, the accuracy was lower (61%) because it was more likely to recommend patients to go to the hospital for routine issues, likely because they reported more severe symptoms (see 2nd paragraph in the discussion); this was still much more accurate than patient decision making. The algorithm correctly identified 10 of 10 medical emergencies in the sample size.

Reviewer #2:

Minor revisions:

1) Clarification is required in the handling of cases where there was more than 1 diagnosis. It is stated that “Symptom checker responses were then assessed to determine the most appropriate physician diagnosis.” What were the criteria for determining the appropriate one – whether the data collected by the symptom checker was more supportive of one of the physician’s diagnoses? What if the data included questions relevant to both? What number/percentage of cases had more than one physician diagnosis?

2) Patient were excluded “if there was a very significant mismatch between patient and physician concerns for visit”. What number/percentage of cases were excluded by this criteria?

3) In listing the 3 key criteria for quality of studies the authors may wish to add a 4th, data entered by patients on their own symptoms, which is missing from most existing studies of SCs, that use existing or made up cases.

Response to reviewer #2:

1) We have adjusted the relevant paragraph of the methods for added clarity. The paragraph is copied here for clarity: “Some physicians documented multiple diagnoses or issues for the healthcare visit. In these cases, each diagnosis was separately categorized as an emergency, urgency, routine, or home appropriate. For example, a patient encounter with dual diagnoses of “1) vaginal discharge not yet diagnosed and 2) lower back pain” were categorized as routine and home appropriate, respectively, because investigations were ordered by the physician for the vaginal discharge and home-treatment solutions were recommended for the lower back pain. Symptom checker responses were then assessed to determine the most appropriate physician diagnosis. In this specific encounter, the patient selected on the symptom checker that the concern was not related to pain; thus the diagnosis “vaginal discharge not yet diagnosed” was used for the purposes of assessing symptom checker performance.”

2) The number of excluded patients are presented in Figure 1. Additional information is provided in the supplementary data, where further commentary is made about the discrepancies between the patient’s symptom checker responses and notes written by physicians.

3) This was a useful comment, and a sentence was added to the introduction to reflect this.

Journal requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming.

-Done

2. Please state in your methods section whether you obtained consent from parents or guardians of the minors included in the study or whether the research ethics committee or IRB approved the lack of parent or guardian consent.

-Done

3. In your Methods section, please provide additional information about the participant recruitment method and the demographic details of your participants. Please ensure you have provided sufficient details to replicate the analyses such as: a) the recruitment date range (month and year), b) a description of any inclusion/exclusion criteria that were applied to participant recruitment, c) a table of relevant demographic details, d) a statement as to whether your sample can be considered representative of a larger population, e) a description of how participants were recruited, and f) descriptions of where participants were recruited and where the research took place.

-All of the information was already there, but we have changed some words for added clarity.

4. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

-Please see our cover letter for further comments.

11 Nov 2021

PONE-D-20-39920R1Performance of a New Symptom Checker in Patient Triage: Canadian Cohort StudyPLOS ONE

Dear Dr. Chan,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We have now received one review on your revised submission. While the reviewer is largely happy with the revisions, they also raise one concern about the exclusions that it would be helpful to see addressed. Please submit your revised manuscript by Dec 26 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Andrea Gruneir

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: Thank you for your responses to my comments. The figure 1 indicates that about 22% of patients were excluded due to "very significant mismatch between patient and physician concerns for visit" which impacts on the overall usability of the system. It is a lot higher proportion than what we have seen in our own studies. Any indication as to why these patients didn't use the symptom checker for the complaint that brought them to the ED or primary care center?

One other comment relates to the question of the algorithm type (raised by reviewer 1). It is very helpful to reviewers and readers to know broadly what type of algorithm is involved. It presumably was not created by any machine learning technique - useful point to clarify. If it uses a Bayesian network for example (like some other symptom checkers) that would be helpful to know.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: Yes: Hamish Fraser

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

11 Nov 2021

Regarding reviewer #2 and PLOSONE editor comments:

Thank you very much for your thoughtful review of our paper!

Regarding "4. Have the authors made all data underlying the findings in their manuscript fully available?",

-the PLOS one editors specifically did not further comment upon this, and we presume that our detailed response from the prior review cycle was satisfactory.

Regarding "Reviewer #2: Thank you for your responses to my comments. The figure 1 indicates that about 22% of patients were excluded due to "very significant mismatch between patient and physician concerns for visit" which impacts on the overall usability of the system. It is a lot higher proportion than what we have seen in our own studies. Any indication as to why these patients didn't use the symptom checker for the complaint that brought them to the ED or primary care center?"

-Unfortunately, we can only hypothesize about why patients did not respond truthfully. Our foremost hypothesis is that patients wanted to participate and use our symptom checker application out of personal interest, but did not want to reveal their own personal health information. We have now noted this hypothesis in the paper's discussion "The physician visit diagnosis and patient’s symptom checker responses were significantly mismatched for 135 patients and excluded from analysis (S1 Table); this may have been because patients wanted to utilize the symptom checker out of personal interest, but did not wish to reveal details about their personal health information."

-In the Supporting Documents, the management of patient data for every individual patient is provided for transparency. Specifically, it does details why a mismatch was flagged for the patient. For example, for patient in row A17, the patient using the symptom checker responded that they were not experiencing any skin related issues or pain; however, their hospital diagnosis was "cellulitis of the hand" which inherently is a skin related issue that almost always presents with pain.

Regarding "One other comment relates to the question of the algorithm type (raised by reviewer 1). It is very helpful to reviewers and readers to know broadly what type of algorithm is involved. It presumably was not created by any machine learning technique - useful point to clarify. If it uses a Bayesian network for example (like some other symptom checkers) that would be helpful to know.":

-A phrase has now been added to clarify that the algorithm was "manually compiled into a decision tree with recursive elements".

16 Nov 2021

Performance of a New Symptom Checker in Patient Triage: Canadian Cohort Study

PONE-D-20-39920R2

Dear Dr. Chan,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Andrea Gruneir

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

19 Nov 2021

PONE-D-20-39920R2

Performance of a New Symptom Checker in Patient Triage: Canadian cohort study

Dear Dr. Chan:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Andrea Gruneir

Academic Editor

PLOS ONE