Personalized risk stratification through attribute matching for clinical decision making in clinical conditions with aspecific symptoms: The example of syncope.

BACKGROUND: Risk stratification is challenging in conditions, such as chest pain, shortness of breath and syncope, which can be the manifestation of many possible underlying diseases. In these cases, decision tools are unlikely to accurately identify all the different adverse events related to the possible etiologies. Attribute matching is a prediction method that matches an individual patient to a group of previously observed patients with identical characteristics and known outcome. We used syncope as a paradigm of clinical conditions presenting with aspecific symptoms to test the attribute matching method for the prediction of the personalized risk of adverse events.
METHODS: We selected the 8 predictor variables common to the individual-patient dataset of 5 prospective emergency department studies enrolling 3388 syncope patients. We calculated all possible combinations and the number of patients in each combination. We compared the predictive accuracy of attribute matching and logistic regression. We then classified ten random patients according to clinical judgment and attribute matching.
RESULTS: Attribute matching provided 253 of the 384 possible combinations in the dataset. Twelve (4.7%), 35 (13.8%), 50 (19.8%) and 160 (63.2%) combinations had a match size ≥50, ≥30, ≥20 and <10 patients, respectively. The AUC for the attribute matching and the multivariate model were 0.59 and 0.74, respectively.
CONCLUSIONS: Attribute matching is a promising tool for personalized and flexible risk prediction. Large databases will need to be used in future studies to test and apply the method in different conditions.

Introduction

Clinical decision tools (CDT) combine different predictors (from patients’ history, clinical examination and tests results) to assess the probability of a diagnosis, prognosis, or response to treatment of an individual patient [1]. The statistical techniques used in this process are usually based on multivariate models such as logistic regression [2]. Other approaches include recursive partitioning analysis and artificial neural networks [3-5]. As they are based on models, CDTs are able to predict the risk of any hypothetical patient, even those with a combination of risk factors different from all the patients of the derivation cohort. Therefore, we do not know how the CDT will perform in subjects with specific clinical presentations or needs. Indeed, they lack the ability to provide personalized estimates as required in the era of precision medicine. For example, patients with uncommon diseases are likely not to be correctly risk stratified by CDTs. In addition, the risk estimates of composite outcomes that are usually provided by CDTs cannot always be applied to all patients, as the definition of “acceptable risk” depends on the patient at risk. Hence the need to assess a personalized risk rather than providing a simple binary answer [6].

Moreover, risk stratification is challenging in conditions (as chest pain, shortness of breath and syncope) presenting with aspecific symptoms that can be the manifestation of many possible underlying diseases. In these cases, decision tools are unlikely to accurately identify all the different adverse events related to the possible etiologies. In syncope, which is a paradigm of the above conditions, the traditionally derived risk stratification tools have failed in predicting adverse events [7-12]. Here, an individualized risk assessment would allow an estimate of not only the probability of a composite endpoint, but rather a detailed risk profile that provides the individual risk of each specific outcome (e.g. arrhythmia or pulmonary embolism).

Attribute matching (AM) is a prediction approach that differs considerably from the regression models and has shown promising results in ruling out acute coronary syndrome and pulmonary embolism in patients with chest pain [13-15]. Instead of considering each clinical characteristic as an individual predictor and deriving a risk estimate based on the sum of their regression coefficients, each individual patient is matched to a group of patients with the same combination of the relevant clinical characteristics (or attributes) from a large reference database. Therefore, each patient is matched to a group of patients with identical risk profile and known outcomes. This approach results in a proportion (i.e. the number patients who had the outcome of interest on the number of previously studied matched patients) that provides the probability (with confidence interval) of the single adverse event. This process resembles the definition of pre-test probability by an expert clinician, which, having seen many patients who had similar clinical characteristics as the patient under consideration, could provide an estimate of the probability of something bad happening. In this case, the computer does so with less variability and without the clinician having to be experienced nor an expert. The aim of this study was to explore the use of AM to predict the personalized risk of adverse events and to compare it to multivariate logistic regression to analyze the possible similarities, differences, strengths and weaknesses of the two methods using syncope in the Emergency Department (ED) as an example.

Materials and methods

To apply AM in a large database, we used an individual-patient dataset from a previous international collaboration that involved 3388 patients prospectively included in 5 studies enrolling syncope patients in the ED from 2000 to 2014 [8,16-20]. The dataset was analyzed to detect demographic and clinical variables among those considered to be relevant for syncope risk stratification as have shown to be related to adverse events [16,17,19,21]. Each single dataset was re-analyzed to create homogeneously defined variables for abnormal electrocardiogram (ECG) and 7–10 day serious outcomes [7,12,22]. We finally identified the variables that were available in all 5 datasets.

The AM estimates of the probability of serious adverse is based upon computer assisted, database-derived system. The clinician puts in a predefined set of clinical attributes for a subject for whom the probability of a serious outcome is unknown. A computer program queries a large patient database, and returns only the patients who share the identical attribute profile as the patient under consideration. The proportion of these attribute-matched subjects who had a clinical outcome of interest is the probability of adverse events.

According to the “Standardized reporting guidelines for emergency department syncope risk-stratification research” serious outcomes included any of the following [22]: 1) all-cause and syncope-related death, 2) ventricular fibrillation, 3) sustained and symptomatic non-sustained ventricular tachycardia, 4) sinus arrest with cardiac pause > 3 s, 5) sick sinus syndrome with alternating bradycardia and tachycardia, 6) second-degree type 2 or third-degree AV block, 7) permanent pacemaker (PM) or implantable cardioverter defibrillator (ICD) malfunction with cardiac pauses, 8) aortic stenosis with valve area ≤ 1 cm2, 9) hypertrophic cardiomyopathy with outflow tract obstruction, 10) left atrial myxoma or thrombus with outflow tract obstruction, 11) myocardial infarction, 12) pulmonary embolism, 13) aortic dissection, 14) occult hemorrhage or anemia requiring transfusion, 15) syncope or fall resulting in major traumatic injury (requiring admission or procedural/surgical intervention), 16) PM or ICD implantation, 17) cardiopulmonary resuscitation, 18) syncope recurrence with hospital admission, and 19) cerebrovascular events.

To explore the potential application of AM in this context, we calculated 1) all the unique combinations of the selected variables (or attributes); 2) the number of combinations verified in at least one patient in the database; 3) the number of combinations with a match size ≥50, ≥ 30, ≥20 and <10 patients.

The potential predictors of short-term severe outcomes were first individually evaluated and then analyzed by multivariate logistic regression analysis with a stepwise selection strategy. In case of one predictor was missing in one patient, it was considered as absent.

The overall diagnostic performance of both multivariate logistic regression and AM was assessed with Receiver Operating Characteristics (ROC) curves and their area under the curve (AUC).To exemplify how the AM would work in the real world, we considered 10 random patients who presented with syncope, as defined according to the main international guidelines and consensus papers [11,12], to the ED of Fondazione IRCCS Ca’ Granda, Ospedale Maggiore Policlinico, Milano from September 2015 to February 2017 [23]. For each patient we recorded the presence or absence of any of the above attributes and calculated the risk of adverse events according to the AM approach. For this purpose we paired the patient of interest to the patients with an identical combination of attributes in the database and calculated the probability of adverse events as the percentage of the matched previously studied patients who had the outcome of interest [13]. A 95% confidence interval (CI) was constructed using the binomial distribution. As part of a larger study on syncope ED risk stratification, we asked the ED physician to assess the patient’s risk of short-term adverse events (low, intermediate or high) according to his/her clinical judgement.

The data for this study were collected and analyzed anonymously. The 10 random example patients had given written informed to have their data collected and the Internal Review Board of L. Sacco Hospital (approval number 608/2015) had approved their use for this study purpose. IRB approval was obtained by the single primary study authors.

Analyses were performed using the SAS (release 9.4) statistical software.

Results

The main characteristics of the 3388 patients included in the individual-patient database are reported in Table 1. We identified 8 common predictors: sex, age (considered as a 3-level categorical variable: < 45 year, ≥ 45 and < 65 years, ≥ 65 years), trauma following syncope, presence of abnormal ECG, history of cerebrovascular disease, history of cardiac disease, history of syncope and absence of prodrome.

Table 1

Characteristics of the included patients.

Variables	EGSYS [18,24]	SFSR [19]	STePS [16]	ROSE [17]	Sun 2007 [20]	Total
Total number of patients	465	684	695	1067	477	3388
Age, median (IQR)	70(45–81)	70(42–81)	64(41–78)	69(48–81)	58(35–79)	67(43–80)
N of admitted patients (%)	178 (38)	364 (53)	265 (38)	538 (50)	286 (60)	1631 (48)
N of men (%)	253 (54)	281 (41)	306 (44)	480 (45)	210 (44)	1530 (45)
N of patients with history of syncope (%)	195 (42)	124 (18)	389 (56)	176 (16)	160/457 (34)	1044/2931 (36)
N of patients without prodrome (%)	122 (26)	260 (38)	195 (28)	410 (38)	141 (30)	1128 (33)
N of patients with trauma following syncope (%)	133 (29)	45 (7)	162 (23)	316 (30)	n.a.	656/2911 (23)
N of patients with abnormal ECG (%)	178 (38)	222 (32)	202 (29)	665 (62)	170 (36)	1437 (42)
N of patients with a history of cardiovascular disease (%)	153 (33)	139 (20)	178 (26)	284 (27)	150 (31)	904 (27)
N of patients with a history of cerebrovascular disease (%)	166 (36)	115 (17)	227 (33)	n.a.	169 (35)	677/2321 (29)
N of patients with serious outcomes at 10 days (%)*	93 (20)	81 (12)	44 (6)	49 (5)	62 (13)	329 (10)
N of deaths	6	6	7	6	1	26 (1)
N of arrhythmias	31	30		20	32
N of cardiopulmonary resuscitations			5	2
N of myocardial infarctions	6	33			1
N of structural cardiopulmonary diseases	9	10		14	6
N of PM insertions or malfunctions	43		25	11	2
N of ICD insertions or malfunctions	5		2
N of haemorrhages		24		7	8

IQR: interquartile range; ECG: electrocardiogram; PM: pacemaker; ICD: Implantable Cardioverter Defibrillator; n.a.: not available.

*Some patients had more than one outcome.

The AM method provided 253 of the 384 possible combinations. No patient in the database matched the remaining 131 combinations of predictors. Only 12 of the 253 (4.7%) combinations had a match size ≥50 patients, 35 (13.8%) had a match size ≥30 patients, 50 (19.8%) had a match size ≥20 patients, and most (160, 63.2%) had a match size <10 patients.

At univariate analysis, the risk factors significantly associated with severe short-term outcomes were age, male gender, syncope during exertion, abnormal ECG, history of cardiovascular disease, history of cerebrovascular disease, absence of prodrome, and history of arterial hypertension (Table 2).

Table 2

Risk factors for severe short-term outcomes within 10 days (univariate analysis).

	Severe Outcomes
	Yes (%) (n = 329)	No (%) (n = 3059)	p-value*
Male gender, n (%)	196 (60)	1334 (44)	<0.0001
Age, n (%)			<0.0001
< 45 years	24 (7)	869 (28)
≥ 45 and < 65 years	56 (17)	658 (22)
≥ 65 years	249 (76)	1532 (50)
Syncope during exertion, n (%)	31 (9)	187 (6)	0.0211
Trauma following syncope, n (%)	64 (19)	592 (19)	0.9651
Abnormal ECG, n (%)	229 (70)	1208 (39)	<0.0001
Medical history, n (%)
Cardiovascular disease	161 (49)	743 (24)	<0.0001
Cerebrovascular disease	132 (40)	545 (18)	<0.0001
Arterial hypertension	154 (47)	1104 (36)	0.0001
Previous syncope	109 (33)	964 (31)	0.5491
Absence of prodrome, n (%)	126 (38)	1002 (33)	0.0430

*Chi-square test; ECG: electrocardiogram

At multivariate analysis, male gender, age between 45 and 65 years, age over 65 years, an abnormal ECG, and a past medical history of cerebrovascular disease were independent risk factors for the development of severe adverse outcomes in the short term (Table 3).

Table 3

Risk factors for severe short-term outcomes within 10 days at logistic multivariate regression (stepwise selection).

	Adjusted Odds Ratio	95% Confidence Interval	p-value*
Male gender	1.6	1.3–2.0	0.0001
Age			<0.0001
< 45 years	1.0
≥ 45 and < 65 years	2.3	1.4–3.8
≥ 65 years	3.5	2.3–5.5
Abnormal ECG	2.6	2.0–3.3	<0.0001
Medical history of cerebrovascular disease	1.9	1.5–2.5	<0.0001

*Chi-square test

ECG: electrocardiogram

The AUC for the AM and the multivariate model were 0.59 and 0.74, respectively.

The predicted probabilities for each of the 10 patients, together with the ED physician’s perceived risk are reported in Table 4. To note, none of these patients had an adverse event at 7–30 days of follow-up according to standardized criteria [22]. The detailed case description of the 10 patients is reported in S1 Table.

Table 4

Predicted probabilities according to attribute matching and clinical judgement in the 10 example patients.

Case n	Attribute matching		ED physician
	patients at risk*	10-day SAE, % (95% CI)
1	15	20 (7–45)	High risk
2	70	4 (1–12)	Intermediate risk
3	42	5 (1–16)	Intermediate risk
4	12	0 (0–24)	Intermediate risk
5	84	4 (1–10)	Intermediate risk
6	34	6 (2–19)	Low risk
7	42	5 (1–16)	High risk
8	6	16 (3–56)	High risk
9	6	0 (0–39)	High risk
10	3	33 (6–79)	High risk

ED: Emergency Department; SAE: serious adverse events

*: number of patients with the same combination of risk factors

CI: Confidence Interval.

Discussion

In this paper, to assess the potential value of AM and to compare it to multivariate logistic regression we used syncope as a paradigm of those conditions, such as chest pain and shortness of breath, in which the creation of accurate CDTs is particularly challenging. If the condition under consideration is the manifestation of many possible underlying diseases, CDTs are unlikely to accurately identify all the different adverse events related to the possible etiologies [25]. In syncope, CDTs are usually designed to identify multiple diagnoses (i.e. pulmonary embolism, aortic dissection, high grade atrioventricular block) and adverse events that might be related to a high number of conditions (i.e. bleeding requiring transfusion, trauma, pacemaker implant). To increase complexity, the reference standard for diagnosis is sometimes missing.

This study explores a method to estimate the probability of serious adverse events based on AM. This approach allows the clinician to determine the probability of a serious outcome of a patient based on the presence of predefined risk predictors (or attributes). This patient is matched to all patients with the same combination of attributes included in a large reference database. The proportion of these attribute-matched patients who had the outcome of interest represents the estimate, with its 95% confidence interval, of the probability that such outcome might occur in the patient under consideration [15]. This process resembles the definition of pre-test probability by an expert clinician, which, having seen many patients who had similar clinical characteristics as the patient under consideration, could provide an estimate of the probability of something bad happening. In this case the computer does so with less variability and without the clinician having to be experienced nor an expert.

The inclusion of a large number of attributes would result in very specific and detailed clinical risk profiles at a cost of requiring a very large reference database. In the present work, we used an eight-attribute profile and a 3388-patient database. Among the 384 possible combinations, only 12 had a match size ≥50 patients and most had a match size <10 patients. Therefore, our data do not offer a clinically useful prediction tool at this stage and the AUC shows that logistic regression is superior if derived from the dataset we used, but this method seems promising, as it has some advantages as compared to model-derived clinical decision tools. Indeed, the successful use of a model to predict the probability of a serious outcome requires that the results are reproduced in an external validation so that both the external validity and robustness of the model are verified. Moreover, models require that the predictors are assigned a weight that allow to estimate the risk of adverse events in every patient, also in those that had no matching subject in the derivation database (for example for patients that have a rare condition). Attribute matching differs from scoring systems derived from logistic regression, which use predictor variables expressed by an individual patient under consideration to guide that patient into a predefined category that predicts a probability. This outcome probability is estimated from knowledge (i.e., the magnitude of importance of predictor variables) manifested by the patients that were used to construct the model. On the other hand, attribute matching works in reverse fashion. Instead of placing the patient under consideration into a category, the computer program finds the patients from a reference database who ‘‘look like” the patient insofar as they are identical on the binary predictor variables. Therefore, the risk of patients with an uncommon combination of predictors, might not be able at all to find a match in the derivation dataset. However, being aware that the patient’s estimated probability might be based on very limited evidence, will allow both the clinician and the patient to take a decision conscious that it might be based on uncertainty, rather than deciding on the false confidence provided by models.

Several thousands of subjects need to be enrolled for acceptable AM risk prediction. If this was the case, only administrative databases could be used to use AM for risk prediction. In the era of big data and with the increase in the availability and accuracy of population-based databases, this might not be a barrier to the use of AM for risk prediction in several conditions [26].

AM has several advantages: 1) The possibility to have as output not only the probability of a composite serious outcome, but a detailed patient specific risk profile based on the probability of different outcomes allowing for a more personalized decision making. Also, the possibility to make the risk profile explicit and more personalized could allow for more meaningful shared decision making with the patient; 2) as there is no need for model fitting, patients could be always added to the dataset thus increasing the probability estimate precision; 3) the flexibility of AM would allow to consider different predictors in different patients, thus allowing an individualized estimate; 4) as there is no statistical modelling, the reliability of the results is based on the similarity between the population of the reference database and every-day patients rather than on complex statistical calculations; 5) the prediction tools based on models, such as logistic regression and neural networks provide a risk estimate in every case, also in patients whose combination of clinical characteristics are different from each patient’s combination in the derivation cohort, giving the physician a false confidence. Conversely, AM would allow both the clinician and the patient to make a decision being aware that it might be based on uncertainty, rather than deciding on the false confidence provided by models. This is crucial in the perspective of a modern medicine increasingly based on personalized and shared decision making.

AM has also some important limitations: 1) to be used in clinical practice the reference database should include a large number of patients; 2) the choice of predictors is crucial for the successful application of the method; 3) AM will promote personalized medicine, providing the probability of events, rather than a clear indication of what to do (i.e. admit vs discharge). However, the need to interpret and apply the estimated probability to the context may be felt as a limitation due to lack of certainty; 4) a score is easy to remember and apply, while AM requires data collection and computer input ideally through a computer/smartphone app. Furthermore, the value of CDT as early and necessary work to determine the choice of predictors to be considered should not be under estimated as they help determine what attributes and factors should be collected and used for AM.

Some limitations of the present study should be acknowledged. The database we used was collected for different purposes and, although we did our best to homogenize the data, we could not overcome some heterogeneity among the single studies’ dataset. Also, we used as predictors the eight variables in common between the original datasets with no a priori decision on the number of predictors to be selected. However, this number strongly influences the sample size of the population to be included in the AM database. Nonetheless, it must be pointed out that syncope and this database were used only as a working example to show the possible applications of AM.

Conclusions

In conclusion, our study shows that the AM is a promising method to predict the risk of adverse events in clinical practice and could offer some advantages as compared with standard methods based on logistic regression. However, large datasets are required to obtain a precise and informative estimate. Future studies should explore the use of administrative databased or big data in conditions in which there is less clinical heterogeneity to use AM and to compare it with the traditional risk stratification tools.

Example clinical cases with the probabilities predicted by attribute matching and clinical judgement.

BP: blood pressure; HR: heart rate; ECG: electrocardiogram; ED: Emergency Department; CI: Confidence Interval.

(DOCX)

Click here for additional data file.

20 Sep 2019

PONE-D-19-19894

Personalized risk stratification through attribute matching using syncope as a clinical example

PLOS ONE

Dear Dr Casazza

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript in 60 days. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Sandro Pasquali, M.D., Ph.D.

Academic Editor

PLOS ONE

This is a good study with some interestign findings. However it is poorly decribed and reported. The Authors should make any effort to increase the quality of their manuscript. Here are some issues to be addressed.

The study is aimed at investigating a new methodology rather than a clinical condition. The title should be reflect the study aim and reworded. For instance, a possibility is “Personalized risk stratification through attribute matching for clinical decision making in clinical conditions with aspecific symptoms: the example of syncope”.

The clinical challenge of predicting patient risk in an aspecific condition is well presented in the letter to the Editor. However this remains quite unclear in the abstract. It should be made clear in the abstract that the Authors are investigating a promising methodology for quite generic symptoms and that they picked up syncope as an example.

Introduction. The sentence “While CDTs provide information that would be applicable to the nonspecific patient, they lack useful and precise prediction in subjects with specific clinical presentations or needs.” Is probably inaccurate. Prediction tools can be accurate also on a specicific patient, they just need to be informative in terms of variables. For instance CTDs are sometimes very accurate in cancer medicine and they are replacing AJCC TNM staging system as highlighted in the 8th edition of the TMN staging manual.

Introduction. Introduction is also likely missing the point. The Authors stated that “Since the well-known limitations of the traditionally derived risk stratification tools in predicting adverse events after syncope”. Are the Authors looking at predicting the cause of a condition that underlie a symptom, that is here syncope?

Introduction. It should be stated which is the difference between standard predicting tools and attribute matching. The description about AM is in the method but it seems more appropriate to move it to the Intro to facilitate the reading.

Introduction. Study hypothesis and aims should be clearly stated. It should be stated that the Authors compared AM prediction and pre-probability defined by an expert clinican. This is quite interesting as the Authors probably expected that a large dataset works as accurately as an expert clinicians though in a more reporducible way when compared to a clinician.

Introduction. AM should be spelled out.

Methods. Time frame of the study is needed.

Methods. Statistical methodology for AM should be biefly reported and referenced.

Methods. Outcome variables and measures not reported.

Results. Which are the 8 selected variables?

Results. ED physician. This should be reported in the method section and criteria for each category described.

Methods/results. The authors mentioned that AM works better than traditional prediction, which may well be the case. However, in the manuscript such comparison has not been made. Since the Authors have a pretty large dataset they should be able to run this comparison. For instance, they can fit a predictive model based on regression for predicting SAE and test in on their 10 prospective patients. Then they should compare AM and traditional prediction based on regression.

Method/results. Are 10 patients enough for this study? Was a sample size calculation performed? If not, which is the reason?

Discussion. This is a well balanced dìscussion.

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for including your ethics statement:  "The data for this study were collected and analyzed anonymously.

The 10 patients in Table 2 had given written informed to have their data collected and the Internal Review Board had approved their use for this study purpose.

IRB approval was obtained by the single primary study authors.".

a.Please amend your current ethics statement to include the full name of the ethics committee/institutional review board(s) that approved your specific study.

b.Once you have amended this/these statement(s) in the Methods section of the manuscript, please add the same text to the “Ethics Statement” field of the submission form (via “Edit Submission”).

For additional information about PLOS ONE ethical requirements for human subjects research, please refer to http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research.

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

4. One of the noted authors is a group or consortium SYNERGI (SYNcope Expert Research Group International). In addition to naming the author group, please list the individual authors and affiliations within this group in the acknowledgments section of your manuscript. Please also indicate clearly a lead author for this group along with a contact email address.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

3 Nov 2019

Dear Dr. Pasquali,

Thank you for having considered our manuscript entitled “Personalized risk stratification through attribute matching using syncope as a clinical example” for publication in PLOS ONE (PONE-D-19-19894). We would also like to thank you for the thoughtful comments: we have revised the manuscript accordingly and we believe that this process has improved its quality.

Please find attached a marked-up copy of the manuscript that highlights changes made to the original version and an unmarked version of the revised paper without tracked changes, together with point-by-point responses to the points raised during the review process.

Sincerely yours,

Giovanni Casazza, on behalf of all Co-Authors

Here are our responses to the specific points raised during the review process.

This is a good study with some interesting findings. However it is poorly decribed and reported. The Authors should make any effort to increase the quality of their manuscript. Here are some issues to be addressed.

The study is aimed at investigating a new methodology rather than a clinical condition. The title should be reflect the study aim and reworded. For instance, a possibility is “Personalized risk stratification through attribute matching for clinical decision making in clinical conditions with aspecific symptoms: the example of syncope”.

We appreciate the editor’s suggestion. We changed the title as suggested.

The clinical challenge of predicting patient risk in an aspecific condition is well presented in the letter to the Editor. However this remains quite unclear in the abstract. It should be made clear in the abstract that the Authors are investigating a promising methodology for quite generic symptoms and that they picked up syncope as an example.

We appreciate the opportunity to clarify. We changed the abstract accordingly.

Introduction. The sentence “While CDTs provide information that would be applicable to the nonspecific patient, they lack useful and precise prediction in subjects with specific clinical presentations or needs.” Is probably inaccurate. Prediction tools can be accurate also on a specific patient, they just need to be informative in terms of variables. For instance CTDs are sometimes very accurate in cancer medicine and they are replacing AJCC TNM staging system as highlighted in the 8th edition of the TMN staging manual.

We agree with the editor that this statement was inaccurate. We tried to explain better our point of view. The sentence now is the following: “As they are based on models, CDTs are able to predict the risk of any hypothetical patient, even those with a combination of risk factors different from all the patients of the derivation cohort. Therefore, we do not know how the CDT will perform in subjects with specific clinical presentations or needs. Indeed, they lack the ability to provide personalized estimates as required in the era of precision medicine. For example, patients with uncommon diseases are likely not to be correctly risk stratified by CDTs. In addition, the risk estimates of composite outcomes that are usually provided by CDTs cannot always be applied to all patients, as the definition of “acceptable risk” depends on the patient at risk. Hence the need to assess a personalized risk rather than providing a simple binary answer [6].”

Introduction. Introduction is also likely missing the point. The Authors stated that “Since the well-known limitations of the traditionally derived risk stratification tools in predicting adverse events after syncope”. Are the Authors looking at predicting the cause of a condition that underlie a symptom, that is here syncope?

We appreciate the opportunity to clarify. We changed the text as follows: Moreover, risk stratification is challenging in conditions (as chest pain, shortness of breath and syncope) presenting with aspecific symptoms that can be the manifestation of many possible underlying diseases. In these cases, decision tools are unlikely to accurately identify all the different adverse events related to the possible etiologies. In syncope, which is a paradigm of the above conditions, the traditionally derived risk stratification tools have failed in predicting adverse events [7-12]. Here, an individualized risk assessment would allow an estimate of not only the probability of a composite endpoint, but rather a detailed risk profile that provides the individual risk of each specific outcome (e.g. arrhythmia or pulmonary embolism).”

Introduction. It should be stated which is the difference between standard predicting tools and attribute matching. The description about AM is in the method but it seems more appropriate to move it to the Intro to facilitate the reading.

We appreciate the editor’s suggestion. We moved the description about the AM method and the difference between standard predicting tools and AM in the introduction.

Introduction. Study hypothesis and aims should be clearly stated. It should be stated that the Authors compared AM prediction and pre-probability defined by an expert clinican. This is quite interesting as the Authors probably expected that a large dataset works as accurately as an expert clinicians though in a more reporducible way when compared to a clinician.

We appreciate the opportunity to clarify. In the present study, we did not perform a formal comparison between AM and clinical judgement. Indeed, the purpose of the study was to describe how the AM method could work in conditions like syncope. A comparison between AM and clinical judgement would require a much larger reference dataset. We just reported some examples of how it could work in real practice. We tried to explain this in the last paragraph of the methods section.

Introduction. AM should be spelled out.

We appreciate the editor’s suggestion. We added spelled out AM as suggested.

Methods. Time frame of the study is needed.

We appreciate the editor’s suggestion. We added the study time-frame as required.

Methods. Statistical methodology for AM should be biefly reported and referenced.

We appreciate the opportunity to clarify. We now added some details on the AM methodology and a reference in the methods section.

Methods. Outcome variables and measures not reported.

We appreciate the opportunity to clarify. We added the outcomes of interest in the methods section.

Results. Which are the 8 selected variables?

We appreciate the editor’s suggestion. We moved the list of the 8 selected variables from the methods to the results section.

Results. ED physician. This should be reported in the method section and criteria for each category described.

We appreciate the editor’s suggestion. We reported the ED physician role in the methods section. As described, there were no criteria to assess risk, as this was left to the ED physician’s judgement.

Methods/results. The authors mentioned that AM works better than traditional prediction, which may well be the case. However, in the manuscript such comparison has not been made. Since the Authors have a pretty large dataset they should be able to run this comparison. For instance, they can fit a predictive model based on regression for predicting SAE and test in on their 10 prospective patients. Then they should compare AM and traditional prediction based on regression.

We appreciate the opportunity to clarify. The purpose of this study was not to make a comparison between AM and the traditional prediction methods. Indeed, we cannot state that AM works better, because a formal comparison would require thousands of patients in the reference database for AM to work. As stated in the study aim, we only wanted to test how AM could work with a real life example and to show that it could allow a different approach.

Method/results. Are 10 patients enough for this study? Was a sample size calculation performed? If not, which is the reason?

We would like to thank the editor for this comment. As the 10 patients were only an example and we did no attempt to “validate” the AM method on them, no sample size calculation was performed. If the editor feels that this in confusing rather than useful, we could remove the 10 example patients from the manuscript.

Discussion. This is a well balanced discussion.

We appreciate the editor’s comment

Submitted filename: responses to reviewers - New.docx

Click here for additional data file.

21 Nov 2019

PONE-D-19-19894R1

Personalized risk stratification through attribute matching for clinical decision making in clinical conditions with aspecific symptoms: the example of syncope

PLOS ONE

Dear Prof Casazza,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Jan 05 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Sandro Pasquali, M.D., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (if provided):

Authors have replied to comments raised in the previous review. Although they agreed in principle with the comments, only minor changes have been made throughout the text. More substantial changes are needed to improve the manuscript. The main idea behind this manuscript should be that the presented methodlogy is interesting and promising and a pilot has been conducted to show this. A larger study is clearly needed to validate the method, either looking at syncope or other conditions. In other words, findings are hypothesis generating rather than conclusive. In this regards, authors should make very clear what their next step will be.

The manuscript has been sent for additional review and comments of Reviewer#2 which are now available need to be carefully addressed in order to meet requirement for publication in PLOS ONE.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors quote the following sentence in the conclusions: "our study shows that the AM method could be used to predict the risk of adverse events in clinical practice". However, the paper makes no systematic comparison with the state of the art to support this view. Without this comparison the paper turns out to be nothing more than an exercise in style.

Several times the authors have been asked for this by the reviewers, with an answer "we did not perform a formal comparison

between AM and clinical judgement. Indeed, the purpose of the study was to describe how the AM method could work in conditions like syncope" and "The purpose of this study was not to make a comparison between AM and the traditional prediction methods. Indeed, we cannot state that AM works better, because a formal comparison would require thousands of patients in the reference database for AM to work. As stated in the study aim, we only wanted to test how AM could work with a real life example and to show that it could allow a different approach". These answers, from my personal point of view, indicate a lack of attention in the normal scientific approach in which, if a method is to be shown, all the specifications for comparison with the other methods must also be provided.

Please, provide:

- variable selection criterion and cut-points justification

- prediction accuracy measure, in order to enable the reader to judge the method.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

5 Jan 2020

Sandro Pasquali, M.D., Ph.D.

Academic Editor - PLOS ONE

Dear Dr. Pasquali,

Thank you for having considered our revised manuscript entitled “Personalized risk stratification through attribute matching for clinical decision making in clinical conditions with aspecific symptoms: the example of syncope” for publication in PLOS ONE (PONE-D-19-19894R1). We would also like to thank you and the reviewer for the thoughtful comments: we have revised the manuscript accordingly and we believe that this process has improved its quality.

Please find attached a marked-up copy of the manuscript that highlights changes made to the original version and an unmarked version of the revised paper without tracked changes, together with point-by-point responses to the points raised during the review process.

Giovanni Casazza, on behalf of all Co-Authors

Here are our responses to the specific points raised during the review process.

Additional Editor Comments (if provided):

Authors have replied to comments raised in the previous review. Although they agreed in principle with the comments, only minor changes have been made throughout the text. More substantial changes are needed to improve the manuscript. The main idea behind this manuscript should be that the presented methodlogy is interesting and promising and a pilot has been conducted to show this. A larger study is clearly needed to validate the method, either looking at syncope or other conditions. In other words, findings are hypothesis generating rather than conclusive. In this regards, authors should make very clear what their next step will be.

We appreciate the editor’s suggestion. We tried to further clarify that this is a pilot study and that studies on much larger datasets and comparing this method to the traditionally used methods for risk prediction are needed. We added what the next step will be, namely the application of AM to larger datasets through the use of administrative data in conditions in which there is less clinical heterogeneity and to compare AM with the traditional risk stratification tools.

The manuscript has been sent for additional review and comments of Reviewer#2 which are now available need to be carefully addressed in order to meet requirement for publication in PLOS ONE.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

________________________________________

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

________________________________________

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

We appreciate the reviewer’s comment. We have now added logistic regression and updated the methods paragraph.

________________________________________

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

We have indicated that data from this study are available upon request. The database analyzed during the current study involves individual-level patients’ data from 5 studies (Costantino G – JACC 2008; Reed MJ – JACC 2010; Del Rosso A – Heart 2008; Quinn JV – Ann Emerg Med 2004; Sun BC – Ann Emerg Med 2007). Public access to the 5 databases is closed and permission to use data for this study had been obtained from each participating center in occasion of the database creation (Costantino G, Am J Med 2014;127:1126.e13-25). Therefore, the authors of the current paper do not have the permission to share de-identified data, which should be asked to the IRB of each corresponding author’s institution.

________________________________________

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

________________________________________

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors quote the following sentence in the conclusions: "our study shows that the AM method could be used to predict the risk of adverse events in clinical practice". However, the paper makes no systematic comparison with the state of the art to support this view. Without this comparison the paper turns out to be nothing more than an exercise in style.

Several times the authors have been asked for this by the reviewers, with an answer "we did not perform a formal comparison between AM and clinical judgement. Indeed, the purpose of the study was to describe how the AM method could work in conditions like syncope" and "The purpose of this study was not to make a comparison between AM and the traditional prediction methods. Indeed, we cannot state that AM works better, because a formal comparison would require thousands of patients in the reference database for AM to work. As stated in the study aim, we only wanted to test how AM could work with a real life example and to show that it could allow a different approach". These answers, from my personal point of view, indicate a lack of attention in the normal scientific approach in which, if a method is to be shown, all the specifications for comparison with the other methods must also be provided.

We appreciate the opportunity to clarify. As it is the first study on the use of AM in the context of decision making in clinical conditions with aspecific symptoms, this study is not intended to show that AM works better or worse than logistic regression or clinical judgement, but only to assess its potential applications. Indeed, the intrinsic characteristics of AM and regression make it unlikely that a method based on data such as AM could work better than a method based on a model derived from a small dataset. Therefore, we felt that a systematic comparison with the state of the art would not be appropriate at this stage. However, as the editor and the reviewers deemed it important, we added a comparison with a model based on logistic regression and we are open to any suggestions to improve such comparison.

Please, provide:

- variable selection criterion and cut-points justification

We appreciate the opportunity to clarify. The database we used was collected for different purposes and we used as predictors the eight variables in common between the original datasets with no a priori decision on the number of predictors to be selected. We acknowledged this in the limitations of the study.

- prediction accuracy measure, in order to enable the reader to judge the method.

We appreciate the reviewer’s suggestion. We added an assessment of the overall diagnostic performance of both multivariate logistic regression and AM with ROC curves and their AUC.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

23 Jan 2020

Personalized risk stratification through attribute matching for clinical decision making in clinical conditions with aspecific symptoms: the example of syncope

PONE-D-19-19894R2

Dear Dr. Casazza,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Sandro Pasquali, M.D., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Also this time the reviewers comments are well adressed. The manuscript now offers a more balanced view compared to previous versions and is more formally robust also from a methodological viewpoint (i.e. addiction of a logistic regression analysis and changes in discussion/conclusions).

21 Feb 2020

PONE-D-19-19894R2

Personalized risk stratification through attribute matching for clinical decision making in clinical conditions with aspecific symptoms: the example of syncope

Dear Dr. Casazza:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Sandro Pasquali

Academic Editor

PLOS ONE