Using natural language processing to identify acute care patients who lack advance directives, decisional capacity, and surrogate decision makers.

The prevalence of patients who are Incapacitated with No Evident Advance Directives or Surrogates (INEADS) remains unknown because such data are not routinely captured in structured electronic health records. This study sought to develop and validate a natural language processing (NLP) algorithm to identify information related to being INEADS from clinical notes. We used a publicly available dataset of critical care patients from 2001 through 2012 at a United States academic medical center, which contained 418,393 relevant clinical notes for 23,904 adult admissions. We developed 17 subcategories indicating reduced or elevated potential for being INEADS, and created a vocabulary of terms and expressions within each. We used an NLP application to create a language model and expand these vocabularies. The NLP algorithm was validated against gold standard manual review of 300 notes and showed good performance overall (F-score = 0.83). More than 80% of admissions had notes containing information in at least one subcategory. Thirty percent (n = 7,134) contained at least one of five social subcategories indicating elevated potential for being INEADS, and <1% (n = 81) contained at least four, which we classified as high likelihood of being INEADS. Among these, n = 8 admissions had no subcategory indicating reduced likelihood of being INEADS, and appeared to meet the definition of INEADS following manual review. Among the remaining n = 73 who had at least one subcategory indicating reduced likelihood of being INEADS, manual review of a 10% sample showed that most did not appear to be INEADS. Compared with the full cohort, the high likelihood group was significantly more likely to die during hospitalization and within four years, to have Medicaid, to have an emergency admission, and to be male. This investigation demonstrates potential for NLP to identify INEADS patients, and may inform interventions to enhance advance care planning for patients who lack social support.

Introduction

Honoring patient preferences is a tenet of medical ethics and a measure of healthcare quality [1]. Still, the majority of Americans do not have advance directives or healthcare proxies to guide their care when they lack the capacity to make decisions for themselves [2, 3]. Nearly 65% of adults in the United States have no advance directives documenting their wishes or appointing surrogate decision makers [2]. At the same time, 33–65% of hospitalized adults lack decisional capacity [4-6]. This leaves patients vulnerable to receiving care that is unaligned with their values, goals, and preferences, and places care teams in the difficult position of having to search for patient advocates and navigate an ethically fraught plan of care [7-9]. In the absence of advance directives and appointed healthcare proxies, care teams often identify family or friends who can provide information about patients’ wishes and values [10]. However, sometimes patients become decisionally incapacitated without any advance directives or personal contacts known to the healthcare team [11]. We refer to such patients as being Incapacitated with No Evident Advance Directives or Surrogates (INEADS).

The number of patients meeting these criteria is expected to rise in the coming decades due to an aging population, a growing number of patients with dementia, and a rise in the number of seniors who live alone [12, 13]. However, estimating the number of patients who are INEADS has been challenging due to a near total lack of data describing this phenomenon. A single study conducted in 2003–2004 found that 16% of ICU patients were INEADS, though these findings reflect the experience of a single unit in an urban public hospital over a short period before Medical/Physician/Provider Orders for Life Sustaining Treatment (MOLST/POLST) programs were widely adopted in the United States [14, 15]. The scarcity of information around this topic may be due in part to the fact that these characteristics are not routinely nor accurately captured in structured healthcare data, meaning that efforts to collect such information have historically required resource intensive methods.

The availability of modern data science methods like natural language processing (NLP) provide a novel opportunity to understand the characteristics and prevalence of INEADS patients using the wealth of information documented in electronic clinical narrative notes. NLP is a method of identifying and extracting information from bodies of text, thereby automating the process of reviewing documents. With the growing volume of electronic health record documentation, NLP is increasingly used for identifying information from free text clinical notes for research and practice applications in healthcare. While no previous studies have reported the use of NLP to identify patients who are INEADS, this method has been used to identify a variety of social factors related to health [16].

Healthcare providers spend up to 50% of their time generating and reviewing documentation [17]. Clinical notes bring together subjective and objective findings, observations, diagnoses, and treatment plans, and also serve as a way of communicating detailed information about patients to other clinicians [18]. Therefore, clinical notes may provide additional information, insight, and details related to advance directives, surrogate decision makers, and decisional capacity beyond what is included in structured data or legal documents, which may not exist or be available immediately to the clinical team at the time of care [19]. The aim of this study was to build and validate an NLP algorithm to identify information about patients in the acute care setting that could be used to assess whether they may be INEADS.

Materials and methods

Study dataset and population

The Icahn School of Medicine at Mount Sinai Institutional Review Board (IRB) approved this single IRB study and issued a waiver of informed consent. The study was conducted using the publicly available Medical Information Mart for Intensive Care III (MIMIC-III) database, which included de-identified data for 53,423 distinct hospital admissions among 46,520 unique patients who were admitted to an intensive care unit at Beth Israel Deaconess Medical Center, an academically affiliated tertiary/quaternary care hospital with a Level 1 Trauma Center in Boston, MA, between 2001 and 2012 [20]. After excluding patients <18 years, 23,904 distinct hospital admissions among 17,886 patients were included in our analyses.

The database contains demographic information, vital signs, laboratory results, procedures, medications, discharge disposition, and electronic clinical notes. We included electronic clinical notes in the following categories: case management, consultation, discharge summary, nursing, nutrition, physician, rehabilitation, respiratory, social work, and general, which encompassed admission, death, procedure, and progress notes. We excluded notes in categories that were unlikely to contain information related to INEADS status, such as radiology, pharmacy, and echocardiogram. Of the 418,393 clinical notes included, 70% were nursing, 23% were physician, and remaining note types collectively comprised 7% of the total notes.

NLP algorithm pipeline

depicts the workflow for developing and evaluating the algorithm in six steps.

(1) Identifying concepts related to INEADS status

Our interdisciplinary team of medical, nursing, social work, bioethics, public health, and informatics researchers identified concepts related to the three domains of INEADS: decisional capacity, advance directives, and surrogate decision makers. We then categorized these concepts based to create subcategories within each domain (). The 17 subcategories were annotated based on whether they indicated the reduced or elevated potential for being INEADS. At each stage of development (identifying concepts, classifying concepts into subcategories, and determining potential for INEADS), decisions were reached following a comprehensive literature review and group discussion, and culminated in full team consensus.

(2) Creating a preliminary list of terms and expressions for each subcategory

To develop the preliminary lexicon for each subcategory, we used a combination of sources including standard clinical terminologies such as the Systemized Nomenclature of Medical Terms (SNOMED) [21], International Classification of Diseases version 10 (ICD-10) [22], and International Classification for Nursing Practice (ICNP) [23]; review of relevant literature; and our team’s expertise in nursing, medicine, social work, bioethics, public health, and informatics. Four team members (JS, BC, AL, MT) developed the initial list of terms, which was iteratively revised and finalized by all team members.

(3) Creating language model: Word embedding model (Word2Vec)

Language models, which are vectorized (numeric) representations of texts in specific domains, allow multiple NLP tasks, such as synonym detection and lexicon creation. We used NimbleMiner [24], an open-source publicly available NLP application in RStudio (Foundation of Statistical Computing, Vienna), to create a language model called Word2Vec algorithm [25] and identify synonyms for the list of preliminary terms in each subcategory within a large body of MIMIC-III clinical notes. This software was selected for its ability to quickly and efficiently identify synonyms from large bodies of text. The system can be downloaded from http://github.com/mtopaz/NimbleMiner under General Public License v3.0.

Briefly, the process of building vocabularies within NimbleMiner is to 1) import a large body of relevant text (clinical notes from MIMIC-III) and a preliminary list of terms for a concept of interests (list of terms within each INEADS subcategory), and 2) perform text preprocessing including deleting punctuations or stop words such as “the” or “is.”

(4) Implementing an interactive vocabulary explorer to identify synonymous expressions

Using NimbleMiner, we implemented an interactive rapid vocabulary explorer in which the software suggested similar words and expressions based on the language model built in the previous step. For example, when the system was queried for synonyms of the term “palliative care,” the system output included terms such as “palliative approach” and “pursue palliative care.” Two reviewers (JS and BC) decided whether to accept or reject the words by selecting them in the interactive vocabulary explorer user interface. This process was repeated until no new relevant synonyms were identified. lists examples of terms in each category. The complete list is available in S1 Appendix.

(5) Developing an expert generated reference standard for evaluation of NLP algorithm performance

To evaluate NLP algorithm performance, we identified all admissions in which patients had ICD-10 diagnoses that could be associated with being INEADS, including psychiatric disorders, tuberculosis, hepatitis C, and HIV/AIDS. We created a dataset that included all notes from these patients and then randomly sampled a gold standard testing set of 300. Two members of the research team with expertise in nursing, social work, public health, and informatics (BC and AL) manually reviewed each note to identify information that reflected one or more subcategories. Discrepancies between reviewers were resolved through discussion following input from a third reviewer (JS). The observed interrater agreement at the note level for determining whether the note included any documentation indicating elevated potential for being INEADS was good (Cohen’s kappa = 0.7).

(6) Evaluating NLP algorithm performance

We applied the NLP algorithm on the gold standard testing set to calculate precision (akin to positive predictive value, calculated as [true positives] / [true positives plus false negatives]), recall (akin to sensitivity, calculated as [true positives] / [true positives plus false positives]), and F-score (the weighted harmonic mean of the precision and recall).

Application of the NLP algorithm and descriptive data analysis

We applied the NLP algorithm to the full dataset of 418,393 clinical notes to identify notes with mentions of the previously described synonyms lexicon. The software also detected negations such as "no" and "denying" to determine the semantic meaning of a sentence. Next, we calculated the proportion of clinical notes that included words or expressions any subcategory and calculated the frequency of each subcategory at the hospital admission level. To establish a cohort of “possibility of being INEADS” patient admissions, we identified admissions that included at least one of the five social subcategories indicating elevated potential for being INEADS (unmarried, living alone, transitionally situated, surrogate decision maker unidentified, and advance directives unavailable). We also established a “high likelihood of being INEADS” cohort consisting of admissions that included at least four of these five subcategories. We further bifurcated this group based on whether or not the admission included documentation of at least one subcategory indicating reduced likelihood of being INEADS. We validated the “high likelihood of being INEADS” classification by conducting a case study validation on eight admissions in each segment of this cohort, in which two reviewers (BC and AL) reviewed all notes documented during the admission to determine whether the patient appeared to meet the definition of INEADS. Lastly, we compared demographic characteristics of the “possibility of being INEADS” cohort and the “high likelihood of being INEADS” cohort against the full cohort using t-tests for continuous data or chi-square tests for categorical data. All analyses were performed using R software version 4.1.0 (Foundation of Statistical Computing, Vienna).

Results

Cohort demographics

The average patient age was 61 years, 54% were female, and 70% were white. The most common type of insurance was Medicare (59%). During the period captured in the database, 45% of patients died within 4 years after discharge, and 10% died during hospitalization (.

Validation of NLP algorithm on gold standard testing set

The algorithm demonstrated good performance overall in identifying subcategories compared with gold standard manual review (average F-score = 0.83), with best results for the transitionally situated, palliative care, and hospice subcategories (F-score = 1) and poorest results for surrogate decision maker unidentified (F-score = 0.4; ).

NLP detection of subcategories in full cohort

Examples of notes containing each subcategory are provided in . At the note level, 50% of clinical notes (209,697/418,393) contained at least one subcategory. Consistent with the proportion of each type of note in the dataset overall, 68% of notes containing at least one subcategory were nursing notes, 18% were physician notes, and 14% were other note types. However, subcategories indicating elevated potential for being INEADS were most likely to be documented in consult notes (57%) and least likely to be documented in nursing notes (48%).

At the admission level, 81% of admissions (19,380/23,904) had clinical notes that included at least one subcategory. The most frequently documented subcategory was living relatives with unknown involvement (72%) followed by lacking capacity (57%), surrogate decision maker identified (41%), and advance directives available (36%). shows the frequency of admissions with clinical notes that included each subcategory. Seventy-six percent of admissions (18,192/23,904) included at least one subcategory indicating the presence of advance directives or surrogates.

Thirty percent of admissions (7,134/23,904) contained at least one social subcategory indicating elevated potential for being INEADS (unmarried, living alone, transitionally situated, surrogate decision maker unidentified, and advance directives unavailable) and were included in the “possibility of being INEADS” cohort. Less than one percent of admissions (n = 81) met the criteria for the “high likelihood of being INEADS” cohort, with 79 containing four and two containing all five of the subcategories. Among the “high likelihood of being INEADS” cohort, 10% (n = 8) contained no subcategories indicating reduced likelihood of being INEADS. In our case study validation, all admissions in this group appeared to meet the definition of INEADS following manual note review. Among the remaining 90% (n = 73) of the “high likelihood of being INEADS” cohort, in which admissions contained at least one subcategory indicating reduced likelihood of INEADS, our case study validation of eight cases found that n = 2 (25%) appeared to meet the definition of INEADS.

compares admission characteristics across the three cohorts. Admissions in the “possibility of being INEADS” cohort were similar to the full cohort but significantly more likely to die during hospitalization and during the follow-up period. Admissions in the "highest likelihood of INEADS" cohort were significantly more likely to die during hospitalization and during the follow-up period, to have an emergency hospitalization, to have Medicaid (a public insurance for low-income patients), and to be male.

Discussion

In this study, we investigated whether NLP could be applied to identify INEADS characteristics in clinician notes documenting over 23,000 patient admissions to critical care over a 12-year period. To our knowledge, this is the first study to examine the prevalence of INEADS in a large cohort of patients, and the first to do so using electronic health records. Based on the information identified using our NLP algorithms, we found that about half of clinical notes contained at least one subcategory of information that could help determine INEADS status by indicating either reduced or elevated potential for being INEADS.

While the algorithms performed well in most categories (F-score = 0.83), they struggled to accurately categorize living relatives with unknown involvement and surrogate decision makers unidentified, both of which are critically important factors in determining INEADS status. Acknowledging these challenges, our findings indicate that 30% of admissions included at least one characteristic that could indicate risk of being INEADS (“possibility of being INEADS”), while <1% included at least four (“high likelihood of being INEADS”). Patients in both groups were significantly more likely to die during hospitalization and during follow-up, though this may be, in part, an artifact of documentation practices, (i.e., if a patient appears likely to die, the clinical team may document more information regarding advance directives and family or friends who might serve as surrogate decision makers). Male patients were also overrepresented in both groups, which is consistent with previous findings that men are more likely to be socially isolated than women and women have more regular healthcare providers and greater likelihood of having advance directives [26, 27]. Patients in the latter group were significantly more likely to have Medicaid insurance and emergency admissions. This is consistent with previous work showing an inverse relationship between income and advance directive completion, and lower completion rates among patients who are not anticipating critical illness in the near term [2, 28].

Though we found only a small percentage of patients to be INEADS, the absolute number of patients meeting these criteria in the United States is significant [9]. This phenomenon is worthy of further study, especially considering that decision-making processes for INEADS patients are not well characterized, and available data suggest they may be ethically troubling. Two studies by White et al. [14, 15] describe how clinical decisions are made for INEADS patients. In these data, physicians consulted an ethics committee, multidisciplinary review committee, patient advocate, or ombudsman in only 8 of 61 life support withdrawal decisions. Though limited to ICU patients for whom life-support withdrawal was being considered, and collected only from the perspective of the attending physicians caring for them, these studies suggest that physicians may make unilateral treatment decisions for INEADS patients despite recommendations for ethics consultations.

Ideally, when patients lack decisional capacity and advance directives, the care team is able to identify a default surrogate who knows the patient’s values, beliefs and preferences, and can offer a substituted judgment of what the patient would want [29, 30]. However, with more adults living in social isolation [31], the healthcare system may face a growing number of patients who will have no voice in determining which life-prolonging or life-limiting care they will receive. Our findings provide important insight into the scope of social considerations among critically ill patients and the ethical challenges that may arise when patients lack decisional capacity, surrogate decision makers, and advance directives. Further studies should work to clarify subcategories and their associations with care needs, and potentially provide early warning signals for intervention, such as social work involvement or ethics consultation, which could have some benefit for patients who have decisional capacity when they enter the hospital and later become incapacitated. The findings of this study also highlight the need for further research to identify patients at highest risk of becoming INEADS and design targeted interventions to discuss goals of care and potential surrogate decision makers in other care settings, such as home care, community-based organizations, or shelters. The goal of all such efforts should be to enhance patients’ autonomy and quality of life by respecting their wishes.

This study has several potential limitations. First, although it contains a diversity of patients, the cohort was from a single academic medical center in a northeastern city. Therefore, NLP algorithms might not be generalizable to other sites and settings due to differences in local patient population, documentation practices, and terminology differences. While the algorithm is intended to have applicability across health systems, local validation would be required to determine the algorithm’s performance and identify modifications for improvement in each new setting. Second, the study is limited by its relatively small number of annotated notes used for validation, which may have resulted in our models missing rare but relevant expressions in the entire set of clinical notes. Third, though the subcategories were developed by a team of experts and iteratively refined during the development process, there may be some ambiguity and heterogeneity among the types of expressions identified within each category. For example, the most prevalent category, living relatives with unknown involvement, likely includes mention of relatives who are actively involved in care decisions and relatives who are uninvolved, but there was not enough documentation for the NLP algorithm to further specify the level of involvement. In addition, documentation in the palliative care and hospice subcategories were classified as indicating reduced potential for being INEADS because generally these decisions occur with input and consent from the patient or surrogates. However, in some instances, the NLP algorithm may have detected documentation in which the care team referred a patient to these services due to a need for more in-depth coordination of advanced care planning. Fourth, our method of identifying patients who are INEADS is inherently limited to what the healthcare team documents in clinical notes. Given the diagnostic and physiological emphasis of documentation in the acute care setting, it is possible that factors related to INEADS status were not documented routinely. This could have led to an overestimation or underestimation of the number of patients who may be INEADS. Lastly, our study was limited to information contained within clinical notes and excluded data contained within structured electronic health record fields. Future studies should explore the use of NLP in conjunction with structured or semi-structured data as complementary approaches.

Conclusion

This investigation demonstrates the potential for NLP to identify patients who may be INEADS, and may inform interventions to enhance advance care planning for patients who lack social support. Efforts to increase completion and portability of advance directives should include strategies for reaching patients who may not have obvious default surrogates, such as those who live alone or in transitional situations. Our novel application of NLP to gather information about social isolation from electronic clinical notes may have implications for healthcare organizations striving to collect and utilize data on social determinants of health.

Complete list of terms and expressions in each subcategory.

(XLSX)

Click here for additional data file.

18 Mar 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Giovanni Ottoboni, Psy, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please ensure that you include a title page within your main document. We do appreciate that you have a title page document uploaded as a separate file, however, as per our author guidelines (http://journals.plos.org/plosone/s/submission-guidelines#loc-title-page) we do require this to be part of the manuscript file itself and not uploaded separately.

Could you therefore please include the title page into the beginning of your manuscript file itself, listing all authors and affiliations.

3. Thank you for stating the following in the Acknowledgments Section of your manuscript:

"This study was funded by National Institute of Nursing Research [NINR] (R21 NR019319), “Improving ethical care for patients who are incapacitated with no evident advance directives or surrogates (INEADS)”. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Nursing Research. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

"Our study was funded by National Institute of Nursing Research [NINR] (R21 NR019319), “Improving ethical care for patients who are incapacitated with no evident advance directives or surrogates (INEADS)”. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

5. Please amend either the abstract on the online submission form (via Edit Submission) or the abstract in the manuscript so that they are identical.

6. Your ethics statement should only appear in the Methods section of your manuscript. If your ethics statement is written in any section besides the Methods, please move it to the Methods section and delete it from any other section. Please ensure that your ethics statement is included in your manuscript, as the ethics statement entered into the online submission form will not be published alongside your manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a very interesting and important study on the use of natural language processing to identify acute care patients who lack advance directives, decisional capacity, and surrogate decision makers. The following points would enhance the study:

Methods:

Study dataset and population: could the authors please provide more information regarding the hospital in which the data is collected (e.g. university or general hospital etc.)?

If patients for instance completed an advance directive with the general practitioner or another primary care provider, might there be a chance that this is included in the clinical hospital notes? Are there shared electronic patient files available between different settings?

Results:

The authors report that 45% of patients died within 4 years after discharge, did the authors collect any information regarding the setting where patients died? Was this data collected in the clinical notes of the hospital? Was this data also collected for those who died outside the hospital? This might need some clarification in the methods section.

Discussion:

The authors found that only a small percentage of patients were characterized as INEADS, could the authors please provide explanations for this low percentage? Might there be an underestimation of this phenomenon?

Could the authors please provide more information on whether this algorithm can be used in other hospitals and healthcare contexts? Should the algorithm be validated again prior using it in for instance another hospital?

Reviewer #2: The manuscript “Using Natural Language Processing to Identify Acute Care Patients Who Lack Advances Directives, Decisional Capacity, and Surrogate Decision Makers” presents an application of NLP on electronic health records to identify patients who are incapacitated with no evident advance directives or surrogates (INEADS).

The authors used an existing open-source dataset including electronic health records and used an NLP algorithm to identify patients who are Incacapacitated with No Evident Advance Directives or Surrogates (INEADS). They developed the NLP algorithm starting from the identification of concepts related to INEADS status (made by three researchers, authors of the manuscript), and, from these concepts, they extracted different subcategories, including different lists of terms. Each subcategory was labeled based on the meaningfulness of that information to classify a patient as an INEADS or not, using labels “reduced” and “elevated”. Then the NimbleMiner algorithm was applied to create a language model, and an interactive vocabulary explorer was implemented to identify synonyms. After a labeling phase from three experts, NLP algorithm performances were evaluated on a (very reduced) gold standard testing set. Once the performances have been assessed, the NLP algorithm has been used on the full cohort to detect the presence of the pre-identified subcategories. Two groups have been detected: admissions with the possibility of being INEADS, having words belonging to more than 1 subcategory, and high likelihood of being INEADS, having words belonging to more than 4 subcategories.

As stated by the authors, this is the first study using NLP to identify INEADS patients. The methodology to use NLP algorithms on electronic health records is well known. The authors of this research used a simple NLP method but other more complex methods could have been investigated and applied. The authors should justify the reason why other methods were not tested and only NimbleMiner was used, or at least present a more comprehensive state-of-the-art regarding NLP and electronic health records.

Secondly, authors stated that “the purpose of the study was to develop and validate a natural language processing (NLP) algorithm to identify patients who are INEADS using documentation in electronic clinical notes”. This is partially true: the applied NLP method is meant to find words belonging to specific categories whose presence is likely to identify INEADS patients. Authors use an arbitrary threshold for the number of the identified categories and, based on the number of identified categories, patients are labeled with a high or low risk of being INEADS. The validation has been made on a very small dataset (10% sample, corresponding to 8 patients), and the authors stated that those patients “appeared to meet the definition of INEADS”, but it is not possible to be completely sure about it, since this information was not reported in the original dataset.

Also, the authors stated that their findings underline the importance of encouraging patients to complete advance directives, particularly when they lack social support. This is not that straightforward from the results, although it can be deduced by comparing the high likelihood of being INEADS demographic with the full cohort. However, this does not seem to be the core of the study, based on what the authors stated (purpose of the study: develop and validate an NLP algorithm to identify patients who are INEADS using documentation …). I would suggest the authors clarify the purpose of the study (develop and validate an NLP algorithm, or investigate the characteristics of INEADS patients?) and be consistent with the conclusions.

Abstract and introduction

Key findings are well reported, but no literature regarding the same topic (i.e., NLP and INEADS patients) was provided. Later in the manuscript, the authors stated that this is the first study attempting to develop NLP algorithms to detect INEADS patients. However, the introduction should include more information about the application of NLP using electronic health records to make the reader better understand what NLP is meant to do. Please add this information in the abstract and the introduction and revise accordingly.

Figures and tables

All the figures and tables are clear and readable. The captions of both figures and tables are complete and accurate. I would suggest adding a different graphical representation for the t-test and chi-square test for clarity.

Methods

I would not use “NLP methods” as the name of the subsection. Instead, I would use “NLP algorithm pipeline”. The word “methods” can be misleading since it seems that more than one method is developed when only one is presented.

Page 10, line 15: “We then created subcategories within each domain”. How were the subcategories created? No information is given. Please, explain in a more extensive way revise accordingly.

Page 10, line 16: “The 17 subcategories were grouped based on whether they indicated reduced or elevated potential for being INEADS”. Instead of “grouped”, I would suggest to the authors to use “annotated” or something similar, since “reduced” or “elevated” is a characteristic of that subgroup. Also, please change “they indicated reduced…” to “they indicated the reduced …”.

Page 11, line 1: “Four team members (JS, BC, AL, MT) developed the initial list of terms, which was iteratively revised and finalized by all team members. Table 1 lists examples of terms in each category”. I would suggest giving the full list of terms within each category as supplementary materials for reproducibility purposes, since only a few examples are available in the main text.

Page 12, line 3: “To evaluate NLP algorithm performance, we randomly sampled a gold standard testing set of 300 clinical notes from all clinical notes of patients with diagnoses that could be associated with being INEADS”. How has the gold standard testing been made? The 300 clinical notes were chosen randomly, or did the authors use a specific rule? Please add information about the note selection in the manuscript.

NLP algorithm performances were evaluated in detecting the 17 subcategories, not in detecting a potential INEADS patient. For this reason, maybe the main research question should be rephrased (INEADS patients are detected based on the arbitrary number (4) of identified categories with an elevated potential of being INEADS).

Page 12, line 23: “…, we identified admissions that included at least one of the five subcategories indicating elevated potential for being INEADS”. The subcategories indicating the elevated potential of being INEADS presented in Table 1 are 6. The “lacking capacity” subcategory is missing in this part of the methods. Only 5 subcategories indicating elevated potential of being INEADS are mentioned throughout the papers. Change it accordingly to what is reported in Table 1.

Page 13, line 2: “surrogate decision maker unknown”. Be consisted of what is written in Table 1, “surrogate decision maker unidentified”.

Page 13, line 4: “We validated the “high likelihood of being INEADS” classification by conducting a case study validation on 10% of this cohort”. Here the 10% means 8 hospital admissions. It is a very small number for the results to be significant. I would suggest validating the classification on a higher number of hospital admissions.

Page 13, line 8: “Lastly, we compared demographic characteristics of the “possibility of being INEADS” cohort and the “high likelihood of being INEADS” cohort against the full cohort using t-tests or chi-square tests”. For the sake of clarity, I suggest to the authors to add specifications on which kind of data (i.e., continuous variables or categorical data) these two statistical methods are applied.

Results

Page 13, line 17: “The most common type of insurance was Medicare”. For a non-American audience, it may be helpful to better explain what “Medicare and Medicaid” stand for (the Medicaid is mentioned in the discussion section).

Page 13, line 21: I would change “NLP algorithm performance against gold standard” in “Validation of NLP algorithm on gold standard testing set”.

Page 14, line 2: “Consistent with the proportion of each type of note in the dataset overall, 68% of notes containing at least one subcategory were nursing notes and 18% were physician notes”. What about the remaining 14%? Which is the source of the remaining notes?

Page 14, line 4: “However, subcategories indicating elevated potential for being INEADS were most likely to be documented in consult notes (57%) and least likely to be documented in nursing notes (48%)”. Is “consult notes” a synonym of “physician notes”? If yes, I suggest remaining consistent using the same terminology throughout the paper.

Page 14, line 17: “Less than one percent of admission (n=81) met the criteria for the “high likelihood of being INEADS” cohort, with 79 containing four and two containing all five of the subcategories”. Are the subcategories indicating elevated potentials of being INEADS six?

Discussion

Page 16, line 21: “However, with more adults living in social isolation”. Please add a reference for this sentence.

Page 17, line 15: “Second, the study is limited by its relatively small number of annotated notes used for training and testing”. Is it right to use “training” for this type of application? The method has pre-defined rules (words within subcategories) and is manually enhanced by researchers by adding some synonyms, so the algorithm does not seem to be “trained” following the strict definition used in machine learning field (https://docs.aws.amazon.com/machine-learning/latest/dg/training-ml-models.html). The algorithm does not seem to “learn” something from the data, rather the authors set rules to identify words.

Conclusion

Page 18, line 10: “Our results underscore the importance of encouraging patients to complete advance directives, particularly when they lack social support”. This conclusion is not so straightforward, based on the implemented methods. Authors should try to justify better their conclusions based on their results.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Kim de Nooijer

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

26 May 2022

Thank you for your careful review and thoughtful comments. We included a table with point-by-point responses to each comment as an attachment.

Submitted filename: INEADS response to reviewers.docx

Click here for additional data file.

7 Jun 2022

Using Natural Language Processing to Identify Acute Care Patients Who Lack Advance Directives, Decisional Capacity, and Surrogate Decision Makers

PONE-D-22-04209R1

Dear Dr. Cohen,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Giovanni Ottoboni, Psy, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: Dear Authors,

thank you for having revised the manuscript. My comments have been addressed and, in my opinion, the quality of the paper has been improved.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Kim de Nooijer

Reviewer #2: Yes: Serena Moscato

**********

30 Jun 2022

PONE-D-22-04209R1

Using natural language processing to identify acute care patients who lack advance directives, decisional capacity, and surrogate decision makers

Dear Dr. Cohen:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Giovanni Ottoboni

Academic Editor

PLOS ONE

Table 1

Domains and subcategories of terms related to INEADS (Incapacitated with No Evident Advance Directives or Surrogates) status.

Domain	Subcategory (sample terms)	Examples of notes containing subcategories	Potential for being INEADS
Surrogate decision makers	Married(husband, wife, spouse)	“…surgical consent signed by husband”	Reduced
	Partnered(girlfriend, significant other, fiancé)	“Pt [patient] fiancé phoned x 1 over noc [nurse on call]…”	Reduced
	Living relatives with unknown involvement(son, aunt, complicated family dynamics)	“Family history: brothers with [name] in their 50s…”	Reduced
	Caregiver support(primary caregiver, sole caregiver, main caregiver)	“social daughter in to visit daughter / lives with pt [patient] and is primary caregiver…”	Reduced
	Living with or close to others(daughter lives nearby, roommate, lives with)	“Social history: lives with roommate works in group home…”	Reduced
	Community connection(supportive neighbors, friend who assists, senior center)	“skin grossly intact no breakdown noted. Social mother and many friends in to visit and supportive…”	Reduced
	Religious connections(their rabbi, clergy involved, practicing catholic)	“current plan of care call chaplain in am and set up time for tomorrow evening for sacrament of the sick…”	Reduced
	Surrogate decision maker identified(family meeting occurred, spoke with daughter, family decided)	“patients critically ill state at this time the patient family was called…the family decided to withdraw support…”	Reduced
	Unmarried(recently widowed, estranged from his wife, never married)	“social history: works as police officer lives alone never married no children…”	Elevated
	Living alone(lives alone, lives independently, without caregiver)	“…new trauma pt [patient] on tsicu [trauma/surgical intensive care unit] / pt is sp [status post] motorcycle collision with car / pt is a 42 years old man who lives alone / in hospital pt did not have demographic information…”	Elevated
	Transitionally situated(currently homeless, half-way house, incarcerated)	“social ETOH abuse ivda [intravenous drug abuse] homeless now…”	Elevated
	Surrogate decision maker unidentified(cannot find family, family member did not respond, family unreachable)	“social dispo [dispoision] full code no contact from family per liver notes patient is not a candidate for transplant…”	Elevated
Advance directives	Palliative care(ethics palliative care, palliative care consulted, palliative services)	“called out to floor plans to have family meeting c [with] team gi [gastroenterology] oncology palliative care present emotionally support pt and family”	Reduced
	Hospice(moving toward hospice, arrange home hospice, hospice nurses)	“…currently cmo [care management organization] plan to discharge home with hospice‥”	Reduced
	Advance directives available(sister hcp, contact daughter hcp, code status DNR/DNI)	“mother telephone fax 1 health care proxy appointed yes…”	Reduced
	Advance directives unavailable(no living will, no advance directives, no healthcare proxy)	“…‥no hcp [health care proxy] lack copy provided…”	Elevated
Decisional capacity	Lacking capacity(aox disoriented, loss of executive function, impaired judgement)	“she suffered from severe dementia along with inability to engage in activities of daily living”	Elevated

Table 2

Characteristics of full, “possibility of being INEADS” and “high likelihood of being INEADS” cohorts.

	All admissions	Admissions with possibility of being INEADS (≥1 social subcategory indicating elevated potential for being INEADS)		Admissions with high likelihood of being INEADS (≥4 social subcategories indicating elevated potential for being INEADS)
N (% among all admissions)	23,904 (100)	7,134 (29.8)		81 (0.3)
Expired during follow-up, n (%)	10,772 (45)	3,743 (52)	**	53 (65)	**
Expired in hospital, n (%)	2,479 (10)	981 (14)	**	14 (17)	*
Age, mean (SD)	60 (18)	61 (19)	*	61 (15)	*
Gender, n (%)			**		**
Female	13,000 (54)	3,793 (53)		34 (42)
Male	10,904 (46)	3,341 (47)		47 (58)
Ethnicity, n (%)			**		**
Black	3,304 (14)	880 (12)		11 (14)
Hispanic	1,014 (4)	275 (3)		2 (2)
White	16,899 (71)	5,162 (72)		56 (69)
Other	920 (4)	245 (3)		2 (2)
Unknown	1,767 (7)	572 (8)		10 (12)
Admission Type, n (%)			**		**
Elective	2,788 (12)	741 (10)		5 (6)
Emergency	20,629 (86)	6,200 (87)		74 (91)
Urgent	487 (2)	193 (3)		2 (2)
Insurance, n (%)			**		**
Medicaid	3,243 (14)	1,006 (14)		16 (20)
Medicare	14,124 (59)	4,271 (60)		49 (60)
Private	5,341 (22)	1,568 (22)		12 (15)
Self-pay	308 (1)	75 (1)		2 (2)
Unspecified government	888 (4)	214 (3)		2 (2)

INEADS, Incapacitated with No Evident Advance Directives or Surrogates.

T-tests (continuous variables) or chi-square tests (categorical variables) compare admissions in the “possibility of being INEADS” and “high likelihood of being INEADS” cohorts with the full cohort.

*p<0.05

**p<0.001

Table 3

Evaluation of natural language processing algorithm performance through gold-standard manual review (n = 300 clinical notes).

Subcategory	Frequency (%) of documentation	Precision	Recall	F-score
Living relatives with unknown involvement	128 (69.6%)	0.52	0.97	0.68
Lacking capacity	119 (64.7%)	0.67	0.99	0.8
Surrogate decision maker identified	53 (28.8%)	0.94	0.75	0.83
Advance directives available	33 (17.9%)	0.94	0.92	0.93
Community connection	20 (10.9%)	0.85	1	0.92
Married	11 (6%)	0.64	1	0.78
Unmarried	7 (3.8%)	1	0.64	0.78
Living alone	7 (3.8%)	0.86	0.75	0.8
Living with or close to others	7 (3.8%)	1	0.64	0.78
Surrogate decision maker unidentified	7 (3.8%)	0.29	0.67	0.4
Partnered	6 (3.3%)	1	0.86	0.92
Transitionally situated	5 (2.7%)	1	1	1
Religious connections	3 (1.6%)	0.67	1	0.8
Palliative care	2 (1.1%)	1	1	1
Hospice	1 (0.5%)	1	1	1
Caregiver support	-	-	-	-
Advance directives unavailable	-	-	-	-
Overall performance	184 (100%)	0.83	0.88	0.83