Organizational characteristics and perceptions of clinical event notification services in healthcare settings: a study of health information exchange.

OBJECTIVE: Event notification systems are an approach to health information exchange (HIE) that notifies end-users of patient interactions with the healthcare system through real-time automated alerts. We examined associations between organizational capabilities and perceptions of event notification system use.
MATERIALS AND METHODS: We surveyed representatives (n = 196) from healthcare organizations (n = 96) that subscribed to 1 of 3 Health Information Organizations' event notification services in New York City (response rate = 27%). The survey was conducted in Fall 2017 and Winter 2018. Surveys measured respondent characteristics, perceived organizational capabilities, event notification use, care coordination, and care quality. Exploratory factor analysis was used to identify relevant independent and dependent variables. We examined the relationship between organizational capabilities, care coordination, and care quality using multilevel linear regression models with random effects.
RESULTS: Respondents indicated that the majority of their organizations provided follow-up care for emergency department visits (66%) and hospital admissions (73%). Perceptions of care coordination were an estimated 57.5% (β = 0.575; P < 0.001) higher among respondents who reported event notifications fit within their organization's existing workflows. Perceptions of care quality were 46.5% (β = 0.465; P < 0.001) higher among respondents who indicated event notifications fit within existing workflows and 23.8% (β = 0.238; P < 0.01) higher where respondents reported having supportive policies and procedures for timely response and coordination of event notifications. DISCUSSION AND
CONCLUSION: Healthcare organizations with specific workflow processes and positive perceptions of fit are more likely to use event notification services to improve care coordination and care quality. In addition, event notification capacity and patient consent procedures influence how end-users perceive event notification services.

INTRODUCTION

Event notification systems are an approach to health information exchange (HIE) that notifies end-users of patient interactions with the healthcare system through real-time automated alerts., The basic features of an event notification system include: (1) end-users who subscribe to the service; (2) a defined list of patient events that trigger notifications, such as a hospital admission, discharge, or an emergency department (ED) visit; (3) a list of patients for whom events are monitored; and (4) the technical monitoring and routing components necessary to enable the exchange of information. End-users, healthcare providers and organizations, leverage event notification systems to facilitate further care for patients in the form of contact telephone calls, scheduling postdischarge follow-up visits, and referrals to service programs. Event notification services have been associated with end-user satisfaction, organizational efficiencies, improved care coordination, and care quality.,

However, organizational capabilities may influence the usage and utility of event notification systems. Health information technology (IT) success, overall, is highly dependent upon the extent to which an organization is willing to commit financial and staff resources., Likewise, to support adoption of these systems, end-users must be made aware of technology functionalities, fit within the process of care, and potential uses. Specific to event notification systems, studies of single sites suggest that clinical and business workflows, internal policies and procedures, and staff attitudes determine how, when, and for which patients event notification systems are most useful.,, The role of organizational capabilities in event notification systems, and health IT in general, will likely become even more pronounced as larger and more complex organizations see the potential applications of event notifications. For example, health systems and Accountable Care Organizations (ACOs) have begun applying event notifications to population health activities for risk stratification and service referrals.

This study sought to examine end-user perceptions of event notifications systems provided by three health information organizations (HIO). Drawing on prior work with a single HIO and with a grounding in a sociotechnical perspective, we examined associations between perceptions of organizational capabilities, care quality, and care coordination. Thus, the results of this study expand the current literature to provide further understanding of how organizational capabilities influence perceptions and use of HIE services.

MATERIALS AND METHODS

Setting

We surveyed healthcare professionals at 160 healthcare organizations that had subscribed to event notification services through 1 of 3 HIOs in New York State (NY): The Rochester Regional Health Information Organization (RHIO) in Western NY; Healthix in the New York City metropolitan area; and HEALTHeLINK in Buffalo, NY. All three offer event notification services that use admit-transfer-discharge (ADT) feeds from participating health systems and organizations. These organizations represented all settings of the health delivery system including home health organizations, hospitals and health systems, and population health-related organizations.

Sample

We identified potential survey respondents with the support of the RHIOs and key points of contact at participating healthcare organizations. We asked each point of contact to provide contact information for clinical and nonclinical personnel that received event notification alerts as part of their job or managed personnel who respond to notifications. Additionally, we sought contact information for personnel who set organizational policies, procedures, or workflows for event notification services. We identified a total of 722 individuals across the 160 healthcare organizations for inclusion in the sample. The overall response rate for the survey was 27.1% (n = 196) with 60% (n = 96) of healthcare organizations participating in event notification services represented by at least one respondent (mean = 2.3).

Data collection

We administered the survey online and obtained informed consent from each respondent using RedCap. To account for nonresponse, multiple respondents were recruited from each site. The study was approved by the Indiana University Institutional Review Board (IRB).

Questionnaire

The questionnaire was administered in Fall 2017 and Winter 2018 using items adapted from existing survey instruments that measured usage and perceptions of event notifications of laboratory examinations and tests, electronic patient messaging, care transitions, and prescription drug alerts with similar automated features. Organizational capabilities and end-user perception questionnaire items were derived from the health services research, information systems, and management information system literature. The full survey is available in Supplementary Appendix S1. Response options used 5-point Likert-scale responses ranging from “strongly disagree” or “never” to “strongly agree” or “always.” Researchers piloted the questionnaire on personnel working in clinical and hospital administration settings to assess survey length, time, content, and comprehension.

Analysis

Respondent characteristics and organizational capabilities were examined using frequencies and percentages. For presentation purposes, we collapsed the 5-point Likert-scales into the three categories of disagree, neutral, or agree using the “top 2 box” approach. We conducted exploratory factor analysis (EFA) to reduce survey items into relevant independent and dependent variables for further analyses (Supplementary Appendix S2). We used oblique rotation to account for correlated items (Supplementary Appendix S3). Through EFA, we identified two factors for use as dependent variables: (1) care quality and (2) care coordination. We also identified 5 factors to use as independent variables: (1) workflow specificity, (2) event notification characteristics, (3) patient consent procedures, (4) perceived fit of event notification services with existing workflows/processes, and (5) perceived organizational capacity (Supplementary Appendix S4). Factor definitions are provided in Table 1. Cronbach’s alphas for each factor ranged between 0.88 and 0.95. We extracted factor scores for each identified variable for use in regression modeling. We estimated separate multilevel linear regression models with random effects for each outcome of interest. In each model, the respondents’ healthcare organization was entered as a random intercept to account for the clustered nature of survey responses.

Table 1.

Factor definitions

Factor	Definition	Variable type
Perceived fit of event notifications	Clinical and nonclinical processes and workflows that align with event notification services and associated tasks to improve care quality and coordination	Independent variable
Perceived organizational capacity	Organizational procedures that enable management and integration of event notifications to improve healthcare service provision	Independent variable
Workflow specificity	The presence of policies and procedures that supported timely and coordinated use of event notifications	Independent variable
Patient consent procedures	Perceived structural barriers and facilitating procedures for providing consent to subscribed patients	Independent variable
Event notification characteristics	Perceived organizational and event notification limitations that inhibits effective response and use of patient health information	Independent variable
Care quality	Perceived organizational efficiency, communication, and patient satisfaction facilitated by use of event notification services	Dependent variable
Care Coordination	An organization's ability to share timely information to improve care coordination, including transitions of care across clinical settings	Dependent variable

RESULTS

The most common respondents were nonclinical staff involved in direct patient interactions such as care navigators, patient navigators, and care coordinators (46%). Clinician respondents (ie, physicians, nurses, physician assistants) made up a fifth of the total sample (Table 2). In terms of job types, personnel tasked with serving in a patient engagement role constituted a majority of the respondents (58%). About half of respondents (49%) reported the primary mode of receiving event notifications within their organization was through secure email and 27% indicated that event notifications came through their electronic health record (EHR) work queues. Most respondents reported receiving notifications for ED encounters (60%) and inpatient admissions (56%). Generally, providers received event notifications for adult patients (63%), but some respondents indicated a focus on select demographic or risk groups (ie, high utilizers, patients with chronic conditions, home health patients, behavioral health patients, geriatric patients, and pediatric patients).

Table 2.

Characteristics of respondents and organizations that subscribe to event notification services

	n	%
Respondents
Clinician (physician, nurse, PA)	41	20.9
Care navigator/patient navigator/care coordinator	91	46.4
Manager/director/supervisor	28	14.3
Other	36	18.4
Job type
Patient engagement	114	58.2
Managerial/administration	46	23.5
Unknown	36	18.4
Gender
Male	26	13.3
Female	124	63.3
Age
<30	17	8.7
31–49	90	45.9
>50	49	25
Unknown	40	20.4
Organizational
How notifications are received
EHR work queue	52	26.5
Secure email	97	49.5
Other	20	10.2
I do not receive notifications as part of my job	25	12.8
Owner of organization
Physicians	17	8.7
Non-physician managements in your group	31	15.8
Hospital, hospital system or healthcare system	14	7.1
Health Maintenance Organization (HMO) or other insurance entity	10	5.1
Federally-Qualified Health Center (FQHC)/Community Health Center (CHC)	20	10.2
Solo practice	13	6.6
Some other entity such as a government entity	55	28.1
Number of facilities under operation by organization
1	35	17.9
2	10	5.1
>3	104	53.1
Do not know	8	4.1
Organization description
Mainly primary care providers	29	14.8
Multispecialty group (specialists and primary care physicians)	22	11.2
Mainly nonprimary care specialists	41	20.9
Other	65	33.2
Organization has…
Care managers	142	72.5
Social workers	101	51.5
Patient navigators	52	26.5
Health coaches	46	23.5
Notification services
Type of notifications organization receives…a
Any clinical event	54	27.6
ED encounters	119	60.1
Inpatient admissions	110	56.1
Patient types included in notifications…a
All patients	123	62.8
High utilizers	14	7.1
Patients with chronic conditions	18	9.2
Home health patients	19	9.7
Behavioral health patients	19	9.7
Geriatric patients	7	3.6
Children/adolescents	1	0.5
Patient population
Mainly adult	87	44.4
Mainly pediatric	5	2.6
Both	65	33.2
Medicaid patient
None	2	1
Some	22	11.2
Quite a bit	75	38.3
All	54	27.6
Do not know	5	2.6
Person primarily responsible for event notification
Clinical staff (MD, DO, NP/RN/LPN, PA)	57	29.0
Other office staff/nobody specific	90	52.5

Event notification services are not mutually exclusive.

Perceived fit of event notifications

Respondents generally reported that event notifications improved job performance and clinical care. Specifically, respondents indicated that event notification services were “clinically useful” (67% agree), and enabled the identification of undiagnosed patient conditions (41% agree) as well as patients who were high utilizers of care (61% agree) (Table 3). These perceptions were shared among both administrative (46% agree) and clinical staff (42% agree). A majority of respondents also indicated that their organizations had effective policies and procedures for responding to event notifications (56% agree), but a lower percentage of organizations had them in place for notifications after normal business hours (39% agree).

Table 3.

Respondent perceptions of event notification services, organizational capacity, and indicators of use

Factors/items	n	%	n	%	n	%
Perceived fit of event notifications	Disagreea		Neutrala		Agreea
The administrative staff believe the organization alert services help them get their job done effectively.	17	8.7	35	17.9	90	45.9
The physicians/clinical staff in our organization believe the organization alert services are an essential component of high-quality care.	11	5.61	32	16.3	83	42.4
The leaders in our organization have emphasized the importance of the organization's alert services in high-quality care	11	5.6	24	12.2	120	61.2
Organization's alert services are clinically useful.	7	3.6	19	9.7	131	66.8
Organization alert services identify patients' healthcare encounters that our organization was not aware of.	10	5.1	16	8.2	130	66.3
As a result of organization alert services, we have identified clinical conditions we did not realize patients had.	20	10.2	44	22.5	81	41.3
As a result of organization alert services, we have identified patients who are high utilizers of medical services.	12	6.1	22	11.2	120	61.2
Our organization has effective written policies and procedures for responding to organization alerts.	36	18.4	24	12.4	109	55.6
Our organization has effective policies and procedures in place to respond to organization alerts arriving after normal business hours	52	26.5	33	16.8	77	39.3
Perceived organizational capacity	Disagreea		Neutrala		Agreea
Our organization effectively manages patient information.	3	1.5	13	6.6	146	74.5
Our organization effectively transfers relevant patient information among staff members.	6	3.1	11	5.6	145	73.9
Our organization effectively integrates information across individual information records to learn more about our entire patient panel.	7	3.6	38	19.4	117	59.7
Our organization effectively leverages patient information to improve our services.	3	1.5	31	15.8	128	65.3
Workflow specificity	Neverb		Sometimesb		Alwaysb
If we receive an alert that a subscribed patient is at the ED, we contact the ED while the patient is in the ED.	57	29.1	42	21.4	58	29.6
If we receive an alert that a subscribed patient is at the ED, we contact (by phone or in-person) the patient about their ED visit.	21	10.7	25	12.8	112	57.1
If we receive an alert that a subscribed patient has been admitted to the hospital, we contact the hospital while the patient is in the hospital.	30	15.3	34	17.4	94	47.9
If we receive an alert that a subscribed patient has been admitted to the hospital, we contact (by phone or in person) the patient about their hospitalization.	15	7.7	21	10.7	123	62.8
Consent procedures	Disagreea		Neutrala		Agreea
Obtaining patient consent is a serious barrier to subscribing to patient alerts. (R)	37	18.9	29	14.8	85	43.4
Our organization has effective procedures for obtaining our patients' consent for inclusion in the RHIO's alert services.	5	2.6	8	4.1	140	71.4
Patients have difficulty understanding consent for alerts. (R)	73	37.2	43	21.9	31	15.8
Most patients who are asked for consent for alerts refuse to grant consent. (R)	127	64.8	12	6.1	12	6.1
Administering consent for alerts represents additional workload for our organization. (R)	53	27.0	37	18.9	65	33.2
Event notification characteristics	Disagreea		Neutrala		Agreea
The organization alert service does not provide enough information. (R)	43	21.9	40	20.4	70	35.7
We often receive organization alerts for patients that are not ours. (R)	93	47.5	17	8.7	35	17.9
The information received from the organization alert service is clear and understandable.	21	10.7	40	20.4	93	47.5
The number of organization alerts our organization receives exceeds what we can effectively manage. (R)	90	45.9	30	15.3	30	15.3
We receive too many organization alerts to easily focus on most important ones. (R)	91	46.4	30	15.3	31	15.8
Care quality	Disagreea		Neutrala		Agreea
Organization alert services have improved our ability to provide high quality of care	12	6.1	25	12.8	124	63.3
Organization alert services have improved our efficiency.	15	7.6	34	17.4	112	57.1
Organization alert services have improved patient satisfaction.	18	9.2	82	41.8	61	31.1
Organization alert services have improved communication with our patients.	10	5.1	30	15.3	121	61.7
Organization alert services have improved our ability to obtain information about our patients from other organizations.	18	9.2	28	14.3	115	58.7
Care coordination	Disagreea		Neutrala		Agreea
Organization alert services facilitate our patients' transitions across different settings of care.	11	5.6	37	18.9	103	52.6
Organization alert services have improved our ability to coordinate care.	10	5.1	21	10.7	125	63.8
Organization alert services have prompted changes in care for many of our patients.	16	8.2	54	27.6	76	38.8
Organization alert services have improved communication with other providers.	20	10.2	48	24.5	83	42.4
Organization alert services help us create a comprehensive medical record for all of our patients.	14	7.1	43	21.94	93	47.5

On a 5-point scale: 1, 2 = Disagree; 3 = Neutral; 4, 5 = Agree.

On a 5-point: 1, 2 = Never; 3 = Sometimes; 4, 5 = Always.

(R), reverse scored for analysis.

Perceived organizational capacity

Majorities of respondents reported that their organizations had the capacity to receive and prioritize patient information to improve healthcare services. Respondents generally agreed that their organizations were equipped to “manage patient information” (75% agree), “transfer relevant patient information among staff members” (74% agree), and “integrate information across individual information records to learn more about the entire patient panel” (60% agree).

Workflow specificity

Respondents somewhat agreed that event notifications prompted their organizations to provide follow-up communication with patients and the healthcare setting in which they were seen. For example, respondents indicated that their organizations contacted the hospital (48% agree) and, to a lesser degree, the ED (30% agree) where patients received treatment during or shortly after the encounter. Respondents also reported that event notifications enabled their organization to initiate postdischarge contact with patients as a result of an ED (57% agree) or hospital (63% agree) visit.

Consent procedures

New York State requires affirmative patient consent for certain types of HIE, including patient enrollment in alert services. A majority of respondents reported (71% agree) that their organizations have “effective procedures for obtaining our patients’ consent for inclusion in the RHIO’s alert services.” However, 43% of respondents reported that obtaining consent was a “serious barrier to subscribing to patient alerts.” Furthermore, a small percentage of respondents indicated (33% agree) that “administering consent for alerts represents additional workload for our organization.”

Event notification characteristics

Respondents generally agreed that there are some limitations of event notifications, such as information incompleteness. Specifically, respondents indicated that “organization alert service does not provide enough information” (36% agree) about subscribed patients. Respondents disagreed (46%) with the item, “the number of alerts our organization receives exceeds what we can effectively manage.” Approximately half of all respondents (48% agree) indicated that information contained within notifications was “clear and understandable.”

Care coordination and care quality

Sixty-three percent of all respondents reported that their organization’s “alert services have improved our ability to provide high quality care.” A majority of respondents indicated that patient information used among their organizations “improved our efficiency” (57%), “improved communication with our patients” (62%), and “improved our ability to coordinate care” (64%). Similarly, 53% reported that event notification services “facilitate our patients’ transitions across different settings of care.”

Associations with care quality and care coordination

In unadjusted models, the proportion of respondents who had favorable perceptions of care quality was an estimated 56.1% (β = 0.561; P < 0.001) higher among respondents who reported better fit of event notifications in their organization’s workflows (Table 4). Similarly, respondents that reported positive perceptions of care coordination were 64.3% (β = 0.643; P < 0.001) higher for organizations that fit event notifications in their existing workflows. The magnitude and direction of these outcomes persisted in adjusted models. After adjusting for respondent characteristics and organizational capabilities, perceived care quality was 46.5% (β = 0.465; P < 0.001) higher among respondents who indicated favorable perceptions of event notification services fit within existing workflows. Results from adjusted models also showed perceived care coordination was an estimated 57.5% (β = 0.575; P < 0.001) higher among organizations where event notification services fit was evident. Among respondents who reported having an increasing perception of workflow specificity, that is, the presence of policies and procedures that supported timely and coordinated use of event notifications, 22.8% (β = 0.228; P < 0.001) and 23.8% (β = 0.238; P < 0.01) perceived better care quality in unadjusted and fully adjusted models, respectively.

Table 4.

Associations between perceived respondent and organizational characteristics and perceived care quality and care coordination

	Care quality		Care coordination
	Unadjusted (95% CI)	Adjusted (95% CI)	Unadjusted (95% CI)	Adjusted (95% CI)
Perceived fit of event notifications	0.561 (0.429–0.692)***	0.465 (0.321–0.608)***	0.643 (0.513–0.775)***	0.575 (0.432–0.717)***
Perceived organizational capacity	0.020 (−0.101–0.141)	−0.014 (−0.149–0.121)	0.024 (−0.097–0.145)	0.013 (−0.123–0.148)
Workflow specificity	0.228 (0.096–0.361)***	0.238 (0.077–0.398)**	0.749 (−0.051–0.201)	0.081 (−0.075–0.237)
Consent procedures	−0.086 (−0.219–0.047)	−0.038 (−0.186–0.111)	−0.007 (−0.135–0.122)	0.025 (−0.119–0.169)
Event notification characteristics	0.039 (−0.092–0.170)	0.043 (−0.100–0.187)	0.008 (−0.119–0.135)	−0.004 (−0.143–0.135)
Respondent
Organization respondents
Clinician (physician, nurse, PA)		0.180 (−0.256–0.617)		−0.136 (−0.569–0.297)
Care manager		0.090 (−0.273–0.454)		0.289 (−0.071–0.650)
Managerial		Ref		Ref
Other		−0.009 (−0.330–0.311)		−0.022 (−0.340–0.296)
Gender
Male		0.341 (0.022–0.659)*		0.297 (−0.022–0.616)*
Female		Ref		Ref
Organizational
Number of clinics/facilities
1		0.075 (−0.212–0.362)		0.007 (−0.277–0.291)
2		−0.262 (−0.756–0.232)		−0.304 (−0.795–0.187)
≥3		Ref		Ref
Don't know		−0.143 (−0.704–0.418)		−0.138 (−0.693–0.418)
Organization description
Primary care providers		Ref		Ref
Multispecialty providers		0.027 (−0.414–0.467)		0.266 (−0.171–0.702)
Mainly nonprimary care specialists		0.075 (−0.354–0.503)		0.345 (−0.081–0.770)
Other		0.053 (−0.333–0.440)		0.184 (−0.199–0.567)
How event notifications are received by organization
EHR work queue/Inbox		0.180 (−0.256–0.617)		0.035 (−0.247–0.316)
Secure email		Ref		Ref
Other		−0.302 (−0.715–0.110)		0.136 (−0.274–0.546)
I don't directly receive event notifications as a part of my job		0.169 (−0.210–0.549)		0.242 (−0.136–0.619)

P < 0.05;

P < 0.01;

P < 0.001.

DISCUSSION

Event notifications are more likely to be perceived as improving care coordination and care quality by respondents who reported positive perceptions of event notification fit and the procedures that facilitate the use of these HIE services. We found that most end-users in this study were in care navigator or care coordinator roles and that they reported being primarily responsible for responding to event notifications. Furthermore, the findings from this study are generally consistent with sociotechnical frameworks that emphasize the role of workplace characteristics in shaping end-user acceptance of technology.,

In three different communities and across multiple organizational and provider types, most end-users reported positive perceptions and agreed that event notification services improve care coordination and care quality. As an example of HIE that enables end-users to access information about patients from other settings of care, event notifications have a clear application to communication and coordination processes., In terms of quality, prior research has found that event notifications reduced the likelihood of hospital readmission, improved follow-up care after hospital discharge, and increased prophylactic measures for patients at risk of thromboembolism. In addition, survey results suggest that event notification services can contribute practical improvements to existing workflow processes by identifying and responding to previously unknown patient events and utilization patterns.

Nevertheless, these results indicate challenges with event notification services. First, end-users had middling perceptions of the completeness of information contained in event notifications. Because they are generally leveraging ADT systems to identify events, event notifications systems include limited data elements, such as patient identifiers and location information. For nearly any HIT, information quality concerns can be barriers to end-user acceptance. HIO organizations could potentially enrich event notifications with additional information from their clinical data repositories that might better meet end-user needs., Alternatively, the limited information content of event notification alerts reinforces the need to align organizational workflows and end-user expectations to the information available. For example, because a single alert may not be sufficient to support medical decision-making, appropriate end-users are those who are best positioned to collect more information or intervene to support care transitions, such as a patient navigator or care coordinator.

Additionally, end-users indicated that obtaining patient consent for alerts created a barrier to the use of HIE services. Healthcare organizations operating in an opt-in policy environment, as were all the HIOs in this study, face greater administrative burdens. Problematically, healthcare organizations and HIOs have limited options for eliminating consent challenges as patient consent requirements are defined at the state and federal levels. Mitigation is a more likely path through such efforts as simplifying consent language, consenting through the patient portal and EHR, and incorporating broader members of the care team in the process., Evidence of such designs suggests that patient consent procedures can facilitate or represent a barrier to inclusion in HIE services., Overcoming administrative burdens is critical to success as patients, when asked, nearly universally consent to HIE services.,

Although health information management within healthcare organizations was not the focus of this study, we asked about it to help us understand organizational capabilities. It is notable that, 10 years after the launch of the federal “Meaningful Use” incentive program to promote adoption of health IT, about a quarter of respondents indicated that their organizations still faced challenges with basic patient information management and intraorganizational data sharing. Notably, 40% reported challenges with panel-level data integration and analytics. It may be important for health IT policymakers to recognize that some organizations are still encountering difficulties with basic health information tasks considered necessary for high healthcare quality.

Limitations

Our findings are subject to several limitations due to the sample and cross-sectional nature of the study. While this study is one of the few to include multiple HIOs, our results may not be generalizable to all providers of event notification services. All three communities were in a single state that has a long history of development and experience offering technology services. Second, while we had representation from nearly every organization in our sample that participated in event notification services in one of the HIOs, our response rate among individuals was low. It is possible that respondents had different motivations for sharing perceptions about the topic than nonrespondents. Additionally, given the cross-sectional nature of the survey, we are unable to establish the temporal relationship between each of our individual domains and perceptions of our dependent variables. It is possible that organizations changed policies and procedures in response to favorable experiences with event notification services instead of first establishing policies and procedures before usage.

CONCLUSION

Organizational characteristics were associated with favorable perceptions of the impact of event notification services on care coordination and quality. Achieving fit between technology and the organization is critical as larger and more complex organizations see the potential applications of event notifications to healthcare delivery.

FUNDING

This work was supported by the National Library of Medicine of the National Institutes of Health (NIH) under award T15LM012502 and the Health Policy Research Scholars program at the Robert Wood Johnson Foundation (RWJF).

AUTHOR CONTRIBUTIONS

JA, KW, KEH, and JRV conceived the study. KW and JRV analyzed the data. All authors were involved in interpretation and drafting the manuscript. All authors made substantial contributions to manuscript revisions and approved the final version.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Click here for additional data file.