Organizational Supports for Research Evidence Use in State Public Health Agencies: A Latent Class Analysis.

OBJECTIVE: Use of research evidence in public health decision making can be affected by organizational supports. Study objectives are to identify patterns of organizational supports and explore associations with research evidence use for job tasks among public health practitioners.
DESIGN: In this longitudinal study, we used latent class analysis to identify organizational support patterns, followed by mixed logistic regression analysis to quantify associations with research evidence use.
SETTING: The setting included 12 state public health department chronic disease prevention units and their external partnering organizations involved in chronic disease prevention. PARTICIPANTS: Chronic disease prevention staff from 12 US state public health departments and partnering organizations completed self-report surveys at 2 time points, in 2014 and 2016 (N = 872). MAIN OUTCOME MEASURES: Latent class analysis was employed to identify subgroups of survey participants with distinct patterns of perceived organizational supports. Two classify-analyze approaches (maximum probability assignment and multiple pseudo-class draws) were used in 2017 to investigate the association between latent class membership and research evidence use.
RESULTS: The optimal model identified 4 latent classes, labeled as "unsupportive workplace," "low agency leadership support," "high agency leadership support," and "supportive workplace." With maximum probability assignment, participants in "high agency leadership support" (odds ratio = 2.08; 95% CI, 1.35-3.23) and "supportive workplace" (odds ratio = 1.74; 95% CI, 1.10-2.74) were more likely to use research evidence in job tasks than "unsupportive workplace." The multiple pseudo-class draws produced comparable results with odds ratio = 2.09 (95% CI, 1.31-3.30) for "high agency leadership support" and odds ratio = 1.74 (95% CI, 1.07-2.82) for "supportive workplace."
CONCLUSIONS: Findings suggest that leadership support may be a crucial element of organizational supports to encourage research evidence use. Organizational supports such as supervisory expectations, access to evidence, and participatory decision making may need leadership support as well to improve research evidence use in public health job tasks.

Research evidence use is a key component of evidence-based decision making (EBDM) in public health. EBDM involves the use of evidence of disease burden to identify and prioritize issues and to select programs and policies previously shown to improve population health. EBDM also involves assessment of community strengths and needs, incorporation of community preferences, implementation of selected evidence-based programs and policies, sound evaluation, and use of evaluation findings to improve implementation.1,2 With funders increasingly requiring use of evidence-based policies and programs (EBPPs) shown to improve population health, and evidence increasingly available, public health practitioners are expected to use evidence in grant applications, selection and justification of strategies, priority setting, implementation, and evaluation.

In previous cross-sectional studies with public health practitioner decision makers, several organizational supports facilitated evidence use for public health decision making if present and, if lacking, served as barriers. These included the following: organizational culture and climate; access to evidence; on-the-job opportunities to build skills in acquiring, assessing quality of, and interpreting evidence; leadership; clear organizational processes for evidence use, such as performance management systems; provision of time to gather and use evidence; positive attitudes of managers; manager sponsorship of the project; opportunity to generate locally relevant evidence; and monitoring of internal decision-making processes.3–10 Studies in mental health and human services organizations had similar findings.11,12 In one of the few longitudinal studies, Dobbins and colleagues13 found that tailored targeted research evidence messaging increased the number of evidence-informed strategies used by health departments.

Leadership support and access to research evidence may be essential to promote and ensure research evidence use.3–8,14–16 The presence of supportive organizational climates and cultures was also identified by interview participants as key to evidence use.3,17,18 In addition to positive manager attitudes, leaders can develop policies and set criteria to monitor evidence use processes, provide staff time for evidence use, and ensure staff receive training in use of evidence.5 Leaders can also facilitate needed organizational access to evidence.4,10,19,20 Many public health agencies lack access to full-text journal articles.16,19 Several reviews noted limited availability of relevant, clear, timely, and reliable research evidence.4,8,10

The purpose of this study was to identify patterns of organizational supports and associations with research evidence use for job tasks among public health practitioners working in chronic disease prevention and health promotion in 12 US state health departments and partnering organizations. While earlier cross-sectional studies measured organizational supports and tested relationships with evidence use, as described earlier, the present study is one of the first to longitudinally test relationships of organizational supports with research evidence use. Latent class analysis (LCA), a model-based approach, is a useful tool to identify groups with homogeneous response patterns in a heterogeneous population with observed variables.21 The latent class model can identify categorical latent variables from a complex array of observed categorical data, comparable with factor analysis.

Methods

The present study was part of a larger 12-state study.22–24 The Washington University in St Louis institutional review board approved the study. Organizational context and resources are hypothesized to facilitate staff adoption of EBDM processes, including research evidence use.22,25

Study sample

Six state public health department chronic disease prevention units were randomly selected and invited to participate, and 6 state health department chronic disease units were selected from states closest in state population to reduce bias from differences in state chronic disease funding. Via e-mail recruitment, the study team invited all professional staff members in the 12 state health department chronic disease units and a purposive sample of the staff in partnering organizations to participate in a baseline survey in 2014 and a second survey in 2016. Partners were included because they are key to implementation of EBPPs and to increase the sample size. Chronic disease directors identified key organizations and staff their programs partner with for chronic disease prevention, including local and district health departments; universities; health care facilities; and coalitions and voluntary and community-based health organizations (see Supplementary Digital Content, available at http://links.lww.com/JPHMP/A480). State enrollment and survey timing were staggered to accommodate study team schedules. Follow-up via e-mail and phone increased the postsurvey response to 70.5% of baseline; a total of 909 participants completed both surveys. A small portion of survey participants (16.2%) attended in-state EBDM training shortly after the initial survey, which included encouragement and training on how to use research evidence.

Measures

Survey development was based on a literature review,26 the study team's prior research, 5 rounds of expert input, cognitive response interviews with 11 former chronic disease directors, and test-retest reliability with 75 state health department staff members working in chronic disease prevention in a variety of states. Details and the conceptual framework have been provided previously.22–24

The 65-item survey addressed demographics, research evidence use, and organizational supports. Participants were asked “how often do you use research evidence to” do each of 6 job tasks, where 0 = seldom or never, 1 = sometimes, 2 = often, 3 = always, and 8 = not relevant to my role. The 6 job tasks were as follows: “write a grant application”; “plan or conduct a needs assessment”; “select policies, programs, or other types of interventions”; “justify selection of interventions to funders, agency leadership, or external partners”; “evaluate interventions”; and “develop materials for local public health agencies or external partners.” After removal of “not relevant to my role” responses, mean frequencies were calculated for each job task at time 1 (see the Supplementary Digital Content Table, available at http://links.lww.com/JPHMP/A480). Research evidence use at time 1 (2014) and time 2 (2016) summary variables were calculated for each individual as the mean of research evidence use frequencies in 6 job tasks after removal of “not relevant to my role” responses. Dichotomous summary variables were then created as the highest tertile of summary research evidence use scores versus all else (vs the middle and lowest tertile scores) because the dependent variable was skewed to the right (to the “agree” side), and odds ratios are readily interpretable. Perceived organizational supports addressing 4 domains identified through a literature review26 were assessed in thirteen 7-point Likert items, where 1 = strongly disagree and 7 = strongly agree. Table 1 shows item wording for the 4 domains: supervisory EBDM expectations; access to evidence and resources for EBDM; participatory decision making; and agency leadership encouragement. Organizational support item scores were dichotomized to strongly agree/agree (6 or 7) versus others.

TABLE 1

Public Health Practitioner Perceptions of Agency Characteristics (N = 872)

Organizational Supports for EBDM	Percentage Strongly Agree or Agreea
Supervisory expectations
SE1: My direct supervisor expects me to use EBDM	59.9
SE2: My direct supervisor recognizes the value of management practices that facilitate EBDM	61.9
SE3: My performance is partially evaluated on how well I use EBDM in my work	28.7
Access to evidence and resources for EBDM
ACC1: My work unit has access to current research evidence for EBDM	57.8
ACC2: Informational resources (eg, academic journals, guidelines, and toolkits) are available to my work unit to promote the use of EBDM	52.2
ACC3: My work unit currently has the resources (eg, staff, facilities, partners) to support application of EBDM	36.1
ACC4: The staff in my work unit has the necessary skills to carry out EBDM	40.5
Participatory decision making
PDM1: When decisions are made within my work unit, program staff members are asked for input	60.9
PDM2: Information is widely shared in my work unit so that everyone who makes decisions has access to all available knowledge	50.1
PDM3: My work unit engages a diverse external network of partners that share resources for EBDM	44.9
Agency leadership encouragement
ALE1: Top leadership in my agency (eg, agency head, state health officer, deputies) recognizes the value of EBDM	63.4
ALE2: Top leadership in my agency encourages use of EBDM	55.1
ALE3: My agency is committed to hiring people with relevant training in the core disciplines in public health	48.2

Abbreviations: ACC, access to evidence and resources; ALE, agency leadership encouragement; EBDM, evidence-based decision making; PDM, participatory decision making; SE, supervisory expectations.

aPercentage that agreed or strongly agreed (6 or 7 on a Likert scale of 1 = strongly disagree to 7 = strongly agree).

Statistical analyses

Data management was conducted using SPSS 24. LCA was employed in SAS 9.4 to identify distinct, mutually exclusive subgroups of participants based on the 13 dichotomous variables of organizational supports.27,28 Six models were fitted to the data for 2 to 7 latent classes. Model selection took parsimony, interpretability, fit statistics, and model identification into account. Akaike information criterion, Bayesian information criterion, and adjusted Bayesian information criterion were used to compare the relative fit of the models. Lower values indicate a better fitting model. To check model identification, parameters were estimated from 100 random starting values for each model.21 Model identification was the proportion of the maxima in all log-likelihood solutions. Inclusive LCA is a recent method that includes the outcome variable (mean research evidence use at time 2 across tasks) as a covariate and may be superior to noninclusive LCA.28 By adjusting for the outcome variable and other characteristics, inclusive LCA can produce more accurate parameter estimates and improve estimate attenuation of the association between latent class membership and the outcome. After model selection, the optimal latent class model was then adjusted for the outcome variable, the 12 participating states, and job position.

To investigate the relationship between research evidence use at time 2 and class membership, 2 commonly used classify-analyze approaches were employed.27,28 The first approach is the maximum probability assignment, which assigns the class membership to individuals based on their maximum posterior probabilities. The second approach is referred to as multiple pseudo-class draws, which accounts for the uncertainty of class membership by randomly assigning class membership based on the distribution of posterior probabilities. Typically, the random assignment is repeated 20 times, and subsequent analysis is conducted in each repeat (as done here). Results are then combined according to rules of multiple imputation for missing data.29 Class membership was then incorporated as a predictor into a generalized linear mixed model in PROC GLIMMIX, with dichotomous research evidence use at time 2 as the outcome variable. The model included state as a random effect because of the sampling design and adjusted for individual characteristics (eg, research evidence use at baseline, education, position). Compared with the maximum probability assignment approach, the multiple pseudo-class draws approach takes the uncertainty of class membership into account27 and was therefore assumed to be less biased. However, some studies indicated that the maximum probability assignment approach had better performance on eliminating biases of point estimation.27,28 It has also been suggested that the performance of both approaches would be similar in the inclusive LCA.28 All analyses were conducted in SAS 9.4 with PROC LCA package.30

Results

Participants

Analyses were conducted with 872 of the 909 participants with complete research evidence use and organizational support data (70.5% of baseline sample). Most (79.2%) were female, nearly two-thirds (65.6%) had a graduate degree, and nearly half (48.9%) were 50 years or older. Of the 872 survey participants, 49.3% worked in state health departments, 11.4% in local or district health departments, 9.8% in universities, and 29.5% in other partnering organizations. On average, participants had worked in their current position for 5.8 ± 5.5 years, in their agency 10.1 ± 7.8 years, and in public health 16.4 ± 12.7 years. Nearly half (47.6%) worked in program manager positions, 19.6% in leadership positions, 28.1% in specialist positions such as health educators, epidemiologists, or program evaluators, and 4.7% in other positions. Participants most commonly used research evidence to select or justify interventions (see the Supplemental Digital Content Table, available at http://links.lww.com/JPHMP/A480). Perceptions of organizational supports are shown in Table 1.

Latent class model selection

The LCA model fit statistics are presented in Table 2. The models with 5 or more latent classes had poor identification and therefore could not be the optimal models. The model with 4 latent classes was selected as the best model, as it had adequate model information criteria, good identification, and interpretability. Entropy was 0.86, which indicates low error in class membership assignment based on the maximum posterior probabilities. The inclusive LCA was adjusted for the outcome variable research evidence use at time 2 (P < .001), state (P < .001), and job position (P < .001).

TABLE 2

Fit Statistics for Latent Class Models of Organizational Supports Reported by Public Health Practitioners (N = 872)

n	LL	G2	AIC	BIC	aBIC	Entropy	df	Identification of Latent Classes
2	−6453.95	3316.68	3370.68	3499.79	3414.05	0.84	8164	100%
3	−6158.42	2725.61	2807.61	3003.68	2873.48	0.84	8150	75%
4	−5994.24	2397.24	2507.24	2770.26	2595.59	0.86	8136	94%
5	−5919.26	2247.28	2385.28	2715.25	2496.12	0.87	8122	13%
6	−5850.18	2109.13	2275.13	2672.05	2408.46	0.86	8108	24%
7	−5790.66	1990.08	2184.08	2647.95	2339.90	0.87	8094	9%

Abbreviations: aBIC, adjusted Bayesian information criterion; AIC, Akaike information criterion; BIC, Bayesian information criterion; df, degrees of freedom; G2, Likelihood ratio statistic; LL, log likelihood.

Classes’ description

The item response probabilities of organizational support indicators among 4 classes are shown in the Figure. Class 1 (24.1% of the total sample) was labeled as unsupportive workplace, with low probabilities in all domains of organizational supports. Class 2 (18.2% of the total sample) was labeled as low agency leadership support, with low probabilities in agency leadership support and moderate probabilities in the other 3 domains of organizational supports. Class 3 (31.5% of the total sample) was labeled as high agency leadership support, with high probabilities in agency leadership support and moderate probabilities in the other 3 domains of organizational supports. Class 4 (26.2% of the total sample) was labeled as supportive workplace, with high probabilities in all 4 domains of organizational supports. The proportion of participants in each class scoring in the highest tertile evidence use was 22.8%, 31.4%, 42.8%, and 44.5%, respectively. We found the same classes and patterns of organizational supports in separate basic LCAs in the subsample of only state health department and local health department staff and in the subsample of only state health department staff.

Item Response Probability of Organizational Supports for Evidence-Based Decision-Making Among Latent Classesa

Abbreviations: ACC, access to evidence and resources; ALE, agency leadership encouragement; PDM, participatory decision-making; SE, supervisory expectations. aAdjusted for evidence use at time 2, state, and job position.

Relationship between research evidence use at time 2 and class membership

The odds ratios of scoring the highest tertile research evidence use at time 2 with 2 different classify-analyze approaches are summarized in Table 3. The unsupportive workplace class was set as the reference. In the generalized linear mixed modeling, research evidence use at baseline (P < .001) and having a master or doctorate degree in any field (P = .007) were incorporated as covariates. Other individual characteristics, such as gender (P = .42), job position (P = .99), and public health master's or doctorate (P = .13), and having a nursing degree (P = .15), were not incorporated in the final models. According to the results from the maximum probability assignment approach, participants in the high agency leadership support class were 2.08 (95% CI, 1.35-3.23) times more likely to score the highest tertile research evidence use at time 2 than the unsupportive workplace class. Participants in the supportive workplace class were 1.74 (95% CI, 1.10-2.74) times more likely to score the highest tertile research evidence use at time 2 than unsupportive workplace class members. However, the low agency leadership support class does not differ from the unsupportive workplace class in research evidence use at time 2 with odds ratio = 1.15 (95% CI, 0.69-1.92). The multiple pseudo-class draws approach produced similar results (Table 3).

TABLE 3

Odds Ratios of Highest Time 2 Research Evidence Usea Among Classes of Public Health Practitioners

Latent Classes	Adjusted Odds Ratios of Highest Research Evidence Use at Time 2 (95% Confidence Interval) (N = 872)
	Maximum Probabilityb	Multiple Pseudo-Class Drawsb
Unsupportive workplacec	1.00	1.00
Low agency leadership support	1.15 (0.69-1.92)	1.25 (0.73-2.15)
High agency leadership support	2.08 (1.35-3.23)d	2.09 (1.31-3.30)d
Supportive workplace	1.74 (1.10-2.74)d	1.74 (1.07-2.82)d

aHighest research evidence use at time 2 is the highest tertile of the mean of frequencies of research evidence use in 6 job tasks at time 2.

bThe 2 approaches were adjusted for baseline research evidence use and education, with state as a random effect.

cThe class unsupportive workplace was set as the reference.

dBoldface indicates statistical significance (P < .05).

Discussion

In the present study, one of the few longitudinal inquiries on this topic, LCA methods were useful in identifying meaningful patterns of organizational supports for research evidence use with actionable implications. Participants working in organizational environments with high perceived agency leadership encouragement, supervisory expectations, access to evidence and resources, and participatory decision making at baseline were nearly twice as likely to frequently use research evidence for multiple job tasks at time 2 compared with participants with low perceived supports in these 4 domains.

Those in organizations with high perceived agency leadership encouragement, middle-high supervisory expectations, and middle other supports were also twice as likely to have the highest use of research evidence at time 2 compared with those in unsupportive workplaces. This finding implies that leadership support at the agency level in combination with moderate levels of work unit supervisory expectations for EBDM, access to evidence and resources for EBDM, and participatory decision making can further the use of research evidence in practice, including selection and justification of interventions.

Supervisory expectations alone, without high agency leadership encouragement, did not predict research evidence use. In a related interview study, state health department middle managers and program managers working in chronic disease prevention deemed it essential to also set up internal procedures and policies within agencies to support use of EBDM processes including use of research evidence in decision making.31 Examples included centralized surveillance databases with easy-to-use data access platforms readily available to staff and external agencies, internal performance management systems, agency-wide digital access to intervention evidence, internal policies and procedures requiring evidence-based justification of intervention selections, and use of research evidence in grant applications, requests for proposals, and contracts with external agencies.

High agency leadership in combination with high or moderate additional organizational supports was associated with increased research evidence use at time 2 in this study. Earlier studies also found leadership support a key facilitator of research evidence use, broader evidence use, and EBDM.3–8,14–16 In interviews in 6 European Union countries, Van de Goor and colleagues5 found leadership essential. As found in the present study, interview participants in Victoria, Australia, deemed senior managers the most influential. They stated that senior management sponsorship of projects helped ensure that research evidence would be used in decision making.7 In Denmark, a survey with 98 local health managers found health manager emphasis on evidence use was significantly correlated with evidence use for prioritization.32

In addition to leadership support, earlier studies identified organizational access to evidence as a facilitator to evidence use or a barrier if lacking.4,10,19,20 In the present study, perceptions of moderate or high access to evidence and other resources in combination with other domains were associated with increased evidence use. Van de Goor and colleagues5 also found lack of locally useful evidence and intervention cost-effectiveness evidence was a barrier.5 Ability and confidence to assess, interpret, and use the research also affected the gathering and use of research evidence. These aspects were not addressed in the present study but are important. Recommendations included collaborative learning approaches to build skills in research use,20 practitioner engagement in creation of local evidence,33 increased collaboration between researchers and practitioners so that research is geared to meet practitioner decision-making needs,8 and use of stories to highlight key messages in succinct and memorable ways.8,33

Research alone is not enough to drive the complex and politically influenced processes of EBDM.3,4,17 Political priorities, policymaker beliefs, community views, social norms, budget constraints, influences of external funder organizations and special interest groups, and opinions of selected experts and managers all play a part in public health decisions, including policy development.3,8,17,32 During the study period, federal funding agencies increasingly required evidence-based practice and the national movement toward accreditation of public health departments accelerated. Accreditation applications required health assessment, strategic planning, and a stronger focus on evidence-based practice. A better understanding of policy-making processes and influences, more practice-based research, more locally relevant research, and stronger trust, relationships, and linkages between researchers and decision makers are recommended in the literature to better address the political contexts in which public health decision making takes place.4,17,20,34

Additional influences noted in the literature include individual factors such as skills and confidence in one's ability to gather and use evidence3,6,7,14; characteristics of the available research itself, including complexity, lack of timeliness, and especially relevance for the specific locality, population group, or programmatic purpose4,5; and relationships and collaboration of practitioners with researchers such as through knowledge brokers.4,10 Interview participants also noted that the high staff turnover in health departments is disruptive to using evidence in decision making.4

Type of organization and job position were not associated with the frequency of public health research evidence use in the present study. However, in an Australian study with agencies that develop public health policies, Zardo and Collie6 found that senior managers were more likely to use research evidence than those in middle manager or nonmanager positions.6 Similar to the findings of Zardo and Collie,6 the present study found those with a master's or doctoral degree were more likely to frequently use public health research evidence than those without graduate degrees.

This study has several limitations. The data were self-reported responses from an online survey. With funders’ emphasis on EBPPs, there could be social desirability bias in participants’ responses. Frequency of evidence use was only asked for research evidence, although practitioners use a variety of types of evidence.23,34,35 Many participants responded that 1 or more of the 6 listed job tasks were not relevant to their job positions; other uses of evidence may have been missed. Additional organizational supports for research evidence use such as departmental quality improvement sections and performance management data systems were not asked. A more ideal study design would have been to survey staff from a single agency type, but our sample of state health department staff or any other single agency type was too small for inclusive adjusted LCA. Models were not adjusted for organizational demographics (such as the number of employees) and structures, although the staff from diverse organizations were included in the sample. However, we found the same classes and patterns of organizational supports in basic LCAs, with the subsample of only state and local health department staff and with the subsample of only state health department staff. And 37 individuals with missing data in research evidence use or organizational supports were excluded from analyses. In addition, the latent class model cannot handle missing data in covariates. In generalized linear mixed modeling, individuals with missing data in any predictors were excluded from analyses by PROC GLIMMIX. This study did not explore how research evidence use in decision-making processes may be affected by external political influences such as special interest advocacy groups or programmatic requests from elected officials.

Four distinct latent classes of organizational supports were identified with LCA in a sample of chronic disease prevention staff from 12 state public health departments and partnering organizations. The results suggest that agency leadership support may be a crucial element of organizational supports to encourage research evidence use. Other organizational supports such as supervisory expectations, access to evidence, and participatory decision making may need the backing of leadership to further research evidence use in public health job tasks.

There are several implications for research from this study and from the literature. Because of the limitations of self-report close-ended organizational support measures, next steps for research could include an expert practitioner panel to review and operationalize the classes found in this survey study to further identify gaps in organizational supports for EBDM and management practices to address gaps. Increased agency leadership encouragement to use research evidence in decision making can further public health practice. Availability of directly relevant and simply presented research evidence (including economic evaluation data on cost-effectiveness) as requested by interview participants in other studies5 could increase evidence use in public health decision making. Oliver and colleagues4,34,36 emphasize the need to learn more about public health decision making, policy-making processes, and how evidence is used at different stages of decision making. Learning more about these processes could help tailor approaches to increase research evidence use.

Organizational supports identified in this study (eg, leadership encouragement, access to evidence, participatory decision making) are modifiable at relatively low cost.26

Organization support measures used in this study can also be applied in practice settings to identify strengths and gaps in agency processes and commitment to EBDM.

Public health agencies can increase the use of research evidence in grant applications, selection and justification of EBPPs, and evaluation through agency leaders valuing and encouraging EBDM and supervisors communicating expectations to staff.24

Leaders can also increase access to evidence and resources for EBDM, provide skill-building opportunities, and ensure staff participation in decision making.

Organizations supporting the public health workforce can disseminate evidence (including practice-based and locally relevant) through governmental Web sites, national guidelines, archived webinars, and workshops, which earlier studies found participants preferred as evidence sources.23,34