Do different uses of performance measurement systems in hospitals yield different outcomes?

BACKGROUND: Inspired by the new public management movement, many public sector organizations have implemented business-like performance measurement systems (PMSs) in an effort to improve organizational efficiency and effectiveness. However, a large stream of the accounting literature has remained critical of the use of performance measures in the public sector because of the inherent difficulty in measuring output and the potential adverse effects of performance measurement. Although we acknowledge that PMSs may indeed sometimes yield adverse effects, we highlight in this study that the effects of PMSs depend on the way in which they are used.
PURPOSE: The aim of this study was to investigate various uses of PMSs among hospital managers and their effects on hospital outcomes, including process quality, degree of patient-oriented care, operational performance, and work culture.
METHODOLOGY: We use a survey sent to 432 Dutch hospital managers (19.2% response rate, 83 usable responses). For our main variables, we rely on previously validated constructs where possible, and we conduct ordinary least squares regressions to explore the relation between PMS use and hospital outcomes.
RESULTS: We find that the way in which PMSs are used is associated with hospital outcomes. An exploratory use of PMS has a positive association with patient-oriented care and collective work culture. Furthermore, the operational use of PMSs is positively related to operational performance but negatively related to patient-oriented care. There is no single best PMS use that positively affects all performance dimensions. PRACTICE IMPLICATIONS: The way in which managers use PMSs is related to hospital outcomes. Therefore, hospital managers should critically reflect on how they use PMSs and whether their type of use is in line with the desired hospital outcomes.

Inspired by the new public management movement (Hood, 1995), many public hospitals in countries such as the United Kingdom, the Netherlands, and New Zealand have implemented business-like performance measurement systems (PMSs) in an effort to improve efficiency and effectiveness (Pollitt & Bouckaert, 2011). Although PMSs may, under some circumstances, indeed contribute to achieving organizational goals, the literature also shows that PMSs are sometimes ill suited for organizations in the public domain and may even yield adverse effects (cf. Frey, Homberg, & Osterloh, 2013; Speklé & Verbeeten, 2014). Examples of adverse effects include more formalization and bureaucratization, increased stress, and lower work satisfaction (Diefenbach, 2009). Although it has been suggested that the strength or existence of effects on various outcomes may be related to the way in which PMSs are used (Abernethy & Brownell, 1999; Naranjo-Gil & Hartmann, 2007a; Speklé & Verbeeten, 2014), there is no conclusive understanding about the use of PMSs in health care settings and the potential effects they have on hospital outcomes. Addressing this gap in the literature is important for at least two reasons. First, setting up and maintaining a PMS require substantial human, technological, and financial resources, and health care organizations might assign lower priority to PMSs when faced with resource constraints if it is not clear what the added value of such a system would be (cf. Zidarov, Poissant, & Sicotte, 2017). Second, given the delicate setting and the potential impact of negative effects on hospital outcomes such as quality of care, it is of paramount importance to understand the relations among different types of PMS use and key hospital outcomes.

In this article, we extend the line of research that studies the use of PMSs in the public sector by exploring how different PMS uses by hospital managers relate to various hospital outcomes, such as process quality (PQ) and operational performance (OP). By doing so, we respond to a recent call by Zidarov et al. (2017, p. 149), who stated that “[f]uture studies are needed to evaluate the real impact of the use of such performance indicators by health organizations on the quality of care.” Our findings indicate that PMS use may lead to positive outcomes for hospitals, if used in an appropriate manner, which matches the specific purpose. We find that an operational use of the PMS (e.g., for operational planning, budgeting and monitoring) is associated with increased OP but also with lower degrees of patient-oriented care (POC). Using the PMS for exploratory purposes (e.g., to learn and to develop goals and policies) is positively associated with the collective work culture (CWC), as well as with POC. Hospital managers thus need to make choices regarding PMS use that provide a proper fit with the objectives they want to achieve.

The remainder of this article is organized as follows. In the next section, we discuss the theoretical background of our study. The third section presents our research methods, and in the fourth section, we present our main findings. In the final section, we discuss our findings and conclude with our main contributions, limitations, and suggestions for further research.

Background

PMSs in the Public Sector

PMSs have become commonplace in many public sector organizations, partly replacing the rule-and-procedure-driven management control on which public sector organizations traditionally relied. Prior literature, for instance, reported the use of PMSs in a wide range of public sector organizations, such as municipalities (Verbeeten, 2008), provinces (ter Bogt et al., 2015), and hospitals (Lachmann, Trapp, & Wenger, 2016). One of the main reasons for this increased use of PMSs in public sector organizations is that it allegedly results in higher efficiency, effectiveness, and service levels, as advocated by the new public management movement (Hood, 1995; Pollitt & Bouckaert, 2011). The importance of PMSs has also been pronounced in the health care management literature. For instance, Naranjo-Gil (2009) studies the use of a very specific PMS, the balanced scorecard, and finds that an effective use of a PMS stimulates managerial dialogue about financial and nonfinancial performance.

Naranjo-Gil and Hartmann (2007b) studied the Spanish hospital setting and found that having performance measures that are future oriented and include nonfinancial information can positively contribute to the extent of strategic change. They also found that a more heterogeneous management team likely leads to an interactive (learning-oriented) use of performance measurement, which also relates to the extent of strategic change. Furthermore, Demartini and Trucco (2017) show that the strategic use of PMSs is related to process improvement in a health care context.

Although PMSs can indeed be associated with positive attributes such as organizational learning, strategy implementation, psychological empowerment, and role clarity (cf. Franco-Santos, Lucianetti, & Bourne, 2012), critical studies have also pointed to potential adverse effects of the implementation of PMSs, particularly in the public sector. For example, on the organizational level, PMSs have been associated with more formalization and bureaucratization (Diefenbach, 2009), and on the employee level, the use of PMSs may result in lower intrinsic motivation (Frey et al., 2013), cognitive dissonance (van der Kolk & Kaufmann, 2018), stress, and lower work satisfaction (Diefenbach, 2009). Employee-level effects such as lower work satisfaction and stress are likely to translate in time to departmental-level and hospital-level effects on performance (cf. Schermerhorn, Hunt, & Osborn, 2002). In other words, the introduction of a PMS is not always beneficial for an organization operating in the public sector, and its effectiveness likely depends on a range of mediating variables and characteristics of the PMS. Furthermore, various studies have added that the way in which performance information is used should also be on the research agenda because this also strongly influences its potential effects on organizational outcomes (e.g., Abernethy & Brownell, 1999; Naranjo-Gil, 2009; Speklé & Verbeeten, 2014).

Different Types of PMS Use

The extant accounting literature distinguishes between the design of a PMS and the way in which it is used (cf. Naranjo-Gil & Hartmann, 2007a, 2007b; Simons, 1995). Design entails aspects related to the types of performance metrics that are used, how many performance metrics are used, and what the targets are for each of the performance metrics. Although decisions about the design of PMSs are often made at a higher level in an organization, decisions about how PMSs are used are frequently made by unit managers, lower in the organization. Our focus is on the way in which a PMS is used, for instance, how often results from performance measures are discussed by a unit manager with a subordinate and whether the performance metrics are also used for internal learning (cf. Naranjo-Gil & Hartmann, 2007a).

Although evidence on the relation between PMS use and hospital performance is relatively limited, there are a few studies in the accounting literature that suggest that PMS use indeed matters for hospital outcomes. For instance, Abernethy and Brownell (1999) showed in a study based on Australian hospital managers that the way in which a PMS is used relates to hospital performance. This study is helpful for understanding PMS in a hospital context, by suggesting a relationship between PMS use and performance, and by proposing to use Simons' (1995) framework to assess different PMS uses. Abernethy and Brownell (1999) focus in their study, however, on the (financial) budget-related aspects of a PMS and do not distinguish between the different dimensions of performance (for instance, quality of care, reputation, and cost-effectiveness are all combined in the same performance construct), leaving the question open whether different uses of a PMS lead to different hospital outcomes.

Simons (1995) distinguishes between the “interactive” and “diagnostic” uses of a PMS and argues that the different uses of a PMS work together to manage tensions in an organization, such as the tension between creativity and control (see also Speklé, van Elten, & Widener, 2017). Speklé and Verbeeten (2014) interpret Simons' (1995) “interactive” use as a form of “exploratory use,” aimed at organizational learning, while further splitting “diagnostic” use into “operational use” and “incentive-oriented use,” which are respectively used to pursue operational efficiency and to align the motivation of employees with organizational goals. This results in three clearly defined types of PMS use in a public sector setting: exploratory use, operational use, and incentive-oriented use. In the remainder of this article, we will draw on this categorization to distinguish between the different types of PMS use.

Exploratory PMS use

An exploratory use of PMS can best be understood as the use of a PMS to learn and improve—as an organization or as a professional. This typically involves the discussion of results with organizational members to understand why they are as they are and stimulates organizational dialogues about whether the right aspects of performance are indeed measured. Tucker, Nembhard, and Edmondson (2007, p. 4) argue that organizational learning is crucial for health care organizations that need to “adjust their practices to reflect advances in knowledge and technology.” Abernethy and Brownell (1999, p. 189) find that the exploratory use of PMS “as a learning machine” may mitigate disruptive performance effects of a strategy change in Australian hospitals. Naranjo-Gil and Hartmann (2007a, 2007b) find that the implementation of strategic policies in Spanish hospitals aimed at improving quality are specifically supported by this type of interactive, exploratory use of performance measurement information. Furthermore, Speklé and Verbeeten (2014) report a positive relation between exploratory use and public sector performance, irrespective of the type of tasks workers carry out.

Operational PMS use

Operational PMS use refers to managerial reliance on performance measures for operational planning, budget allocation, and process monitoring. This relates to what Naranjo-Gil and Hartmann (2007a, 2007b) describe as the “typical” (administrative) way of dealing with performance information, that is, aimed at supporting operational-, business-, and finance-related objectives such as cost reduction and budget control. Prior research maintains that an operational use of PMS is common in most organizations and is positively associated with performance (cf. Hansen & Van der Stede, 2004). Furthermore, Speklé and Verbeeten (2014) indicated that an operational use of PMS is more in line with “single-loop learning” and a shorter-term orientation, whereas an exploratory use would allow for “double-loop learning” and a longer-term orientation.

Incentive-oriented PMS use

This type of PMS use aims to align the motivation of the employees with the organizational goals. The underlying assumption of the incentive-oriented use of PMS in health care organizations is that such a use would lead to more effective and efficient delivery of services and a higher level of service quality, in line with the expectations of new public management movement (Hood, 1995). Incentive systems that couple performance and (implicit) rewards are increasingly used in health care organizations (Smalarz, 2006), and Cardinaels (2009) found that such pay-for-performance practices and OP correlated in a hospital setting. Verbeeten (2008) only finds support for a positive relation between incentive-oriented uses of PMS and quantitative performance (e.g., productivity) and not for qualitative performance (e.g., accuracy and reputation). Speklé and Verbeeten (2014), however, found a strong negative direct relation between incentive-oriented use and public sector performance, and in a similar vein, a strict use of PMS to incentivize employees has been associated with “lower employee motivation and more opportunistic behavior” (van der Kolk, ter Bogt, & van Veen-Dirks, 2015, p. 956), which may harm the collective culture within a unit as employees may focus more on their individual incentives and performance than on the collective.

Taken together, the overall claim of the PMS literature seems to be that the very same PMS system can be used in different ways, each of which potentially has different effects on organizational outcomes. Hence, because the effects of PMSs depend on the way they are used, prior literature called for “a more situation-dependent approach to performance measurement” (Speklé & Verbeeten, 2014, p. 143). Therefore, it is essential to take the peculiarities of the health care setting into account with the multidimensional conceptualization of performance leading the way.

Performance Is Multidimensional

Performance in health care organizations is multidimensional because multiple—potentially interrelated—objectives are pursued. Although the extant literature makes various distinctions between these dimensions, a central tenet is the distinction between financial and nonfinancial performance. Nonfinancial performance involves employee interests, internal processes, and client orientation, which is reflected in performance measures such as diversity climate and workforce diversity (Weech-Maldonado et al., 2018), operational efficiency and effectiveness (Al-Amin, Makarem, & Rosko, 2016), quality of care (Upadhyay et al., 2018), and perception of care performance (Oppel, Mohr, & Benzer, 2017). Similarly, Bos, Boselie, and Trappenburg (2017) distinguish financial performance, employee well-being, and client well-being. Their systematic review of for-profit versus not-for-profit organizations shows that different performance dimensions can conflict and that financial performance would fit a profit-focused environment, whereas nonfinancial, employee and patient performance dimensions seem more relevant in a not-for-profit setting.

The purpose of the current article is to contribute to the literature by further exploring how different types of PMS use in a not-for-profit hospital affect hospital outcomes. Although we acknowledge that not-for-profit hospitals may set specific budgeting or revenue targets for themselves, “profitability” as such is considered of less importance in this setting compared to a for-profit hospital. Therefore, we will particularly pay attention to four outcomes that are considered important for public sector hospitals: first, PQ, capturing the way health care activities are executed, due to the growing importance of implementing and adhering to evidence-based guidelines (cf. Lugtenberg, Burgers, & Westert, 2009); second, POC, referring to the empathic and patient-centered degree of health care delivery, because there is an increasing trend toward involvement of patients within the care process (cf. Sacristán, 2013); third, OP, focusing on production targets and efficiency concerns, because production targets are important in selective contracting between health purchasers and health care providers (cf. van de Ven & Ferry, 1980); fourth and finally, the CWC in an organization as it has intrinsic value to the employees of the organization.

Given the current absence of (conclusive) evidence and hence the exploratory nature of our study, we do not formulate hypotheses here but rather empirically examine how the three types of PMS use (exploratory use, operational use, and incentive-oriented use) relate to the four hospital outcomes defined above. The following section conveys information about our research method.

Method

Setting

The empirical context of our study is the Dutch hospital setting, which has been at the forefront of implementing business-like practices such as PMSs (Hood, 1995; Pollitt & Bouckaert, 2011). The Dutch health care system and the hospital sector, in particular, rely on competitive elements to incentivize affordable but high-quality care. Health insurers, for instance, are able to engage in selective contracting based on the price and the quality of care hospitals provide. Hospital performance is therefore an important determinant during the purchasing negotiations between health insurer and health provider.

In the Netherlands, there are 113 hospitals: 83 general hospitals, 22 specialist hospitals, and 8 university medical centers. Together, these hospitals serve the 17 million inhabitants of the Netherlands and employ approximately 130,000 full-time equivalent (FTE) employees, according to Dutch Hospital Association. Dutch hospitals are private organizations with a nonprofit orientation. Generally, hospitals consist of cure and care units, as well as support departments (such as accounting or human resources management). In our study, we focus on the use of PMS at the unit level, and we do so by surveying hospital managers who are in charge of a medical unit.

Study Design

For our empirical analyses, we use a purpose-developed survey to collect data from hospital managers. We are interested in intraunit control, that is, we study whether differences in PMS use by hospital managers (to control their own unit) yield different outcomes at the unit level. We survey hospital unit managers using a convenience sample, a commonly used survey sampling method in an effort to “generate insights that help us further develop the theory” (Speklé & Widener, 2018, p. 4). Five research assistants with a background in health care management identified and approached 432 managers who were at the time of the study (April 2018 to June 2018) responsible for a medical unit in a Dutch hospital. These hospital unit managers were invited to participate in our survey study and received a link to the online survey instrument. Potential respondents were informed about the content of the questionnaire as well as the purposes of the study. Verbal informed consent was obtained from all participants, and all participants agreed to participate voluntarily; they were free to quit at any time during the research. We sent up to three reminders to managers who did not respond in the first and second calls, which eventually resulted in a response rate of 19.2% (83 responses). Note that our study was outside the scope of the Netherland’s Medical Research Involving Human Subjects Act because it did not concern “medical/scientific research” about illness and health, nor did the content or methods cause “an infringement of the physical and/or psychological integrity” of the participants. Therefore, according to Dutch law, no formal ethical approval was needed.

The surveys were conducted in Dutch to ensure that our respondents understood the questions and answers well, and we pretested the survey to increase the reliability and face validity of the survey instrument. The three pretest interviews with experts in hospital management and survey research led to only minor changes in the wording of the items.

Because we rely on one survey instrument for the collection of all our data included in this study, we followed generic recommendations (e.g., Speklé & Widener, 2018) to reduce the risk of common method bias, including guaranteeing anonymity and separating the measurement of predictor and criterion variables. We conducted Harman’s single-factor test to address this. Exploratory factor analyses on all included survey items yielded five factors with eigenvalues greater than 1, of which the first factor explained 27.3% of the variance of all included items. This suggests that common method bias is not a significant threat to the reliability of our findings (cf. Podsakoff & Organ, 1986).

Variable Measurement and Data Analysis

The survey instrument draws on previously validated constructs where possible (e.g., King, Clarkson, & Wallace, 2010; Speklé & Verbeeten, 2014) and questionnaire items that are tailored to the specific situation of hospital management. All main variables are measured using a 5-point Likert scale, where a higher score means “more” of the measured construct (e.g., more PMS use).

PMS Use

We use the instrument developed by Speklé and Verbeeten (2014) to measure exploratory PMS use, operational PMS use, and incentive-oriented PMS use. These three types of PMS use are not mutually exclusive (i.e., it is possible to score high on both operational use and incentive-oriented use), and a score for each type of use is calculated by averaging the item score of questions related to the exploratory, operational, and incentive-oriented use of various aspects of performance.

The respondent indicated the perceived importance of the input measures for eight different purposes (e.g., “How important are input measures for operational planning within your unit?” “How important are input measures for career decisions within your unit?”). Equivalently, and for the same eight purposes, the respondent indicated the perceived importance of output measures (e.g., “How important are output measures for operational planning within your unit?”), process measures (e.g., “How important are process measures for operational planning within your unit?”), and quality measures (e.g., “How important are quality measures for operational planning within your unit?”). All items are assessed on a 5-point Likert scale. The four different types of performance measures (input, output, process, quality) are averaged into a first-order construct for each of the eight different purposes. The reliability and dimensionality statistics for each purpose are reported in Table 1, which shows, on each row, the first order factor loadings for the perceived importance.

Table 1

Importance of input, output, process, and quality measures for eight different purposes: Factor analysis for first-order constructs for performance measurement system use

	Input measures (loading)	Output measures (loading)	Process measures (loading)	Quality measures (loading)	α	Factor analysis (AVE)
Operational planning	.802	.832	.848	.638	.777	61.6%
Budget allocation	.814	.871	.822	.801	.844	68.5%
Monitoring of processes	.801	.872	.804	.641	.786	61.5%
Career-related decisions	.805	.870	.884	.701	.822	66.9%
Financial rewards	.864	.858	.834	.686	.806	66.2%
Goal communication	.821	.766	.816	.699	.778	60.4%
Assessing objectives/policies	.745	.748	.749	.755	.736	56.2%
Revising the unit’s policies	.778	.619	.816	.700	.702	53.6%

Note. Factor analysis based on N = 83 surveys. The importance of input (output, process, and quality measures) is assessed with one item per purpose, that is, Cronbach’s α is based on four items. AVE = average variance extracted.

In our analyses, the first-order constructs are combined into second-order PMS uses (“exploratory PMS use, operational PMS use, and incentive-oriented PMS use”), which are included in Table 2. Consistent with Speklé and Verbeeten (2014), factor loadings and values for Cronbach’s α indicate that the individual survey items load consistently and reliably on the three types of PMS use. The second-order constructs included in Table 2 are used in the analyses of the regression model.

Table 2

Main constructs used in regression analyses (N = 83)

Construct	Items	Factor loading
Process quality	Quality of care	.779
Factor analysis AVE: 63.5%	(Process) innovations and new ideas	.757
Cronbach’s α: .697	Evidence-based care	.851
Patient-oriented care	Empathic care	.725
Factor analysis AVE: 58.0%	Patient-centered care	.817
Cronbach’s α: .636	Patient satisfaction	.739
Operational performance	Productivity	.735
Factor analysis AVE: 56.8%	Realization of production targets	.678
Cronbach’s α: .609	Efficiency	.840
Collective work culture	Employees share norms and values	.691
Factor analysis AVE: 57.9%	Importance of joint meetings	.815
Cronbach’s α: .757	Aligning shared goals in meetings	.803
	Clarity of behavior expectations	.728
PMS exploratory use	Goal communication	.878
Factor analysis AVE: 82.8%	Assessing objectives and policies	.950
Cronbach’s α: .885	Revising the unit’s policies	.900
PMS operational use	Operational planning	.835
Factor analysis AVE: 63.6%	Budget allocation	.713
Cronbach’s α: .695	Monitoring of processes	.837
PMS incentive-oriented use	Career-related decisions	.908
Factor analysis AVE: 82.4%	Financial rewards	.908
Cronbach’s α: .775

Note. The three types of PMS use (exploratory, operational and incentive-oriented use) are second-order constructs. PMS = performance measurement system; AVE = average variance extracted.

Hospital outcomes at the organizational unit

We measure four different dimensions of hospital outcomes at the organizational unit: (a) PQ, (b) POC, (c) OP, and (d) the CWC. We relied on items from prior research, and we tailored some of these measures to fit the specific hospital context. We discuss the four outcomes below, and we present them together in Table 2. We measure hospital outcomes at the level of the organizational unit, because this is the relevant theoretical level to assess the potential impact of different uses of PMS by hospital managers.

PQ and POC

Campbell, Roland, and Buetow (2000) distinguish between clinical care and interpersonal care, where the former type of care is more related to internal processes at the hospital, and the latter is more related to the patient experience. Following this distinction, we consider health care services from two angles. The first angle, PQ, refers to the quality of the medical/technical process and includes three items that are related the overall medical/technical quality of care, the amount of (process) innovations and/or new ideas in the unit, and the evidence-based nature of the provided care. The second angle, POC, relates to the patient orientation and includes three questions regarding the unit’s contribution to patient satisfaction and the patient-centered and empathic nature of the provided care.

Operational performance

OP is measured using three survey items about the unit’s performance related to productivity, realization of production targets, and efficiency aspects. These three items have been developed by van de Ven and Ferry (1980) for the public sector and focus specifically on the operational, efficiency-oriented part of performance.

Collective work culture

CWC is the extent to which the culture of a unit is collectivist (rather than individualist). The collectivist work culture signals good relations among colleagues and is sometimes viewed as a predictor of future (medical) performance, and moreover, this measure is likely to relate to work satisfaction (cf. Belias & Koustelios, 2014; Schermerhorn et al., 2002). The four items that together make up CWC are adapted from Stouthuysen, Slabbinck, and Roodhooft (2017) and Kaptein (1998).

An overview of the constructs used and the items that are included in each construct can be found in Tables 1 and 2. The factor component loadings for each construct are well above .500 (with average variance extracted exceeding 50%), and the reported Cronbach’s αs are satisfactory, as they are all above .600. Together these form an indication of reliable constructs.

Control variables

We include two single-item measures for hospital type and unit size as control variables. We differentiate between hospital types by incorporating dummy variables for the types of hospitals in the Netherlands—general hospitals (22% of our sample), specialist hospitals, “top clinical” hospitals (56% of our sample), and academic hospitals (22% of our sample), with general hospitals serving as the reference category. Unit size is measured using the log-transformed number of FTE employees. We measure size from a human resource perspective because the number of employees is more indicative of control system design difficulties than capacity elements (such as the number of beds). Although we acknowledge that multi-item constructs have benefits over single items in terms of predictive validity (Diamantopoulos, Sarstedt, Fuchs, Wilczynski, & Kaiser, 2012), we preferred single-item measures for these control variables because of the factual nature of the related questions and to promote adequate response rates and brevity of the survey instrument. For the main variables of interest in this study, however, we do rely on multi-item constructs. Table 3 shows the mean, standard deviation, and correlations of all the main variables in this study.

Table 3

Correlation table and descriptive statistics

		M	SD	1	2	3	4	5	6	7	8	9
1.	Process quality	3.791	.643	—
2.	Patient-oriented care	4.004	.571	.286 ***	—
3.	Operational performance	3.735	.577	.258 **	.201 *	—
4.	Collective work culture	3.994	.605	.644 ***	.343 ***	.454 ***	—
5.	PMS exploratory use	3.871	.639	.157	.107	.244 **	.404 ***	—
6.	PMS operational use	3.791	.661	.145	−.130	.408 ***	.326 ***	.621 ***
7.	PMS incentive-oriented use	1.855	.661	.002	.076	.090	.146	.281 **	.418 ***	—
8.	Size	204.6	218.8	.142	−.117	−.035	.062	.136	.027	−.266 **	—
9.	Academic hospital	22%	n/a	.461 ***	.151	−.046	.370 ***	.226 **	.019	055	.289 ***	—
10.	Specialist hospital	57%	n/a	−.159	−.051	.104	−.140	−.265 **	.004	−.005	−.129	−.601 ***

Note. N = 83 observations (N = 81 for size). The descriptive statistics for size are based on the absolute number of full-time equivalents, and the correlations for size are based on the transformed (natural log) of size to approach normality of the distribution. Instead of the mean for the manager and hospital-type dummies, percentages are presented. PMS = performance measurement systems; n/a = not applicable.

*p = .10 level (two-tailed). **p = .05 (two-tailed). ***p = .01 (two-tailed).

Results

We conduct four ordinary least squares models to explore the relationship between PMS use and the four various hospital outcomes (PQ, POC, OP, and CWC). The results are presented in Models 1–4 in Table 4. No models show indications of high multicollinearity; all Variance Inflation Factor scores are well below 1.9. In addition, case-wise diagnostics did not reveal any outliers (>3 SDs), and the visual inspection of the residuals did not show indications of nonnormality.

Table 4

Main regression model (unstandardized coefficients)

	Model 1	Model 2	Model 3	Model 4
	DV: PQ	DV: POC	DV: OP	DV: CWC
PMS exploratory use	−.013	.280**	.034	.248**
PMS operational use	.118	−.348***	.340**	.094
PMS incentive-oriented use	−.105	.077	−.105	−.038
Size	−.020	−.086	−.039	−.049
Academic hospital dummy	.897***	.267**	−.045	.625**
Specialist hospital dummy	.276	.181	.170	.261*
Intercept	3.351***	4.349***	2.603***	2.712***
ANOVA F	4.112***	1.934*	2.454**	4.267***
R2 (Adj R2)	.250 (.189)	.136 (.065)	.166 (.098)	.257 (.197)

Note. N = 81 observations. Model dependent variables (DV): PQ = process quality, POC = patient-oriented care, OP = operational performance, CWC = collective work culture; PMS = performance measurement systems; ANOVA = analysis of variance.

*p = .10 (two-tailed). **p = .05 (two-tailed). ***p = .01 (two-tailed).

Model 1 shows that none of the PMS use types is significantly (two-tailed) related to PQ. We find that operational use of PMS is negatively associated with POC (Model 2: −.348, p < .01) but positively associated with OP (Model 3: .340, p < .05). Our results suggest that an (arbitrary) one-unit increase in operational PMS use increases OP almost by the same magnitude (+.340) as it decreases POC (−.348). In addition, our findings indicate that incentive-oriented use is not significantly related to any of the considered outcomes. Finally, for an exploratory use of PMS, our results indicate a positive relationship with POC (Model 2: .280, p < .05) and a positive relationship with CWC (Model 4: .248, p < .05). Notably, the positive effects of exploratory PMS use are comparable in magnitude, and our results do not indicate a significant difference in effect size (the 95% CIs [0.020, 0.541] and [0.011, 0.485] are overlapping).

In addition to our main results, we find that academic hospitals are associated with higher PQ (Model 1: .897, p < .01) and higher POC (Model 2: .267, p < .05) than general hospitals. Specialist and academic hospitals also score higher in terms of CWC than general hospitals (Model 4: .261, p < .10; .625, p < .05).

Discussion and Conclusions

We began this article by posing the question whether different uses of PMS would yield different effects on different hospital outcomes. Drawing on prior literature from the field of accounting, we distinguish between three types of PMSs, that is, operational use, incentive-oriented use, and exploratory use. Our findings indicate that different uses are associated with different hospital outcomes and contribute to the literature on PMS use in public sector settings (Abernethy & Brownell, 1999; Naranjo-Gil, 2009; Naranjo-Gil & Hartmann, 2007a, 2007b; Speklé & Verbeeten, 2014; Van der Kolk & Kaufmann, 2018; Verbeeten, 2008).

Specifically, our findings indicate that an operational use of PMS may harm the degree of POC but simultaneously improve the OP of a medical unit. This finding is in line with prior studies that acknowledged that the operational use of PMS is primarily short-term oriented (cf. Hansen & Van der Stede, 2004, p. 416), and we add to this literature that this type of (short-term oriented, operational) PMS use may work to enhance OP, but there might be the chance that it comes at a “cost”—a lower level of POC. Furthermore, we find that an exploratory use of PMS is positively associated with POC and with a CWC at the level of the organizational unit in a hospital. The latter association is relevant for hospitals, as work culture is often associated with higher job satisfaction (Belias & Koustelios, 2014), which in turn may positively impact other hospital outcomes, although we did not specifically examine this effect in our study.

We do not find positive or negative relations between incentive-oriented PMS use and the hospital outcomes at the level of the organizational unit. This result is, to some extent, in line with Verbeeten (2008), who also does not find a relation between an incentive-oriented use of a PMS and qualitative performance (e.g., accuracy and reputation) of public sector organizations such as hospitals. Our finding can be explained by our relatively small sample size, but alternatively, it is also likely that this type of PMS use is less prominent in the hospitals included in our study. Relative to the other types of PMS use, incentive-oriented PMS use seems to play a less pronounced role in our sample, although we do find sufficient variance to examine its role in different hospital settings. Furthermore, we did not examine financial performance in our study: A PMS that allows an incentive-oriented use may have a stronger effect on financial performance (cf. Verbeeten, 2008) than on the four types of hospital performance examined in this study. Lastly, it has been suggested in the literature that employees in the public sector are less motivated by extrinsic incentives than their private-sector counterparts (Frey et al., 2013), which may also explain the nonsignificant relation of incentive-oriented use with the four performance dimensions.

What our analyses show is that not only the presence or design of PMSs explains relations with hospital outcomes, but the type of PMS use also may affect the extent to which objectives are realized. The identified connections between PMS use and various hospital outcomes indicate that a choice of a certain type of use may thus be “value-laden,” which means that this choice may enhance specific hospital outcomes (perhaps at the expense of others). For instance, although a focus on operational PMS use may enhance OP, it seems to come at the expense of POC. This is in line with prior research that has found that PMSs may play an “expressive” role and bear organizational values (cf. Chenhall, Hall, & Smith, 2017). We complement this literature by showing how the use (not the design as such) of a PMS may play this expressive role and implicitly (or explicitly) prioritize potential organizational outcomes.

The results of this study have to be interpreted in the light of the limitations inherent to the chosen research method. Our study relies on a survey filled out by hospital managers working in the Dutch hospital context. Although this allowed us to control for cultural and institutional differences at the national level, this also means that we should be cautious in generalizing findings from this study. Furthermore, our survey yielded a response rate of 19.2%. Although this may seem relatively low, survey response rates for high-level managers have been declining over the years, indicating that it is increasingly challenging to obtain high response rates when targeting high-level managers (cf. Cycyota & Harrison, 2006). Closely related to the response rate is the absolute number of observation used in this study. The regression analyses in this study are based on 81 observations, which limit our ability to rely on more demanding econometric models (such as structural equation modeling) or to robustly test interactions between types of PMS use. We acknowledge that more data have to be collected in this area to make stronger claims regarding the antecedents, interactions, contingencies, and effects of different types of PMS use in the public sector. Also, relying on survey data introduces the risk of common method bias. We followed suggestions from previous literature to mitigate the potential effects of common method bias (cf. Speklé & Widener, 2018), and the results from Harman’s single-factor test suggest the absence of common method bias in our study. Lastly, as we rely on data from a cross-sectional survey, we cannot completely rule out reverse causality or endogeneity due to omitted variables. More research is therefore needed to validate our conclusions regarding the directionality of the effects described and to explore the role of variables not included in the current study. Field experiments and longitudinal case studies seem appropriate methods to explore these issues in more detail, and we hope that our findings can serve as an inspiration and a stepping stone for future research.

Practice Implications

Our research shows that the way in which PMSs are used relates differently to different hospital outcomes, and we thus highlight the important role of (middle) managers, who have an active role and discretion in deciding on the use of PMS. Depending on the current situation of a department, a manager can choose to use the PMS in a way that focuses attention on the most important issues at that moment. For instance, if a manager wants to improve the unit’s performance in the dimension of POC, our study suggests that the manager uses the PMS in a more exploratory way—focused on organizational learning (see also Naranjo-Gil & Hartmann, 2007a). Alternatively, in order to improve OP, a manager could choose to use the PMS in a more operational way—a focus on the ability of the PMS to serve as a more formal tool for planning and budgeting (cf. Hansen & Van der Stede, 2004). More in general, our study highlights the importance and potential of an exploratory use of PMSs in hospitals, that is, a use of PMS that is aimed at organizational learning—rather than using it as an instrument for incentivizing purposes. We show that an exploratory PMS use may contribute to increasing the CWC, which in itself has the potential to affect the job satisfaction of employees and future performance (cf. Schermerhorn et al., 2002). In addition, if unit managers rely on a mix of PMS uses, exploratory PMS use could have the potential to counteract some adverse side effects of operational PMS use, and we consider these insights to be important for hospital managers. We hope that the insights provided in this article contribute to better decisions by hospital managers and inspire further research on the impact of different PMS uses on organizational outcomes.