"Assessing the impact of COVID-19 on the efficiency of Portuguese state-owned enterprise hospitals.

This paper uses Value-Based Data Envelopment Analysis (VBDEA), to assess the impact of the COVID-19 pandemic on the efficiency of 37 state-owned enterprises (SOE) hospitals by employing data publicly available from the Portuguese Health Service database between January and November 2019 and 2020, respectively. Furthermore, a productivity index (specifically adjusted to the VBDEA approach) is also used that allows identifying which factors are behind the relative efficiency changes of these hospitals. The factors considered to perform the efficiency assessment of the Portuguese SOE hospitals include labour, capacity, and activity-related indicators. Out of the 37 SOE hospitals, 21 and 17 were efficient in 2019 and 2020, respectively. <Irrespective of the value functions considered, the hospitals more often viewed as a reference for best practices were Santa Maria Maior, Tâmega e Sousa and Entre Douro e Vouga. Santa Maria Maior and Algarve were the only hospitals found to be robustly efficient for both years. Overall, the majority of SOE hospitals showed negative productivity (except for Évora and Santa Maria Maior) and all of them presented negative technological change, thus highlighting the massive impact that the COVID-19 outbreak has had on the performance of these hospitals. An additional conclusion is that inefficient hospitals substantially increased all their resources in 2020 as compared to inefficient hospitals in 2019, suggesting that the inefficiency of these hospitals was not due to the lack of resources. Finally, irrespective of the model employed, the hospitals located in the Portuguese northern region were more resilient to the COVID-19 crisis. All in all, to become more resilient (even for future COVID-19 outbreaks), hospitals should undertake changes that are advantageous irrespective of the obstacles they face and that are even beneficial during normal times. A culture of cooperation within and across hospitals should also be cultivated, which allows exchanging resources where they can be used more efficiently.

Introduction

Cases of the novel coronavirus disease 2019 (COVID-19), caused by a new beta coronavirus (SARS-CoV-2), were first reported in Wuhan, China, in December 2019 [1]. Since then, Covid-19 has rapidly spread all around the world [2]. As such, prevention and containment became the priority against COVID-19, along with an ongoing search for disease characteristics that would allow for early detection and provide management and triage information [3].

In this context, the efficiency of countries impacted by COVID-19, considering their population density and health system infrastructure, became the focus of attention of many scholars. In this vein, the first study to employ the initial data available on COVID-19 (from February 2 to April 12 of 2020) in the assessment of governance performance of 19 countries was reported by Ghasemi et al. [4]. These authors concluded that Australia, Finland, Japan, Malaysia, Singapore and Thailand showed higher efficiency in preventing deaths caused by COVID-19 when compared to other countries. Shirouyehzad et al. [3] also found that Singapore, Vietnam, and Belgium were the countries that showed highest efficiency both in contagion control and medical treatment in the first wave of the pandemic. Curiously, Singapore, which has one of the largest population densities in Southeast Asia, managed to attain the highest efficiency scores among all countries. In Europe, Italy and Belgium were the least and most efficient countries, respectively. Finally, in the Middle East, Egypt was found to be the least efficient in contagion control but the most efficient in medical treatment, whereas Iran was considered the most efficient in contagion control. Overall, [5] concluded that the countries that performed less effectively and efficiently in the outbreak response management were Iran, the United States, Iraq, and San Marino, while Singapore, Malaysia, Vietnam, and Macau were the most effective and efficient.

In general, it can be ascertained that health systems around the world were not designed to deal with this type of crisis, i.e., a large-scale, unpredictable health challenge that required urgent mobilization of resources and affected the entire population.

In Portugal, like in any other country, in addition to the substantial burden on the National Healthcare System (NHS), COVID-19 ended up dictating the pace of the Portuguese economy, posing great challenges to economic and policy decision-makers (DMs). As a result, the NHS may once again be forced to contain costs, but not without its caveats. On the one hand, eventual cuts in this sector might preclude its preparedness for handling future COVID-19 outbreaks. On the other hand, the severe economic and financial costs inflicted by the virus may limit the capacity of further funding the NHS. Therefore, the efficiency assessment of the NHS public hospitals is more than ever timely and relevant.

In this context, the Data Envelopment Analysis (DEA) methodology has been broadly accepted and applied to the efficiency assessment of healthcare systems and specifically to hospitals [[6], [7], [8], [9]].

DEA enables the detection of potential sources of inefficiency delivering public DMs information on ways to surpass them. Another advantage of the application of this non-parametric tool in efficiency assessment is the capacity of identifying the benchmarks of inefficient Decision-Making Units (DMUs) – the SOE hospitals in our case, presenting managers with information on the best practices that can be pursued to reach efficiency.

However, and to the best of our knowledge, none of the studies reviewed to date and revised in other works (e.g. [7,8,10], explicitly incorporated in efficiency analysis real DMs and addressed the translation of their preferences into value functions in the efficiency assessment of hospitals. Insofar, this is one of the novelties introduced by our study, which applies the VBDEA method in the assessment of the efficiency and productivity of SOE hospitals prior to and after the COVID-19 emergency. This modelling framework can be particularly suitable for supporting policymakers (when compared to other DEA models) since it enables to bring into the analysis different political priorities with repercussions on the identification of distinct benchmarks. This potentiality of the model can be explored by setting a ranking order on the weights consistent with the political concerns of DMs. Hence, the capacity of including the preferences of a policymaker either by using weight constraints or by translating them into value functions enables to refine the evaluation, leading to better support in actual decision-making. Furthermore, the VBDEA approach has several innovative characteristics when compared with the traditional DEA models because it converts inputs and outputs into value scales. This can be particularly useful not only for incorporating the preferences of the DMs but also for easily tackling negative or null data. In addition, the application of the DEA approach in the efficiency assessment of hospitals rarely considers the robustness assessment or uncertainty handling of the efficiency scores obtained. The robustness assessment in VBDEA enables us to foresee the impact that potential adjustments on the evaluation factors might have on the efficiency scores attained. This can be particularly important to evaluate the impact that disruptive events, such as a pandemic, might have on efficiency. Finally, a novel way of addressing Productivity Analysis in the framework of VBDEA is also developed and used.

Overall, the main research questions that we want to answer with our work are.

“Which factors have influenced the efficiency and productivity of Portuguese SOE hospitals before and after the COVID-19 pandemic hit the country? ”

“Which Portuguese SOE hospitals are more often viewed as benchmarks before and after COVID-19 crisis?”

“Which Portuguese SOE hospitals showed higher levels of resilience to the COVID-19 crisis”?

Our study is devoted to the assessment of SOE hospitals since these absorb 6042 M Euros, representing 59.9% of the public budget dedicated to the NHS [11]. In addition, these hospitals are one of the pillars of the public NHS which is aimed at providing universal healthcare and their assessment is particularly relevant under the current sluggish economic growth and public health crisis.

All in all, the main novelties of our proposal are the following: 1) it suggests the use of the VBDEA approach, which allows performing a robustness evaluation of the results obtained according to stability intervals, and also enables exploring distinct decision-making settings through the imposition of constraints on the ranking order of the weights; 2) it assesses these hospitals in the periods prior and after the pandemic, i.e. between 2019 and 2020, enabling to understand and suggest ways of correcting the inefficiencies detected; 3) it evaluates the productivity gains/losses of these hospitals within this time frame through a novel VBDEA productivity index; 4) unlike any other studies published so far on the matter, this work involved a real decision-maker (DM) in the analysis.

The remaining parts of this paper are structured as follows. Section 2 presents a literature review of studies specifically addressing the application of DEA to Portuguese hospitals also contemplating the evaluation of healthcare systems and governance performance during the pandemic. Section 3 provides a description of the VBDEA methodology and its main underpinning assumptions. Section 4 describes the data gathered and the main premises followed. Section 5 discusses some illustrative results and, finally, the last section delivers the main conclusions and future work developments and potential improvements.

Literature review

Despite its shortcomings and limitations, 31 out of the 57 studies reviewed in [7] measured hospitals' performance through the single use of the DEA approach. Furthermore, according to the review conducted by [10]; healthcare, including hospitals, is found to be one of the top five most popular application areas of the DEA methodology. More recently, [8] reviewed 262 papers on DEA applications in healthcare with a special focus on hospitals.

In this context, the next two Sections provide a literature review on studies, which employ DEA in the Portuguese healthcare system and examine the governance performance during COVID-19 epidemics.

Literature on efficiency between health care facilities in Portugal

From the point of view of strict efficiency analysis, [12] evaluated the performance of 68 Portuguese hospitals in 2005 through DEA, also using a double-bootstrap procedure to account for the influence of the operational environment on efficiency. The inputs considered were CAPEX, the number of Staff and OPEX, whereas the outputs were the number of patients and the number of emergency and outpatient visits. Overall, they found that administrative public sector hospitals had better performance than SOE hospitals, both under Constant Returns to Scale (CRS) and Variable Returns to Scale (VRS) models. In the same vein, [13] applied DEA to the efficiency evaluation of Internal Medicine services of most hospitals in the Portuguese NHS. The model considers as inputs the number of doctors and nursing staff weekly hours, the number of medical exams, the number of materials used, the number of beds and the number of hours devoted to surgical operations. The model contemplates as outputs the number of inpatient discharges (corrected by the case-mix), the number of outpatient visits, the number of first outpatient visits, the number of sessions in day care hospitals, the number of patients in the day care hospitals and the number of ambulatory surgeries (corrected by the case-mix). One novelty of their model lies in the imposition of constraints on weights, to avoid obtaining unreasonable outcomes. Overall, they managed to conclude that it was possible to reach expressive savings in medicine and clinical materials. Also, in the same line of work, [14] studied the technical efficiency levels of 27 Portuguese public hospitals in 2016 through DEA and Order-α methodologies. The factors considered for this efficiency assessment were total expenses, risk-adjusted treated patients, delivered healthcare services’ quality, and the epidemiologic and demographic conditions in which hospitals operate. They were able to conclude that inefficient hospitals showed average technical inefficiency levels of about 10%. Later, [15] also evaluated the efficiency levels of 27 public hospitals in Portugal through DEA, considering as inputs total costs, services provided measured by the number of patients attended by case-mix, quality of service provided and access to it, as well as the external environment (demography and epidemiology). Their results showed that the average level of inefficiency, only for inefficient hospitals, is approximately 6%, having concluded that Portuguese public hospitals exhibit a considerable average performance. However, they also suggested that there is some disparity across regions in terms of the efficiency scores obtained. In the same context, [16] employed the DEA approach to quantify and compare the efficiency of 27 hospitals in 2017. The inputs used were the overall costs of providing care, the number of beds, the number of full-time equivalent (FTE) doctors and nursing staff, while the outputs considered were the number of patients with hospitalization episodes, the number of medical appointments, the number of patients who went through the emergency service, and the number of surgeries. Their results showed a large average performance across the country, but also interregional disparities that deserved special attention from the policymakers and hospital managers.

To examine whether the technology of Portuguese hospitals works under CRS or VRS, [17] employed DEA to estimate the efficiency of 159 Portuguese hospitals between 2005 and 2008, considering the most reliable technology. Two models to assess the Portuguese hospitals’ efficiency were constructed, the first encompasses CAPEX, staff expenses, and operation and maintenance expenses minus staff expenses as inputs, whereas the second considers the number of employees and number of available beds as inputs. The outputs, which are common to both models, are the number of patients treated, number of emergency visits, and total number of medical appointments. Overall, they were able to establish that most groups of hospitals had efficiency improvements, on average, and both models indicated that single hospitals and those located in the North were the most efficient. In terms of their returns to scale, conclusions suggest that hospitals should reach an optimal scale by reducing their cost levels and increasing their assets.

Another line of research seeks to incorporate quality in hospital efficiency analysis. In this context, [18] developed a methodology to compute DEA technical efficiency scores adjusted for output quality, for 37 Portuguese NHS hospitals in 2009. In their work they have used as inputs the number of doctors, the number of nurses and the number of other remaining staff, the number of beds and total costs. The output measures included inpatient visits, outpatient visits, emergency episodes, and surgery interventions (ambulatory plus non-ambulatory). These authors concluded that efficiency gains may be achieved without significantly sacrificing service quality. However, they also found that analysing hospital efficiency without considering disparities in quality of service can produce biased results. Later, [19] also considered a set of Portuguese public hospitals from the years 2013–2016 to explore the possible trade-off between technical efficiency and social performance. Their research considers three different models to study the relationship between quality, access, and technical efficiency. Hence, inputs were clustered into a financial model, which only accounts for monetary resources – costs –, and two mixed models, which consider both physical and monetary resources – doctors, nurses, hospital days, and other operational costs but staff. Their conclusions indicate that promoting efficiency improvements will likely decrease the patients' clinical safety due to the existence of observed perverse effects [9]. also investigated the relationship between technical efficiency, access, and quality of services by using a sample of Portuguese public hospitals (operating between 2013 and 2016) and a dataset composed of financial resources, hospital services, appropriateness and timeliness of care, patients’ clinical safety, access to health care services, demographics, and epidemiology variables. Their findings suggest that Portuguese public hospitals exhibit low performance in terms of quality and that quality and access can be improved with no efficiency sacrifice and vice versa.

With the aim of understanding if the market structure reforms in the Portuguese health system have improved hospital performance and productivity, [20] employed the Malmquist index. In their assessment, they scrutinized 216 non-corporatized and 176 corporatized Portuguese hospitals for the period 2002–2009. Five models were thus employed, based on three dimensions (internment, emergencies and doctor visits). Their findings show that although corporatized hospitals presented the highest efficiency consistency, they also had the lowest levels of productivity, while the hospitals under the traditional administrative public management system were the ones with the best average performance.

Overall, it can be ascertained that in Portugal, there has been proficuous scholarly attention on the assessment of hospital efficiency, but the resilience of Portuguese hospitals to the COVID-19 outbreak has not been addressed so far. In addition, from the review conducted it can be concluded that studies that evaluate the efficiency of Portuguese hospitals through the DEA approach rarely encompass the policy DMs' preferences and attend the examination of potential productivity gains/losses of Portuguese hospitals between different periods. This analysis is particularly relevant in the current context to shed light on the impact of COVID-19 on hospitals' productivity, thus implying the consideration of the periods prior to and after the pandemic. Finally, despite its merits (see e.g. [[21], [22], [23], [24]], i.e., the possibility of incorporating the preferences of the policymaker through the use of value functions and weight constraints, the VBDEA methodology has not hitherto been employed in the efficiency assessment of hospitals. Hence, an empirical application of this modelling framework is suggested which enables assessing Portuguese SOE hospitals’ efficiency and productivity.

Literature on COVID-19 epidemics and efficiency

In general, there has been a consensus amidst several epidemiology studies that the containment of the COVID-19 pandemic, particularly in the first waves, was of critical relevance [25]. In effect, [25] found that the health system of every country ends up setting an upper limitation on the number of patients that can obtain suitable treatment (according to the available number of hospital beds, nursing and medical staff, number of equipped intensive care units, etc.). Therefore, if the spread of the coronavirus could not be contained rapidly, the health systems could easily collapse.

With the foregoing in mind, [26] assessed the effectiveness of nonpharmaceutical interventions (policies) to mitigate the spread of SARS-CoV-2 (effectiveness of each policy to result in fewer infections) and [27] quantified the effectiveness of governmental interventions at an early stage on slowing down or reversing the growth rate of deaths. From the point of view of technical efficiency with which health systems were able to contain the spread of COVID-19 infections (i.e., flattening the curve), Breitenbach et al. [2] analysed a sample of 31 countries concluding that inefficient countries could have reached the same outcomes in terms of flattening the curve by using fewer resources such as locking down the economy earlier than they did. Specifically, among the worst performers were some of the richest countries in the world, such as Germany, Canada, the United States and Austria. These countries were even more inefficient in applying their available resources to flatten the curve than countries like Italy, France and Belgium, who were some of those who were hit harder by the early spread of the virus. More recently, Lupo and Tiganasu [28] showed that, especially in the first phase of the pandemic, the inefficiency of health systems was quite high, especially in Western countries, such as Italy, Belgium, Spain, and the United Kingdom. In the relaxation phase, and the second wave, western countries severely affected at the beginning of the pandemic, began to take appropriate measures and improved the efficiency of their health systems. On the contrary, Eastern European countries were hit hard by inefficient health systems (e.g., Bulgaria, Greece, Hungary, and Romania). All in all, these findings show that there is a multiplicity of factors that might influence the efficiency of health systems, which are also associated with economic, social, and governmental aspects. For example, these authors revealed that the evident psychological crisis made the system even more vulnerable. In fact, in many situations, the fear of contracting the disease in the hospital caused the postponement of doctor's appointments, leading to an increase in chronic diseases. Moreover, most countries verified that the reduction of inpatient hospital activity during 2020 was attributed not only to people's reluctancy to seek hospital care but also to lockdowns and quarantines, reorganization of hospital operations and rationing of the medical workforce [29,30]. Overall, without the possibility of changing the size and structure of the health systems in the short run, most countries did not improve their efficiency against COVID-19, and in some cases, even worsened. At least for European countries, the inefficiency of the public health sector was not due to a lack of medical resources, but to the incorrect use of available resources [[2], [28]].

Finally, since the studies reviewed so far revealed that Portugal was unable to use its resources efficiently [[2], [28]], we wanted to further understand which factors might help explain these findings, particularly considering the stringent resource allocation faced by Portuguese hospitals during the pandemic.

The methodological approach

The VBDEA model has been developed by [31] and tackles both the scaling bias and the absence of interpretation of the scores obtained through the weighted additive model [32]. This is accomplished by coupling into a single framework the DEA and MCDA approaches embedding the DMs’ preferences, also translating the input and output values into value scales.

In the field of The MCDA methods, the Multiple Attribute Value Theory (MAVT) can be used to solve problems that involve a finite and discrete set of alternative policies that must be evaluated based on conflicting objectives [33]. For any objective, one or more different attributes or criteria are used to measure the performance against that objective. These attributes or criteria are generally measured at different measurement scales. In this context, it is also worth mentioning that although the VBDEA methodology allows considering distinct criteria in the analysis, the number of criteria should be consistent with the rule of thumb established by [34] for traditional DEA models in the choice of the number of input and output factors used in the analysis, i.e. the number of units under assessment should be at least the double of the number of criteria considered (for further details please also see [35]).

In VBDEA we have the set of n DMUs under evaluation considering their performance on q criteria. Each () is thus characterized by a performance vector (, … , … ), where q = m + p, such that are being minimized and are being maximized. We use the simplest form of the value function (.), the additive form, where the can be decomposed into different partial value functions . In this way, every criterion c has its value function, which simplifies the assessment procedure for the global function V(.). The functions in the additive form of the MAVT can be used to transform the different measurement scales of the criteria into an identical scale. To do this, each value function remains within the range [0,1] because it is established that the performance of DMU j in criterion c () has the value 0 in the worst case and the value 1 in the best case, resulting in a problem where all criteria are considered to be maximized, and translating the preferences of DMs. Therefore, we can say that MAVT allows the compensation of bad performance in one criterion for the good performance of another criterion, since the MAVT aggregates the performance of the options in all criteria to form an overall assessment. In other words, each of the single-dimensional integrates the global value function, , where , ∀c = 1, …,q and (by convention). The weights of the additive value function correspond to the scale coefficients and are specified for each single dimensional value function once they are computed for each to minimize the value difference to the best of all , following the min-max regret rule [36]. We can also, instead of letting each DMU freely choose the weights associated with these value functions, restrict them according to the preferences of the DMs. In addition, weight restrictions can be incorporated into the efficiency assessment process, allowing to deal with the fact that, otherwise, important criteria could be ignored in the analysis.

After converting all the criteria into a value scale, the VBDEA tailored to fit the concept of super-efficiency [37] to allow the discrimination of efficient DMUs has the following phases [38]:

Phase 1: Solve problem (1) and obtain the efficiency score, , and the corresponding weighting vector for each (k = 1, …,n).

Phase 2: If then solve problem (2), using the optimal weighting vector computed in Phase 1, , and compute the corresponding projected point of the under evaluation.

The optimal value of the objective function, , is the value difference to the best of all DMUs (note that the best DMU will also depend on w), excluding itself from the reference set. If is negative, then under evaluation is efficient. Then, it is possible to rank the efficient DMUs by considering that the more negative the value , the more efficient is .

If is non-negative, then is inefficient and a projection target can be obtained through problem (2):

The set of efficient DMUs (it can be a single one) that defines a convex combination with >0 (j = 1, …,k-1,k+1, …,n) is called the set of “benchmarks” of . This convex combination leads to a point on the efficient frontier that is better than by a difference of (slack) value, , in each criterion c.

Robustness analysis

If we want to solve real-life decision problems, we have to be aware that they are dynamic, and that the information is often unavailable or uncertain. There may be changes in initial data, for example, resources may vary, or there may be a dispute over the nature of specific criteria value functions. Such inaccuracies can be checked to see whether the DMU is robust.

We propose to do a robustness analysis according to what was done in [38]. For each criterion, we establish the observed performance ranges plus or minus the highest tolerance value considered (in this case δ = 10%). This analysis will allow each DMU to be classified as robustly efficient as surely efficient, potentially efficient, or surely inefficient for the defined tolerance value.

The value (performance of DMU j in criterion c) is considered uncertain but bounded within the range and a common tolerance δ is applied to all performances, in original values, such that . It is assumed that the value functions are monotonous and the previous inequalities lead to , if criterion c is to be maximized, or if criterion c is to be minimized.

The optimistic efficiency measure is computed considering the best value of the intervals for the under evaluation and the worst value of the intervals for all the other DMUs. The reverse is considered to compute the pessimistic efficiency measure. To compute the optimistic efficiency measure for we solve linear problem (3):

To compute the pessimistic efficiency measure for , we solve problem (4).

Note that linear problems (3) and (4) are solved after the original performances of the criteria are converted into value scales. A robust DMU is the one that after the changes in its criteria remains efficient (or inefficient). Thus, it can be said that the DMU is robustly efficient (or robustly inefficient) for the considered tolerance.

The productivity index

The Malmquist Index and the Luenberger Productivity Indicator have been established to appraise changes in efficiency over time [39].

There is a plethora of prior studies that applied DEA to measureproductivity in different contexts, such as agriculture (e.g., [40]), banks (e.g., [[41], [42], [43]]), countries (e.g., [44]), the electricity sector (e.g. [45]), and hospitals (e.g., [46]).

These productivity indices are obtained from the efficiency scores computed with DEA models and allow measuring total factor productivity (TFP). TFP can be decomposed into Technical Change and Efficiency Change [47]. Technical Change evaluates shifts in the production frontier, also known as frontier shifts. Efficiency Change assesses changes in the position of a DMU regarding the efficient frontier, being also called the catching-up effect.

The proposal in this work is based on the Luenberger Productivity Indicator as a TFP measure because several authors suggest that this indicator has advantages over the Malmquist Index [39,48] and it matches better the non-oriented and non-radial nature of VBDEA.

However, until now, the approaches available in the literature do not apply to VBDEA since this approach is based on value functions and not on the original units of the factors. This means that if a DMU consumes 20% more inputs and produces 20% more outputs, then the overall value of the DMU will change according to the way the DM values these differences.

Hence, the TFP assessed by the VBDEA model productivity indicator is defined as follows:where:

t and t+1 denote two different years;

and denote the (in)efficiency scores of DMU k in years t and t+1, respectively, considering the efficiency frontier in year t;

and denote the (in)efficiency scores of DMU k in years t and t+1, respectively, considering the efficiency frontier in year t+1.

To compute and formulation (1) must be changed. This allows the analysis of efficiency changes over time. For instance, is calculated considering that the under evaluation has the values of the year t+1 and the values of the year t are considered for all the other DMUs (see linear problem (6)). The reverse is considered to compute (see linear problem (7)).

To better evaluate the structure of TFP change, Färe et al. [47] proposed the decomposition of TFP into technical change (TECHCH) and efficiency change (EFFCH).

Then, the TFP (5) can be decomposed into TECHCH (8) and EFFCH (9) for the VBDEA as well and it can be rewritten as in (10).

The value of TECHCH reached for any DMU only corresponds to frontier shifts from the standpoint of that DMU and does not necessarily mean that the corresponding DMU actually shifts the frontier of production into a more desirable direction.

Thus, in order to identify which DMUs are responsible for shifting the frontier line, the so-called ‘‘innovators’’ (see [47]), the following three conditions must be met (for a given DMUk):

Data and variables

Despite the prolific number of studies on the application of DEA to hospital efficiency, there is no globally accepted model. Hence, the choice of inputs and outputs must be made by adjusting, as best as possible, the model to the reality under evaluation. In this framework, [49] established some guidelines to help researchers choose the appropriate measures, proposing three main input categories for hospitals: 1) capital investment, 2) labour, and 3) operating expenses. As for outputs, [49] also proposes the incorporation of inpatients as well as outpatient visits. In our study, we have selected the inputs and outputs identified as the top used factors in hospital efficiency by [8]. Therefore, we have selected as an input the number of beds, which might be viewed as a proxy for capital investment (see also [13,[16], [17], [18]]). Besides, [50] highlight the importance of labour in a hospital environment, advocating the distinction between different types of personnel. In this sense, we have used the number of medical staff and nursing staff in FTE [49] and other non-medical staff as inputs (see e.g., [8,16,18,19]). Finally, for outputs we have selected the number of inpatient discharges (e.g. [13,18], and the number of emergency and outpatient visits (see e.g., [12,13,[16], [17], [18],20]).

We have used a cross-sectional dataset from a sample of 37 Portuguese hospitals for 2019 and 2020 gathered from the database publicly available at https://transparencia.sns.gov.pt/and https://benchmarking-acss.min-saude.pt/. Subsequently, we have removed from the analysis oncology hospitals because of their specificity. Besides, we have eliminated from the sample hospitals with missing data.

Since we did not have access to the required information to build the case-mix index (CMI) from the System of Patient classification in Diagnose Related Group (DRG), we use a length of stay case-mix index for 2019 and 2020 following the proposal of [18,51] by considering the length of stay of each inpatient visit as a proxy for the complexity of each case.

From the analysis of Table 1 it might be concluded that when contrasting the years 2019 and 2020 until November (the most recent data obtained in 2020), there has been an increase in the average number of doctors, nurses, and operational staff, of 2%, 5% and 10%, respectively, whereas the average number of beds has suffered a slight decrease (−2%). Finally, there has been a significant drop in in-person outpatient visits and inpatient services as a result of COVID-19, with the average number of inpatient discharges, and outpatient and emergency visits reducing to 15%, 11% and 27%, respectively.

Table 1

Descriptive statistics regarding the factors evaluated for Portuguese SOE hospitals.

	Number of physiciansx_1		Number of nursing staffx_2		Number of Operational Staffx_3		Number of bedsx_4		Number of inpatient discharges (until November)y_1		Number of outpatient visits (until November)y_2		Number of emergency visits (until November)y_3
	2019	2020	2019	2020	2019	2020	2019	2020	2019	2020	2019	2020	2019	2020
Mean	620	633	914	956	621	684	523	514	16651	14182	272014	242724	145420	105433
Standard Error	81	84	109	115	69	72	59	54	1721	1407	33303	30286	10742	7742
Median	438	457	686	738	516	581	400	412	14755	12354	230394	203509	144439	102729
Standard Deviation	494	511	666	697	418	438	362	329	10467	8558	202577	184224	65341	47095
Minimum	105	89	195	211	141	157	117	117	4481	4406	64268	59359	52703	41999
Maximum	1877	1903	2939	3096	1886	1975	1702	1550	51360	38021	828134	689600	332791	236356

Value functions

Unlike the Multiple Attribute Utility Theory (MAUT), MAVT does not seek to model the DMs’ attitude towards risk [33]. As a consequence, it is based on simpler elicitation procedures, which are more widely accepted by DMs in practice.

In the VBDEA method, the purpose of converting factors into a value scale (piecewise linear or non-linear value functions) is to reflect the DMs' preferences while putting the performances of the criteria in the same range [0,1].

In this work, we had the collaboration of an SOE hospital clinical board member of direction to build the value functions. The procedures used to elicit the partial value functions are the same ones that have been used in previous works [21,22]. With the answers of the DM, instead of constructing piecewise linear value functions, it was possible to adjust these to non-linear predefined curves (logarithmic functions).

In order to assess the impact of the DM's preferences (incorporated into the value functions) on the results, we decided to consider two models: the model in which the value functions are all linear, resulting from the normalization of all the criteria performances, typically used in MAVT problems, through transformation (14), converting them into the range [0,1], and another model that suits the preferences of the DM, with non-linear value functions.

Before building any of the value functions, two limits were determined for each criterion, and , such that and , for each , already taking into account the tolerance δ = 10% assigned to the performance values.

By doing this, we have assumed two perspectives: one which considers neutral value functions (i.e., linear), and another which uses value functions defined by encompassing the preferences of a real DM, named Conservative. The latter received this designation because of the functions that best fit the answers of the DM, which in the case of inputs it is assumed that “the less the better” but for outputs, in the current pandemic context, from a certain point onwards “more is less good”, since having more outputs comes at the cost of having fewer resources dedicated to COVID-19.

For the Neutral perspective, the values for each DMU were computed using:

For the Conservative model, the non-linear value functions were obtained by making the corresponding adjustment of a known function to the preferences revealed by the DM. These are similar convex logarithmic functions for all the inputs and similar concave logarithmic functions for all the outputs. In Fig. 1 we present an example function for one of the inputs and one of the outputs. It is possible to observe that for the number of physicians FTE: (800)–(400) > (1200)–(800), all other performance levels being equal, and for the number of inpatient discharges corrected by Normalized Length of Stay Case-Mix Index (NLCMI) – see [18] – that: (25000)–(15000) > (35000)–(25000), translating that successive increases in inpatient discharges result in increasingly smaller additions of merit.

Fig. 1

Two of the value functions elicited.

Table 2 presents the original data and the corresponding value functions for the Neutral and Conservative perspectives, respectively.

Table 2

Performances of DMUs in original scales and in value scales for Neutral and Conservative perspectives (in 2020).

		criteria in original scales							criteria in value scale (Neutral)							criteria in value scale (conservative)
DMU	Hospital	x_1	x_2	x_3	x_4	y_1	y_2	y_3	v_1	v_2	v_3	v_4	v_5	v_6	v_7		v_1	v_2	v_3	v_4	v_5	v_6	v_7
1	Centro Hospitalar Barreiro/Montijo, EPE	341	697	490	388	7717	144552	102729	0.880	0.842	0.822	0.848	0.070	0.106	0.196		0.580	0.533	0.515	0.546	0.260	0.352	0.439
2	Centro Hospitalar de Leiria, EPE	385	820	551	509	15042	272798	122983	0.860	0.805	0.793	0.785	0.210	0.254	0.257		0.546	0.479	0.475	0.455	0.510	0.574	0.518
3	Centro Hospitalar de Lisboa Ocidental, EPE	1010	1452	1071	735	16805	390177	104345	0.564	0.615	0.543	0.666	0.250	0.389	0.201		0.245	0.290	0.246	0.333	0.550	0.700	0.446
4	Centro Hospitalar de Setúbal, EPE	582	842	727	376	11356	194616	93955	0.767	0.798	0.708	0.855	0.140	0.163	0.169		0.417	0.471	0.379	0.557	0.410	0.456	0.400
5	Centro Hospitalar do Baixo Vouga, EPE	457	747	460	412	12204	206441	112572	0.830	0.827	0.837	0.836	0.160	0.177	0.225		0.490	0.510	0.537	0.526	0.430	0.477	0.479
6	Centro Hospitalar do Médio Ave, EPE	291	429	295	279	6818	127653	88940	0.900	0.922	0.916	0.906	0.060	0.086	0.154		0.630	0.693	0.690	0.656	0.220	0.308	0.376
7	Centro Hospitalar do Oeste, EPE	376	614	456	316	10525	119653	108423	0.860	0.867	0.838	0.886	0.130	0.077	0.213		0.550	0.575	0.540	0.615	0.380	0.286	0.463
8	Centro Hospitalar e Universitário de Coimbra, EPE	1873	3096	1880	1550	38021	689600	195017	0.150	0.121	0.154	0.237	0.650	0.734	0.473		0.050	0.039	0.053	0.083	0.850	0.899	0.720
9	Centro Hospitalar Entre Douro e Vouga, EPE	398	592	732	377	13776	230854	137863	0.850	0.873	0.706	0.854	0.190	0.205	0.301		0.540	0.587	0.377	0.556	0.480	0.516	0.568
10	Centro Hospitalar Médio Tejo, EPE	344	761	638	441	12354	143203	94508	0.880	0.823	0.751	0.821	0.160	0.104	0.171		0.580	0.504	0.424	0.503	0.440	0.349	0.403
11	Centro Hospitalar Póvoa de Varzim/Vila do Conde, EPE	154	264	174	150	5046	81363	49233	0.970	0.972	0.974	0.974	0.020	0.033	0.035		0.830	0.855	0.871	0.863	0.110	0.150	0.117
12	Centro Hospitalar Tâmega e Sousa, EPE	541	733	609	464	17192	268339	151660	0.790	0.831	0.765	0.808	0.250	0.248	0.343		0.440	0.517	0.440	0.486	0.560	0.569	0.610
13	Centro Hospitalar Tondela-Viseu, EPE	940	1104	581	629	14971	200431	111273	0.600	0.719	0.778	0.722	0.210	0.170	0.221		0.270	0.381	0.457	0.385	0.510	0.466	0.474
14	Centro Hospitalar Trás-os-Montes e Alto Douro, EPE	614	1078	721	553	20199	263571	120516	0.750	0.727	0.711	0.762	0.310	0.243	0.249		0.400	0.389	0.382	0.428	0.620	0.562	0.509
15	Centro Hospitalar Universitário Cova da Beira, EPE	242	435	317	311	7531	116778	44558	0.930	0.920	0.905	0.889	0.070	0.074	0.021		0.690	0.689	0.665	0.620	0.250	0.277	0.074
16	Centro Hospitalar Universitário de Lisboa Norte, EPE	1746	2063	1529	1016	27534	587355	159290	0.220	0.431	0.323	0.518	0.450	0.616	0.366		0.070	0.174	0.124	0.225	0.730	0.843	0.631
17	Centro Hospitalar Universitário de São João, EPE	1679	2510	1384	1082	35452	677584	189148	0.250	0.297	0.392	0.483	0.600	0.720	0.456		0.090	0.109	0.158	0.203	0.830	0.893	0.707
18	Centro Hospitalar Universitário do Algarve, EPE	1048	1706	1123	911	20893	256357	236356	0.546	0.539	0.518	0.573	0.320	0.235	0.598		0.233	0.237	0.230	0.261	0.630	0.553	0.804
19	Centro Hospitalar Universitário do Porto, EPE	1348	1625	1015	811	21689	601247	115053	0.400	0.563	0.570	0.626	0.340	0.632	0.233		0.150	0.253	0.265	0.300	0.650	0.851	0.489
20	Centro Hospitalar Universitário Lisboa Central, EPE	1903	2748	1975	1238	31667	615264	158189	0.141	0.226	0.108	0.401	0.529	0.649	0.363		0.046	0.079	0.036	0.159	0.786	0.859	0.628
21	Centro Hospitalar Vila Nova de Gaia/Espinho, EPE	1020	1287	1030	578	18127	424511	128043	0.559	0.664	0.563	0.748	0.272	0.429	0.272		0.242	0.330	0.260	0.413	0.579	0.729	0.536
22	Hospital da Senhora da Oliveira, Guimarães, EPE	496	759	409	753	14960	213079	92291	0.808	0.823	0.861	0.656	0.212	0.185	0.164		0.467	0.505	0.577	0.325	0.508	0.488	0.392
23	Hospital de Braga, EPE	908	1086	785	686	21365	402567	149216	0.612	0.725	0.680	0.692	0.333	0.403	0.336		0.278	0.386	0.353	0.356	0.640	0.711	0.603
24	Hospital Distrital da Figueira da Foz, EPE	142	211	161	154	4466	75815	48407	0.975	0.988	0.980	0.972	0.013	0.026	0.032		0.858	0.928	0.898	0.855	0.060	0.126	0.110
25	Hospital Distrital de Santarém, EPE	356	631	484	401	11970	113205	78751	0.874	0.862	0.825	0.842	0.155	0.069	0.124		0.571	0.566	0.519	0.535	0.425	0.266	0.323
26	Hospital Espírito Santo de Évora, EPE	405	576	431	167	8288	137651	49569	0.851	0.878	0.850	0.965	0.085	0.098	0.036		0.531	0.596	0.559	0.828	0.289	0.335	0.120
27	Hospital Garcia de Orta, EPE	695	1042	673	564	14083	253592	103909	0.713	0.738	0.734	0.756	0.195	0.231	0.199		0.362	0.400	0.406	0.421	0.486	0.549	0.444
28	Hospital Professor Doutor Fernando Fonseca, EPE	813	1071	803	789	19294	268694	167633	0.657	0.729	0.672	0.637	0.294	0.249	0.391		0.313	0.391	0.345	0.309	0.602	0.569	0.654
29	Hospital Santa Maria Maior, EPE	89	213	157	117	4805	63178	41999	1.000	0.987	0.982	0.991	0.019	0.012	0.013		1.000	0.925	0.907	0.946	0.087	0.062	0.048
30	Unidade Local de Saúde da Guarda, EPE	226	662	615	349	6814	76120	65642	0.935	0.852	0.762	0.869	0.057	0.027	0.084		0.713	0.550	0.437	0.581	0.217	0.127	0.243
31	Unidade Local de Saúde de Castelo Branco, EPE	189	375	332	218	6337	68582	43613	0.953	0.939	0.898	0.938	0.048	0.018	0.018		0.769	0.739	0.649	0.739	0.190	0.091	0.064
32	Unidade Local de Saúde de Matosinhos, EPE	563	693	479	360	11218	210077	71001	0.776	0.843	0.827	0.863	0.141	0.181	0.100		0.427	0.535	0.523	0.571	0.401	0.483	0.278
33	Unidade Local de Saúde do Alto Minho, EPE	487	738	631	419	12605	203509	102355	0.812	0.830	0.754	0.832	0.167	0.174	0.195		0.473	0.514	0.428	0.520	0.445	0.472	0.438
34	Unidade Local de Saúde do Baixo Alentejo, EPE	212	459	455	210	5156	71215	67557	0.942	0.913	0.839	0.942	0.026	0.021	0.090		0.733	0.672	0.541	0.751	0.113	0.104	0.256
35	Unidade Local de Saúde do Litoral Alentejano, EPE	132	252	241	135	4406	59359	62830	0.980	0.975	0.942	0.982	0.011	0.007	0.076		0.882	0.870	0.759	0.899	0.055	0.040	0.224
36	Unidade Local de Saúde do Nordeste, EPE	234	540	494	370	7967	78306	73097	0.932	0.889	0.820	0.858	0.079	0.029	0.107		0.703	0.618	0.512	0.562	0.275	0.137	0.290
37	Unidade Local de Saúde do Norte Alentejano, EPE	196	457	392	216	6079	73476	58481	0.950	0.914	0.869	0.939	0.043	0.024	0.063		0.757	0.673	0.592	0.742	0.174	0.115	0.193

Discussion of results1

After running the Neutral and Conservative perspectives we found 21 and 17 efficient SOE hospitals in Portugal in 2019 and 2020, respectively (efficiency score <0) in both models – see also Tables A1, A.2, A.3 and A.4 from the Appendix. Under the Neutral perspective, the mean efficiency score of the whole sample in 2019 and 2020 is −0.01299 and −0.00510, whereas under the Conservative perspective it is −0.00540 and −0.00001, revealing a decrease in the overall efficiency level according to both settings. Nevertheless, while under the Neutral perspective the average efficiency score of efficient SOE hospitals decreased 61% from 2019 to 2020, it decreased by 100% in the same period according to the Conservative perspective. This outcome is the result of using non-linear value functions consistent with the principle “more is less good”, especially because having more outputs comes at the expense of having fewer resources devoted to COVID-19 (Conservative). As a result, the hospitals which belong to the top five ranking of efficient units in both models are different.

The descriptive statistics of the evaluation factors regarding efficient and inefficient hospitals in both models are provided in Table 3 . Overall, it can be concluded that, when compared to 2019, efficient hospitals had, in 2020, a mean lower number of physicians, which had to be compensated with a higher number of nurses and operational staff. In contrast, between these two years, inefficient hospitals increased the number of physicians and all the other hospital staff. Curiously, the negative impact of COVID-19 on the mean outpatient visits, inpatient discharges and emergency visits is more evident between these two years in efficient hospitals, than in inefficient hospitals. These findings suggest that these latter units had their activity levels less compromised by COVID-19, but at the expense of a substantial increase in all resources.

Table 3

Descriptive statistics of the evaluation factors of SOE Portuguese hospitals

	Number of physiciansx_1		Number of nursing staffx_2		Number of Operational Staffx_3		Number of bedsx_4		Number of inpatient discharges (until November)y_1		Number of outpatient visits (until November)y_2		Number of emergency visits (until November)y_3
Years covered	2019	2020	2019	2020	2019	2020	2019	2020	2019	2020	2019	2020	2019	2020
Mean (efficient)	698	678	998	1014	662	688	568	546	19046	15914	322830	285031	165994	116419
Mean (inefficient)	518	596	803	906	568	680	463	488	13507	12709	205319	206762	118415	96094
Standard Deviation (efficient)	534	550	737	823	441	479	386	393	11709	10110	223655	208078	71516	57109
Standard Deviation (inefficient)	433	487	562	587	394	412	331	271	7848	6908	153082	157739	45352	35442
Minimum (efficient)	105	89	195	211	141	157	117	117	4481	4406	64268	59359	57595	41999
Minimum (inefficient)	155	189	264	375	163	317	143	210	6193	5156	80444	68582	52703	43613
Maximum (efficient)	1873	1873	2939	3096	1740	1880	1702	1550	51360	38021	828134	689600	332791	236356
Maximum (inefficient)	1877	1903	2603	2748	1886	1975	1541	1238	38219	31667	664375	615264	224199	167633
Count (efficient)	21	17	21	17	21	17	21	17	21	17	21	17	21	17
Count (inefficient)	16	20	16	20	16	20	16	20	16	20	16	20	16	20

From the Neutral perspective, except for São João hospital (which ranked 8th in 2019) and Leiria hospital (which climbed from the 18th place in 2019 to the 5th place in 2020), all the other top 5 hospitals in the ranking remain the same in both years – see the top half of Fig. 1.

In the Conservative perspective except for the Évora hospital which climbs from the 29th place to the 3rd place, and the Coimbra hospital, which loses the 1st place in 2019, occupying the 11th place in 2020, all the other top 5 hospitals in the ranking remain unchanged in both years – see the bottom half of Fig. 2 .

Fig. 2

Ranking of efficient SOE hospitals in 2020 vs. 2019.

According to the Neutral perspective, the hospital more frequently viewed as a benchmark in 2020 is Entre Douro e Vouga (12 times in 2019 and 25 times in 2020), being subsequently followed by Tâmega e Sousa (25 times in 2019 and 14 times in 2020) and Santa Maria Maior (20 times in 2019 and 14 times in 2020) – see the top half of Fig. 2. The hospital more often selected as a reference for best practices under the Conservative perspective in both years is Santa Maria Maior (22 times), which is also the first of the efficiency ranking (climbing two positions from the previous year), followed by Tâmega e Sousa (15 times in 2019 and 12 times in 2020) and Entre Douro e Vouga (10 times in 2019 and 15 times in 2020) – see the bottom half of Fig. 3 . Overall, it is curious to see that the value functions used do not lead to substantial changes in the benchmarks selected by inefficient hospitals. In addition, among the biggest hospitals, Coimbra stands out since it is viewed as a reference hospital 10 times according to the Conservative perspective and 6 times following the Neutral perspective in 2020.

Fig. 3

Nomination of each efficient SOE hospital as a benchmark in 2020 vs. 2019.

The main factors which help explain the efficiency of these four hospitals in both years are depicted in Fig. 4 (see also Tables A.1, A.2, A.3 and A.4 from the Appendix for the remaining hospitals).

Fig. 4

Factors held responsible for the efficiency of SOE hospital in 2020 vs. 2019.

Table A.1

Score, weights, slacks and reference countries obtained according to VBDEA method for the year of 2019 – Neutral

DMU	d*	w1	w2	w3	w4	w5	w6	w7	s1	s2	s3	s4	s5	s6	s7	DMU2	DMU6	DMU8	DMU9	DMU12	DMU14	DMU18	DMU19	DMU22	DMU23	DMU29	DMU32	DMU35	DMU37
1	0.015	0.498	0.000	0.207	0.000	0.000	0.010	0.284	0.013	0.054	0.000	0.047	0.026	0.036	0.028	0.000	0.567	0.000	0.433	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
2	−0.003	0.497	0.000	0.109	0.000	0.299	0.000	0.095
3	0.047	0.475	0.000	0.009	0.000	0.082	0.434	0.000	0.061	0.092	0.067	0.115	0.000	0.039	0.058	0.000	0.000	0.000	0.478	0.000	0.000	0.000	0.522	0.000	0.000	0.000	0.000	0.000	0.000
4	0.021	0.000	0.000	0.000	0.605	0.207	0.188	0.000	0.071	0.065	0.000	0.013	0.038	0.030	0.172	0.000	0.000	0.000	0.960	0.000	0.000	0.000	0.000	0.000	0.000	0.040	0.000	0.000	0.000
5	−0.002	0.050	0.000	0.547	0.000	0.047	0.056	0.300
6	−0.013	0.000	0.000	0.414	0.224	0.000	0.137	0.225
7	−0.012	0.000	0.000	0.327	0.285	0.090	0.000	0.298
8	−0.249	0.000	0.000	0.000	0.000	1.000	0.000	0.000
9	−0.041	0.113	0.507	0.000	0.000	0.000	0.008	0.372
10	0.003	0.604	0.000	0.000	0.000	0.337	0.000	0.059	0.003	0.014	0.000	0.014	0.000	0.044	0.023	0.000	0.000	0.000	0.270	0.000	0.344	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.386
11	0.001	0.000	0.000	0.237	0.349	0.400	0.014	0.000	0.011	0.009	0.000	0.000	0.002	0.004	0.037	0.000	0.000	0.000	0.000	0.047	0.000	0.000	0.000	0.024	0.000	0.929	0.000	0.000	0.000
12	−0.015	0.023	0.029	0.224	0.303	0.268	0.056	0.098
13	0.039	0.000	0.000	0.399	0.104	0.310	0.000	0.186	0.152	0.114	0.078	0.025	0.018	0.005	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.096	0.000	0.904	0.000	0.000	0.000	0.000	0.000
14	−0.030	0.477	0.000	0.000	0.010	0.514	0.000	0.000	0.000	0.041	0.027	0.000	0.000	0.021	0.041	0.428	0.000	0.068	0.000	0.504	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
15	0.005	0.567	0.000	0.019	0.000	0.247	0.166	0.000	0.007	0.022	0.000	0.054	0.000	0.007	0.114	0.000	0.000	0.000	0.306	0.000	0.026	0.000	0.000	0.000	0.000	0.668	0.000	0.000	0.000
16	−0.021	0.000	0.264	0.000	0.122	0.000	0.386	0.227
17	−0.029	0.000	0.000	0.192	0.207	0.000	0.342	0.259
18	−0.220	0.000	0.000	0.000	0.182	0.000	0.000	0.818
19	−0.068	0.000	0.185	0.133	0.173	0.000	0.509	0.000
20	0.072	0.000	0.422	0.000	0.000	0.146	0.337	0.095	0.083	0.000	0.156	0.010	0.161	0.111	0.113	0.000	0.000	0.828	0.000	0.000	0.000	0.000	0.000	0.000	0.172	0.000	0.000	0.000	0.000
21	−0.002	0.000	0.000	0.000	0.482	0.000	0.310	0.208
22	−0.048	0.000	0.000	0.583	0.000	0.417	0.000	0.000
23	−0.032	0.000	0.300	0.131	0.000	0.062	0.292	0.215
24	−0.006	0.000	0.500	0.153	0.000	0.000	0.246	0.101
25	0.015	0.469	0.061	0.074	0.000	0.325	0.000	0.071	0.017	0.041	0.000	0.047	0.000	0.094	0.069	0.000	0.000	0.000	0.325	0.341	0.000	0.000	0.000	0.000	0.000	0.334	0.000	0.000	0.000
26	0.013	0.000	0.000	0.032	0.557	0.264	0.147	0.000	0.036	0.030	0.000	0.020	0.000	0.011	0.092	0.000	0.000	0.000	0.218	0.000	0.000	0.000	0.131	0.000	0.000	0.651	0.000	0.000	0.000
27	0.052	0.000	0.000	0.421	0.173	0.040	0.289	0.078	0.088	0.096	0.100	0.051	0.036	0.000	0.000	0.000	0.000	0.000	0.000	0.460	0.000	0.000	0.014	0.526	0.000	0.000	0.000	0.000	0.000
28	−0.005	0.000	0.395	0.110	0.000	0.262	0.000	0.234
29	−0.017	0.357	0.000	0.000	0.643	0.000	0.000	0.000
30	0.041	0.000	0.000	0.000	0.594	0.394	0.000	0.012	0.029	0.055	0.000	0.007	0.086	0.147	0.214	0.000	0.000	0.000	0.758	0.000	0.000	0.000	0.000	0.000	0.000	0.242	0.000	0.000	0.000
31	0.010	0.585	0.027	0.000	0.000	0.388	0.000	0.000	0.017	0.020	0.041	0.030	0.000	0.023	0.031	0.000	0.000	0.000	0.000	0.000	0.116	0.000	0.000	0.000	0.000	0.884	0.000	0.000	0.000
32	−0.001	0.000	0.000	0.301	0.286	0.226	0.187	0.000
33	0.013	0.000	0.000	0.020	0.573	0.385	0.000	0.021	0.038	0.029	0.000	0.017	0.007	0.000	0.000	0.000	0.000	0.000	0.379	0.118	0.257	0.000	0.000	0.000	0.000	0.246	0.000	0.000	0.000
34	0.010	0.596	0.000	0.000	0.160	0.000	0.000	0.244	0.000	0.030	0.035	0.002	0.054	0.072	0.042	0.000	0.000	0.000	0.382	0.000	0.000	0.000	0.000	0.000	0.000	0.618	0.000	0.000	0.000
35	−0.017	0.000	0.420	0.000	0.354	0.000	0.000	0.226
36	0.002	0.545	0.065	0.000	0.000	0.351	0.000	0.040	0.001	0.000	0.000	0.046	0.000	0.069	0.028	0.000	0.000	0.000	0.170	0.000	0.170	0.000	0.000	0.000	0.000	0.203	0.000	0.000	0.458
37	−0.008	0.755	0.000	0.000	0.000	0.000	0.000	0.245

Table A.2

Score, weights, slacks and reference countries obtained according to VBDEA method for the year of 2020 – Neutral

DMU	d*	w1	w2	w3	w4	w5	w6	w7	s1	s2	s3	s4	s5	s6	s7	DMU2	DMU8	DMU9	DMU12	DMU14	DMU17	DMU18	DMU19	DMU22	DMU23	DMU24	DMU26	DMU29	DMU35
1	0.012	0.409	0.000	0.177	0.000	0.000	0.000	0.414	0.000	0.060	0.029	0.045	0.061	0.025	0.016	0.000	0.000	0.000	0.511	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.489
2	−0.035	0.409	0.000	0.115	0.000	0.013	0.463	0.000
3	0.048	0.206	0.000	0.000	0.299	0.181	0.313	0.000	0.000	0.033	0.105	0.016	0.048	0.110	0.041	0.351	0.000	0.000	0.000	0.000	0.000	0.000	0.649	0.000	0.000	0.000	0.000	0.000	0.000
4	0.026	0.000	0.000	0.000	0.537	0.216	0.183	0.065	0.039	0.047	0.082	0.000	0.061	0.040	0.083	0.000	0.000	0.000	0.704	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.296	0.000	0.000
5	0.003	0.000	0.000	0.509	0.034	0.065	0.253	0.139	0.019	0.051	0.000	0.011	0.019	0.000	0.012	0.000	0.000	0.000	0.629	0.000	0.000	0.000	0.000	0.070	0.000	0.302	0.000	0.000	0.000
6	−0.012	0.000	0.000	0.591	0.000	0.000	0.000	0.409
7	0.005	0.000	0.000	0.109	0.497	0.000	0.000	0.394	0.009	0.029	0.006	0.000	0.017	0.063	0.010	0.000	0.000	0.000	0.550	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.450
8	−0.049	0.000	0.000	0.000	0.000	1.000	0.000	0.000
9	−0.021	0.509	0.079	0.000	0.000	0.000	0.000	0.412	0.000	0.000	0.087	0.000	0.027	0.006	0.000	0.000	0.000	0.000	0.845	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.155
10	0.010	0.372	0.000	0.000	0.174	0.454	0.000	0.000	0.000	0.011	0.071	0.000	0.021	0.106	0.042	0.767	0.000	0.000	0.000	0.053	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.179	0.000
11	−0.001	0.000	0.000	0.244	0.326	0.000	0.313	0.117
12	−0.032	0.000	0.110	0.278	0.106	0.193	0.000	0.313
13	0.025	0.000	0.000	0.401	0.121	0.478	0.000	0.000	0.192	0.110	0.000	0.066	0.036	0.069	0.097	0.000	0.000	0.000	0.860	0.000	0.000	0.000	0.000	0.140	0.000	0.000	0.000	0.000	0.000
14	−0.019	0.347	0.000	0.000	0.058	0.595	0.000	0.000
15	0.014	0.077	0.220	0.254	0.000	0.202	0.247	0.000	0.016	0.026	0.020	0.054	0.010	0.000	0.079	0.000	0.000	0.000	0.261	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.739	0.000
16	0.016	0.000	0.353	0.000	0.000	0.000	0.373	0.274	0.122	0.000	0.160	0.035	0.033	0.043	0.000	0.000	0.000	0.000	0.000	0.000	0.553	0.000	0.352	0.000	0.095	0.000	0.000	0.000	0.000
17	−0.075	0.000	0.000	0.052	0.355	0.478	0.114	0.000
18	−0.162	0.141	0.000	0.000	0.000	0.000	0.000	0.859
19	−0.063	0.000	0.426	0.000	0.000	0.000	0.574	0.000
20	0.072	0.000	0.385	0.000	0.000	0.615	0.000	0.000	0.106	0.071	0.284	0.082	0.072	0.072	0.093	0.000	0.000	0.000	0.000	0.000	1.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
21	−0.008	0.000	0.000	0.000	0.571	0.000	0.322	0.107
22	−0.030	0.000	0.000	0.520	0.000	0.480	0.000	0.000
23	−0.017	0.000	0.424	0.000	0.000	0.359	0.217	0.000
24	−0.006	0.000	0.481	0.146	0.000	0.000	0.177	0.196
25	0.009	0.316	0.214	0.021	0.000	0.449	0.000	0.000	0.000	0.033	0.029	0.042	0.003	0.082	0.084	0.000	0.000	0.000	0.590	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.410	0.000
26	−0.017	0.000	0.000	0.000	0.567	0.260	0.173	0.000
27	0.040	0.046	0.000	0.315	0.173	0.229	0.237	0.000	0.073	0.093	0.031	0.053	0.059	0.017	0.144	0.000	0.000	0.000	1.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
28	0.004	0.000	0.325	0.000	0.000	0.417	0.000	0.258	0.047	0.012	0.024	0.108	0.000	0.037	0.000	0.000	0.000	0.000	0.764	0.000	0.084	0.151	0.000	0.000	0.000	0.000	0.000	0.000	0.000
29	−0.023	0.885	0.000	0.115	0.000	0.000	0.000	0.000
30	0.018	0.609	0.000	0.000	0.000	0.200	0.000	0.191	0.029	0.107	0.152	0.088	0.004	0.033	0.000	0.000	0.000	0.247	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.753	0.000
31	0.014	0.356	0.028	0.000	0.161	0.455	0.000	0.000	0.022	0.029	0.037	0.030	0.000	0.027	0.044	0.000	0.000	0.171	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.829	0.000
32	0.010	0.000	0.000	0.187	0.361	0.238	0.214	0.000	0.069	0.031	0.000	0.000	0.042	0.000	0.137	0.000	0.000	0.000	0.666	0.000	0.000	0.000	0.015	0.000	0.000	0.000	0.029	0.289	0.000
33	0.024	0.129	0.000	0.050	0.372	0.154	0.296	0.000	0.002	0.022	0.039	0.000	0.057	0.044	0.106	0.000	0.000	0.000	0.870	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.130	0.000
34	0.018	0.038	0.000	0.000	0.565	0.267	0.000	0.130	0.019	0.043	0.070	0.012	0.039	0.042	0.000	0.000	0.000	0.267	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.733	0.000
35	−0.014	0.000	0.000	0.000	0.678	0.000	0.000	0.322
36	0.012	0.609	0.000	0.000	0.000	0.200	0.000	0.191	0.017	0.058	0.065	0.085	0.000	0.051	0.008	0.000	0.000	0.352	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.648	0.000
37	0.017	0.609	0.000	0.000	0.000	0.200	0.000	0.191	0.026	0.054	0.065	0.029	0.005	0.021	0.000	0.000	0.000	0.172	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.828	0.000

Table A.3

Score, weights, slacks and reference countries obtained according to VBDEA method for the year of 2019 – Conservative

DMU	d*	w1	w2	w3	w4	w5	w6	w7	s1	s2	s3	s4	s5	s6	s7	DMU6	DMU8	DMU9	DMU12	DMU14	DMU19	DMU21	DMU22	DMU23	DMU24	DMU29	DMU32
1	0.024	0.334	0.000	0.202	0.000	0.000	0.007	0.457	0.067	0.148	0.005	0.103	0.054	0.041	0.000	0.000	0.000	0.712	0.000	0.000	0.000	0.000	0.000	0.000	0.288	0.000	0.000
2	−0.003	0.406	0.000	0.092	0.000	0.343	0.000	0.158
3	0.039	0.410	0.000	0.053	0.000	0.063	0.474	0.000	0.000	0.039	0.030	0.068	0.038	0.073	0.009	0.000	0.000	0.224	0.000	0.000	0.776	0.000	0.000	0.000	0.000	0.000	0.000
4	0.032	0.000	0.000	0.000	0.532	0.096	0.359	0.013	0.140	0.137	0.047	0.054	0.022	0.000	0.110	0.000	0.000	0.891	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.109	0.000
5	−0.002	0.122	0.000	0.369	0.000	0.107	0.023	0.379
6	−0.023	0.000	0.000	0.320	0.195	0.000	0.117	0.368
7	−0.012	0.000	0.000	0.261	0.234	0.128	0.000	0.377
8	−0.109	0.000	0.000	0.000	0.000	1.000	0.000	0.000
9	−0.048	0.233	0.255	0.000	0.000	0.000	0.044	0.468
10	0.009	0.451	0.052	0.000	0.000	0.408	0.000	0.089	0.000	0.070	0.014	0.053	0.000	0.097	0.059	0.000	0.000	0.538	0.000	0.308	0.000	0.000	0.000	0.000	0.000	0.154	0.000
11	0.005	0.000	0.000	0.179	0.330	0.491	0.000	0.000	0.066	0.047	0.016	0.008	0.000	0.002	0.079	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.116	0.000	0.000	0.884	0.000
12	−0.015	0.003	0.000	0.202	0.291	0.318	0.020	0.166
13	0.046	0.000	0.000	0.244	0.187	0.415	0.000	0.154	0.119	0.104	0.083	0.000	0.063	0.070	0.000	0.000	0.143	0.000	0.103	0.000	0.000	0.000	0.754	0.000	0.000	0.000	0.000
14	−0.023	0.423	0.000	0.000	0.048	0.530	0.000	0.000
15	0.018	0.489	0.000	0.008	0.000	0.282	0.221	0.000	0.013	0.016	0.058	0.106	0.042	0.000	0.141	0.000	0.000	0.000	0.000	0.413	0.000	0.000	0.000	0.000	0.000	0.587	0.000
16	−0.008	0.000	0.214	0.000	0.155	0.000	0.471	0.160
17	−0.013	0.000	0.000	0.000	0.444	0.380	0.132	0.044
18	−0.109	0.000	0.000	0.000	0.191	0.000	0.000	0.809
19	−0.038	0.000	0.000	0.257	0.209	0.000	0.534	0.000
20	0.036	0.000	0.371	0.000	0.000	0.037	0.497	0.095	0.027	0.000	0.062	0.002	0.079	0.052	0.073	0.000	0.889	0.000	0.000	0.000	0.000	0.000	0.000	0.111	0.000	0.000	0.000
21	−0.007	0.000	0.000	0.000	0.426	0.000	0.411	0.164
22	−0.074	0.000	0.000	0.508	0.000	0.492	0.000	0.000
23	−0.021	0.000	0.291	0.110	0.000	0.000	0.431	0.168
24	−0.025	0.000	0.399	0.147	0.000	0.000	0.254	0.200
25	0.030	0.392	0.039	0.069	0.000	0.368	0.000	0.132	0.055	0.063	0.000	0.090	0.000	0.142	0.050	0.000	0.000	0.441	0.000	0.292	0.000	0.000	0.000	0.000	0.000	0.267	0.000
26	0.027	0.000	0.000	0.033	0.477	0.301	0.190	0.000	0.096	0.065	0.055	0.051	0.002	0.000	0.087	0.000	0.000	0.000	0.000	0.000	0.410	0.000	0.000	0.000	0.000	0.590	0.000
27	0.058	0.214	0.000	0.257	0.000	0.000	0.461	0.069	0.000	0.063	0.009	0.024	0.099	0.105	0.106	0.000	0.000	0.000	0.430	0.000	0.000	0.000	0.000	0.570	0.000	0.000	0.000
28	−0.001	0.000	0.330	0.068	0.000	0.314	0.000	0.287
29	−0.085	0.378	0.000	0.000	0.622	0.000	0.000	0.000
30	0.091	0.000	0.000	0.000	0.508	0.461	0.000	0.031	0.177	0.165	0.230	0.106	0.082	0.143	0.000	0.000	0.000	0.000	0.000	0.438	0.000	0.000	0.000	0.000	0.000	0.562	0.000
31	0.038	0.502	0.008	0.000	0.000	0.490	0.000	0.000	0.074	0.078	0.126	0.102	0.000	0.082	0.086	0.000	0.000	0.000	0.000	0.223	0.000	0.000	0.000	0.000	0.000	0.777	0.000
32	−0.008	0.000	0.000	0.384	0.154	0.005	0.456	0.000
33	0.020	0.000	0.000	0.019	0.487	0.448	0.000	0.045	0.060	0.079	0.000	0.037	0.000	0.022	0.040	0.000	0.000	0.501	0.403	0.000	0.000	0.000	0.000	0.000	0.000	0.096	0.000
34	0.027	0.376	0.000	0.000	0.197	0.000	0.000	0.427	0.004	0.078	0.086	0.000	0.148	0.194	0.059	0.000	0.000	0.534	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.466	0.000
35	−0.063	0.000	0.279	0.000	0.315	0.000	0.000	0.407
36	0.010	0.451	0.052	0.000	0.000	0.408	0.000	0.089	0.000	0.040	0.093	0.125	0.020	0.174	0.000	0.000	0.000	0.068	0.000	0.458	0.000	0.000	0.000	0.000	0.000	0.473	0.000
37	−0.025	0.566	0.000	0.000	0.000	0.000	0.000	0.434

Table A.4

Score, weights, slacks and reference countries obtained according to VBDEA method for the year of 2020 – Conservative

DMU	d*	w1	w2	w3	w4	w5	w6	w7	s1	s2	s3	s4	s5	s6	s7	DMU2	DMU8	DMU9	DMU12	DMU14	DMU17	DMU18	DMU19	DMU22	DMU23	DMU24	DMU26	DMU29	DMU35
1	0.027	0.333	0.000	0.166	0.000	0.000	0.049	0.451	0.030	0.026	0.018	0.000	0.154	0.113	0.019	0.795	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.205
2	−0.041	0.381	0.000	0.097	0.000	0.101	0.417	0.003
3	0.043	0.200	0.000	0.000	0.285	0.140	0.375	0.000	0.000	0.015	0.067	0.003	0.063	0.088	0.050	0.231	0.000	0.000	0.000	0.000	0.000	0.000	0.769	0.000	0.000	0.000	0.000	0.000	0.000
4	0.045	0.000	0.000	0.077	0.425	0.000	0.349	0.148	0.119	0.116	0.000	0.003	0.069	0.057	0.162	0.000	0.000	0.987	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.013	0.000	0.000
5	0.005	0.000	0.000	0.492	0.015	0.019	0.287	0.187	0.038	0.091	0.000	0.036	0.021	0.000	0.023	0.039	0.000	0.000	0.753	0.000	0.000	0.000	0.000	0.000	0.000	0.208	0.000	0.000	0.000
6	−0.019	0.000	0.000	0.522	0.000	0.000	0.000	0.478
7	0.012	0.000	0.000	0.344	0.195	0.102	0.000	0.359	0.052	0.072	0.022	0.023	0.000	0.092	0.000	0.000	0.000	0.000	0.637	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.039	0.324
8	−0.026	0.000	0.000	0.000	0.000	1.000	0.000	0.000
9	−0.023	0.324	0.193	0.000	0.000	0.000	0.175	0.308
10	0.020	0.373	0.000	0.000	0.103	0.524	0.000	0.000	0.044	0.052	0.125	0.037	0.000	0.137	0.034	0.827	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.173	0.000
11	−0.007	0.000	0.000	0.222	0.306	0.000	0.379	0.093
12	−0.035	0.000	0.085	0.287	0.110	0.219	0.000	0.299
13	0.034	0.000	0.000	0.359	0.145	0.494	0.000	0.001	0.248	0.147	0.035	0.151	0.000	0.017	0.000	0.000	0.000	0.000	0.402	0.434	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.163	0.000
14	−0.012	0.300	0.000	0.000	0.148	0.552	0.000	0.000
15	0.036	0.072	0.202	0.244	0.000	0.205	0.277	0.000	0.120	0.049	0.060	0.120	0.011	0.000	0.172	0.420	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.580	0.000
16	0.011	0.000	0.371	0.000	0.000	0.000	0.476	0.153	0.071	0.031	0.103	0.036	0.009	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.540	0.000	0.241	0.000	0.219	0.000	0.000	0.000	0.000
17	−0.039	0.000	0.000	0.028	0.402	0.422	0.148	0.000
18	−0.104	0.166	0.000	0.000	0.041	0.000	0.000	0.793
19	−0.032	0.000	0.310	0.000	0.120	0.000	0.571	0.000
20	0.033	0.000	0.225	0.000	0.000	0.000	0.775	0.000	0.039	0.030	0.122	0.045	0.042	0.034	0.078	0.000	0.000	0.000	0.000	0.000	1.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
21	−0.012	0.000	0.000	0.000	0.473	0.000	0.393	0.134
22	−0.043	0.000	0.000	0.503	0.000	0.490	0.006	0.000
23	−0.011	0.000	0.423	0.000	0.000	0.262	0.315	0.000
24	−0.029	0.000	0.532	0.033	0.000	0.000	0.435	0.000
25	0.021	0.304	0.179	0.000	0.000	0.518	0.000	0.000	0.029	0.066	0.053	0.082	0.000	0.158	0.127	0.000	0.000	0.000	0.716	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.284	0.000
26	−0.055	0.000	0.000	0.000	0.556	0.000	0.444	0.000
27	0.047	0.000	0.000	0.292	0.224	0.194	0.276	0.014	0.100	0.133	0.053	0.083	0.055	0.000	0.144	0.000	0.000	0.000	0.961	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.039	0.000
28	0.010	0.000	0.316	0.000	0.000	0.365	0.000	0.319	0.031	0.000	0.002	0.079	0.000	0.014	0.032	0.000	0.000	0.000	0.579	0.000	0.063	0.358	0.000	0.000	0.000	0.000	0.000	0.000	0.000
29	−0.126	0.852	0.000	0.148	0.000	0.000	0.000	0.000
30	0.061	0.511	0.000	0.000	0.000	0.113	0.000	0.376	0.115	0.248	0.271	0.218	0.017	0.105	0.000	0.000	0.000	0.376	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.624	0.000
31	0.052	0.304	0.179	0.000	0.000	0.518	0.000	0.000	0.113	0.098	0.156	0.108	0.000	0.081	0.106	0.000	0.000	0.000	0.217	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.783	0.000
32	0.015	0.000	0.000	0.243	0.272	0.223	0.262	0.000	0.099	0.012	0.023	0.012	0.000	0.023	0.018	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.563	0.000	0.000	0.000	0.000	0.437	0.000
33	0.037	0.145	0.000	0.051	0.306	0.191	0.308	0.000	0.102	0.101	0.000	0.069	0.000	0.004	0.084	0.000	0.000	0.871	0.037	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.092	0.000
34	0.056	0.016	0.000	0.000	0.515	0.269	0.000	0.200	0.084	0.118	0.154	0.039	0.130	0.140	0.000	0.000	0.000	0.400	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.600	0.000
35	−0.048	0.000	0.000	0.000	0.571	0.000	0.000	0.429
36	0.040	0.494	0.000	0.000	0.000	0.244	0.000	0.262	0.066	0.077	0.172	0.131	0.031	0.189	0.000	0.516	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.484	0.000
37	0.057	0.016	0.000	0.000	0.515	0.269	0.000	0.200	0.117	0.158	0.167	0.096	0.022	0.074	0.000	0.000	0.000	0.279	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.721	0.000

Either in the Neutral or in the Conservative perspective, Santa Maria Maior was efficient because of its doctors (w1 = 0.36 and w1 = 2 0.38, in the first and second models, respectively), and beds (w4 = 0.64 and w4 = 0.62, in the first and latter model, respectively) in 2019. In 2020 it became efficient mainly because of its doctors and operational staff. The fact that the number of doctors was reduced by 15% from 2019 to 2020 in this hospital has had a significant contribution to this outcome. In the case of Tâmega e Sousa all factors are important in the Neutral perspective while in the Conservative perspective almost all factors (except for nursing staff) have a positive impact on efficiency in 2019, whereas in 2020 almost all factors (except for doctors and outpatient visits) have a positive contribution to efficiency (in both models). In the case of Entre o Douro e Vouga, its efficiency in 2019 is mainly explained because of its capacity of handling emergency visits and because of its nursing staff and doctors. In this same hospital, in 2020, and from the Neutral perspective, doctors and emergency visits are the most important factors, while in the Conservative perspective, doctors, emergency visits, nursing staff and outpatient visits are the most relevant factors for efficiency. In the case of Coimbra, the number of inpatient discharges has been elected for explaining its efficiency in both years according to the Conservative perspective, but under the Neutral perspective beds are the factor elected by these hospitals to explain efficiency. It is worth mentioning that for this latter hospital these results are obtained without increasing the number of doctors in FTE and with a slight increase of 5% and 8% of the nursing and operational, respectively, between 2019 and 2020. Furthermore, while in 2019 these results could be considered positive, in 2020 these findings might not reflect a good situation because many patients were discharged from hospitals to free up beds during the pandemic [52].

From the analysis of Fig. 5 , it can be concluded that efficient hospitals in 2020 had to work with an average of fewer doctors and beds than efficient hospitals in 2019 with a small increase in the nursing and operational staff, while inefficient hospitals increased substantially their resources in 2020 when compared to inefficient hospitals in 2019. The activity of efficient hospitals in 2020 has been substantially impacted by COVID-19 if we contrast the activity levels attained by efficient hospitals in 2020 against that of efficient hospitals in 2019. These findings are supported by the fact that the COVID-19 pandemic radically changed the delivery of outpatient care in 2020, leading to the deferral of elective visits, and increasing the use of telemedicine. Furthermore, many patients were also avoiding visits because of risk exposure.

Fig. 5

Variation of the average values attained for the factors of efficient and inefficient hospitals between 2019 and 2020.

Finally, in the case of inefficient hospitals, outpatient visits had a mild increase of 1% in 2020 but at the expense of a substantial increase in the hospital staff when compared to 2019.

From the examination of Fig. 6 , it can be concluded that when we contrast efficient with inefficient SOE hospitals, in 2019 efficient hospitals had, on average, access to a higher volume of resources but could also produce much more. The same is also particularly true in 2020, i.e., efficient hospitals had on average more resources than inefficient hospitals (particularly, doctors, nursing staff and beds), but could also have much higher activity levels than that of inefficient hospitals (although the situation became worse when compared to 2019).

Fig. 6

Difference between the average factors of efficient vs inefficient and efficient vs all hospitals in 2019 and 2020.

If we consider the results obtained with Neutral value functions, it can be seen from the left-hand half of Fig. 7 that, in order to become efficient, beds are the resources requiring higher adjustments, when compared to 2019. Nevertheless, according to the Conservative perspective, from the analysis of the right-hand half of Fig. 7 it becomes evident that to become efficient like their peers, in 2020, bigger adjustments are required in all resources (when compared to 2019), suggesting that this latter model leads to more balanced improvements.

Fig. 7

Average improvements required from inefficient hospitals to become efficient between 2019 and 2020.

We have performed a robustness analysis of the results in face of uncertain information through the use of linear problems (3) and (4) and considering a tolerance δ equal to 5% and 10%. This type of analysis enabled us to assess whether each SOE hospital is surely efficient, potentially efficient, or surely inefficient.

Table 4 illustrates the robustness results of some hospitals according to the Neutral and Conservative perspectives, respectively (the remaining hospitals are all potentially efficient across all tolerances in both years). Algarve and Santa Maria Maior are the only hospitals that become surely efficient for all the tolerances considered (i.e. 5% and 10%), for both years of the analysis and both models. Guimarães only loses this same status in 2020 for a tolerance of 10% (in both models). It is also possible to see that Coimbra becomes potentially efficient in 2020, in spite of being surely efficient in 2019 for both tolerances, showing one of the possible impacts of COVID-19 on the efficiency of this hospital (in both models).

Table 4

Robustness assessment of efficiency scores.

		Conservative		Neutral		Conservative		Conservative		Neutral		Neutral
		Efficiency Score	Efficiency Score	Efficiency Score	Efficiency Score	5%	10%	5%	10%	5%	10%	5%	10%
DMU	Hospital	2019	2020	2019	2020	2019	2019	2020	2020	2019	2019	2020	2020
18	Centro Hospitalar Universitário do Algarve,EPE	−0.1085	−0.1042	−0.2203	−0.1620	E++	E++	E++	E++	E++	E++	E++	E++
17	Centro Hospitalar Universitário de São João, EPE	−0.0127	−0.0394	−0.0288	−0.0750	E + -	E + -	E++	E + -	E + -	E + -	E++	E + -
19	Centro Hospitalar Universitário do Porto, EPE	−0.0377	−0.0322	−0.0678	−0.0626	E++	E + -	E + -	E + -	E++	E + -	E++	E + -
8	Centro Hospitalar e Universitário de Coimbra, EPE	−0.1095	−0.0259	−0.2494	−0.0487	E++	E++	E + -	E + -	E++	E++	E + -	E + -
2	Centro Hospitalar de Leiria, EPE	−0.0029	−0.0410	−0.0031	−0.0353	E + -	E + -	E++	E + -	E + -	E + -	E++	E + -
12	Centro Hospitalar Tâmega e Sousa, EPE	−0.0152	−0.0350	−0.0150	−0.0325	E + -	E + -	E + -	E + -	E + -	E + -	E++	E + -
22	Hospital da Senhora da Oliveira, Guimarães, EPE	−0.0738	−0.0432	−0.0484	−0.0296	E++	E++	E++	E + -	E++	E++	E++	E + -
29	Hospital Santa Maria Maior, EPE	−0.0846	−0.1263	−0.0172	−0.0226	E++	E++	E++	E++	E++	E++	E++	E++
9	Centro Hospitalar Entre Douro e Vouga, EPE	−0.0483	−0.0226	−0.0412	−0.0207	E++	E + -	E + -	E + -	E++	E + -	E++	E + -
26	Hospital Espírito Santo de Évora, EPE	0.0267	−0.0552	0.0128	−0.0171	E + -	E + -	E++	E + -	E + -	E + -	E++	E + -
23	Hospital de Braga, EPE	−0.0211	−0.0108	−0.0316	−0.0169	E + -	E + -	E + -	E + -	E + -	E + -	E + -	E + -
35	Unidade Local de Saúde do Litoral Alentejano, EPE	−0.0630	−0.0483	−0.0174	−0.0137	E++	E + -	E++	E + -	E++	E + -	E++	E + -
36	Unidade Local de Saúde do Nordeste, EPE	0.0101	0.0399	0.0018	0.0118	E + -	E + -	E−	E + -	E + -	E + -	E + -	E + -
31	Unidade Local de Saúde de Castelo Branco, EPE	0.0379	0.0519	0.0105	0.0136	E−	E + -	E−	E + -	E + -	E + -	E−	E + -
15	Centro Hospitalar Universitário Cova da Beira, EPE	0.0184	0.0356	0.0049	0.0140	E + -	E + -	E−	E + -	E + -	E + -	E−	E + -
37	Unidade Local de Saúde do Norte Alentejano, EPE	−0.0246	0.0571	−0.0079	0.0167	E + -	E + -	E−	E + -	E + -	E + -	E−	E + -
34	Unidade Local de Saúde do Baixo Alentejo, EPE	0.0267	0.0564	0.0104	0.0181	E + -	E + -	E−	E + -	E + -	E + -	E−	E + -
30	Unidade Local de Saúde da Guarda, EPE	0.0915	0.0610	0.0407	0.0183	E−	E−	E−	E + -	E−	E−	E−	E + -
33	Unidade Local de Saúde do Alto Minho, EPE	0.0198	0.0372	0.0129	0.0239	E + -	E + -	E−	E + -	E + -	E + -	E−	E + -
13	Centro Hospitalar Tondela-Viseu, EPE	0.0463	0.0344	0.0392	0.0252	E−	E + -	E + -	E + -	E−	E + -	E + -	E + -
4	Centro Hospitalar de Setúbal, EPE	0.0323	0.0452	0.0212	0.0258	E + -	E + -	E−	E + -	E + -	E + -	E−	E + -
27	Hospital Garcia de Orta, EPE	0.0581	0.0467	0.0525	0.0397	E−	E + -	E−	E + -	E−	E + -	E−	E + -
3	Centro Hospitalar de Lisboa Ocidental, EPE	0.0386	0.0426	0.0465	0.0481	E−	E + -	E−	E + -	E + -	E + -	E−	E + -
20	Centro Hospitalar Universitário Lisboa Central, EPE	0.0358	0.0329	0.0717	0.0717	E−	E + -	E + -	E + -	E + -	E + -	E−	E + -

Litoral Alentejano is surely efficient for a tolerance of 5% in both years (in both models), while Douro e Vouga and Porto are also surely efficient in the same conditions for the Neutral perspective but become surely efficient for this same tolerance only in 2019 (in the Conservative perspective). Curiously, despite COVID-19, some hospitals only became robustly efficient in 2020. This is the case of Leiria, Évora and São João which become surely efficient for a tolerance of 5% (in both models), whereas Tâmega e Sousa is in this same situation under the Neutral perspective. Guarda only becomes potentially efficient in 2020 for a tolerance of 10%, whereas its behaviour is always surely inefficient for the remaining cases. Lisboa Ocidental, Garcia da Orta and Castelo Branco are robustly inefficient in both years for a tolerance of 5%. Overall, it can be established that regardless of the model used, the hospitals located in the northern region of the country are more robustly efficient (other studies had also suggested these findings early – see [16]) in the years studied.

Productivity Analysis

Next, we analyse the results regarding the evolution of TFP after solving problems (6) and (7). Changes in TFP are also split into TECHCH and EFFCH. While the former illustrates shifts in the production frontier, the latter assesses changes in the position of a DMU relative to the frontier. A positive TFP value means that the SOE hospital achieved a higher level of productivity across the period under analysis.

The illustration of the TFP drivers of the SOE efficient hospitals according to distinct value functions is provided in Fig. 8 . From its evaluation it is possible to ascertain that only two hospitals show productivity gains from 2019 to 2020 (Évora and Santa Maria Maior), mainly explained by efficiency gains. Despite of their negative productivity Leiria, São João do Porto, Tâmega e Sousa, Vila do Conde, Espinho and Figueira da Foz showed efficiency gains. As would be expected, COVID-19 caused a general technological deterioration (TECHCH <0) for all 37 hospitals. These findings highlight the massive impact that the COVID-19 outbreak has had on SOE hospitals, which needed to free up enough resources to deal with the pandemic, being forced to shut down or significantly reduce many non-COVID care areas in 2020. At the same time, the fact that fewer patients were seeking care led to a significant drop in elective procedures, and outpatient visits.

Fig. 8

Contribution to TFP - efficient hospitals in 2020.

Weight restrictions

In section 3 when the VBDEA method was introduced it was said that the weights used in the global value function are the scale coefficients (weights) of the partial value functions and that they translate the possible trade-off values between the different criteria (see [53]). To avoid null weights, we can also restrict them according to the preferences of the DMs. For that purpose, we use the swing method, like what has already been used with real DMs in previous studies [21,22]. At first, a ranking of weights is obtained and after that, a limit to the ratio between the weights ranked first and last is defined. According to this procedure (“swinging” from the value 0 to 1), the weight restrictions were elicited.

We have considered two scenarios established by the DM. According to the first scenario, the criterion with the highest scaling constant is “Number of Beds” and after this one comes “Number of Inpatient Discharges” (an alternative way of obtaining additional beds), followed by “Number of emergency and outpatient visits”, “Number of outpatient visits”, “Number of nursing staff”, “Number of physicians” and in the last position we have the “Number of operational staff”. Therefore, the following ranking of the coefficients of scale was obtained: w4 ≥ w5 ≥ w7 ≥ w6≥ w2≥ w1≥ w3. Hence, our DM aimed at understanding which hospitals could tackle the highest number of emergency and outpatient visits with the smallest number of beds possible, reducing the hospital staff to the least possible.

In line with the second scenario, the following ranking of the coefficients of scale was obtained: w5 ≥ w7 ≥ w6≥ w3≥ w1≥ w2≥ w4. Thus, our DM aimed at evaluating which were the hospitals that remained efficient if a higher importance level is given to their activity, with the smallest hospital staff possible, and reducing the number of beds the least possible.

Let W denote the set of weight vectors compatible with the ranking of the scale coefficients obtained. The set W is added to problem (1), which leads to a new formulation including (w1, …, w7) W. With this change in Phase 1, we also had to change the formulation of problem (2) which is solved in Phase 2. It was now necessary to free the slacks; since, otherwise, it might not be possible to keep the optimal value difference resulting from (1) (for further details see [54]).

The results considering the sets of weights added to problem (1) and the adjustment made in problem (2) considering scenario 1 in 2020 are presented in Table 5 . In this new setting, the number of hospitals that remains efficient becomes reduced to 6. Thus, the hospitals which remain efficient are in decreasing order: Santa Maria Maior (15 times as benchmark), Évora (7 times as benchmark), São João (12 times as benchmark), Douro e Vouga (8 times as benchmark), Tâmega e Sousa (21 times as benchmark) and Litoral Alentejano (3 times as benchmark). Then again, the majority of efficient hospitals belongs to the northern region of the country.

Table 5

Score, weights, slacks and reference countries obtained according to VBDEA method with weight constraints – Scenario 1 (2020).

DMUs	d*	w1	w2	w3	w4	w5	w6	w7	s1	s2	s3	s4	s5	s6	s7	DMU9	DMU12	DMU17	DMU26	DMU29	DMU35
1	0.073	0.120	0.120	0.120	0.160	0.160	0.160	0.160	0.031	0.108	0.067	0.080	0.157	0.063	0.000	0.000	0.697	0.000	0.000	0.303	0.000
2	0.014	0.120	0.120	0.120	0.160	0.160	0.160	0.160	−0.065	0.067	0.000	0.065	0.015	−0.043	0.050	0.000	0.926	0.000	0.000	0.074	0.000
3	0.077	0.001	0.001	0.001	0.444	0.184	0.184	0.184	−0.046	−0.054	0.000	−0.041	0.196	0.092	0.230	0.000	0.312	0.688	0.000	0.000	0.000
4	0.047	0.002	0.002	0.002	0.477	0.267	0.126	0.126	0.121	0.115	0.000	0.000	0.070	0.059	0.165	0.977	0.016	0.000	0.007	0.000	0.000
5	0.033	0.120	0.120	0.120	0.160	0.160	0.160	0.160	0.067	0.091	0.000	0.056	0.032	−0.013	0.014	0.000	0.793	0.000	0.000	0.207	0.000
6	0.032	0.104	0.159	0.104	0.159	0.159	0.159	0.159	0.216	0.117	0.086	0.161	0.000	−0.104	−0.171	0.000	0.281	0.000	0.000	0.719	0.000
7	0.017	0.002	0.002	0.002	0.526	0.257	0.002	0.210	0.049	0.060	−0.097	0.000	0.027	0.148	0.046	0.828	0.000	0.000	0.000	0.000	0.172
8	0.020	0.001	0.001	0.001	0.249	0.249	0.249	0.249	0.040	0.069	0.105	0.120	−0.020	−0.006	−0.013	0.000	0.000	1.000	0.000	0.000	0.000
9	−0.013	0.160	0.160	0.001	0.198	0.160	0.160	0.160
10	0.056	0.056	0.056	0.013	0.385	0.377	0.056	0.056	0.000	0.112	−0.002	0.086	0.006	0.128	0.121	0.914	0.000	0.000	0.000	0.086	0.000
11	0.016	0.001	0.151	0.001	0.394	0.151	0.151	0.151	0.076	−0.002	−0.077	0.000	0.061	0.008	0.041	0.212	0.000	0.000	0.000	0.788	0.000
12	−0.012	0.048	0.048	0.048	0.270	0.270	0.048	0.270
13	0.080	0.002	0.002	0.002	0.463	0.463	0.004	0.066	−0.052	−0.130	−0.182	0.000	0.163	0.265	0.062	0.000	0.000	0.710	0.290	0.000	0.000
14	0.006	0.002	0.002	0.002	0.463	0.463	0.004	0.066	−0.067	0.000	−0.024	0.000	0.002	0.090	0.082	0.000	0.561	0.334	0.105	0.000	0.000
15	0.083	0.104	0.159	0.104	0.159	0.159	0.159	0.159	0.075	0.063	0.044	0.131	0.038	0.000	0.213	0.000	0.425	0.000	0.000	0.575	0.000
16	0.028	0.001	0.199	0.001	0.200	0.200	0.200	0.200	0.074	−0.002	0.078	0.023	0.058	0.000	0.060	0.000	0.154	0.846	0.000	0.000	0.000
17	−0.038	0.001	0.001	0.001	0.394	0.394	0.104	0.104
18	0.009	0.001	0.001	0.001	0.389	0.303	0.001	0.303	0.116	0.174	0.137	0.152	0.000	0.100	−0.169	0.000	0.741	0.259	0.000	0.000	0.000
19	0.038	0.001	0.001	0.001	0.444	0.184	0.184	0.184	0.000	−0.074	−0.058	−0.048	0.134	−0.014	0.201	0.000	0.171	0.829	0.000	0.000	0.000
20	0.045	0.001	0.032	0.001	0.436	0.436	0.046	0.046	0.044	0.030	0.122	0.045	0.044	0.034	0.078	0.000	0.000	1.000	0.000	0.000	0.000
21	0.014	0.001	0.001	0.001	0.444	0.184	0.184	0.184	0.108	0.081	0.107	0.000	0.051	−0.077	0.099	0.000	0.741	0.259	0.000	0.000	0.000
22	0.064	0.120	0.120	0.120	0.160	0.160	0.160	0.160	0.035	0.057	−0.086	0.212	0.000	0.025	0.156	0.000	0.890	0.000	0.000	0.110	0.000
23	0.009	0.001	0.199	0.001	0.200	0.200	0.200	0.200	0.000	−0.058	−0.043	0.000	0.045	0.008	0.052	0.000	0.538	0.462	0.000	0.000	0.000
24	0.014	0.104	0.159	0.104	0.159	0.159	0.159	0.159	0.084	−0.048	−0.043	0.041	0.079	−0.008	0.000	0.000	0.111	0.000	0.000	0.889	0.000
25	0.052	0.042	0.042	0.042	0.387	0.387	0.042	0.060	−0.063	0.000	−0.022	0.007	0.077	0.241	0.219	0.000	0.879	0.000	0.000	0.121	0.000
26	−0.038	0.002	0.002	0.002	0.502	0.490	0.002	0.002
27	0.072	0.014	0.014	0.014	0.431	0.431	0.048	0.048	0.038	0.060	0.019	0.135	0.029	0.000	0.000	0.000	0.407	0.213	0.380	0.000	0.000
28	0.038	0.034	0.034	0.034	0.288	0.288	0.034	0.288	0.000	−0.023	−0.008	0.074	0.056	0.118	−0.009	0.000	0.636	0.364	0.000	0.000	0.000
29	−0.048	0.003	0.003	0.003	0.980	0.003	0.003	0.003
30	0.121	0.059	0.059	0.001	0.394	0.368	0.059	0.059	−0.174	0.038	−0.043	0.000	0.245	0.372	0.282	0.905	0.000	0.000	0.095	0.000	0.000
31	0.066	0.053	0.053	0.021	0.383	0.383	0.053	0.053	0.000	0.016	0.063	0.016	0.095	0.183	0.218	0.000	0.418	0.000	0.000	0.582	0.000
32	0.047	0.039	0.054	0.039	0.379	0.379	0.054	0.054	0.086	0.042	0.000	0.108	0.000	−0.056	0.060	0.000	0.451	0.000	0.499	0.050	0.000
33	0.044	0.056	0.056	0.013	0.385	0.377	0.056	0.056	0.111	0.105	0.000	0.073	−0.002	0.000	0.080	0.904	0.000	0.000	0.000	0.096	0.000
34	0.057	0.002	0.002	0.002	0.526	0.257	0.002	0.210	−0.185	−0.066	−0.026	0.000	0.222	0.271	0	0.293	0.000	0.000	0.664	0.000	0.043
35	−0.007	0.002	0.002	0.002	0.533	0.230	0.002	0.230
36	0.095	0.127	0.127	0.090	0.176	0.176	0.127	0.176	0.000	0.089	0.145	0.139	0.065	0.196	0.058	0.000	0.535	0.000	0.000	0.465	0.000
37	0.059	0.007	0.007	0.007	0.512	0.260	0.007	0.200	0.000	0.074	0.032	0.000	0.114	0.180	0.135	0.516	0.000	0.000	0.000	0.422	0.062

The results obtained considering scenario 2 in 2020 are presented in Table 6 . In this new weight ranking scenario, the number of hospitals that remains efficient becomes reduced to 4. These are in decreasing order: São João (9 times as benchmark), Santa Maria Maior (20 times as benchmark), Coimbra (1 time as benchmark), Tâmega e Sousa (32 times as benchmark). Once more, the majority of efficient hospitals belong to the northern region of the country.

Table 6

Score, weights, slacks and reference countries obtained according to VBDEA method with weight constraints – Scenario 2 (2020).

DMUs	d*	w1	w2	w3	w4	w5	w6	w7	s1	s2	s3	s4	s5	s6	s7	DMU8	DMU12	DMU17	DMU29
1	0.069	0.162	0.095	0.162	0.095	0.162	0.162	0.162	−0.049	0.049	0.000	0.014	0.224	0.135	0.080	0.000	0.840	0.000	0.160
2	0.000	0.194	0.028	0.194	0.000	0.194	0.194	0.194	−0.106	0.037	−0.034	0.031	0.050	−0.006	0.092	0.000	1.000	0.000	0.000
3	0.087	0.103	0.103	0.103	0.103	0.196	0.196	0.196	0.000	−0.001	0.037	−0.004	0.161	0.050	0.218	0.000	0.442	0.558	0.000
4	0.083	0.129	0.129	0.129	0.129	0.226	0.129	0.129	0.202	0.176	0.209	0.076	0.000	−0.048	0.031	0.000	0.683	0.000	0.317
5	0.030	0.162	0.095	0.162	0.095	0.162	0.162	0.162	0.067	0.091	0.000	0.056	0.032	−0.013	0.014	0.000	0.793	0.000	0.207
6	0.029	0.161	0.161	0.161	0.036	0.161	0.161	0.161	0.000	−0.039	−0.092	−0.015	0.181	0.090	0.045	0.000	0.664	0.000	0.336
7	0.042	0.130	0.130	0.130	0.130	0.175	0.130	0.175	0.205	0.169	0.161	0.129	−0.084	0.000	−0.167	0.000	0.442	0.000	0.558
8	−0.020	0.001	0.001	0.001	0.001	0.994	0.001	0.001
9	0.005	0.130	0.130	0.130	0.130	0.160	0.160	0.160	−0.058	−0.040	0.098	−0.035	0.045	0.015	0.000	0.000	0.926	0.000	0.074
10	0.066	0.129	0.129	0.129	0.129	0.226	0.129	0.129	0.068	0.163	0.188	0.153	−0.054	0.033	0.000	0.000	0.632	0.000	0.368
11	0.016	0.120	0.120	0.160	0.120	0.160	0.160	0.160	0.105	0.020	−0.022	0.026	0.036	−0.026	0.000	0.000	0.124	0.000	0.876
12	−0.019	0.098	0.098	0.152	0.098	0.201	0.152	0.201
13	0.083	0.076	0.076	0.193	0.076	0.193	0.193	0.193	0.170	0.136	−0.016	0.102	0.050	0.102	0.136	0.000	1.000	0.000	0.000
14	0.020	0.107	0.107	0.107	0.107	0.358	0.107	0.107	−0.032	0.044	0.000	0.000	−0.004	0.073	0.121	0.000	0.794	0.206	0.000
15	0.074	0.161	0.161	0.161	0.036	0.161	0.161	0.161	0.022	0.025	0.000	0.088	0.082	0.047	0.266	0.000	0.518	0.000	0.482
16	0.041	0.103	0.103	0.103	0.103	0.196	0.196	0.196	0.074	−0.002	0.078	0.023	0.058	0.000	0.060	0.000	0.154	0.846	0.000
17	−0.026	0.053	0.053	0.210	0.053	0.210	0.210	0.210
18	0.021	0.096	0.096	0.096	0.096	0.261	0.096	0.261	−0.034	0.000	0.017	0.031	0.115	0.239	−0.128	0.000	0.314	0.686	0.000
19	0.042	0.076	0.076	0.193	0.076	0.193	0.193	0.193	−0.015	−0.091	−0.070	−0.060	0.145	0.000	0.205	0.000	0.129	0.871	0.000
20	0.048	0.046	0.046	0.046	0.046	0.724	0.046	0.046	0.004	−0.039	0.017	−0.075	0.064	0.040	0.092	1.000	0.000	0.000	0.000
21	0.045	0.103	0.103	0.103	0.103	0.196	0.196	0.196	0.025	−0.015	0.041	−0.067	0.115	0.000	0.122	0.000	0.505	0.495	0.000
22	0.033	0.167	0.167	0.167	0.000	0.167	0.167	0.167	−0.027	0.012	−0.137	0.162	0.052	0.081	0.218	0.000	1.000	0.000	0.000
23	0.006	0.106	0.106	0.197	0.000	0.197	0.197	0.197	0.009	−0.048	−0.036	0.007	0.038	0.000	0.049	0.000	0.562	0.438	0.000
24	0.012	0.161	0.161	0.161	0.036	0.161	0.161	0.161	0.076	−0.054	−0.050	0.034	0.087	0.000	0.009	0.000	0.126	0.000	0.874
25	0.067	0.129	0.129	0.129	0.129	0.226	0.129	0.129	0.000	0.045	0.029	0.058	0.025	0.185	0.156	0.000	0.767	0.000	0.233
26	0.073	0.129	0.129	0.129	0.129	0.226	0.129	0.129	0.000	−0.014	−0.044	−0.268	0.195	0.153	0.400	0.000	0.840	0.000	0.160
27	0.075	0.193	0.016	0.193	0.016	0.193	0.193	0.193	−0.022	0.000	−0.046	−0.016	0.152	0.113	0.193	0.000	0.714	0.286	0.000
28	0.023	0.126	0.126	0.126	0.000	0.247	0.126	0.247	0.073	0.062	0.051	0.133	0.000	0.050	−0.029	0.000	0.844	0.156	0.000
29	−0.021	0.135	0.135	0.135	0.135	0.192	0.135	0.135
30	0.130	0.130	0.130	0.130	0.130	0.175	0.130	0.175	0.095	0.233	0.308	0.205	0.035	0.111	0.000	0.000	0.348	0.000	0.652
31	0.085	0.129	0.129	0.129	0.129	0.226	0.129	0.129	0.113	0.098	0.156	0.108	0.000	0.081	0.105	0.000	0.217	0.000	0.783
32	0.067	0.129	0.129	0.129	0.129	0.226	0.129	0.129	0.117	0.057	0.003	0.000	0.072	−0.008	0.229	0.000	0.816	0.000	0.184
33	0.054	0.129	0.129	0.129	0.129	0.226	0.129	0.129	0.008	0.033	0.047	0.000	0.080	0.059	0.131	0.000	0.926	0.000	0.074
34	0.095	0.130	0.130	0.130	0.130	0.175	0.130	0.175	0.062	0.102	0.193	0.025	0.149	0.146	0.000	0.000	0.370	0.000	0.630
35	0.026	0.130	0.130	0.130	0.130	0.175	0.130	0.175	0.064	0.013	0.099	0.000	0.081	0.074	−0.118	0.000	0.104	0.000	0.896
36	0.094	0.161	0.161	0.161	0.036	0.161	0.161	0.161	0.000	0.089	0.145	0.139	0.065	0.196	0.058	0.000	0.535	0.000	0.465
37	0.085	0.130	0.130	0.130	0.130	0.175	0.130	0.175	0.144	0.177	0.229	0.120	0.000	0.040	−0.041	0.000	0.184	0.000	0.816

Conclusions

This paper provides an additional understanding of the main factors that have influenced the efficiency and productivity of Portuguese SOE hospitals before and after the COVID-19 pandemic hit the country, using data made publicly available by the NHS. To the best of our knowledge, this is the first work that proposes the use of the VBDEA framework in the hospital context.

The use of VBDEA goes beyond traditional DEA approaches by converting inputs and outputs into value scales, being particularly useful for handling the DMs’ preferences. In this study, we had a real DM to elicit value functions, an SOE hospital clinical board member. To explore the advantage of our approach, we have considered two types of value functions, which reflected two perspectives towards the pandemic context: neutral value functions defined linearly and conservative value functions non-linearly defined, consistent with the preferences of the DM in the context of the COVID-19 turmoil. This is particularly relevant for the outputs for which the DM followed the principle “the more the less good”. In the first stage, resource consumption and services production were converted into multiple value functions that were aggregated using a weighted sum (MAUT additive model), which allows each DMU to select the weights associated with these value functions in a way that minimizes the value difference for the best DMU, according to the min-max regret rule. Then, these weights were restricted according to the preferences of the DM, enabling him/her to deal with the fact that otherwise important factors could be ignored in the analysis. In fact, under the Hospital's efficiency evaluation perspective, this feature can be particularly appropriate to support management decisions, as it allows obtaining the reference hospitals that should be considered benchmarks in terms of good practices following different scenarios. This was done by imposing a weight sort order according to the political concerns of the DM assuming two scenarios. In the first scenario, we consider that the DM is more concerned with the resources available, whereas, in the second scenario, we consider that the DM is more affected by the services activities of the hospitals. In addition, this approach also encompasses robustness analyses, allowing the DM to understand how sensitive the efficiency of these hospitals is to the variation of data.

According to our findings and regardless of the value functions considered, out of the 37 SOE hospitals, 21 and 17 were efficient in 2019 and 2020, respectively. Nevertheless, it is interesting to see that whereas under a Neutral perspective the average Hospitals’ efficiency decreased 61% from 2019 to 2020, under a Conservative perspective (adopted by a real DM considering the COVID scenario underway) the average efficiency score of efficient SOE hospitals decreased even further by 100% between 2019 and 2020. Overall, without the possibility of changing the size and structure of the NHS in the short run, the Portuguese hospitals, like in many other countries, did not manage to improve their efficiency in face of COVID-19. Our findings are in line with other recent studies, which classified the Portuguese health system during the pandemic as inefficient in terms of the use of its resources [[2], [28]].

Another important factor refers to the negative impact of COVID-19 on hospitals’ activity levels, which was more pronounced in efficient hospitals than in inefficient hospitals during these two years. These findings imply that COVID-19 had less of an impact on these latter units' activity levels but at the price of a significant increase in their level of total resources. While efficient hospitals in 2020 had to operate with fewer physicians and beds on average than efficient hospitals in 2019, with a slight rise in nursing and operational personnel, inefficient hospitals substantially increased all their resources in 2020 as compared to inefficient hospitals in 2019. Hence, the results obtained seem to corroborate those of [2] and [28], who concluded, at least for European countries, that the inefficiency of the public health sector was not due to a lack of medical resources but to their inefficient use. Hence, to become more resilient (even for future COVID-19 outbreaks), hospitals should operate adjustments, which are beneficial regardless of the hurdles they face and are even useful during normal times. A culture of cooperation within and across hospitals should also be fostered, which allows the allocation of resources where they can be used more efficiently.

Under a more stringent scenario that considers a higher concern of the DM with the hospital resources available, i.e., if we evaluate which hospitals manage to remain efficient by tackling the highest number of emergency and outpatient visits with the smallest number of beds possible, reducing the hospital staff the least possible, the number of efficient hospitals becomes reduced to 6. However, if we evaluate which hospitals manage to remain efficient if a higher importance level is given to their activity, with the smallest hospital staff possible, and reducing the number of beds to the least possible, the number of efficient hospitals becomes even further reduced to 4. These latter results reflect the impact of the changes operated in the hospital admission criteria during the COVID-19 crisis. Some of the adopted measures involved the cancellation of elective admissions, which included surgical procedures and medical and interventional procedures. These outcomes were further exacerbated by the lower number of patients seeking care because of the fear of getting infected.

The fact that in the previous scenarios, the majority of efficient hospitals belonged to the northern region of the country and that, irrespective of the model employed, the hospitals located in this same region were more robustly efficient over the years analysed, highlights the need for undertaking specific measures to address these interregional divergences by policymakers and hospital administrators.

Curiously, the hospitals more often viewed as a reference for best practices were Santa Maria Maior, Tâmega e Sousa and Entre Douro e Vouga, also irrespective of the value functions considered. Overall, the majority of SOE hospitals showed negative productivity (except for Évora and Santa Maria Maior) and all of them presented negative technological change, thus highlighting the massive impact that the COVID-19 outbreak has had on the performance of these hospitals.

Authors statement

Carla Oliveira Henriques – Conceptualization; Methodology; Formal Analysis; Investigation; Validation; Resources; Writing - Review & Editing; Maria do Castelo Gouveia – Methodology; Formal analysis; Investigation; Writing - Review & Editing.