System impact research - increasing public health and health care system performance.

BACKGROUND: Interventions directed to system features of public health and health care should increase health and welfare of patients and population. AIMS: To build a new framework for studies aiming to assess the impact of public health or health care system, and to consider the role of Randomized Controlled Trials (RCTs) and of Benchmarking Controlled Trials (BCTs).
METHODS: The new concept is partly based on the author's previous paper on the Benchmarking Controlled Trial. The validity and generalizability considerations were based on previous methodological studies on RCTs and BCTs.
RESULTS: The new concept System Impact Research (SIR) covers all the studies which aim to assess the impact of the public health system or of the health care system on patients or on population. There are two kinds of studies in System Impact Research: Benchmarking Controlled Trials (observational) and Randomized Controlled Trials (experimental). The term impact covers in particular accessibility, quality, effectiveness, safety, efficiency, and equality.
CONCLUSIONS: System Impact Research - creating the scientific basis for policy decision making - should be given a high priority in medical, public health and health economic research, and should also be used for improving performance. Leaders at all levels of health and social care can use the evidence from System Impact Research for the benefit of patients and population. Key messages The new concept of SIR is defined as a research field aiming at assessing the impacts on patients and on populations of features of public health and health and social care systems or of interventions trying to change these features. SIR covers all features of public health and health and social care system, and actions upon these features. The term impact refers to all effects caused by the public health and health and social care system or parts of it, with particular emphasis on accessibility, quality, effectiveness, adverse effects, efficiency, and equality of services. SIR creates the scientific basis for policy decisions. Leaders at all levels of health and social care can use the evidence from SIR for the benefit of the patients and the population.

Background

The foremost aims in public health and in health and (integrated) social care are to increase effectiveness, safety, efficiency, and equality of the system (1,2). In clinical research most studies aim at assessing effectiveness of a particular intervention targeted at the patient or the population. However, systems guiding the clinical work, as well as the (policy related) interventions to make changes in the system, pursue the same aim: to generate beneficial impact for the patients and for the population. Thus, besides interventions directed to the individuals, also interventions targeted at the public health or at the health and social care system should be subjected to systematic research in order to get evidence of their impact (3). This evidence should be considered when solving the complex issues related to evidence based policy making (4–6).

The aims of this paper are firstly, to consider the need for the new concept of System Impact Research (SIR); secondly, to provide a definition of SIR, and to describe the main categories of SIR; thirdly, to provide examples of SIR, and fourthly, to find a simple way to formulate the study question in SIR, and finally, to consider the validity and generalizability of findings from the two methods within SIR: the Benchmarking Controlled Trials (BCTs) and the Randomized Controlled Trials (RCTs).

Methods

The author has in a previous paper presented the novel concept BCT, which covers all observational effectiveness research, and can be used to answer clinical or system related study questions (3). However, also RCTs may provide evidence of impacts of health and social care system on patients or population. Usually cluster randomization is needed to ensure similarity in system features other than that which is under study between the study arms (7).

Evidence-based medicine framework was used for the formulation of a simple way of presenting the study question in SIR (8,9). In this framework for clinical interventions a PICOS-type study question is formed. PICOS stands for Patient or Population, Intervention, Control intervention, Outcome, and Study design.

The issues of methodological validity and generalizability of the results in SIR were based on previous methodological considerations of RCTs (10–13) and of recommendations for cluster RCTs (7) and for the BCTs (3).

Results

Need to a novel concept and its definition

Hitherto there has been no common concept and its operationalization for studies assessing the impact of public health and health and social care system interventions to the patient or population. Thus there is an obvious need for a novel concept for this purpose.

SIR is defined as a research field aiming at assessing the impacts on patients or on population of features of public health and health and social care systems, or of interventions trying to change these features. SIR covers all features of public health and health and social care system and actions upon these features. The term impact refers to all effects caused by the public health and health and social care system or parts of it, with particular emphasis on accessibility, quality, effectiveness, safety, efficiency, and equality of services. The term research refers to the conceptual basis of the concept, and the BCTs and RCTs are the means to achieve as non-biased evidence as possible to broaden the evidence base of SIR.

SIR includes all studies assessing performance of the health care or public health systems (Figure 1) (13). In many of the categories there are study questions that can be tested both with BCTs and with RCTs. However, some study objects are not feasible for RCTs, but all can be studied by BCTs. In most cases of RCTs the unit of randomization is that of an organization, not the patient or client, and the design is accordingly of a cluster RCT. Similarly also in BCTs, the study design is targeted at organizations, not the individuals, even when the goal is to obtain individual level outcome data.

Figure 1.

System Impact Research includes all studies assessing performance of the health care or public health systems. All study objects are feasible for Benchmarking Controlled Trials, while many cannot be studied using a Randomized Controlled Trial design. The Clinical Impact Research is placed in the bottom right corner of the figure only to illustrate another category of impact research; i.e. that of assessing impact of interventions targeting individuals.

Figure 2 illustrates how to formulate the study question of SIR using the PICOS (Population, Index System, Comparator System, Outcome, Study design) – framework. The most important impact measures are related to these concepts: accessibility, quality of the services (particularly congruence with the current scientific evidence), effectiveness (including patient experience), safety, efficiency and equality (of obtaining effective services of uniform quality).

Figure 2.

Shaping the study question of the System Impact Research (SIR) according to PICOS (Population, Index System, Comparator System, Outcome, Study design) -framework. The most important outcome measures are related to six concepts: accessibility, quality of the services (particularly according to scientific evidence), effectiveness (including patient experience), safety, efficiency and equality (of obtaining effective services of uniform quality).

The methodological issues in SIR are related to the study question and design, clinical characteristics and system features (Table 1). The first five items of the system features are related to the health and social care services and costs. The sixth item contains four levels; staff competence, use of current scientific evidence (evidence based medicine framework), assessment of performance and quality improvement, and benchmarking one’s performance with the peers to improve the performance. These items originate from the real-effectiveness medicine framework (2). Examples of BCTs and of a cluster RCT using the PICO framework are shown in Table 2 (14–20).

Table 1.

Methodological issues in the system impact research (SIR) in benchmarking controlled trials (BCTs) and (cluster) randomized controlled trials (RCTs).

A: Issues related to the research question and study design

1. Benchmarking Controlled Trial or (cluster) Randomized Controlled Trial

2. Operationalized according to PICOS: Population, Index System, Comparator System, Outcome, Study design

B: Issues related to the clinical data:

1. Selection of patients or population to the study and measures to increase comparability

2. Validity and completeness of clinical baseline data, and comparability of study subjects at baseline

3. Validity and completeness of clinical process data throughout the clinical pathway

4. Validity and completeness of clinical outcome data

5. Statistical and data issues

C: Issues related to the features of the health and social care system:

1. Financing of the care system

2. Organization of the care system

3. Available resources

4. Reimbursement and incentives

5. Regulations

6. Competence, evidence-based-medicine, quality improvement, benchmarking (real-effectiveness medicine framework)

7. Other system related issues

Table 2.

PICOS (population, index system, comparator system, outcome, and study design) in system impact research. Some examples from benchmarking controlled trials (BCTs) and (cluster) randomized controlled trials (RCTs).

System features/changes	Population	Index system	Comparator system	Outcomes (primary)	Study design
1. Related to the financing of the care system (e.g. tax based or insurance based system)	Hypertensive patients at community health centres (Li et al. Medicine (Baltimore), 2015;94;e455)	Government-funded system	Hospital- or private-funded system	Quality of hypertension management and control of hypertension	Benchmarking controlled trial (BCT)
2. Related to the reimbursement and incentives (e.g. pay for performance)	Population of north-west region of England vs rest of England (Sutton et al. NEJM 2012;367:1821–1828)	‘Advancing quality’ – a hospital pay for performance program	No hospital pay for performance program	Mortality	Benchmarking controlled trial (BCT)
3. Related to the way how and by whom the services are organized/provided (e.g. centralized vs decentralized)	Patients with chronic obstructive pulmonary disease (Kruis et al. BMJ 2014; 349:g5392)	Integrated disease management delivered in primary care	Usual care	Quality of life	Randomized controlled trial (RCT; multicentre, pragmatic cluster randomized controlled trial)
4. Related to the regulations (e.g. on uptake of new technology)	Surgical patients in the NSQIP of database of American College of surgeons (Rajaram et al. JAMA 2014;312:2374–2384)	Restricted resident duty hours at surgical departments: two years after the 2011 duty hour reform	Previous duty hours at surgical departments: two years before the 2011 duty hour reform.	Composite measure of death or serious morbidity within 30 days of surgery	Benchmarking controlled trial (BCT)
5. Related to the amount of available resources for health care (e.g. amount of personnel, GDPs of the countries)	Intensive care patients (Wallace et al. NEJM 2012;366:2093–2101)	Nighttime intensivist physician staffing	No nighttime intensivist physician staffing	Mortality	Benchmarking controlled trial (BCT)
6. Related to competence of the staff, use of up-to-date evidence, quality improvement and benchmarking (real-effectiveness medicine framework)	Bariatric surgery patients (Birkmeyer et al. NEJM 2013;369:1434–1442)	Bariatric surgery performed by surgeons belonging to the top quartile (of all participating surgeons) in their surgical skills	Bariatric surgery performed by surgeons belonging to the bottom quartile (of all participating surgeons) in their surgical skills	Complication rates after bariatric surgery	Benchmarking controlled trial (BCT)
7. Related to other system or structure related features	Hospital personnel; patients (Pittet et al. Lancet 2000; 356:1307–1312)	Hospital before implementation of a hand-hygiene campaign	Hospital during implementation of a hand-hygiene campaign	Compliance with hand hygiene during routine patient care; infection rates among patients	Benchmarking controlled trial (BCT)

Discussion

The author’s idea was that there is a need for a framework aiming at assessing the impact of public health features, and health and social care system or parts of it, as well as that of interventions intended to change the system for the benefit of the patients or population. To authors’ knowledge no such framework has been hitherto published.

This paper presents the concept of SIR, which covers both impact research categories, the observational, i.e. BCT and the experimental, i.e. RCT. The strengths and weaknesses of RCTs have been studied extensively and current recommendations on planning, conducting and reporting RCTs have an ample scientific background (21). On the contrary, observational effectiveness studies do not have this background, as prior to the recent paper on BCTs (3) no definition for these studies have existed, and neither criteria on how to appraise them.

RCTs are acknowledged as studies providing the least biased information of the effectiveness of interventions; usually of single interventions under optimal (experimental) circumstances (10). Observational intervention studies aim at assessing effectiveness under ordinary (non-experimental) health care circumstances. The latter studies utilize comparisons between peers, health care providers treating similar patients, and therefore these studies are named the BCTs (1,2).

Major differences between BCTs and RCTs in the risk of bias of the study are related to the consequences of selection of patients. In the former, patients entering the study in each treatment arm may differ at baseline due to selection, while in the latter random allocation to treatment arms leads (regardless of selection) usually to comparable treatment groups (3,10).

Because the unit of randomization in studies aiming at assessing effectiveness of a system is usually that of an organization, not the patient, the design in RCTs is accordingly usually of a cluster randomized trial. Similarly also in BCTs, the study objects are usually organizations, not the individuals, even when the goal is to obtain individual level outcome data. The hypothesis is that changes in the organization lead to favorable impacts for the patients.

When assessing impact of interventions targeting the public health or health care system there are four major challenges. Firstly, particularly in case of BCTs, sufficient data is needed to obtain information indicating whether (particularly) the health care system factors (e.g. related to an economic incentive) may have led to selection of patients and thus to differences in baseline characteristics. The second challenge, principally in case of BCTs, is to adjust for differences in baseline characteristics between the comparators. Third challenge, again particularly for BCTs, is to obtain data of the patients’ clinical pathways to tell in what degree the intervention targeting the system may have changed the pathway and the way patients are treated. The fourth challenge, both for BCTs and RCTs, is to ensure valid and comprehensive outcome measurements in the study arms. Fifth challenge, particularly for BCTs, is to ensure comparability of the system related features between the health care providers. For example the baseline comparability of the patients may be adequate but the competence of the staff of the health care provider using the studied intervention may be better than the competence of the personnel of the health care provider using the control intervention. The sixth challenge, both for BCTs and RCTs, is to document all the effects the intervention causes to the health care system including unintended unfavorable effects. This major challenge of observing often unanticipated consequences in a complex system is most difficult to document, and may become evident only after considerable time lapse after the (policy related) intervention (3).

Conclusions

The new concept of SIR is intended to provide guidance for conducting and assessing studies aiming at providing evidence of comparative impacts of public health and health care system features or policy interventions. BCTs and RCTs cover the whole domain of SIR.

When the experiment is directed to the public health or health care system or part of these, the study questions are often similar than questions posed by the decision makers. SIR – creating the scientific basis for policy decisions – should be given a high priority in clinical, public health and health economic research and should be used for improvement activities. The leaders at all levels of public health and health and (integrated) social care can use the evidence from SIR for the benefit of the patients and the population.