Towards evidence-based, quality-controlled health promotion: the Dutch recognition system for health promotion interventions.

Registration or recognition systems for best-practice health promotion interventions may contribute to better quality assurance and control in health promotion practice. In the Netherlands, such a system has been developed and is being implemented aiming to provide policy makers and professionals with more information on the quality and effectiveness of available health promotion interventions and to promote use of good-practice and evidence-based interventions by health promotion organizations. The quality assessments are supervised by the Netherlands Organization for Public Health and the Environment and the Netherlands Youth Institute and conducted by two committees, one for interventions aimed at youth and one for adults. These committees consist of experts in the fields of research, policy and practice. Four levels of recognition are distinguished inspired by the UK Medical Research Council's evaluation framework for complex interventions to improve health: (i) theoretically sound, (ii) probable effectiveness, (iii) established effectiveness, and (iv) established cost effectiveness. Specific criteria have been set for each level of recognition, except for Level 4 which will be included from 2011. This point of view article describes and discusses the rationale, organization and criteria of this Dutch recognition system and the first experiences with the system.

Introduction

Evidence-based medicine aims to apply the best available evidence gained from the scientific method to medical decision making. Striving towards evidence-based practice, in terms of evidence-based decision making, evidence-based guidelines and evidence-based interventions is widely accepted in the medical fields. Different levels of evidence have been recognized, and based on careful compiling and consideration of the available evidence, health care or treatment protocols and directives have been established to promote evidence-based practice.

In other—non-medical—areas of health and medical practice, including health education and health promotion, the evidence-based paradigm is also advocated [1, 2]. However, for health education and health promotion interventions, no formal evidence-based registration, admission or directive systems are in place. In the Netherlands, two important governmental or government-supported agencies for public health promotion, i.e. the Centres for Healthy Living and Youth Health of the National Institute for Public Health and the Environment and the Netherlands Youth Institute, initiated and coordinate a recognition system for health education and health promotion interventions to promote quality assurance and control.

In this Point of View article, the Dutch recognition system, the recognition procedure and the first experiences are introduced and discussed. This article is a Point of View paper in the sense that we believe that more systematic quality assurance and control has been neglected and should be addressed in the health promotion field, but that the way to promote this, the quality criteria and the effects of such quality control and promotion may be controversial and should be debated. We hope that this Point of View article leads to presentations of similar recognition systems or alternative systems for quality assurance and control that are in place or under construction in other countries.

The Dutch health promotion interventions recognition system

Procedure

The Dutch system is as follows: an organization or other entity that wishes to have an intervention formally recognized and submits a description of the intervention according to a standard submission form to the recognition registration desk at the National Institute for Public Health or the Youth Institute. The submission also indicates for what level recognition is requested (see next paragraph). The submission form demands information about the different steps of the model of planned promotion of population health [3] (Fig. 1).

Fig. 1.

A basic model of planned promotion of population health [3].

For steps 1–3, a brief epidemiological analysis is required outlining what the ultimate public health goals of the submitted intervention are, which behavioural or environmental risk factors are addressed and what the important and modifiable determinants of exposure to these risk factors are. This epidemiological analysis should result in a clear, preferably SMART (Specific, Measurable, Attainable, Relevant, Time-bound), description of the goals of the intervention and of the target populations. Regarding Step 4 of the planning model depicted in Fig. 1, the intervention description should detail the intervention methods, strategies and materials and their theoretical and empirical foundation. For Step 5, the intervention implementation and dissemination procedures or protocols should be outlined. Finally, the evidence for efficacy and effectiveness and/or process evaluation results of the intervention should be summarized.

More detailed descriptions of the intervention, intervention handbooks or protocols as well as research reports to underpin the epidemiological analysis, the interventions methods and strategies and/or the (cost) effectiveness of the intervention should be submitted as attachments to the recognition submission form.

The registration desk checks the completeness and quality of the submitted forms and provides initial feedback to improve the submission if necessary. The intervention dossier is then submitted to a committee of experts. Two subcommittees for recognition of health promotion interventions are in place, one for youth interventions and one for interventions for adults. Both subcommittees consist of experts on health promotion and/or youth health care research, practice and policy. Each submitted intervention is pre-evaluated by four committee members guided by a pre-structured evaluation form that lists all criteria for each recognition level. The submitted interventions are then discussed in a live committee meeting to come to a decision about the recognition level and to prepare feedback for further improvement of the intervention's description and evidence base. Decisions are made by consensus. If a consensus cannot be reached, additional information may be requested, a lower level of recognition is considered or recognition is not granted. The committees meet approximately every 2 months for up to 3 h, in which usually a maximum of six interventions are discussed and evaluated. The chair's role is to ensure that all members contribute to the evaluation, to summarize the evaluation results and to propose a conclusion.

The results of the procedure are published online on the websites of the coordinating centres, and admitted interventions are registered in the so-called I-database (http://www.loketgezondleven.nl/i-database/ and http://www.nji.nl/jeugdinterventies), where the intervention is described, the evidence is presented, contact details for the intervention's ownership are provided as well as the recognition level that was granted.

The recognition levels and criteria

Health promotion interventions are almost always ‘complex interventions’ as indicated by the number of interacting intervention components, the number and difficulty of behaviours required by those delivering or receiving the intervention, the number of groups or organizational levels targeted by the intervention, the number and variability of outcomes and the degree of flexibility or tailoring of the intervention to individuals, subgroups or local circumstances [4].

To establish the levels and criteria, three information sources have been used: previous experiences with existing admission systems in the Netherlands [5] in other fields (preschool programs, youth care, forensic care), international literature and current views of researchers, politicians and practitioners on the requirements of a useful admission systems. The resulting admission levels concur strongly with the UK Medical Research Council (MRC)'s framework for design and evaluation of complex interventions to improve health [6]. Four levels of recognition are distinguished: (i) theoretically sound, (ii) probable effectiveness, (iii) established effectiveness and (iv) established cost effectiveness. The recognition criteria for the different levels are detailed in Table 1. For each higher level of recognition, the criteria for the lower levels should also be met. For the first recognition level (theoretically sound), the intervention should be well described, meaning that a formal description and manual for the intervention should be available as well as results of a process evaluation indicating that the intervention can be conducted with the reach, level of adoption, quality of implementation and maintenance that the intervention requires according to the description and manual. Furthermore, the intervention's methodologies and strategies and the process through which these are supposed to impact the targeted determinants or risk factors should be described, based on and with reference to established empirical health behaviour change theory, i.e. theories that describe, explain and/or guide health behaviour change that are supported by empirical evidence. A description and relevance of a range of such theories can be found elsewhere (e.g. 1). This first level is rather similar to Phase I of the MRC framework [5].

Table I.

The criteria for the different levels of recognition

Recognition level		Minimum criteria
I	Theoretically sound	1. Intervention descriptiona. End goals and intermediary goals have been made explicit, preferably in a SMART format.b. The target population and relevant intermediary target groups have been made explicit.c. The intervention's methods, strategies, activities and materials are described in detail.d. The intervention's procedure and timing is described.e. The intervention's ownership and support system is described.2. Theoretical foundationa. Sound scientific epidemiological analysis describing the relevance of the health issue at stake as well as the determinants and risk factors, that defines the goals and target populations.b. The intervention methods, strategies and activities are appropriate for and tailored to the goals and target population, preferably as indicated by published scientific evidence.3. External validity and quality assurancea. The intervention is transferable, as indicated by a transfer system, consisting, for example, of an intervention handbook, protocols, staff training, etc.b. There is a system for monitoring of intervention deliverance and integrity.c. The costs of the intervention and its implementation are outlined.
II	Probable effectiveness	1. Evidence for effectivenessa. At least.i. One study conducted in the Netherlands of strong internal and external validity indicating intervention effectiveness.ii. Or three studies of lower internal or external validity, of which at least two are studies conducted in the Netherlands.b. Effect size is reported; the effect size is relevant and in concordance with the intervention goals.
III	Established effectiveness	1. Evidence for effectivenessa. At least.i. Two studies conducted in the Netherlands with strong research designs and high internal and external validity, orii. One such study conducted in the Netherlands and two studies with strong research designs conducted in other countries.b. Effects and effect sizes are relevant and in concordance with the goals of the intervention.
IV	Established cost-effectiveness	The criteria for this level have not been established yet.

For the second level (probable effectiveness), all criteria for Level I should be met, and additionally there should at least be preliminary evidence for effects of the intervention, based on exploratory studies, not necessarily of the strongest methodological rigor. The third level—established effectiveness—does require more than one study of strong internal validity conducted in the setting in which larger scale implementation is supposed to take place. For recognition at this level also evidence for generalizability of the results should be available. The evidence is not necessarily derived from results of randomized controlled trials (RCTs), despite the fact that RCT-designed studies have the highest internal validity. As has been argued before, RCT's may not be a realistic or the best option for evaluation of certain kinds of complex health promotion interventions [6]. For levels II and III, the studies with the best possible designs in terms of internal and external validity appropriate for the intervention at hand are required [7]. For example, for nationwide mass media interventions, these can be interrupted time series designs; for community interventions, community intervention trials may be most appropriate.

For Level IV (established cost effectiveness), the criteria have not been detailed yet. The methodology to conduct economic evaluations of complex health promotion interventions is in its infancy, and no interventions have been submitted to date for evaluation at this level. The criteria and the possibility to submit for this recognition level will be in place in 2011.

Early experiences with the system

In 2008, the evaluations of recognitions were started, first only at Level I. From June 2009, the levels II and III were also considered. After a slow start in 2008 (18 health promotion interventions were evaluated), the number of evaluations increased in 2009 (36 interventions), and this number is expected to further increase in 2010 (45 interventions). Until December 31, 2009, 54 health promotions interventions have thus been considered by the committees, with some interventions having been submitted more than once because after the first submission the committee regarded the information as insufficient to come to a decision. In Table 2, the topics and the recognition level granted of the interventions for which evaluation was completed in 2009 are presented.

Table II.

Number of submitted interventions, per topic and recognition levels in 2008 and 2009

Topic	Number of submitted interventionsa	Established effectiveness	Probable effectiveness	Theoretically sound	No recognition
Nutrition	3			1	2
Physical activity	5			5
Prevention of obesity	2			2
Mental health (prevention of depression, anxiety)	8	2	1	5
Smoking	4			3	1
Prevention of alcohol misuse	3		1	2
Prevention of drug abuse	1				1
Combination of alcohol and drug abuse	1			1
Sexual health	6			5	1
Safety	5			4	1
Other	4		2	1	1
Total	42	2	4	29	7

Twelve interventions were re-submitted because recognition was not granted the first time, or the submitting organization re-submitted to obtain recognition at a higher level.

Mental health promotion interventions have been submitted most often, and two of these have been admitted at Level III. With 12 and 10 submitted interventions, the fields of sexual health and safety promotion are also well represented. For other topics of established importance for population health, such as healthy nutrition and physical activity promotion, obesity prevention, and prevention of alcohol and drug abuse, fewer interventions have been submitted. Based on the committee's evaluation of the submission forms, it appears to be difficult for health promotion organizations to describe the theoretical basis of their interventions, and interventions often appear not to have been systematically planned and developed based on established health behaviour change theory.

The majority of the submitted interventions is individual counselling and health education-like interventions. Fewer interventions have been submitted that take a broader health promotion perspective, i.e. directed at changing the physical, social–cultural, economical or political environments. Such health promotion interventions are often more difficult to evaluate in internally valid research designs and may therefore have more difficulties in being recognize at the established effectiveness level. Most interventions have been submitted by national health promotion organizations, probably because their interventions are meant to be disseminated on a larger scale and because they have the means to more carefully prepare the submissions. Most interventions have been submitted to be evaluated at the first level of recognition.

Discussion

Promoting quality assessment and control in health promotion is of great importance in order to ensure that the most effective and efficient interventions are implemented and disseminated, i.e. to promote that the relatively meagre means for health promotion are invested in the best possible way. The Dutch recognition system is an attempt to contribute to promotion of dissemination and implementation of the best available interventions in health promotion practice. The recognition system has been based on and inspired by an established health promotion planning model and an outline for evaluation of complex interventions [3-6].

Whether the Dutch system is indeed the way forward is debatable and experience with, discussion about and evaluation of the system and comparison with similar or alternative quality promotion attempts in other countries is needed and will take place within the next few years. One such system is the Guide to Community Preventive Services, from the US Centers of Disease Control and Prevention, which is a free resource to assist choosing programs and policies to improve health and prevent disease in community-based settings. This system has reviewed more than 200 interventions based on which recommendations for their use have been issued [8]. Another established example is the NICE Public Health Guidance of the UK National Institute for Health and Clinical Guidance [9]. This system indeed provides guidance for health promotion institutes and actors to select appropriate well-described, evidence-based, best-practice interventions. We hope this Point of View article will initiate reactions from the international health promotion research and practice community to help to evaluate and improve the Dutch system. The exact criteria as well as the interpretation of the criteria are disputable. For example, the fact that the first step towards recognition is an epidemiological analysis to substantiate that the intervention addresses an important health issue does not recognize arguments made in the health promotion field to conduct a community analysis or a needs assessment that take more into account than epidemiological data to indentify health promotion priorities [1].

The subcommittees evaluate which criteria are met, facilitated by evaluation forms provided by the registration desk personnel. The subcommittee members have been selected by the founding institutes, based on their expertise and experience in the relevant fields. The committees consist of professional experts who have received clear instructions on the procedures and purposes, but representation of lay people or health promotion intervention receivers. Expanding the committee with such representatives may be considered. Furthermore, the recognition process is internally monitored and evaluated, for example based on observations of subcommittee meetings, in order to adjust the procedures and processes when and where needed.

The precision with which theoretical insights have been translated in intervention strategies and activities, the number and consistency of studies with positive results and the balance between internal and external validity in evaluations of complex interventions are issues that have been debated in developing the admission system, but that remains sometimes difficult to evaluate and assess. Other critical points that have been debated include the reliance on probable or presumed determinants of health as effect indicators and what effect size can be regarded as relevant for population health. These issues are being and will be discussed in and between the evaluation committees as well as with the main stakeholders and should be further debated with international experts in the field.

Some issues will need attention in the years to come. First, the criteria for recognition may favour certain interventions. Relatively many mental health promotion interventions have been submitted and recognized. These interventions are often individually tailored counselling-like interventions, sometimes based on e-health approaches. Such mainly individual and educational interventions are better suited for evaluation in RCTs enabling internally valid effect estimates. Interventions for, for example, obesity prevention, physical activity or healthy eating promotion typically require more integral approaches, combining health education with physical and policy environmental changes [3]. Evaluation of such interventions is more complex, requires long-term follow-up and is therefore costly. If the Dutch recognition system leads an imbalance, with more recognition of interventions that can be tested in RCTs, the system needs adjustments. Practice-based research is needed to improve evidence-based practice in health promotion and for evaluation of such interventions, a broader range of evaluation designs need to be considered [10]. The Dutch recognition system accepts that RCTs are often not applicable or not even desirable for evaluation of certain health promotion interventions, and recognition is not dependent on RCT-based evaluation research. However, the strength of evidence from other research designs is often harder to interpret, and funding for evaluation studies with such alternative designs may be harder to obtain. The fact that most interventions to date were recognized at the level of theoretically sound illustrates that evidence for effectiveness is lacking for many interventions, many of which are already implemented and disseminated.

The system has also been criticized by health promotion practitioners who claim that the system favours interventions developed by the larger, more established organization and interventions developed in or in collaboration with academia, where more manpower and more expertise on health promotion theory and systematically describing interventions in theory-based concepts and strategies is available. Therefore, the organizations responsible for the system now offer support and workshops to ‘intervention owners’ to prepare recognition submissions.

A condition for recognition is that the intervention ‘owner’ facilitates implementation and dissemination of the intervention, by providing clear implementation handbooks and protocols, as well as a description of what personnel and other (financial) means are necessary for implementing the intervention. The recognition system does not require active dissemination of interventions that were granted recognition, and careful monitoring should indicate if the recognition system does lead to dissemination of recognized interventions, or if and what other promotion dissemination activities are needed.

Another critique is that the recognition system may hinder intervention innovation. The system may result in restricting implementations to recognized interventions, especially if financial support for intervention implementation is going to depend on the level of recognition. In 2012, the Netherlands Health Care Inspectorate will evaluate the use of recognized interventions by the municipal health services in their inspection task. Efforts are therefore necessary to promote and enable intervention innovations preceding recognition. The main driver of intervention innovation and evaluation research is the Netherlands is the Netherlands Organisation for Health Research and Development and their prevention research funding programs. Closer collaborations between this organization and the National Institute for Public Health and Environment to endorse continuity between development, evaluation and recognition are now being explored and experimented with.

Funding

Funding for open access charge: National Institute for Public Health and the Environment.

Conflict of interest statement

None declared.