Second Global Symposium on Health Systems Research: a conference impact evaluation.

Evaluation researchers have confirmed the importance of conference evaluation, but there remains little research on the topic, perhaps in part because evaluation methodology related to conference impact is underdeveloped. We conducted a study evaluating a 4-day long health conference, the Second Global Symposium on Health Systems Research (HSR), which took place in Beijing in November 2012. Using a conference evaluation framework and a mixed-methods approach that involved in-conference surveys, in-conference interviews and 7-month post-conference interviews, we evaluated the impact of the Symposium on attendees' work and the field of health systems research. The three major impacts on participants' work were new knowledge, new skills and new networks, and many participants were able to provide examples of how obtaining new knowledge, skills or collaborations had changed the way they conduct their work. Participants noted that the Symposium influenced the field of HSR only in so far as it influenced the capacity of stakeholders, but did not lead to any high level agenda or policy changes, perhaps due to the insufficient length of time (7 months) between the Symposium and post-conference follow-up. This study provides an illustration of a framework useful for conference organizers in the evaluation of future conferences, and of a unique methodology for evaluation researchers. Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine

This is one of few studies to draw on a conceptual framework, as well as intra- and post-conference data in evaluating a conference and more such research is required.

Attendees at the Second Global Symposium on Health Systems Research highly valued the Symposium, and many were able to substantiate their claims with concrete examples of changes to work, practice or policy, which aligned with positive results from other conference evaluation studies.

Attendees reported mixed results for the Symposium’s influence on the field of Health Systems Research, with the majority reporting that the major influence of the Symposium on the field was the capacity change of individual stakeholders, which did not align with our indicators for high-level agenda or policy changes.

Introduction

Rigorous conference evaluation has been recognized as an important component of understanding the impact of conferences on a field of work or study, yet there remains a lack of research on how they can be most effectively evaluated (Lee and Back 2005; Malekmohammadi ; Neves ). While continuing education meetings and workshops have been extensively evaluated, when it comes to large, multi-day conferences, there are a number of logistical and financial barriers to conference evaluation (Forsetlund ). The size of such conferences and the varied geographical distribution of attendees make follow-up evaluation difficult, and financial constraints often limit the rigour of conference evaluation (Neves ). In addition, the complexity and breadth of issues discussed at such conferences makes it difficult for organizers to establish clear objectives about how their conference will affect policy and practice (Mathieson 2009; Tepper and Hinton 2003). Key informants interviewed by Neves described three main objectives for planning and participating in multi-day conferences: (1) dissemination of research, (2) networking and professional development and (3) increasing visibility of a specific field of work.

In this article, we use a combined qualitative and quantitative evaluation strategy to evaluate the Second Global Symposium on Health Systems Research (HSR), an event hosted by Peking University Health Sciences Center and co-sponsored by the World Health Organization and the BRAC University School of Public Health. The Symposium is an international 4-day conference, which ran from 31 October 2012 until 3 November 2012, that arose in response to unprecedented focus on the need for evidence-based policies to strengthen health systems (Remme ). The first Symposium, held in 2010 under the theme ‘Science to accelerate Universal Health Coverage’, attracted over 1200 participants, and the Second Symposium, held in 2012 under the theme ‘Inclusion and innovation towards Universal Health Coverage’ attracted more than 1600 participants, from over 100 countries (Team BSO 2012).

The five objectives of the Second Global Symposium, developed by the organizing Secretariat in the initial phases of programme planning, are presented in Figure 1. The structure of the Second Symposium, held in Beijing, followed a similar structure to the First Symposium, with an expanded number of sessions in order to broaden the programme's scope. In total over 130 concurrent and six plenary sessions took place in the 4 days of the Symposium, in addition to ∼500 poster presentations. As well, 16 selected films were shown to provide greater diversity in session types. A poster competition was organized to provide an incentive and collaboration amongst presenters and participants recruited as judges, as a networking strategy.

Figure 1

Objectives of the second Symposium on HSR.

The objectives of this article are to assess the immediate and 7-month post-conference impacts of the Second Global Symposium on: (1) the way attendees conduct their work in HSR and policy, and (2) the field of HSR as a whole. A broader objective is to inform the field of conference evaluation by illustrating a practical and effective method to evaluate large conferences.

Methods

A framework for Symposium evaluation was adapted from a conference specific evaluation framework developed by Neves et al. and is represented in Figure 2 (Neves ). The framework was derived from the Symposium objectives (Figure 1), and clearly defines the methods and indicators used to evaluate the objectives. We used three data collection tools: (1) a survey (available in hard copy and online) with quantitative and qualitative data sections available to all attendees, (2) a series of short qualitative interviews with an opportunity sample of 78 attendees, and (3) a follow-up qualitative phone interview 7 months post-conference, with a purposive sample of 75 attendees, comprised of 44 in-conference interview attendees who had provided a name and contact information, and a random sample of 31 additional conference attendees. Ethics review was completed by the Hamilton Integrated Research Board (Hamilton, Ontario) as this is the primary institutional affiliation of first author E.M. and authors J.N. and J.L. The Symposium executive committee approved of and supported the evaluation of the conference. The detailed methods used for each data collection tool are outlined subsequently.

Figure 2

Conceptual framework for the Symposium evaluation (adapted from Neves et al.).

End-of-conference survey

We distributed printed questionnaires (Supplementary File 1) as attendees entered the morning plenary session the last day of the Symposium and also placed questionnaire forms beside five drop-boxes throughout the venue over the last two days. Survey questions required participants to rate on a Likert scale from 1 to 5 (very poor to very good, respectively) their satisfaction with the programme and logistics of the Symposium, as well as the potential impact of the Symposium on HSR (based on the objectives of the Symposium). The Likert scale was supplemented by boxes for detailed comments on strengths or weaknesses for each category. The survey also asked participants to select from a list of potential benefits that they felt they received and if they would utilize any of these benefits in a meaningful way. In order to increase response rate, the evaluation was publicized in communications prior to the Symposium as well as via announcements during plenary sessions, and participation was incentivized with entry into a prize draw for three e-readers. For a week following the Symposium, the survey was also made available in an electronic format through the Symposium website to provide ample opportunity for attendees to provide feedback. Both the paper and online survey were completed anonymously.

Two authors (M.K.R., A.N.) independently entered the survey data into Microsoft Excel and reconciled any discrepancies. A third author (J.N.) analysed the data using descriptive statistics and reported the tabulated data with means and standard deviations (in brackets) for total survey participants as well as for the following categories: (1) participants who attended the previous Symposium in Montreux; (2) participants funded by the Symposium; (3) years of experience in HSR and (4) gender.

In-conference interviews

To collect in-depth qualitative data from Symposium attendees, we conducted 10-min semi-structured interviews with question prompts addressing each of the five Symposium objectives (see Supplementary File 2). Question prompts were designed using feedback from 10 similar qualitative interviews conducted for the one-year post-Symposium evaluation of the First Global Symposium in Montreux. Interviews were conducted by authors D.W., J.N. and A.N. as well as trained student volunteers from Peking University. We conducted pilot interviews on the first 2 days of the Symposium to ensure logistical feasibility, and controlled for inter-interviewer reliability by conducting several pilot interviews with each interviewer acting as a non-participating observer in turn. Student volunteers also observed D.W. and J.N. interviewing prior to conducting their first interview. Participants were recruited by approaching Symposium attendees at various venues throughout the Symposium site. Attendees who agreed to be interviewed were taken to a seated, semi-private area and the interview was conducted in either English or Mandarin with notes taken by the interviewer but without an audio recording. The in-conference interviews were conducted from mid day on the second last day of the Symposium until the end of the Symposium the following day. Responses were entered into a database within 3 days of the interview. All data were anonymized, but participants were given the option for a post-conference follow-up interview. If they agreed, their contact information was stored in a separate file on a password-protected computer for the post-conference interview. In total 78 in-conference interviews were conducted (4.8% of Symposium attendees), of which 44 consented to giving a name and contact information for post-conference follow-up.

Post-conference interviews

Eight months following the Symposium, we conducted semi-structured telephone interviews (see Supplementary File 3). One participant (E.M.) contacted 75 attendees post-conference. The 44 participants who had agreed to be followed-up with post-conference, and had provided a name and phone number or email, were contacted. If an email was provided but no phone number, we emailed the participant to obtain a phone number at which they could be reached and to schedule an interview. We also contacted an additional 31 participants from the conference list of participants, drawn randomly from the participant list using Microsoft® Excel. Of the individuals on the randomly sampled list, those who did not provide phone numbers, whose numbers did not work, or who were directly involved in organizing the Symposium were excluded from the study.

We conducted telephone interviews during a three week period, from 27 May to 18 June 2013. We contacted all participants up to three times, during three separate weeks between 8:30 AM and 5:30 PM in the participant’s local time, and if a participant was out of office for more than five working days, they were included in the study as a ‘no response’. If a participant was phoned and indicated they were unable to conduct an interview at the moment but would be available on a separate occasion, we scheduled an interview with them at their convenience. Participants were also given the opportunity to email any additional comments after the interview had been conducted.

The interviews were audio-recorded and in-depth notes were taken during the interviews. Interviews were not transcribed with the exception of key quotes. E.M. conducted the qualitative analysis of the post-conference interviews. Qualitative description and constant comparison methods were the main analytic techniques used (Lincoln and Guba 1985). Convergent and divergent perspectives were identified from the interviews and used to code for major themes. In-conference interview responses and post-conference interview responses were coded separately by two authors (E.M. and D.W.) and compared.

Results

Demographic distribution

Attendees at the Symposium and participants in the study represented all six World Health Organization regions (Table 1). A minority of attendees at the Symposium and participants across all three data collection steps were funded by the Symposium to attend the Symposium. The majority of participants in both the in-conference surveys and post-conference interview had 1 to 9 years experience in the field of health systems work. The largest age cohort in terms of participants and all respondents were between the ages of 26 and 40 years. There were roughly equal numbers of female and male attendees and participants from each of the different evaluation steps. Of the 75 individuals contacted for the post-conference interviews, 52 were successfully interviewed. Thirty-nine interviews were conducted in English, 10 were conducted in Mandarin and three were conducted in French.

Table 1

Demographic information of study participants

	Category	Symposium attendees	%	Symposium attendees who participated in in-conference survey	%	Symposium attendees who participated in in-conference interviews	%	Symposium attendees who participated in post-conference interviews	%
WHO region of residencea	AFRO	267	17	–	–	21	27	14	27
	EMRO	21	1	–	–	2	3	2	4
	EURO	273	17	–	–	9	12	5	10
	PAHO	341	21	–	–	8	10	8	15
	SEARO	234	15	–	–	12	15	10	19
	WPRO	468	29	–	–	23	29	13	25
	Not available	1	0	390	–	3	4	0	0
	Total (n=)	1605		–	–	78		52
Symposium Secretartiat funding status	Full	191	12	109	28	22	28	8	15
	Partial	52	3	–	–	4	5	3	6
	None	1363	85	212	54	49	63	41	6
	N/a	–	–	69	18	3	4	0	79
	Total (n=)	1605		390		78		52
Years of experience in HSR	<1	–	–	24	6	–	–	1	2
	1–9	–	–	134	34	–	–	26	50
	10–19	–	–	59	15	–	–	14	27
	>20	–	–	50	13	–	–	9	17
	Not available	1605	–	123	31	78	–	2	4
	Total (n=)	–	–	391		–	–	52
Attended previous symposium	Yes	–	–	89	23	54	69	–	–
	No	–	–	234	60	23	29	–	–
(2010 Symposium)	Not available	1605	–	67	17	1	1	52	–
	Total (n=)	–	–	390		78		–	–
Days attended	1	–	–	1	0	0	0	1	2
	2	–	–	7	2	0	0	2	4
	3	–	–	40	10	7	9	5	10
(2012 Symsposium)	4	–		160	41	49	63	2	4
	5	–	–	79	20	1	1	40	77
	Not available	1605	–	103	27	21	27	2	4
	Total (n=)	–	–	390	100	78		52
Ageb	16–25 (<30)	42	3	17	4	3	4	3	6
	26–40 (30–40)	576	37	163	42	23	29	22	42
	41–50 (41–50)	441	28	67	17	16	21	15	29
	>50	497	32	73	19	9	12	11	21
	Not available	–	–	79	20	27	35	1	2
	Total (n=)	1556		390		78		52
Gender	Male	837	52	142	36	23	29	24	46
	Female	764	48	177	45	31	40	25	48
	Transgender	–	–	1	0	0	0	–	–
	Not available	–	–	70	18	24	31	3	6
	Total (n=)	1601		390		78		52

aWorld Health Organization (2013).

bThe first (non-bracketed) age categories were used for all data collection tools except for the online survey, in which the bracketed age categories were used.

AFRO, African Regional Office; EMRO, Easter Mediterranean Regional Office; EURO, European Regional Office; PAHO, Pan-American Health Organization; SEARO, South-East Asia Regional Office; WPRO, Western Pacific Regional Office.

Symposium programme

Overall, on a scale from one to five (very poor to very good, respectively), participants ranked the programme highly. The most highly rated components were the Symposium dinner (4.3 [0.9]) and satellite sessions (4.1 [0.9]), while the composition of the timetable received the lowest mean rating (3.4 [1.1]) (Table 2). The most positive responses were reported by funded participants and those with less than one year’s experience in HSR. Attendees who attended the first Symposium in Montreux reported the lowest scores in general, especially for the composition of the timetable (3.1 [1.0]).

Table 2

Average participant rating (with standard deviation) of programme components

PROGRAMME scale from 1 (very poor) to 5 (very good)	All Participants		Participants who attended the first Symposium		Participants who were funded by Symposium		Participants with __ years of experience in HSR								Participants who were male
							<1		1–9		10–19		>20
	Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ
Plenary sessions	3.9	[0.9]	3.6	[0.9]	4.1	[0.9]	4.4	[0.6]	3.8	[0.9]	3.7	[0.9]	4.1	[0.8]	3.9	[0.9]
Lunchtime sessions	3.8	[0.9]	3.4	[0.8]	3.9	[0.9]	4.0	[0.7]	3.8	[0.8]	3.6	[0.8]	3.5	[1.0]	3.7	[0.9]
Concurrent sessions	4.0	[0.8]	3.9	[0.9]	4.1	[0.8]	4.3	[0.6]	4.0	[0.8]	3.8	[0.8]	3.9	[0.8]	3.9	[0.8]
Satellite sessions	4.1	[0.9]	3.9	[1.0]	4.1	[0.8]	4.4	[0.7]	4.1	[0.7]	4.1	[1.1]	4.1	[0.8]	4.0	[0.9]
Poster presentations	3.8	[0.9]	3.9	[0.9]	4.0	[0.9]	3.9	[1.0]	3.8	[0.9]	3.7	[1.1]	3.8	[0.9]	3.8	[1.0]
Possibilities for discussion	3.8	[1.0]	3.8	[1.1]	4.0	[0.8]	4.0	[0.9]	3.9	[0.9]	3.6	[1.2]	3.8	[1.1]	3.9	[0.9]
Composition of the timetable	3.4	[1.1]	3.1	[1.0]	3.6	[1.0]	3.8	[1.0]	3.4	[1.2]	3.2	[1.1]	3.3	[1.0]	3.4	[1.2]
Symposium dinner	4.3	[0.9]	4.1	[0.9]	4.3	[1.0]	4.7	[0.5]	4.2	[0.9]	4.0	[1.0]	4.1	[0.9]	4.2	[0.9]
Overall (Programme)	3.9	[0.9]	3.7	[0.8]	4.1	[0.8]	4.4	[0.7]	3.9	[0.9]	3.7	[0.8]	4.0	[0.8]	3.9	[0.9]
Overall (Social events)	4.0	[0.8]	3.8	[0.8]	4.0	[0.8]	4.3	[0.6]	4.0	[0.8]	3.8	[0.9]	3.9	[0.8]	4.0	[0.8]

Impact on and influence of the Symposium on attendees

Responses from the in-conference surveys showed that new knowledge, sharing experiences and new opportunities for collaboration were the top three reported benefits of attending the Symposium. Participants with over 20 years of HSR experience, as well as participants with less than one year of experience, did not report the same number of opportunities for future collaboration, as indicated in Table 3. New knowledge and new opportunities were also most reported as benefits attendees would utilize in their work, both in total numbers, 91 and 98, respectively, and as a percentage of participants reporting that benefit generally (22.9 and 24.6).

Table 3

Number and percentage of participants reporting benefits from attending symposium

Benefits reported (selected from the list of 11 possible benefits in the benefits columns)																	Intention to utilize benefits in a meaningful way (i.e. indicated an intent to utilize one of three benefits from a list of 11 possible benefits)
What benefits did you gain from attending the Second Global Symposium? (select all that apply):	All Participants (n = 398)		Participants who attended first Symposium (n = 89)		Participants who were funded by Symposium (n = 109)		Participants with __ years of experience in HSR								Participants who were male (n = 142)		Total (n = 398)
							<1 (n = 33)		1–9 (n = 159)		10–19 (n = 71)		>20 (n = 61)
New knowledge	312	78.4%	63	70.8%	90	82.6%	24	72.7%	114	71.7%	43	60.6%	37	60.7%	113	79.6%	91	22.9%
Sharing experiences and lessons learned	286	71.9%	69	77.5%	86	78.9%	15	45.5%	99	62.3%	48	67.6%	40	65.6%	103	72.5%	36	9.0%
New opportunities for future collaboration, including professional development	271	68.1%	66	74.2%	68	62.4%	13	39.4%	98	61.6%	50	70.4%	27	44.3%	101	71.1%	98	24.6%
Renewed motivation and sense of purpose	213	53.5%	40	44.9%	68	62.4%	16	48.5%	82	51.6%	33	46.5%	22	36.1%	75	52.8%	46	11.6%
Increased awareness of the challenges to achieving UHC	195	49.0%	39	43.8%	63	57.8%	15	45.5%	72	45.3%	26	36.6%	28	45.9%	75	52.8%	30	7.5%
Affirmation of current work, approach and practice	193	48.5%	44	49.4%	59	54.1%	5	15.2%	78	49.1%	30	42.3%	28	45.9%	65	45.8%	28	7.0%
Better understanding of the meaning and importance of UHC	176	44.2%	25	28.1%	51	46.8%	14	42.4%	64	40.3%	25	35.2%	21	34.4%	62	43.7%	27	6.8%
Better understanding of how HSR can be utilized to improve Universal Health Care (UHC)	162	40.7%	28	31.5%	50	45.9%	16	48.5%	58	36.5%	25	35.2%	21	34.4%	63	44.4%	30	7.5%
New skills	139	34.9%	21	23.6%	55	50.5%	15	45.5%	53	33.3%	19	26.8%	16	26.2%	49	34.5%	36	9.0%
Opportunity to advocate on specific issues	123	30.9%	21	23.6%	42	38.5%	6	18.2%	47	29.6%	14	19.7%	20	32.8%	44	31.0%	16	4.0%
Identification or clarification of priority needs and the ways I can help meet them	81	20.4%	15	16.9%	23	21.1%	8	24.2%	31	19.5%	12	16.9%	12	19.7%	29	20.4%	18	4.5%
No benefits reported	5	1.3%	1	1.1%	2	1.8%	0	0.0%	2	1.3%	1	1.4%	1	1.6%	1	0.7%	N/A	N/A

From both the in-conference and post-conference interviews, the majority of participants reported intending to change (46/78) or currently changing (40/52) the way they conduct their work (reported in Table 4). In-conference interview participants also reported being unsure (11/78) or not knowing (10/78) how they would change their work, and a minority of participants both in-conference and post-conference reported not intending to change the way they conduct their work (11/78) or not having changed the way they conduct their work (11/52) as a result of the Symposium.

Table 4

In-conference and post-conference participant responses on the impact of the symposium to participants’ work

In-conference top responses (n = 78)	Post-conference top responses (n = 52)
1. Yes (46)	1. Yes (40)
(A) Intent to apply new knowledge (10)	(A) Applying new knowledge (18)
Shift focus of work to: ○ Equity (1) ○ Qualitative work (2) ○ Operational research (1) ○ Community level (1) ○ Policy/management (1) Expand focus of work (3) Gained knowledge (1) ○ Used knowledge in a paper (1)	Shift focus of research (1) ○ Shift focus of research to be on HSR (2) Gained knowledge (12) ○ Knowledge of new concepts and vocabulary used in research (3) ○ Applying new knowledge (8) ○ Knowledge used to better organize multidisciplinary teams (1) Changing allocation of research funds (1) New program (1) Using more HSR resources (1)
(B) Intent to apply new skills (20)	(B) Applying new skills (16)
Improved research methods (13) Relating work to policymakers (4) Grant writing skills (1) Operational research skills (1) Improved efficiency in work (1)	Improved research methods (4) Applying research to policymaking and managing (5) Involve policymakers and translate research to policy (3) Applying new skills (4)
(C) Miscellaneous (7) New collaborations (7)	(C) Miscellaneous (0)
2. Not sure (11); Unknown (10)	2. Not sure (0); Unknown (0)
3. No (11)	3. No (11)

The three major ways in which participants reported changing the way they conduct their work were: applying new knowledge to their work, applying new skills to their work and using new networks to establish new collaborations. A lower proportion of participants (10/78) in-conference reported intending to apply new knowledge than the proportion of participants (18/52) post-conference who reported having applied new knowledge to their work (reported in Table 4). In-conference participants reported a number of ways in which they were planning on applying their new knowledge to their work, including shifting the focus of their work, expanding the focus of their work or using new knowledge in a paper.

Post-conference participants reported applying new knowledge to better articulate concepts and vocabulary within their research, shifting the focus of their research to be on HSR, or changing the allocation of research funds. One participant, a researcher from India, shared that her research institution now uses the evaluation framework presented by the World Bank at the Symposium. Another participant commented that: A similar proportion of in-conference participants (20/78) reported the intent to apply new skills to their work compared with proportion of post-conference participants (16/52) who reported having actually used new skills in their work. In-conference participants reported intending to improve their research methods, relating their work to policymakers, and using grant writing skills and operational research skills, as well as improving the efficiency of their work. Post-conference participants reported applying new skills by improving the way they conduct their research, applying research to policymaking and management, involving policymakers in research, and translating research to policy. One researcher commented that: Other participants commented: A small proportion (7 of 78) of in-conference participants said they were planning on utilizing new connections made during the Symposium, when asked how they would change the way they conduct their work as a result of the Symposium. However, in post-conference interviews, no participants responded that they were using new connections as a result of the Symposium when asked how the Symposium had changed their work, but many reported new connections and/or collaborations when specifically asked about new connections that resulted from the Symposium. The majority of post-conference interview participants (36 of 51) made new professional connections at the Symposium. However, only 17 participants of 52 made new connections that resulted in collaborations. Of the collaborations that did result, some were between researchers and funders, and some were between researchers from different geographical locations. A number of participants (12 of 51) did not undertake any new collaborations, but found the Symposium useful for building on existing relationships. A small number of individuals (2 of 51) did not make any new connections, nor did they build on existing connections.

Yes it has [changed the way I conduct my work]. In the way that, I took what I am seeing there about the informal sector information, I had shared with colleagues at home how we are neglecting the informal sector, and yet it was very important, and the lessons that I learned from that Symposium … In a way it has indeed influenced the way that I want to look at the informal sector, and it has influenced the work that I do (Researcher, Malawi, English).

Yes in a sense it [the Symposium] has made me more active to involve people outside of the research arena, other stakeholders, to make sure they are aware not only of the results of my research but the importance of putting research into action. So right now I am starting with a small town where I have already involved the mayor and the city health officer, and they have become more aware of current conditions (Researcher, Philippines, English).

We have re-thought the concept of knowledge translation into policy and how to do so most effectively. After this event, we have had many national debates on the explanation of research results to policymakers and other stakeholders. We asked ourselves how to divide the tasks amongst the stakeholders and the research team. Here, I believe I made valuable contributions that will strongly influence policymakers’ attitudes towards research findings and foster greater impact and quicker, novel decisions (Researcher, Benin, French).

In the past, when we did research, we only focused on the results of the research. We seldom translated our results into health policy making. However, nowadays, we are gradually changing, to apply the results from the research to health policy making. We cannot conduct research only for the sake of research itself; we also need to apply the results to the real world. So we are now trying to connect the epidemiology research with health service and health policy (Researcher, China, Mandarin).

Impact and influence of the Symposium on the field of HSR

Overall, the mean scores for the Symposium’s ability to have positive impact on certain meeting objectives, especially the objectives related to the development of the field, were lower than the other sections of the survey. In-conference survey participants thought the Symposium was most likely to have an impact on improving collaboration between health systems researchers, reporting a mean rating (from 1 to 5) of 4.1 [0.9], as shown in Table 5. The lowest ratings were reported in regards to increasing the likelihood that policymakers will access and understand HSR and similarly, increasing the likelihood that researchers will respond to the needs of policymakers (3.3 [1.0] and 3.5 [1.0], respectively). As noted in other evaluation tables (excluding benefits reported), participants with less than one year of HSR experience gave higher ratings, while participants who attended the Montreux Symposium provided lower ratings. In response to the Symposium’s impact on policymakers’ access and understanding of HSR, past attendees reported the only mean score less than 3.0 over all evaluation categories (2.9 [1.1]).

Table 5

Average participant rating (with standard deviation) of meeting objectives

MEETING OBJECTIVES scale from 1 (very poor) to 5 (very good) How likely is the Second Global Symposium to have a positive impact upon the following:	All Participants		Participants who attended first Symposium		Participants who were funded by Symposium		Participants with __ years of experience in HSR								Participants who were male
							<1		1–9		10–19		>20
	Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ
Knowledge on the latest research findings related to universal health coverage	3.9	[0.9]	3.7	[1.0]	4.1	[0.8]	4.5	[0.7]	3.9	[0.9]	3.7	[0.9]	4.1	[0.9]	3.9	[0.9]
Type of methods used by health systems researchers	3.8	[0.9]	3.6	[1.1]	4.0	[0.8]	4.1	[0.6]	3.8	[1.0]	3.7	[0.7]	3.9	[0.9]	3.7	[0.9]
Quality of methods used by health systems researchers	3.7	[0.9]	3.5	[1.0]	3.9	[0.8]	4.1	[0.7]	3.8	[0.9]	3.4	[0.9]	3.7	[1.0]	3.6	[0.8]
Likelihood that policymakers will access, understand, and use HSR	3.3	[1.0]	2.9	[1.1]	3.5	[1.0]	3.8	[1.0]	3.4	[1.0]	3.1	[0.9]	3.3	[1.1]	3.3	[1.0]
Likelihood that health systems researchers will respond to the demands of policymakers	3.5	[1.0]	3.2	[0.9]	3.7	[0.9]	4.0	[0.8]	3.5	[1.0]	3.4	[0.9]	3.3	[1.1]	3.4	[0.9]
Career support and training available to health systems researchers	3.7	[0.9]	3.5	[1.0]	3.8	[0.8]	3.6	[1.2]	3.8	[0.9]	3.4	[0.9]	3.7	[0.9]	3.6	[0.9]
Collaboration between health systems researchers	4.1	[0.9]	3.9	[0.8]	4.1	[0.8]	4.2	[0.8]	4.3	[0.7]	3.8	[0.8]	4.0	[0.9]	4.1	[0.8]
Number of people engaged in HSR	3.9	[0.9]	3.8	[0.9]	4.1	[0.8]	4.3	[0.8]	3.9	[0.9]	3.7	[0.9]	4.0	[0.8]	3.9	[0.9]

The majority of in-conference (67/78) and post-conference (42/52) participants interviewed felt the Symposium had influenced the field of HSR, as indicated in Table 6. Only two participants in-conference felt the Symposium would not have an impact on the field, and one participant post-conference felt that Symposium had not influenced the field of HSR, specifically stating that from the participant’s context within Kenya, the Symposium had not made a difference within Africa. A number of post-conference participants felt unsure of the impact of the Symposium on HSR (4 of 52), or they felt it was difficult to say (5 of 52). One participant felt that, ‘I guess I haven’t seen much of an impact from the Symposium, this being the second one … There’s been an increase in terms of interest in health systems, but whether or not that’s because of the Symposiums or not, I’m not sure’ (Researcher, Malawi, English). Another expressed that, ‘Yes I think it is a bit difficult because measuring impact needs to be measured in a longer time. As far as I can evaluate that, I think it is mainly bringing people together and sharing ideas and maybe to learn from each other, and that is the most direct impact in the shorter term, but in the longer term, it is too early to tell,’ (Researcher, Belgium, English).

Table 6

In-conference and post-conference responses of the impact of the symposium on the field of HSR

In-conference top responses (n = 78)	Post-conference top responses (n = 52)
1. Yes (67)	1. Yes (42)
(A) Increase capacity of health systems professionals (55)	(A) Increase capacity of health systems professionals (37)
Sharing experiences and research from different countries (24) Building networks (13) Increasing knowledge in the field of HSR (7) Capacity building (2) Increasing skills and methods (9)	Shared experience and research from different countries between researchers (25) Building networks (2) Increasing knowledge in the field of HSR (2) Capacity building (2) Using new research areas (3) Encourages qualitative research (1) Helps motivate attendees (2)
(B) Agenda/policy changes (11)	(B) Agenda/policy changes (6)
Changes the direction of HSR (3) Refine the definition of HSR (3) Creating new research (5)	Helps determine direction of field (3) Helps allocate resources (1) Moving dialogue forward on UHC (2)
(C) Increase profile of the field (10)	(C) Increase profile of the field (4)
2. Difficult to say (0), Do not know (0)	2. Difficult to say (5), Do not know (4)
3. No (2)	3. No (1)

Participants both in-conference and post-conference felt there were three major ways in which the Symposium had influenced the field of HSR: (1) increase capacity of health systems professionals, (2) facilitate agenda or policy changes and, (3) increase the profile of the field (Table 6). The majority of participants (55/78 of in-conference participants and 37/52 of post-conference participants) felt that the Symposium had influenced the field of HSR by increasing the capacity of health systems professionals. One participant interviewed post-conference stated, ‘I think that it has [influenced the field of health systems research] because a lot of people come from different regions and are bringing their research to the Symposium, and they allow people to learn from them’ (Researcher, Thailand, English). Another participant felt that: ‘When they [participants] come back [to their home country], they have to write a report and they have to share on the information that they got from the event to their colleagues, to their bosses … at their organizations. So I think that the immediate impact of this event is to build capacity of health systems researchers’ (Researcher and manager, Vietnam, English). An editor from a journal that attended the Symposium shared that the journal had, ‘set up a new programme as a result of the Symposium. The intention is to reach out to other groups and individuals doing systematic reviews, and try to create a network of people looking at systematic reviews of health systems’.

In particular, some post-conference participants commented on the attendance of young researchers as being important for influencing the future of HSR. Though the geographical diversity was noted as being very beneficial by many participants, one participant felt that the Symposium’s overrepresentation of researchers led to an impact on HSR, but did not influence the translation of that research into practice and policy, stating: A small proportion of in-conference participants (11/78) felt that the Symposium had shifted the agenda of the field of HSR. Some indicated that the definition might be refined, others felt that the direction of HSR might change, and others felt new research may be created. A similarly small proportion of post-conference participants (6/52) felt that the Symposium had helped to determine the direction of the field, to allocate resources and to move the dialogue forward on Universal Health Care (UHC).

I am not sure what kind of impact the symposium would have [on health systems research and capacity building]. I don’t think there were many policymakers attending the symposium. Therefore, I think the major impact of this symposium is on academia. In terms of the practice and capacity building in reality, I think it takes more time for the impact to be observed (Researcher, China, Mandarin).

Finally, a small proportion of in-conference participants (10/78) felt that the Symposium would increase the profile of the field, and a small proportion of post-conference participants (4/52) felt that the Symposium had increased the profile of the field.

Discussion

Key findings

This article presents an evaluation of a large 4-day conference in HSR. Overall, participants across all three data collection steps—survey, in-conference interviews and post-conference interviews—valued the conference highly in terms of gaining new knowledge, skills and networks, which aligned with our indicators for changes to work, practice and policy (Figure 2). That said, many were able to substantiate this claim post-conference with specific examples of how they had changed the way in which they conduct their work. Our evaluation of the Symposium’s influence on the field of HSR showed mixed results. The majority of interview participants in-conference and post-conference indicated that the biggest influence on the field of HSR would result from increased capacity building of individual researchers, which did not align with our indicators for assessing the impact of the Symposium on the field (Figure 2), as there was insufficient data to suggest that any high-level agenda or policy changes resulted due to the Symposium.

The present evaluation holds several strengths and weaknesses. First, to our knowledge, this is one of very few conference evaluations to use both in-conference and post-conference indicators of success by gathering participant feedback, via interviews, longitudinally both during and after the meeting. A notable exception are evaluations that use the tool developed by Phillips et al. for estimating return on investment (Phillips and Breining 2007) the aim of which is to compare the monetary benefits of the meeting to its costs. Second, the evaluation strategy and data collection instruments were based on a conceptual framework derived from a comprehensive systematic literature review. Third, all three data collection tools measured impact of the Symposium on attendees' work and on the field of HSR, allowing us to triangulate our findings. Fourth, gathering post-conference interview data 7 months after the Symposium allowed us to assess actual changes to behaviour rather than anticipated or predicted changes. Finally, qualitative interview data were gathered from a large, diverse and multilingual sample of respondents, which may help reveal the breadth of opinions regarding Symposium outcomes.

One methodological limitation was using multiple interviewers to collect qualitative data, particularly during the in-conference interview, which potentially reduced the uniformity of the data collected. Reporting bias may also factor in as interviewees may have tried to respond in a way that could be perceived as pleasing to the interviewer. Next, sampling bias may have favoured those with strong compliments or complaints that they wished to share with Symposium organizers, particularly in the self-administered survey, for which there was no mechanism to prevent attendees from completing the survey more than once. In addition, due to resource limitations, in-conference interviews were not recorded which conducted our ability to conduct a discourse analysis of the in-conference and post-conference responses and to compare them. Finally, evaluating impact on a field is ostensibly a much longer-reaching endeavour, and data collected 7 months following such an event is likely to be somewhat speculative, as is data from an in-conference assessment.

Our study found that the Symposium objectives positively influenced participants’ work, which aligned with many other studies that reported positive feedback from conference attendees, though the ways in which their work was influenced sometimes differed from the benefits to Symposium attendees. Similar to our findings, other academic conferences reported acquiring new knowledge and engaging in knowledge transfer as being an important conference objective (Karosas ; Haley ; Lund and Gram, 1998). In addition, obtaining new skills by building capacity, as well as building new networks, was also reported as an objective of academic conferences, as well as business conferences (Karosas ; Lund and Gram, 1998; Saha ; Alefsen 2009; Storberg-Walker ). However, some other academic conferences included publication rates as an objective of their conference, and conducted post-conference database searches to measure publication rates, which was not included in our study (James 2001; Scott 2005; Saito ). Building strategies to address sector issues was also an objective reported by other conferences that was not included in our study, though it was an indicator used primarily by business and political conferences (Mathieson 2009; Storberg-Walker ; Foster ). In addition, the relatively short time frame (7 months) between the Symposium and our post-conference interviews seems to have been too short for identifying broad impacts that resulted from the Symposium, which differed from a study conducted by James, evaluating the ‘ALL WELL’ conference, which conducted an evaluation seven years post-conference and is one of few studies to relate a specific impact to the conference (James 2001).

Our framework (Figure 2) and mixed-methods evaluation approach could be used by organizers of large, multi-day conferences as a tool to guide their evaluation framework, objectives and methods. The results from our study regarding the benefits to participants of attending the Symposium—gaining and using new knowledge, skills and networks—could be used by conference organizers to plan conferences that will meet attendees’ expectations. Conference organizers may wish to conduct evaluations in collaboration with an external agency or academic institution (to enhance objectivity). Moreover, they may wish to solicit in-conference feedback using a mixed-methods approach and post-conference feedback using qualitative methods as was done in this study. If resources permit, they may wish to employ a mixed methods approach post-conference (such as including a post-conference survey in addition to telephone interviews), which might allow for a larger post-conference sample size, and if time permits, they may wish to conduct post-conference evaluations more than 7 months post-conference to capture more tangible impacts of the conference on the field of research. In addition, building networking time into an agenda is recommended to allow time that attendees highly value for creating and maintaining new networks.

Future research is recommended concerning how to measure the long-term impact of a conference on a field of research, what methods best suit this type of long-term evaluation, and what duration of time post-conference is sufficient for measuring long-term impacts of a conference on a field. Further analysis on the efficacy of a mixed-methods approach to conference evaluation is needed to confirm whether the three separate data collection steps—in-conference survey, in-conference interview and post-conference interview—are necessary and sufficient for an unbiased evaluation. Finally, cost-benefit analyses of conferences would be useful in reporting to funders, in financial terms, the impact of the conference on a field and on attendees’ work. Phillips et al. have done considerable work in this area (particularly for corporate events), including systematically categorizing benefits—both ‘tangible’ and ‘intangible’ and costs, and converting benefits to monetary values (Phillips and Breining 2007). Their work can surely be built upon for the purposes of relatively more academic conferences.

Conclusions

This study suggests that the 4-day Second Global Symposium on HSR may have influenced attendees’ knowledge, skills and/or collaborations. Respondents ranked the utility of events outside the official agenda (e.g. the Symposium dinner and satellite sessions) high, perhaps reflecting the fact that many respondents reported having made new professional contacts (which is more likely in such settings) or that conference elements like satellite sessions attract individuals with a particular interest in the subset of issues being discussed in these sessions. There was little evidence to suggest that the Symposium influenced more broadly on the field of HSR; however, respondents felt the diversity of participants—in terms of age and country/regional representation—increased the Symposium’s impact on the field, and indicated a desire that a greater diversity of disciplines be represented at future Symposia. The study provides an illustration of a useful framework, and lessons learnt regarding evaluation methodology, for those wishing to assess the impact of future conferences.

Supplementary Data

Supplementary data are available at Health Policy and Planning online.