Quality assesment of mie 2009 sarajevo conference presentations.

INTRODUCTION: XXII European Congress of Medical Informatics (MIe 2009) took place in Sarajevo from August 30th to September 2nd 2009. Assessment of quality of papers presented at MIe 2009 was a process of observation, measurement, comparison and evaluation of the quality of orally presented papers.
METHODOLOGY: For this study, and for the first time since EFMI founding (1976) and MIE congresses, the authors introduced a specially created quality assessment form with five relevant paper quality variables (methodological approach, international influence, scientific content, language quality, technical features) which the first author of this article used in peer-review process of papers submitted for publication in the journal Acta Informatica Medica (as Editor-in-Chief for last 18 years). The survey was conducted on the principle of random sampling of participants of MIE 2009 Conference in Sarajevo, where specially trained interviewers (final year students of medicine and engineering at the University of Sarajevo) interviewed 33 session's chairs and 110 participants/listeners of MIE 2009 paper presentations in 33 sessions (of total 40). Data was collected, entered into a specially created database, analyzed and presented.
RESULTS: From the total of 150 oral presentations at the MIE 2009, 110 oral presentations were graded by both chairs and participants/ listeners. Grading results were compared and we found that in 60% of cases (66 papers) session chairs gave higher ratings than other participants of the congress. The highest rating was 10, and the lowest 3. Only 3 of the papers received all four grades 10 from the session chairs. The most common grade given by chairs of the session was 8 (26.36%), followed by 7 (20%), 9 (19.32%), 6 (13.18%), 10 and 5 (7.50%), 4 (5%) and 3 (1.14%). Significant differences in quality assessment of papers done by chairs and those done by other participants/listeners are observed.
CONCLUSION: This work should demonstrate the importance of introducing universal (uniform) scale for assessment of articles at conferences that would provide objective and relevant assessment, which has not been the practice. Results obtained using a single standardized scale can be compared to each other and thus improve the quality of the articles and the congress. Future congresses can be organized in this manner and become leading events in certain fields of medical science.

1. Introduction

Assessment of quality of papers presented at MIE 2009 Conference in Sarajevo (August 30th to September 2nd 2009) was a process of observation, measurement, comparison and evaluation of the quality of orally presented papers (1, 2). Assessment may be affected by the following factors: the oratory skills (style and manner of expression) of the presenter, active knowledge or ignorance of the official language (in this case English), previous experience of presenting at similar conferences and sessions, subjective impression of listeners/raters (same country, professionally close/distant topic) etc.

It is assumed that the evaluation made by session chairs should be more accurate in comparison to the evaluation made by other session participants/listeners, as it is the practice to have experts from the scientific field covering the session topic as chairs. However, it can happen that among the session participants/ listeners there are those who are more closely and/or professionally related to the topic and are important for the quality assessment and evaluation of the paper presentation.

2. Methodology

During four working days of MIE 2009 Conference there were 40 sessions and 259 papers were presented as oral presentations, poster presentations, tutorials or workshops. This study presents the result of quality assessment evaluation done by session chairs and other randomly chosen participants/listeners of papers presentations in 33 sessions.

For this study, and for the first time since EFMI founding (1976) and MIE congresses, the authors introduced a specially created quality assessment form with five relevant paper quality variables (methodological approach, international influence, scientific content, language quality, technical features), which the first author of this article used in peer-review process of papers submitted for publication in the journal Acta Informatica Medica (as Editor-in-Chief for last 18 years) (Figure 5). The survey was conducted on the principle of random sampling of participants of MIE 2009 Conference in Sarajevo, where specially trained interviewers (final year students of medicine and engineering at the University of Sarajevo) interviewed 33 session’s chairs and 110 participants/listeners of MIE 2009 paper presentations in 33 sessions (of total 40). Data was collected, entered into a specially created database, analyzed and then presented in large number of different tables and charts (Figure 1).

Figure 5.

Quality assessment form for session chairs (based on Acta Medic Informatica peer-review form)

Figure 1.

sample of evaluation analysis results (sessions 1,2,3 - chairs; sessions 1,4,5 - other participants)

In this paper, we will present detailed results of the first and last session, and although this is a relatively small sample, it is methodologically important for two reasons: first and last session presenters were renowned scientists and researchers in the field of medical informatics with a relatively large number of cited references in this area; thematic content of their lectures was covering global, strategic and methodological features in the field of medical informatics.

3. Results

We show detailed comparative evaluation from sessions where chairs were professionally closely related to the topic and where presenters were relatively younger experts in the field of medical informatics (Tables 1,2; Figures 2,3). In the tables and charts presented, we can notice significant differences in quality assessment of papers done by chairs and those done by other participants/ listeners.

Table 1.

Session 1: National eHealth. Chairs: Etienne De Clercq. C.A. – Chair’s Assessment, P.A. – Participant’s Assessment

Oral Paper	Scientific		International		Originality		Presentation		Total
Session 1	C.A.	P.A.	C.A.	P.A.	C.A.	P.A.	C.A.	P.A.	C.A.	P.A.
Mihalas	4	7.00	7	7.50	6	7.00	6	6.25	23	27.75
Bolka	4	7.25	6	8.75	6	7.75	6	8.25	22	32.00
Marcun	4	7.75	6	9.00	6	8.75	7	8.75	23	34.25

Table 2.

Session 40: Learning and education. Chairs: George I. Mihalas, C.A. – Chair’s Assessment, P.A. – Participant’s Assessment

Oral Paper	Scientific		International		Originality		Presentation		Total
Session 40	C.A.	P.A.	C.A.	P.A.	C.A.	P.A.	C.A.	P.A.	C.A.	P.A.
Rognoni	9	6.17	9	5.67	9	5.50	9	7.33	36	24.67
Hercigonja-Szekeres	9	4.11	9	4.22	9	3.89	9	5.11	36	17.33
Lessard	9	7.44	9	6.56	9	6.22	9	5.44	36	25.67

Figure 2.

Total for Oral Paper Session 1

Figure 3.

Total for Oral paper session 40

Analysis of the evaluation also showed that in most cases results match in grading of the scientific content quality, although it was expected that “stricter criteria” will be applied by session chairs, since they are more referent professionals and experts in both Health Sciences and related Medical Informatics field.

From the total of 150 oral presentations at the MIE 2009, 110 oral presentations were graded by both chairs and participants/listeners. Grading results were compared (as shown graphically in tables and graphs), and we found that in 60% of cases (66 papers) session chairs gave higher ratings than other participants of the congress. The highest rating was 10, and the lowest 3. Only 3 of the papers received all four grades 10 from the session chairs (2 papers in the group 11 and one from the group 16). The most common grade given by chairs of the session was 8 (26.36%), followed by 7 (20%), 9 (19.32%), 6 (13.18%), 10 and 5 (7.50%), 4 (5%) and 3 (1.14%) (Figure 4).

Figure 4.

Percent of ratings given by the session’s chairs

4. Discussion

One of the most respectable scientists, researchers and physicians in biomedical literature, which dealt with the concept of quality in general, especially in the field of medicine and health, was Avedis Donabedian, author of most quoted definition of quality assessment: structure - process - outcome. In his fifty published articles in the field of quality he presents the modern approach to the transformation of thinking about health systems. Other authors, such as Louis, Lohr, Maxwel, Goves in their studies promote the importance of evaluation and assessment of quality in medicine and healthcare. The author of this article is closely specialized in the field of measuring the quality of medical education, as one segment in the field of quality health care, and in his research he uses a modified Lickert scale for measuring the quality of education (3, 4, 5, 6).

Assessment can be prospective or retrospective (1, 2, 3, 4). Quality control is a synonym for quality assessment and is increasingly replaced by this term (5, 6). XXII European Congress of Medical Informatics (MIE 2009) took place in Sarajevo from August 30th to September 2nd 2009. Presenters of oral presentations were arranged in 40 sessions and there were total 150 presentations. Chairs of the group sessions and participants of the congress graded the quality of oral presentations. Session chairs grading was done in 5 categories: scientific work, internationality, support to technical characteristics of work, originality (news), and presentations. Other congress participants evaluated papers in 4 categories: scientific work, internationality, originality (news), and presentation. They did not grade the technical characteristics of the work. For these reasons, in this paper we compared grades for scientific work, internationality, originality (news) and the presentation which were given by both the session chairs and participants of the congress. All grades were summarized and compared. The resulting data are presented in tables and charts (Tables 1,2; Figures 1,2.3,4).

5. Conclusion

The specificity of this work, and also we assume its value, is that on the basis of the available literature we did not find any article which compared the evaluation of oral presentations by the session chairs and participants of the congress.

This work should demonstrate the importance of introducing universal (uniform) scale for assessment of articles at conferences that would provide objective and relevant assessment, which has not been the practice. Results obtained using a single standardized scale can be compared to each other and thus improve the quality of the articles and the congress. Future congresses can be organized in this manner and become leading events in certain fields of medical science.

Based on the indicators from the processed survey data (survey forms for quality assessment as those used during the MIE 2009 Conference in Sarajevo) the following could be selected:

Articles which will be declared as the best and rewarded at the end of MIE Conference;

Certain number of articles for publication in indexed journals in the field of Medical Informatics of general educational interest to the general readership and which will carry the appropriate number of ECTS points for the MSc and PhD students.

This kind of selection for the best quality of articles, presented at the congress, can be a sign and stimulus to future participants of MIE Conferences, if they have information that the quality assessment, as a specific method of assessment of articles, is used during the congresses of Medical Informatics at the international level. Such potential participants will be motivated to make better presentation of their research results, because they expect it to be published in one of prestige journals of Medical Informatics.