Nutrition content of summative examinations within an Australian 4-year graduate entry medical course: 2013-2016.

BACKGROUND: Poor nutrition is a major contributor to chronic disease, but the level of nutrition education in medical training is limited. Deakin University Medical School has been working to embed more nutrition into the curriculum since 2009. AIM: To assess the nutrition content of all summative examinations in the Bachelor of Medicine, Bachelor of Surgery over a 4-year period.
METHODS: The type, amount and scope of nutrition-related questions were assessed in all summative examinations delivered to all 4-year levels from 2013 to 2016. These were assessed independently and analysed for nutrition content. The amount of nutrition was quantified, and the nutrition topic areas and nutrition competencies addressed were documented.
RESULTS: Less than 10% of summative examination questions contained any nutrition content. For first-year and second-year students, these examinations included an average yearly total of 433 multiple choice questions (MCQs) (range 337-530) and 47 short answer questions (SAQs) (range 33-62). Third-year and fourth-year students had 150 MCQs on average per year and no SAQs. The percentage of nutrition-related questions across all 4 years ranged from 6% in 2013 to 10% in 2016. The proportion of SAQs with nutrition content ranged from 12% in 2013 to 19% in 2016. Basic nutritional sciences, accounted for 60% of nutrition content and, 25% addressed dietary strategies for prevention and treatment of disease, and skills-based nutrition competencies represented approximately 10% of all questions. SUMMARY AND
CONCLUSIONS: Minimal nutrition was included in the summative examinations. There did not appear to be any consistent increase in the nutrition content of MCQs over the 4-year period but there was some indication of an increase in nutrition content in SAQs. Longer term evaluation is required to confirm this trend. Only a small number of nutrition questions were skills based, most focused on basic nutritional science. Examinations included few skills-based nutrition questions, and consideration of setting a minimum level of nutrition in examinations could assist in ensuring the development of appropriate nutrition competencies in medical graduates. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Up to 10% of exam questions in the medical course at Deakin University were nutrition related

Nutrition related questions were mostly short-answer and asked in the preclinical years of the course

Questions mostly assessed knowledge of basic nutritional science and knowledge of dietary strategies for prevention and treatment of disease

Introduction

Poor nutrition is a major contributor to chronic disease1 2 and it is important for medical graduates to have an understanding of nutrition and modifiable dietary risk factors, as in Australia around 50% of patients who visit a general practitioner have one or more chronic conditions.3 However, recent studies indicate low levels of nutrition-related content in medical curricula around the world4–6 and a similar situation exists in Australia.7

The Australian Medical Council (AMC) is responsible for accrediting medical schools and their programmes of study for the medical profession. The AMC Accreditation Standards and Procedures8 indicate that the content of the curriculum should ‘prepare graduates to protect and advance the well-being of individuals, communities and populations’ which encompasses nutrition but does not specifically mention it. To guide nutrition teaching and learning in medicine, an Australian Nutrition Competencies Framework (NCF) has been developed that includes four knowledge and five nutrition-related skill goals for medical students7 9(see online supplemental appendix 1).

Assessment is a primary driver of learning,10 and quality teaching and learning activities should have clear learning objectives and closely aligned assessment.11 Few studies have examined the nutrition content in summative assessment tasks, with no documentation of this for Australian Medical Schools. Hark et al 12 quantified the amount of nutrition information in step 1 of the US Medical School Licensing Examinations in 1986 and 1993. They found 11% of nutrition-related content in 1993 compared with 9% in the 1986 examinations. A more recent 2015 review, also in the USA, of the nutrition content of the test preparation material (but not quantification) found a large amount of information relating to vitamin and mineral deficiencies and gastroenterology, but not nutrition and lifestyle modification.13

Deakin University’s (DU) graduate entry Bachelor of Medicine, Bachelor of Surgery (BMBS) (now Doctor of Medicine) is based in Geelong, Victoria and has approximately 140 students in each of the 4 years of the course. DU School of Medicine has been actively integrating additional nutrition into the curriculum since 2009.

To assess the nutrition content of summative examinations, between 2013 and 2016, we documented the total number of examination questions in year levels 1–4, the number of examination questions covering nutrition topics, the types of nutrition-related questions, the proportion of marks allocated to nutrition content and the specific nutrition competencies addressed.

Methods

Design and setting of study

The DU School of Medicine was established in 2008 and has a strong focus on public health medicine. The school has an established collaboration with nutrition academics in the much longer established School of Exercise and Nutrition Science (ENS) at Deakin to facilitate the incorporation of nutrition-related learning objectives in the medical course. Collaborative initiatives have included reviewing nutrition-related learning objectives, lectures by ENS staff, development of nutrition competencies and convening student nutrition interest groups.

In years 1 and 2 (preclinical) of the BMBS, teaching is delivered via problem-based learning cases which are supplemented by lectures, workshops and practical classes focusing on human body systems. In the clinical years (years 3 and 4), learning is more experiential with students based at hospitals and general practices throughout Southwest Victoria. Student progress is assessed through written summative examinations, as well as Objective Structured Assessment Clinical Examinations (OSCEs), Workplace-Based Assessments (WBA) and written assignments.

Description of materials

Summative examinations in the BMBS incorporate multiple choice questions (MCQs) and short answer questions (SAQs), and the number of these varies by year level. An extensive question bank has been developed from which summative examination papers are assembled. The bank is continually expanded and reviewed to allow new items to be tested to preserve academic integrity and maintain quality assurance. The MCQs are composed of a question with five possible answers (one correct answer plus four distractors). The MCQs range in complexity from asking simple factual information to interpretative case-based study questions (clinical vignettes). SAQs comprise a brief prompt requiring a written answer, varying in length from one or two words to a short paragraph testing both factual and applied knowledge. All examinations are designed in accordance with the DU Assessment (Higher Education Courses) Procedure.14

Processes, interventions and comparisons

Two dietitians (RP, CN) independently undertook the following steps 1–4. Results were cross-checked and any discrepancies resolved by discussion.

Step 1: documented the total number of summative examinations and number and type of question (MCQ or SAQ) over 2013–2016 in year levels 1–4.

Step 2: identified the total number and type (MCQ or SAQ) of questions that covered or assessed nutrition using the Australian NCF9 (see online supplemental appendix 1).

Step 3: documented the marks allocated to nutrition-related SAQs and assessed the proportion of examination assessment marks allocated to nutrition in these SAQs. To determine the proportion of marks allocated to nutrition, the examiner-set marking scale was used. This ranged between 1 and 13 marks. Any SAQs that were entirely assessing nutrition knowledge/application were allocated the total marks for the question. The proportion of nutrition included in any SAQs that were only partly testing nutrition knowledge was allocated a proportion of nutrition-related marks (calculated as a percentage of the total marks for the question). MCQs were not included as they were all allocated 1 mark each.

Step 4: classified the nutrition-related examination questions according to which nutrition competencies they addressed.9 Each nutrition-related question was reviewed in detail so as to determine the number and type of nutrition-related competencies covered. Where questions were related to more than one competency, they were classified under all relevant competencies. Online supplemental appendix 2 gives an example of a question addressing a number of nutrition competencies. Questions that did not directly test nutritional knowledge were not included in the final analysis.

Results

Number of examinations and types of questions

First-year students in Deakin’s BMBS undertook an average of 14 summative examinations between 2013 and 2016, representing 66%–69% of their total assessment. For year-2 students, the contribution of summative examinations to total assessments was slightly lower (62%–68% of total assessment). Year-3 and 4 students had one final summative examination at the end of each year, representing 38% of their total assessment. For first-year and second-year students, these summative examinations included an average yearly total of 433 MCQs (range 337–530) and 47 SAQs (range 33–62). Third-year and fourth-year students had an average yearly total of 150 MCQs and no SAQs (see table 1).

Table 1

Total number of MCQs and SAQs in year levels 1–4 (2013–2016)

Year	MCQs				SAQs
	1	2	3	4	1	2	3*	4*
2013	523	422	150	150	34	33	–	–
2014	530	422	150	150	51	42	–	–
2015	498	405	150	150	46	41	–	–
2016	487	337	150	150	62	36	–	–

*There were no SAQs administered in years 3 and 4 examinations.

MCQs, multiple choice questions; SAQs, short answer questions.

Questions with nutrition-related content

The percentage of nutrition-related questions across all 4 years ranged from 6% in 2013 to 10% in 2016. The number of nutrition-related questions in year level 1 ranged from 6% to 12%; year level 2 8%–10%; year 3 0%–7%; and year 4 1%–6% (see table 2, figure 1 and online supplemental appendix 3).

Table 2

Total number of nutrition-related MCQs and SAQs in year levels 1–4 (2013–2016)

Year	Total number of MCQs	Total number of nutrition-related MCQs	% of nutrition-related MCQs	Total number of SAQs	Total number of nutrition-related SAQs	% of nutrition-related SAQs	Total number of MCQs and SAQs	Total number of nutrition-related MCQs and SAQs	% of nutrition-related MCQs and SAQs
2013 years 1–4	1245	71	6	67	8	12	1312	79	6
1	523	28	5	34	4	12	557	32	6
2	422	37	9	33	4	12	455	41	9
3	150	4	3	0	0	0	150	4	3
4	150	3	2	0	0	0	150	3	2
2014 years 1–4	1252	94	8	93	20	22	1345	114	8
1	530	52	10	51	9	18	581	61	10
2	422	35	8	42	11	26	464	46	10
3	150	5	3	0	0	0	150	5	3
4	150	1	3	0	0	0	150	1	1
2015 years 1–4	1203	62	5	87	16	18	1290	78	6
1	498	28	6	46	7	15	544	35	6
2	405	27	7	41	9	19	446	36	8
3	150	0	0	0	0	0	150	0	0
4	150	5	3	0	0	0	150	5	3
2016 years 1–4	1164	104	9	98	19	19	1262	123	10
1	487	51	10	62	14	23	549	65	12
2	377	33	9	36	5	14	413	38	9
3	150	10	7	0	0	0	150	10	7
4	150	9	6	0	0	0	150	9	6

MCQs, multiple choice questions; SAQs, short answer questions.

Figure 1

Percentage of nutrition-related questions (MCQ and SAQ), 2013–2016. MCQ, multiple choice question; SAQ, short answer question.

In years 1 and 2, the percentage of nutrition-related content in the MCQs ranged from 7% in 2013 to 9% in 2016. In years 3 and 4, this range was 2% in 2013 to 7% in 2016 (see figure 2, table 3).

Figure 2

Average of all year levels’ nutrition-related MCQ versus SAQ (2013–2016). MCQ, multiple choice question; SAQ, short answer question.

Table 3

Comparison of amount of nutrition-related content versus marks allocated to nutrition in SAQs, year levels 1 and 2 (2013–2016)

Year	Year level	Total marks allocated to SAQs	Total nutrition-related marks allocated to SAQs	% of nutrition-related content in SAQs	% of nutrition-related marks allocated to SAQs
2013	1	194	26	12	13
	2	205	26	12	13
	Average	199	26	12	13
2014	1	255	27	18	11
	2	148	50	26	34
	Average	201	38	22	22
2015	1	234	29	15	12
	2	203	52	19	25
	Average	218	40	17	18
2016	1	218	43	23	21
	2	174	18	14	10
	Average	196	30	18	15

SAQs, short answer questions.

Nutrition-related SAQs (only in years 1 and 2) ranged from 12% in 2013 to 19% in 2016 (see figure 2, table 2). The percentage of nutrition marks allocated to the nutrition content in SAQs varied from 2013 to 2016 in the range of 13%–21% (see online supplemental appendix 4).

Classification of questions according to the NCF

Evidence-based dietary strategies for prevention and treatment of disease (K3) were the most commonly covered competencies between 2013 and 2016 with over 60% of nutrition-related questions addressing Demonstrate understanding of the basic sciences in relation to nutrition (K1) and 25% addressing K3 (see figure 3). Almost no questions assessed prevention knowledge (K2) and only 11% or less of questions assessed skills-based competencies (S1–5) across the 4 years and S4 and S5 were not assessed at all (see online supplemental appendices 5 and 6).

Figure 3

Individual nutrition competencies addressed each year (2013–2016).

Over the 4 years, knowledge-based competencies were assessed 65–108 times in all year levels compared with skills-based competencies which were only assessed 7–10 times.

Discussion

Less than 10% of summative examination questions in the DU BMBS contained nutrition content over the period 2013–2016. There was some indication of an increase in nutrition content in SAQs, up from 12% in 2013 to 19% in 2016, but longer term evaluation is required to confirm this trend.

We cannot comment on the acceptability of a level of 10% nutrition content as we have no benchmarks against which to assess this. A review of studies investigating nutrition in medical courses found that nutrition has not adequately been incorporated into the medical curriculum and hence medical students are not adequately supported to provide high-quality, effective nutrition care to patients.4 Even in 1985 when the US National Research Council Committee on Nutrition in Medical Education15 recommended a minimum of 25 classroom hours devoted to nutrition in the preclinical years, most medical schools did not achieve this.16 Requirements such as formal hours of nutrition education may produce greater nutrition competency in graduates, but only if the nutrition-related curriculum is assessed accordingly.17 To our knowledge, this is the first time the nutrition-related content of summative examinations in a medical course has been documented.

In year levels 1 and 2, nutrition-related content was more common in SAQs than MCQs. This may be because most of the nutrition teaching and learning came through the Public Health Medicine theme and this theme tends to assess student knowledge and skills using SAQs more commonly than the other themes.18 Some studies indicate that SAQs could be a more effective assessment strategy than MCQs. A 2018 review of 20 UK medical courses indicated that while SAQs actually improved learning,19 MCQs gave a false impression of students’ competence and only indicated the ability to recall information.10 20

The number of MCQs administered decreased with increasing year level which represents the change in assessment from knowledge based to experiential (WBA and OSCE) assessments. In years 1 and 2, MCQ/SAQ style assessment comprised approximately 70% of total assessment, compared with approximately one-third of total assessment in years 3 and 4. The amount of nutrition contained in other types of assessments is unknown and needs to be further explored.21 Few Australian studies have looked at where and when nutrition education occurs in the latter years of medical school training, let alone the best modes of assessment. Many skills-based nutrition competencies would be better assessed outside of written summative assessment tasks: such as with the OSCE that has been widely adopted as a tool to assess students’ competencies as it measures outcomes and allows very specific feedback.

A higher number of knowledge-based nutrition-related questions were assessed in all year levels compared with only one-tenth being skills-based nutrition-related questions. Most of the knowledge-based nutrition questions addressed the basic sciences relating to nutrition such as biochemistry and physiology and fewer addressed the interactive role of nutrition in the prevention of disease and evidence-based dietary strategies for the treatment of disease such as heart disease and diabetes. While knowledge-based questions are necessary, skills-based questions are essential for the application of nutrition knowledge in practice.22

Of interest is that two skills-based nutrition competencies were not assessed over the 4-year period reviewed. These related to being able to ‘apply ethics in nutritional management’ (S4), and to ‘work as part of a team to provide adequate nutritional care’ (S5). Poor communication between caregivers and lack of interprofessional teamwork are causes of poor medical outcomes,23 while ethical and moral nutritional issues are essential considerations in patient care.24–26 These competencies may have been addressed as they may be elements more relevant to later professional practice and possibly not relevant to be assessed in case-based questions. Not all of the nutritional competencies can be addressed in non-clinical teaching.

Consideration of setting minimum levels of nutrition content in all examinations is worthy of discussion by key medical curriculum advisers as this could assist in ensuring adequate nutrition content. Increasing the level of assessment of nutrition competencies in summative assessment tasks begins with increasing the level of nutrition in learning objectives. Significant barriers do exist, such as medical school staff without adequate nutrition knowledge and experience; staff without specific training in drafting nutrition-related examination questions specific to the format required in medical courses and ensuring there are personnel with experience in translating evidence-based nutrition science to be relevant for medical students. Summative assessments for nutrition-related learning ideally would be developed by dietitians or nutritionists who could drive the implementation of nutrition competencies into medical curricula22 as this would facilitate their input on nutrition throughout the curriculum, which could enhance the nutrition education of medical students.27

Strengths and limitations

The strengths of this paper include the extensive documentation of the nutrition content of summative examination questions over a 4-year period and the type and content of questions. Information on competency categories (knowledge and skills based) was also provided. The limitations are that the summative assessment tasks examined here were subjectively assessed and hence we did not conduct a statistical analysis to determine if changes over time were significant.

Summary and conclusions

At DU, the School of Exercise and Nutrition and School of Medicine have worked together to incorporate more nutrition into the curriculum and to ensure nutrition is part of student assessment. This current study provides a ‘snapshot’ of the nutrition content of summative assessment tasks in an Australian medical school in year levels 1–4 between 2013 and 2016.

Incorporation of more assessment tasks focused on skills-based nutrition into the medical curriculum is likely to increase the nutrition competency, rather than just knowledge, of medical graduates.

Wider discussion and the development of guidelines on the minimum amount of nutrition in summative examinations is warranted as this could be an avenue to assist in ensuring the development of appropriate nutrition competencies in medical graduates.