Online formative assessments: exploring their educational value.

INTRODUCTION: Online formative assessments (OFA's) have been increasingly recognised in medical education as resources that promote self-directed learning. Formative assessments are used to support the self-directed learning of students. Online formative assessments have been identified to be less time consuming with automated feedback. This pilot study aimed to determine whether participation and performance in online formative assessments (OFA's) had measurable effects on learning and evaluate the students' experience of using the OFA's in the department of Obstetrics and Gynaecology.
METHODS: This is a cross-sectional study conducted among fourth year medical students (n=92) during their seven week postings in Obstetrics and Gynaecology. Five sets of online formative assessments in the format of one best answers (OBA), Objective structured practical examination (OSPE) and Short answer question (SAQ) with feedback were delivered over five weeks through the online portal. The mean scores of the end of posting summative exam (EOP) of those who participated in the assessments (OFA users) and of those who did not (non-OFA users) were compared, using Students t test. The frequency of tool usage was analysed and satisfaction surveys were utilized at the end of the course by survey questionnaire using the five point Likert scale.
RESULTS: The mean scores of the students in end of posting summative examination marks for students who had participated in the online formative assessment (OFA users) and for those who had not (non OFA users) showed no significant difference in all the three components OBA, SAQ and OSPE (p=0.902, 0.633, 0.248). Majority of the students perceived that OFAs fulfilled the stated aims and objectives and so they would persuade their peers to participate in the OFAs.
CONCLUSIONS: Online formative assessments are perceived as tools that promote self-directed learning, improved knowledge and tailor learning for individual learning needs and style.

Introduction

Online formative assessments (OFAs) have been increasingly recognised in medical education as resources that promote self-directed learning. With the shift from lecture-based to student-based instruction, there is a need to stimulate the inquiry phenomenon and actively engage the student in the learning process. Self-directed learning is identified to be a promising tool in preparing students for self-study and continuing professional education (1). The self-directed learners are able to self-appraise their work, identify their strengths and weaknesses and seek, accept and use feedback from others in order to improve their performance (2). Formative assessment can be defined as one form of self-assessment by the student, which intends to provide feedback to both the teacher and the student (3). The faculty takes cognisance of the feedback to modify teaching and learning to meet the students’ needs and for the students to identify their learning needs. As designers of medical curriculum are looking for strategies to invigorate the teaching and learning delivery methods, formative assessments are considered as means of ensuring deeper learning and understanding (4). Bandura proposed that repeated exposure to successful testing experiences in students with increased anxiety will promote self-efficacy for subsequent tests (5). The use of formative assessments, with no evaluation stress on the students, is the ideal exposure which has been proved to increase the positive experience in future testing event by reducing the cognitive stress anxiety (6).

Formative assessment can be delivered as informal comments made at the end of a case presentation on a ward round to highly complex and formally structured computer-based learning tools (7). Within the clinical context these formative assessment are used to encourage appropriate professional behaviour, to develop clinical competence and to stimulate acquisition of knowledge and clinical reasoning. From the clinician perspective, the time constraint is likely to impact the ability to provide a comprehensive formative assessment task to complement the learning (8). Considering the clinician’s time, production of formative assessment materials for the online medium is an expensive exercise. However, if time, effort and money are to be spent in this direction, there must be an evidence of its benefits and cost-effectiveness (9).

There are studies demonstrating the benefits of web based formative assessment that students voluntarily take part in while preparing for a summative exam (10-12). The potential limitations with paper based formative assessments include time constraint for individualised feedback and the need for the students to be gathered at specific time and place to receive the feedback, which becomes a tedious task in the presence of large class size (13). There is, therefore, an argument to move towards online formative assessments (OFAs).

The postulated advantages of online formative assessments (OFAs) include easy access and availability, utilising interactive features such as images, provision of immediate and individualized feedback, along with the scores allowing timely interventions (12,14). The formative assessments are perceived to assist the students in terms of their extent of understanding the course material and therefore planning their subsequent learning activities (15).

Many studies have investigated the effects of OFAs and the improvement of scores on the subsequent summative assessment. The mechanisms proposed are related to increasing student engagement, increasing time on task, preventing procrastination and identifying learning deficiencies through the formative feedback (12,14,15). . Although the literature shows that the students participating in the OFAs achieved more, often in the form of a grade, all students do not tend to participate in such assessments (16,17). This highlights the need to identify the reasons why some students do or do not use OFAs despite the demonstrated positive effects.

Our study is aimed to explore the educational value of OFAs in the department of Obstetrics and Gynaecology. Our hypothesis is that OFAs will have positive impact on the summative examination scores of the students. Based on this research question, online formative assessments are created using articulate quiz software and students are provided with unrestricted access and they elect to participate in these assessments voluntarily. It is anticipated that our study will identify the reasons why students do or do not participate in OFAs (OFA users and non OFA users). Our particular interest is to identify if there is any difference in the summative performance of the students among the OFA users and non OFA users. Insights into these aspects will provide information on the mechanisms that explain the relation between the OFAs and the final summative examinations and the personal learning styles and learning preferences of students. This would assist in formulating guidelines on designing and implementation of OFAs aligning with the curricular learning outcomes and students’ learning needs.

Methods

The context

Obstetrics and Gynaecology is one of the major disciplines with the students spending a total of 12 weeks with 7 weeks in year 4 and 5 weeks in year 5, which is the senior clerkship year. Currently, there are no online formative assessments in the year 4. The summative/End of posting (EOP) in the year 4 examination includes 30 one best answers (OBA), 3 short answer questions (SAQ) and 3 objective structured practical examination (OSPE) questions. This evaluation is conducted at the end of the 7 weeks of the posting.

The participants in this study were semester 8 students (n=90) in their 7 weeks of obstetrics and Gynaecology posting. This is a cross-sectional study conducted among fourth year students during their seven week postings in Obstetrics and Gynaecology. A convenient sample is taken and the students’ participation is voluntary. Five sets of online formative assessments (OFAs) in the format of one best answers (OBA), Objective structured practical examination (OSPE) and Short answer question (SAQ) with feedback were delivered over five weeks through the online portal. The online formative assessments (OFAs) were prepared, using software -Articulate Quiz maker by the faculty teaching the course, thereby establishing content validity. This also ensured that style and difficulty of the questions were similar in both the formative and summative assessments. The questions present in the formative assessments covered the core content of the course syllabus, which is assessed during summative examinations. The assessments were delivered by university Moodle to which every student has access by their individual username and password. The question format is similar to summative examination that includes one best answer (OBA), OSPE identifying a detail (hot spot) on an image or drag and drop sequence and SAQ (OBA, n=40, OSPE, n=7 and SAQ n=3). Students will be made aware of the OFA’s in the course syllabus by class announcements. The students’ participation was voluntary and taking the OFA’s was based on their own initiative either in the computer lab or in their own personal computer.

The assessments were organized into five assessments each (1/5) being available online starting from the second week of the posting and was open with unrestricted access for 24 hours and 7 days a week. By the end of the 6th week all the five assessments (5/5) were available for the students. Automated feedback was given by the computer program after every attempt of answering the set of questions. The students’ performance on first attempt of each assessment was reported and the mean marks of the 5 tests with the SD were computed.

This study was exempted from ethical clearance as the online formative assessments are ongoing curriculum activity and this pilot project was designed as quality assurance of the curriculum, the results of which will be analysed for feasibility of implementation across the clinical specialities.

Data analysis

The data collected was tabulated and analysed by using the Statistical Package for Social Sciences (SPSS) version 17.0. In this study, a p<0.05 was considered statistically significant. The effectiveness of the OFA’s is assessed by comparing the summative examination scores for students who used OFA’s and did not use (‘non-OFA’s’) by student t-test. The relationship between the scores of students using the OFA’s and the summative assessment was determined by the Pearson’s Correlation Coefficient.

The pattern and frequency of access of modules was analysed by week and time of day usage. The satisfaction surveys were utilized at the end of the posting by qualitative analysis using a survey questionnaire. The survey questionnaire has eleven items of closed ended questions and the response is obtained by five point Likert scale. The reliability and the internal consistency of the test items were measured by Cronbachs alpha. The factor analysis was performed to check the quality and integrity of the questionnaire and to see whether the factors are conceptually fitting together or not. It was based on Keyser-Meyer-Olkins (KMO) measure and Bartlett's Test.

Results

Comparing the summative scores of the OFAs and Non OFAs users

Of the total of 92 students, 48.9% (n=45) participated in the assessments (OFA users) and 51.08% (n=47) did not participate in the assessments (Non OFA users). The End of posting summative examination marks for OFA users and Non OFA users showed no significant difference with the mean scores of 64% & 66% in the OBA (p=0.902) 54% & 50% in the OPSE (p=0.633) and 58% & 57% in the SAQ (p=0.248). The Non OFA users demonstrated similar performance as the OFA users in all the components of the summative examinations, Table 1.

Table 1

Summative examination scores for students who used OFA’s and did not use (‘Non-OFA’s’) compared by independent student t-test

OFA/Non OFA users	EOP/Summative OBA Mean±SD	EOP/Summative OSPE Mean±SD	EOP/Summative SAQ Mean±SD
OFA users (n=45, 48.9%)	64.66±9.5	54.81±12	58.25±9.3
Non OFA users (n=47, 51.08%)	66.38±8.9	50.24±13	57.05±7.6
p	0.902	0.633	0.248

However, there is a moderate positive correlation r=0.46 between the mean scores of the OFA users and the end of posting summative examination in the one best answer (OBA) component (p<0.001). On testing the summative performance on six similar concept questions of formative assessments with different content, the OFA users performed better with 85.4% scored correct responses compared to 46.8% in the formative assessments. Among the Non OFA users, 62% of them had correct responses to these questions (Table 2).

Table 2

Total number of OFA users who answered correct/ Missed of similar concept questions in summative assessments

Concepts tested	Missed in formative assessment (OFA users n=45) 46.8%	Correct in summative assessment (OFA users n=45) 85.4%	Missed in both formative and summative assessment (OFA users n=45)
1. Induction of labour protocol	31 (68.8%)	42 (93.3%)	3 (6.6%)
2. Heart disease risk stratification	12 (26.6%)	40 (88.8%)	5 (11%)
3. IUGR monitoring	24 (53.3%)	36 (80%)	9 (20%)
4. Diagnosis of pre-eclampsia	12 (26.6%)	41 (91.1%)	3 (6.6%)

Pattern and frequency of usage of OFA’s

The five assessments were viewed a total of 247 times by 45 students (48.9%) in 5 weeks’ time. The maximum views were in the fourth week of the posting. Mean time taken for completion was 11:13 minutes. The weekends, Friday (37.3%) and Saturday (48.4%) were the days of maximum completion of the assessments. There was a significant after hour use of the assessments (8:00pm-11:00pm) and an increased access close to summative examination.

Perception of the intervention by survey questionnaire

Quantitative analysis of the survey

The Cronbach’s alpha (an estimated of internal consistency) was 0.96 for the present study. The factor analysis of 11 items was examined. The Kaiser-Meyer-Olkin measure of sampling adequacy was 0.83, above the recommended value of 0.6, and Bartlett’s test of sphericity was significant (X 2=615.64, p<0.000). This suggested that each item shared some common variance with other items and therefore reliability was achieved (Table 3).

Table 3

Perception of the OFA assessment structure and content

Statement	SA/A	Uncertain	SDA/DA
	Frequency (%)	Frequency (%)	Frequency (%)
The OFA’s appropriately tested the intended learning objectives of the task based learning	26 (57.8%)	15 (33.3%)	4 (8.9%)
The OFA’S feedback provided is timely and relevant	25 (55.6%)	16 (35.6%)	4 (8.9%)
The OFA’s feedback presents new knowledge in the content areas	27 (60%)	16 (35.6%)	2 (4.4%)
The OFA’s helped me to identify my area of weakness	27 (60%)	16 (35.6)	2 (4.4%)
The OFA’s are effective learning tools	27 (60%)	16 (35.6%)	2 (4.4%)
The OFA’s motivated me to study	28 (62.2%)	15 (33.3%)	2 (4.4%)
The OFA’s has Improved my ability of self-assessment of performance	28 (62.2%)	15 (33.3%)	2 (4.4%)
I could complete the OFA’s on time	26 (57.8%)	15 (33.3%)	4 (8.9%)
There was no problem with log in and access to the OFA’s	31 (68.1%)	13 (28.1%)	1 (2.2%)
I was able to navigate through the OFA’s effectively	30 (66.7%)	14 (33.1%)	2 (4.4%)
The OBA (One best answer) images of the OFA’s are clear	27 (60%)	16 (35.6%)	2 (4.4%)

Overall, 44.4% of students rated the online formative assessments as good, 31.1% as very good and 24.4% as satisfactory. About 64.2% felt that the OFA’s fulfilled its stated aims and objectives and 77.1% felt that they would persuade their peers to participate in the OFA’s. About 62.2% perceived that participation in the assessments had improved their ability of self-assessment of performance. About 60% strongly agreed that OFA’s had presented new knowledge to the content (Figure 1).

Figure1

Perception of the intervention, Quantitative analysis by survey questionnaire

Discussion

In the pursuit of inventing new strategies to deliver the curriculum, our shift is towards meaningful teaching and learning tools that can challenge students to ‘stretch further than they think they can’ (18). Assessments are one way of challenging the students that enhance the engagement as long as the challenges are associated with swift focused feedback. Formative assessments well fit the model that have the ability to foster student engagement and deliver purposeful learning (19). However, the emphasis must also be on assessing and providing evidence on the educational gains of such creative strategies. The primary purpose of our study was to determine if the OFAs would improve the summative exam scores. There are studies reporting that formative quizzes enhance the summative exam performances in the undergraduate medical and dental students (20-23). However, there are few research findings suggesting that formative assessments do not enhance summative exam scores (24-26).

The results of our study showed that mean summative exam scores for OFA users and non-OFA users were almost equal. The non-OFA users demonstrated similar performance as the OFA users in all the components of the summative examinations. There was a moderate correlation of mean scores of the one best answers component (OBA) of the formative assessments with the summative assessments. Our study did not demonstrate statistically significant improved performance of the summative assessments between OFA users and non-OFA users. These findings contradict the concept that retrieval practice, quizzing in particular, directly boosts exam outcomes, and overall academic performance (27). However, the OFA users performed better on similar concepts questions compared to the non-OFA users in the summative assessments. The format and content of the OFA’s material is similar although they did not mirror the summative assessments, so rote learning is not a reasonable conclusion to draw.

In our study only half of the students (48.5%) participated in the OFA’s. The low participation is perhaps related to the students’ motivation. The key motivating factors for assessments are the perceived relevance of these assessments to preparing them for high stake exams, the peer influences and teachers enthusiasm (28). While the first two items are within the realm of student strategic considerations of what to learn that would benefit them, the third item is an external influence related to teacher’s factors who, due to busy clinical duty, may not find the necessary to time to build the enthusiasm and support system. The students who participated were more curious and motivated and had better study habits. These traits would have contributed to satisfactory performance outcomes rather than the effect of the resources itself.

Nonetheless, one study reported there was no effect of online formative assessments on the students’ final assessments as measured by their entry grade point average (GPA) (29). However, the design of our study could not account for such a causal relationship that would mandate randomisation of group to OFA and non OFA users that would deprive one group of the opportunity to take formative assessments during the trial period. While it could be argued that the aim of formative assessment is not so much to raise the standards of attainment as to foster the spirit of learning, it appears that OFA’s are perceived as a useful resource that enables students to self-regulate their learning process (9). According to Sadler and Hattie and Timperley, the three factors that are motives for students for using formative assessments are (a) feed up, (b) feedback and (c) feed forward (30,31). Our study confirms the findings that students consider feedback function being an important factor to use the OFA’s as it helps them to check their understanding and guide their future learning directions which are again the feed forward function of the OFA’s. Furthermore, the students perceived that OFA’s have given them an idea about what is expected of them in the summative assessment in terms of both content and form. Regarding the pattern of usage of OFA’s, there is a non-uniform temporal fluctuation of usage of the OFA’s, which is more pronounced towards close to summative examinations dates, supporting the notion that students have viewed OFA’s as learning tools that would prepare them for the high-stake exams. This supports the feed up function of OFA’s (31,32). However, with our study findings suggesting multiple attempts on OFAs have not resulted in increased performance in the summative exams, it can be proposed that the summative examination performance is primarily influenced by the inherent properties of the students, rather than the salient effects of formative assessment with feedback itself.

Implications for future research and practice

The evidence from the feedback of the students is encouraging enough to place value on OFA’s as valuable learning tools. While there was considerable time, effort was directed in the generating these OFA’s, the limited uptake by the students’ needs reflection on the teacher’s role on how to stimulate motivation and direction for student centred learning. The findings of the study have implications for postgraduates and specialist trainees who have formidable time constraints and having well-designed OFA’s will be of considerable benefit for non-threatening feedback on their knowledge and clinical decision-making. For future research perspective, it is worth determining whether the learning benefits of OFA’s for junior medical students (semester 8), which we have demonstrated, persist into senior clerkship of medical programs.

Limitations

One of the limitations of the study is that we did not explore information on the reasons why students elected not to participate in the assessments. The small sample size precludes the generalisation of our results. Ours was a convenient sample of all students in the Obstetrics & Gynaecology rotation in year 4. It is also possible that the observed effects would be substantially influenced by the speciality and there is a need to examine the effects in larger samples and across other specialities before we conclude if the OFA’s are robust enough as educational interventions.

Conclusions

Our experience with online formative assessments is a demonstration of utilising technology to supplement traditional assessments to provide additional learning platform for the students. Although we cannot conclusively provide evidence that the OFA’s improve the final summative scores, there is moderate correlation of mean scores of the one best answers component of the formative assessments to the summative assessments. The OFA users performed better on similar concepts with different content questions compared to the non-OFA users. This reinforces that repeated assessments of questions related to those of exams but focussing on various aspects of content can produce consistent improvement in the exam performance. The students perceived that the assessments improved their knowledge and their self- assessment ability to tailor learning for their individual learning needs and style. The usage pattern demonstrates that the flexibility of an e-learning technology that allows the students to access the resource materials at their convenient time. The significantly higher use of assessments towards the final weeks of the posting implies that the students were using the formative assessment as preparation for the exams rather than to facilitate the learning processes. Our future efforts will be directed towards improving the assessment contents to improve the students’ learning experience overall and to include them in the development of online resources to optimize their future usage.