Systematic Review of Computer Based Assessments in Medical Education.

Medical schools, postgraduate training institutes, licensing and certification bodies have developed and implemented many new methods for accurate, reliable, and timely assessments of the competence of medical professionals and practicing physicians. The underlying objective of all these assessments is to not only evaluate the students' learning and educational goals but also to establish the graduating individual's skills and professionalism. Computer based assessment (CBA) has emerged in recent years as a viable alternative to traditional assessment techniques. It has also infiltrated and influenced the medical curriculum where it has been employed for assessment tasks. This study presents how CBA offers pedagogical opportunities and analyzes its usage pattern over the past three decades. We examined 47 CBAs in medical education and analyzed several assessment components, including application area, assessment purpose, assessment type, assessment format, student level, and emphasized the interplay among these components. Our analysis determined that formative assessment is the most frequently used type and 75% of all assessment types employed used the multiple choice questions format.

INTRODUCTION

Measurement of learning competence and performance is an indispensable component of the education process. Computer based assessment (CBA) is an emerging technology that offers a range of advantages over traditional paper-pencil-based testing. These, among others, include rich educational assessment with dynamic sounds visuals, user interactivity, adaptability, improved reliability and impartiality. Near real-time score reporting, instantaneous personalized feedback, time and space independence, and efficient data collection for statistical analysis.[12] The use of computers makes the assessment easier, relieves the faculty of the burdensome task of invigilation and grading.[3] However, some researchers have also discussed the associated disadvantages of using computer technology with the perceived validity of CBA.[4] Universities worldwide have implemented such computer-assisted assessment systems because of the obvious benefits when compared to traditional assessment methods both for formative, summative, and self-assessment purposes.[5] Studies have also been conducted to consider its use for students with disabilities.[6]

CBA has the potential to contribute to different facets of educational and professional testing and to effective learning. It has successfully been implemented for testing basic educational skills, college and university admissions, achievement levels, professional certification and licensing, clinical psychology, life sciences, law, intelligence, language, employment, and adult education. The use of information and communication technologies in medical education is not new as the adoption of CBA techniques has previously been evaluated in the context of medical curriculum teaching and learning, along with the effects of the development of pervasive, high speed information, and knowledge in clinical and medical backgrounds.[7]

An overview of assessments, including computer-based testing approaches in medicine, their advantages, disadvantages and other pertinent questions, has been researched presenting CBA as a qualitative shift away from traditional methods such as paper-based tests and suggests its use for diagnostic purposes for determining students’ prior knowledge as well.[28]

It also discusses assessment question type for medical and health professionals and their content to assess higher order intellectual skills and competences. An investigation into the use of CBA in health education suggested that it presents an alternative approach to paper-pencil based assessment. While both approaches show similar results, it can be concluded that the anxiety of computer use and experience in using computers are not related to student performance.[9] It also emphasized that the strength of multiple choice questions (MCQs) lies in the quality of the items being tested. College level medical students found CBA to be convenient in its accessibility and flexible with regard to time and space.[10]

Reports have indicated that medical students showed a keen interest in and had a positive experience using CBA, prompting a recommendation to introduce formative assessment early in higher education.[11] The preceding research also analyzes the opinions of medical students toward web-based assessment, including their reservations, which resulted in a finding that a high percentage of students showed a positive attitude toward it. A six-step approach for developing CBA for summative assessment in a medical college in Saudi Arabia reported that higher percentages of students approved CBA and suggested that undertaking a CBA pilot to acquaint the students with the new assessment tool would be beneficial.[12]

Different techniques have been employed in medical education assessment ranging from exploration based hypercube to case-based brainstorming and mind map pads, and from random based tests to fixed assessments. One study has identified ten different techniques for assessment and have classified these into three categories namely, exploration based, puzzle-based hierarchy, and case based methods.[13] A taxonomy of the application of CBA has been presented that showed the versatility and potential richness of CBA for educational assessment.[1]

Recently, simulation-based software has also been employed in clinical skills and diagnostics to collect data for assessment of medical students, providing feedback, and executing formative assessments.[1415] CBA realization and assessment related issues for undergraduate medical education, such as hardware requirements, the choice of software, types of test questions, security, integrity, technical knowledge, and skills are of paramount importance and need the utmost attention before undertaking any form of CBA.[16] Assessment has been applied not only to the medical professional learning assessment but also to assessing medical communication skills successfully.[17]

Very recently, medical schools in United Kingdom have developed projects to exploit the "customized Apps for smartphone" concept that not only provides continuous professional development and lifelong learning but also contains features such as recording evidence, assessing clinicians and healthcare professionals in near-patient environments through teacher uploaded exercises.[18] Such tools report the performance and instructor feedback to the students instantaneously. Virtual patient E-assessment systems have also been developed for assessing practical skills similar to those in a real time environment.[19]

The notion of clinical competence and class performance is embraced and articulated in assessments and evaluations both objectively and subjectively in medical education.[20] It has been reported that medical students perceived CBA more favorably than the traditional assessment methodology. Different models of CBA implementation have thus been proposed, ranging from single computer based to multi-purpose PC labs, and models based around personally owned internet-enabled portable devices.[21] Efforts have been made to integrate an assessment model for CBA in science and analyze its validity in the medical sciences.[22] Researchers have laid down guidelines for teachers regarding how to exploit the use of CBA in medical education.[23]

Objectives

The purpose of this review study is to delineate the ways in which current and potential uses of computer technologies are being employed to support assessment activities in undergraduate and postgraduate medical education. We have considered two different aspects of CBA in medical education: Assessments in class and self-assessments. The study also examines the assessment purposes, levels, types, formats, and the areas in which it is applied and the interplay of these components for assessment in medical education.

MATERIALS AND METHODS

Data collection

The focus of the current review is to investigate the use of technology applications in assessments in medical education. With this purpose in mind, we were particularly interested in papers that reported the use of CBA in medical education with empirical findings. To ensure the selection of relevant quality articles, we restricted our search for published papers in peer-reviewed academic journals and excluded conference proceedings, book chapters, unpublished manuscripts, dissertations, project reports, and position papers. The rationale behind such an approach is three-fold. First, the review process for publications other than journal papers are normally not that rigorous which may, in turn, lead to incomplete review and unconvincing conclusions. The journal articles undergo a rolling review schedule, with multiple review phases, ensuring the findings and conclusions about the reported assessments are valid, methodological and comprehensive.[22]

Second, the journal articles are usually longer than conference papers and hence present detailed information about the assessments. Also, these other types of publications are not easy to access and may result in asymmetrical studies. Moreover, journal articles provide detailed and comprehensive information regarding the assessment presented. Although focusing only on journal articles allows a consistent and systematic review, this may omit some important research work in these publications and limit the generalizability of this finding. To gather a sufficiently comprehensive corpus for the study, we undertook extensive research on a number of available sources. This included multiple electronic databases such as Summon Web Scale Discovery, Scopus, Web of Knowledge and the Saudi digital library for relevant journal articles published between 1987 and 2013.

Search criteria

The process of search was initiated with a systematic identification of articles with relevant keywords in journals of educational research, educational technology in medicine, and technology-enhanced medical learning. The journals considered during this study are listed in Table 1. Although the primary search emphasis was CBAs in medical education, we also considered articles that captured other variations of technology-based assessment in medical education, such as comparison of CBAs to traditional paper-pencil versions. Once the screening process was completed, we proceeded to review the references of selected articles with the aim of identifying new resources for further information regarding assessments. There were journal articles related to CBA in general: Some of these were review articles and the rest were analyzed for this review. The whole process as shown in preferred reporting items for systematic reviews and meta-analyses [Figure 1] yielded 47 assessments in 85 articles as these provided sufficient information about the assessment even if the main focus was on other aspects of measurement practices. Each paper was then read and analyzed for the assessment purpose, type and format; participant level, and application area. This synthesis resulted in the coding scheme described in Appendix I.

Table 1

Journals searched

Journal name	Journal name
Academic Emergency Medicine	Journal of General Internal Medicine
Academic Medicine	Journal of Head Trauma Rehabilitation
Acta Ophthalmologica	Journal of Nursing Education
Acta Otorhinolaryngologica Italica	Journal of Surgical Education
Acta pædiatrica	Journal of the American Academy of Pediatrics
Advances in Health Sciences Education	Journal of the American Geriatrics Society
Advances in Physiology Education	Journal of the American Medical Association
American Journal of Obstetrics and Gynecology	Medical Education
American Journal of Roentgenology	Medical Education Quartet
Anesthesia	Medical Teacher
Anatomical Sciences Education	Nurse Education Today
Annals of Internal medicine	Pediatrics
Assessment and Evaluation in Higher Education	Perspectives on Medical Education
Bio Med Central Medical Education	Quality & Safety in Health Care
BMC Medical Education	Teaching and Learning in Medicine
British Journal of Educational Technology	Teaching and Teacher Education
British Medical Journal	Teaching with technology
Computers and Education	The American Journal of Medicine
Critical Care Medicine	The American Journal of Surgery
Education for Health	The Australian And New Zealand Journal of Surgery
International Journal of Clinical Monitoring and Computing	The British Journal of General Practice
International Journal of Human-Computer Interaction	The Clinical Teacher
International Journal of Medical Informatics	The Journal of Laryngology and Otology
Irish Journal of Medical Science	The Journal of Technology, Learning, and
	Assessment
Journal of Allied Health	The Lancet
Journal of Cancer Education	The New England Journal of Medicine
Journal of Continuing Education in the Health Profession

Figure 1

Preferred reporting items for systematic reviews and meta-analyses flow diagram

Coding for potential moderators

The potential moderators that were identified for this research were the characteristics associated with CBA across the study conducted in medical education. All the coded categories carry a common first author and publication year code. The most important category is the area in which such an assessment has been performed. We identified journal articles from diverse areas in medical education for evaluation purposes. The first of the coded moderators is the assessment category. For the sake of making a distinction, we focused on two broad categories: In-class formative or summative assessments and self-assessments. The next category to be coded is a measure of assessment purpose. This includes assessment of conceptual and factual knowledge and synthesis and applied knowledge where an examinee is required to apply prior concepts to the information presented in the question item. Problem-solving items require solutions in the context of the problem. Other types included skills test and suitability testing and those for which the purpose of assessment had not been specified. The third category to be measured is the assessment format. For this category we took a subset of item types presented by Scalise and Gifford[23] and applied these in the context of medical education. Another category is the level of the assessment applied and it spans the duration of the student's course of study. These categories are presented in Table 2.

Table 2

Coding scheme

Category	Code	Meaning[2223]
Assessment category	FA in class	Provides feedback about learning progress and instructional effectiveness during the course of instruction
	SA in class	Offers a summary of instructional effectiveness and student learning after a curriculum or unit of instruction
	SELF	Students check their performance against provided test items and criteria
Assessment purpose	CFK	Assessments with items requesting information retrieval and concept comprehension
	SAK	Using presented information or concepts in an applied question
	ST	Clinical and communication skills testing
	U	The purpose is not specified
Assessment format	T/F	Simple selected response items offering only two choices
	MC	Four or five distractors with a single correct option
	S/I	Multiple true/false with a single/multiple correct response over many items
	R/R	Rearranging/categorizing the multiple choices into an order or sequence
	S/C	Not only identify the correct answer to select, but also whether any of the provided solutions should be used
	B	Fill in the blanks, short-answer and sentence completion, Clozeprocedure, and matrix completion problems
	CON	A task whose solutions are composed of many elements
	P	Performances assessment such as projects, portfolios, demonstrations, experiments
	Checklist	Comprehensive list of important or relevant actions, or steps to be taken in a specific order
Assessment level	Undergraduate, years 1-5 G	Student level

FA – Formative assessment; SA – Summative assessment; CFK – Conceptual and factual knowledge; SAK – Synthesis and applied knowledge; ST – Skills test; U – Unspecified; T/F – True/false; MC – Multiple choice; S/I – Selection/identification; R/R – Reordering/rearrangement; S/C – Substitution/correction; B – Blanks; CON – Construction; P – Presentation; G – Graduate; SELF – Self-assessment

Data analysis

After a rigorous search, we selected 47 assessments in medical journals for study using the following procedures. The foremost consideration given to the assessments were based on the coding criteria defined earlier in Table 2. We observed that multiple codes appeared for some assessments in each category as apparent for the assessment presented in Basu et al.[24] with both conventional multiple choice and the selection/correction assessment format. This was done for both the class-based assessments and self-assessments. Once the coding process was completed, a statistical analysis was performed to identify various emerging patterns in the use of CBAs in medical education. The analysis has been divided into tables for multiple categories in the form of percentages. It represents a holistic picture of how CBAs have proliferated in medical education and the emerging patterns.

RESULTS AND DISCUSSION

In this section, we present findings from our analysis based on the criteria established. These findings focus on specific areas in medicine, assessment category and purpose, and assessment format. We report our analysis in terms of percentages of the assessments considered in Tables 3 and 4.

Table 3

In-class assessments

Assessment purpose	Assessment type (%)
	Formative	Summative	Total
Conceptual and factual knowledge	9 (26.5)	4 (11.8)	38.2
Synthesis and applied knowledge	7 (20.5)	5 (14.7)	35.3
Problem solving	2 (5.8)	0	5.9
Skill testing	1 (3)	3 (9)	11.8
Unspecified	1 (3)	2 (3)	8.8
Total	58.82	41.18

*Percentages calculated based on assessments included in this study

Table 4

Summary of in-class assessments format

Assessment type	In-class assessment format
	T/F	MCQ	S/I	R/R	S/C	B	CON	P	Checklist	Total (%)
Formative	1	14	0	1	1	0	4	1	0	22 (64.4)
Summative	2	8	1	1	1	1	3	1	1	19 (55.9)
Skill testing	0	1	0	0	0	1	0	1	1	4 (11.8)
Problem solving	0	1	0	0	0	0	0	1	0	2 (5.9)
Unspecified	0	1	0	0	0	0	0	0	0	1 (3)
Total (%)	3 (6.3)	25 (52.1)	1 (2.1)	2 (4.2)	2 (4.2)	2 (4.2)	7 (14.6)	4 (8.3)	2 (4.2)

*Percentages calculated based on assessments included in this study. T/F – True/false; MCQ – Multiple choice question; S/I – Selection/identification; R/R – Reordering/rearrangement; S/C – Substitution/correction; B – Blanks; CON – Construction; P – Presentation

Assessment disciplines

It can be seen from Appendix I that CBA has been successfully implemented in almost all disciplines related to medical education. It has been applied to in-class as well as self-assessments. As far as the former assessment is concerned, CBA applications range from assessment, and training in clinical skills and practice, internal medicine, nursing, and diagnostics competence to the communication skills. It reveals the proliferation of technology in assessment and evaluation and equips the teachers with a theoretical and practical steering instrument for measuring competence for continuous development and evaluation of learning outcomes. The same trend has been observed for the self-assessments.

Assessment constructs

Overall, in the 47 assessments considered for analysis, the most common type of assessment performed was formative as shown in Table 3. Nearly 59% of the assessments were formative in nature with 41% summative. This shows that the CBA is mostly employed as an indicator of overall learning and progress. A large proportion (about 38.3%) of the 47 selected assessments (formative = 26.5%, summative = 11.8%) were based on assessment of conceptual and factual knowledge. The second category for which CBA is extensively adopted for assessment is synthesized and applied knowledge with an overall proportion of 35.5%, contributing 20.5% of formative, and 14.7% of summative assessments. Clinical skills testing have been employed for 11.8% while problem solving based assessment has been used the least (5.9%), with 8.8% of the formats not specified. It can be concluded that formative assessment is the preferred mode of assessment in medical education as it reinforces students’ intrinsic motivation toward learning and performance.[3437]

Another notable aspect of the analysis sheds light on the fact that medical faculties are more inclined to strengthen conceptual knowledge and information retrieval. It helps the students to adjust performance based on current understanding and achievements. This use shows the degree of reliance on computers for both the formative and summative assessments. Assessments involving test items based on problem solving and skills testing using computers are not a favored mode.

Assessment format

Assessment in class

The validity and reliability of the assessments are crucial and relates to the type of items being used for assessments. We observe that a range of test items have been used in medical assessments. When we look at Table 4, which lists the assessment formats used for a particular type of assessment, we immediately noticed that multiple choice are favored over other formats, with a high percentage of 72.1%. These could be MCQs with conventional four to five option text answer format or medical context-based figurative MCQs. There are many occasions when extended MCQs were the preferred format type. This is followed by constructed response type format with a proportion of more than 14%. Items based on true/false represent only 6.3% of the total. This indicates that the assessments are more focused on assessing examinees’ learning through MCQs than a simple true/false scenario. Also, very few assessments used the selection/identification format (2.1%) or reordering/rearrangements (4.2%). The same can be said about the checklists and substitution/correction (both 4.2%). The test item of presentation based on images or video clips has also been used and contributed to 8.3% of the item types.

Self-assessments

A similar pattern is observed for item types in self-assessment, where the most commonly used item is ranking/sequencing, with a high percentage of approximately 44% [Table 5]. Another commonly-used item is again the multiple choice type, representing 25%. A new item, based on audio video media type, has also been used for self-assessments. This accounts for 12.5% of the item types.

Table 5

Summary of self-assessments format

	Self-assessment format
	T/F	MC	NM	S/I	R/R	R/S	S/C	B	CON	P	Checklist	Others
Format (%)	0 (0)	4 (25)	2 (12.5)	0 (0)	0 (0)	7 (43.75)	0 (0)	0 (0)	2 (12.5)	0 (0)	0 (0)	1 (6.25)

*Percentages calculated based on assessments included in this study. T/F – True/false; MC – Multiple choice; NM – New media; /I – Selection/identification; R/R – Reordering/rearrangement; S/C – Substitution/correction; B – Blanks; CON – Construction; P – Presentation

Participant level

Our analysis has shown that the CBA has been used at both the undergraduate and graduate levels, though most of the published articles have reported CBA use at the undergraduate level. Also, CBA has been utilized across a spectrum of courses, laboratories, and training. This also indicates that technology is permeating not only in learning but also for assessment and evaluation.

CONCLUSION

Through this study, we have conducted a review of the potential contributions of CBAs to medical education in the last three decades with a special focus on in-class and self-assessments. We have found that CBA is being extensively used for assessments and enhancing learning opportunities. It has a major spillover effect in almost all areas of medical curricula and clinical skills, professional competence testing, and practice. It has been applied both in formative and summative manners to assess factual and applied knowledge. Formative assessments are also being used as a prelude to summative assessments since it motivates students to improve their performance and inspires them to achieve higher professional competence. It has been found from the analysis that MCQ-based assessment formats remain the most commonly used in in-class, self-assessment, and simulation-based assessments. There is a higher percentage of the CBA assessment applied at the undergraduate level in medical education.

We conclude our study by observing that assessment in medical education remains an area of complex competencies. It requires quantitative and qualitative information to be analyzed carefully. When choosing CBA as an assessment instrument, one must ensure that it links the educational objectives with the assessment contents.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

Appendix I

CBAs and in-class codes

First author, reference	Year	Discipline	Assessment purpose		Assessment type	Assessment format									Participant level, year
						T/F	MC	S/I	R/R	S/C	B	CON	P	Checklist
Lee and Weerakoon,[9]	2001	Microbiology	CFK		SA	C									UG,U
Rudland, et al.[10]	2011	U	CFK		FA		C								UG
Deutsch, et al.[11]	2012	U	CFK/SAK		FA		C					E			UG, 4
Hassanien, et al.[12]	2013	U	SAK		SA		C								UG, 3
Devitt and Palmer[15]	1998	Cardiology and neurosurgery	CFK		SA	C									UG, 4
Hulsman, et al.[17]	2004	Preclinic	ST		SA							E			UG, 3
Basu, et al.[24]	2004	Musculoskeletal	CFK		SA		C			M					UG, 1-5
Chen and Chuang[25]	2012	Nursing	CFK		FA										UG, Junior
Wheeler, et al.[26]	2003	Peri-operative medicine	U		SA, FA		T								UG, Final
Beullens,et al.[27]	2002	Anesthesia	SAK		SA				EMCQ						UG, Final
Beullens,et al.[28]	2005	Clinical reasoning	ST		FA		EM								UG,5
Siriwardena,et al.[29]	2009	All disciplines	CFK				C	EMCQ							G
Gilmer,et al.[30]	2003	Nursing					C								G, U
Gordon and Eisenberg[31]	1987	Pulmonary Medicine	SAK			C	C			M					UG, 3
Devitt and Palmer[32]	1998	Clinical skills	CSK											C	UG, 3
Vioreanu,et al.[33]	2013	Musculoskeletal	CFK		SA	C		M				E			UG, 4
Krasne,et al.[34]	2006	Multi topic	SAK		FA,SA		C					E			UG, 1
Paschal[35]	2002	Physiology	SAK		FA		C								UG, 4
Manikam,et al.[36]	2013	Pediatric	SAK		SA		C								UG
Velan,et al.[37]	2008	U	CFK/SAK		FA		C					E			UG
Asman and Lindén[38]	2010	Ophthalmoscopy	SAK		ST		CF								UG
Liebermanet al.[39]	2003	Neurology	SAK		ST		C								UG, 4
Ferenchick,et al.[40]	2013	Internal medicine	SAK		ST									C
Rotthoff,et al.[41]	2006	Haematology and endocrinology	SAK		SA						LMQ				UG, 4
Bernardo,et al.[42]	2004	Surgery	SAK		FA		C					C			UG, 3
El Shallaly and Mekki[43]	2012	Surgery	PS		FA								PV		UG, 5
Humphris and Kaney[44]	2001	Clinical Comm. skills ST	SA										PV	C	UG, 1
Leaf,et al.[45]	2009	Internal medicine	CFK		FA		C								UG, 2
Ganguli,et al.[46]	2009	Radiology	ST		SA						C				UG 1-5
Swagerty,et al.[47]	2000	Geriatrics	CFK		FA		C								UG,3
Feldman,et al.[48]	2006	Clinical skills	CFK				C								UG,3
Bhakta, et al.[49]	2005	Surgery	CFK		FA		EM
Beullens,et al.[28]	2005	Clinical reasoning	ST		FA		EM								UG,5
Peat and Franklin[50]	2002	Biology	CFK		FA		C								UG,1
Inuwa,et al.[51]	2012	Anatomy	SAK		SA		C								UG,1-2
CBAs and self-assessment codes
Author	Year	Discipline	Assessment format											Others	Participant level, year
			T/F	MC	NM	S/I	R/R	R/S	S/C	B	CON	P	Checklist
Antonelli[52]	1997	Clinical medicine		C	AV						C				2
Albanese,et al.[53]	2006	Infection and immunity		C											2
Vivekananda-Schmidt,et al.[54]	2007	Musculoskeletal skills						R							3
Eva, et al.[55]	2004	Multiple		C											1-2
Fitzgerald,et al.[56]	2000	Clinical skills		C											1-3
Weiss,et al.[57]	2005	Obstetrics and gynecology						R							3
Pierre,et al.[58]	2005	Pediatrics						R							3
Hodges,et al.[59]	2001	Family medicine			AV			R							1, residents
Tousignant and DesMarchais[60]	2002	U						R			OS				3
Lind,et al.[61]	2002	Surgery						R							3
Bernard,et al.[62]	2013	Emergency medicine												Qua
Abadel and Hattab [63]	2013	Clinical competency						R							Grad

NM1 – New media; R/S – Ranking/sequencing; AV – Audio Video; Qua – Qualitative; Grad – Graduate; C – Conventional or Standard MC; R/R – Reordering/rearrangement; CFK – Conceptual and factual knowledge; S/C – Substitution/correction; E – Essay; M – Matching; SAK – Synthesis and applied knowledge; LMQ – Long-menu questions; PV – Photo/video; EMCQ – Extended MCQ; MCQ – Multiple choice question; CST – CBA suitability testing; CF – Conventional figurative; MC – Multipla Choice; P – Presentation; ST – Skills Test; PS – Problem Solving; S/I – Selection Identification; Y – Yes; U – Unspecified; OS – Oral structured; CBA – Computer-based assessment; T/F – True/false; MC – Multiple choice; NM – New media; S/I – Selection/identification; R/R – Reordering/rearrangement; S/C – Substitution/correction; B – Blanks; CON – Construction