Blended programmatic assessment for competency based curricula.

The uncertainty in all spheres of higher education due to the COVID-19 pandemic has had an unprecedented impact on teaching-learning and assessments in medical colleges across the globe. The conventional ways of assessment are now neither possible nor practical for certifying medical graduates. This has necessitated thoughtful considerations in making adjustments to the assessment system, with most institutions transitioning to online assessments that so far have remained underutilized. Programmatic assessment encourages the deliberate and longitudinal use of diverse assessment methods to maximize learning and assessment and at present can be utilized optimally as it ensures the collection of multiple low-stake assessment data which can be aggregated for high-stake pass/fail decisions by making use of every opportunity for formative feedback to improve performance. Though efforts have been made to introduce programmatic assessment in the competency-based undergraduate curriculum, transitioning to online assessment can be a potential opportunity if the basic tenets of programmatic assessment, choice of online assessment tools, strategies, good practices of online assessments and challenges are understood and explored explicitly for designing and implementing online assessments. This paper explores the possibility of introducing online assessment with face-to-face assessment and structuring a blended programmatic assessment in competency-based medical education.

Introduction

Effective assessment requires tasks that assess cognitive, psychomotor, and communication skills, while also assessing professional attributes. Students' learning depends largely on how they think they will be assessed, especially when embedded within “authentic” learning activities; hence, the assessment takes the shape of a strategic program for enriching teaching and learning in higher education, thus making way for the implementation of programmatic assessment.

As online education takes center stage, one question continues to linger in the minds of medical educators: “How do I know what my online students have learnt?” For which there indeed are no simple answers, just as there are not in face-to-face teaching–learning; however, with a little creativity and flexibility, an online learning environment can open up a host of new student assessment possibilities. And, just as with conventional courses, the answer lies in finding the right combination that works best for a particular course.

The rise of e-learning and web-based education to provide or improve educational experiences has led to a growth in the use of online assessment, which will further increase as the use of e-learning becomes more widespread. E-assessments when compared with conventional methods possess distinct characteristics due to the asynchronous nature of interactivity among the online participants. Therefore, teachers need to rethink the strategies of online pedagogy and develop effective assessment techniques that can support meaningful learning. Meaningful exchange and cooperation between students, peers, and teachers with a common purpose can create opportunities for ongoing and ample learner support. This can ultimately foster impactful engagement and deep learning in online education. Effective application of assessment in an online learning environment can offer a creative pedagogical strategy to facilitate such opportunities.

Are there any ways and means of introducing programmatic assessment through online modes for competency-based medical curriculum; if not fully at least partially? This review tries to explore the possibilities.

Programmatic Assessment and Competency-Based Assessment—The Linkage

Programmatic assessment (PA) is a program of assessment, wherein information about learner's competence and progress is collected, analyzed, and acted upon longitudinally. It shifts the focus away from individual tools and assessments, which serve as data points, which are used to provide feedback and mentoring but not for summative decision making. Decisions are made on the collection of data points, proportional to the stakes.[1] Since individual assessments are not used for high-stake decisions, it allows tools with lesser reliability but high educational impact to be used. It also allows lowering the stress of examinations. Delinking of decision making from each assessment by using it for formative purposes and making high-stake decisions on the collection of such data points can be considered the hallmark of PA.[2]

PA encompasses enough opportunities for assessment for learning and not merely depends upon the assessment of learning. Originally proposed by van der Vleuten and Schuwirth (2005),[3] PA assessment gels well with the concept of competency-based medical education (CBME). CBME advocates the longitudinal development of the learner by mastering competencies and by incorporating more formative assessment strategies, feedback, and continuous monitoring.[4] Principles of PA and Competency-Based Assessment (CBA) are congruent to a large extent.[5678] Under these circumstances, it would not be an overstatement to claim that CBME can be implemented properly, only when supplemented by PA [Figure 1].

Figure 1

The mutual supplementing nature of the programmatic assessment and competency-based assessment

Online Assessment—Context

Online assessment can be liberally used for the assessment of knowledge through online multiple-choice questions, assessment of skills by using online objective-structured clinical examination or virtual patients, assessment of competency and performance by using e-logbooks and e-portfolios, and assessment of affective domain through contributions and discussion in online team project works.[910]

As CBME is being adopted in the undergraduate medical curriculum in many institutes, the unparalleled fall-out of the COVID 19 pandemic has dealt a heavy blow on teaching–learning and assessments in medical colleges across the globe. The conventional ways of assessment based on high-stake clinical examination(s) are neither possible nor practical in the face of such adversity, as institutions today are faced with a dilemma of certifying medical graduates in the absence of summative examinations.[11]

The current testing times due to COVID-19 have necessitated exceptional modifications to the assessment system, with most institutions adopting e-learning and e-assessments as an indispensable mode of learning and assessment that so far remained underutilized. The triage of resources in medical education at all levels has recently been suggested as a policy to tide over the COVID-19 pandemic.[12] Even in the post-pandemic era, online modes of learning and assessment are going to stay in medical education. We just need to devise ways to tap their full potential.

In such challenging circumstances, a programme of assessment incorporating frequent online assessment activities is the most feasible option fit for the purpose of assessment. The programmatic approach to assessment encourages the deliberate and longitudinal use of diverse assessment methods to maximize learning and assessment. Such an assessment approach ensures the collection of multiple low-stake assessment data which can be aggregated for higher stake pass/fail decisions by making use of every opportunity for formative feedback to improve performance.[13] But can both CBA and PA be connected through online mode, via e-assessment strategies, particularly so in the face of the COVID 19 crisis? For this to happen, the basic principles of programmatic assessment, choice of online assessment methods, and their relevance to the curriculum need to be prioritized while developing e-assessment strategies for CBME[14] that can enrich the quality and range of student assessment [Table 1].

Table 1

Principles of programmatic and/or competency-based assessment as addressed by e-assessment strategies

Programmatic and/or Competency Based Assessment Principles	Online assessment strategy
Multiple low-stake assessments	Multiple low-stake assessments can be carried out online coupled with feedback such as MCQs, case-based exercises, simulated OSCE, collaborative projects etc.,
Multiple assessment tools; quantitative and qualitativeClinical competence necessitates a diverse set of assessment tools and strategies	E-assessment of different domains mandates use of different assessment tools. Aligning assessment with objectives e.g., paper based for knowledge and observation based for practical skills.Most of these tools (MCQs, Rating scales, Objective Structured Video Exam, MSF) can be made use of in an online environment. However, at times, conduct of workplace-based assessments (direct observation of clinical skills) may not be possible in an online setup.Triangulation of student data from previous assessments, clinical rotations may be done to take a progress decision on student learning.
Feedback is essential and helps students to monitor their own progress	E-assessments should be followed by feedback that is specific, timely, informative, and supportive to aid improvement in the quality of student learning.The purpose of each individual assessment is not just to pass or fail a student but also to create a window of opportunity to provide feedback and allow learners to reset goals and work towards improvement.
Assessment designed to drive learning	Students focus their study strategies on concepts they know will be examined. E-assessment should be designed keeping all expected competencies in mind (e.g., professionalism), some of which may be difficult to assess but nevertheless are important. When students see that all features of being a physician are being assessed, they try to master the nonacademic attributes of a good physician like communication, ethics, and professionalism.
Assessments should be valid	Validity implies the appropriateness of inferences based on test scores for a specific purpose.For example, using MCQs as an assessment tool to test knowledge does not provide enough evidence to support the purpose of attaining clinical competence in medical students. Therefore, for e-assessments to be valid, multiple sources of evidence should be available to support or refute meaningful test score interpretation.
Assessments should be reliable	Acceptable levels of reliability are achievable in e-assessments by increasing the number of assessments, using different methods of assessment (including standardised and nonstandardized assessment) and multiple assessors.
High-stakes assessment;Reproducibility (statistically reliable test) and Equivalence (every student is tested in the same way)	Programmatic assessment attempts to pursue a longitudinal and continuous approach of assessment to achieve a more meaningful and self-directed way of learning. Information on learning and performance for knowledge as well as clinical and professional skills can be gathered using a variety of e-assessment tools.Summative e-examinations for high-stake decisions can be conducted using a hybrid model to complement the conventional methods of assessment.Secure online communication software can assist in collection and electronic storage of past assessments for easy online access, compilation, and declaration of high-stake decisions.

Online Assessment – Practical Considerations

We need to reconsider and re-review the assessment modalities in medical education looking into the current pandemic.[15] Though the prospects of introducing online assessment may look bright, some practical aspects do need consideration.

First, online assessment cannot be introduced for the entire undergraduate medical course. As experiential learning through patient contact is the focus of medical training, so is the assessment. Second, online teaching must precede online assessment, which literally means one need to develop modules for imparting online teaching and assessment in areas of feasibility. Third, online assessment must be aligned with competency objectives and online teaching-learning methods. Fourth, while implementing online assessment at the institutional level, the opportunities provided by the CBME curriculum, like self-directed learning and facility of virtual classrooms and skill labs must be tapped.

The emphasis within online assessment should be placed on the employment of multimodal tools for formative and summative assessments.[9] These tools must focus upon mastering clinical reasoning, problem-solving, and decision-making skills, enabling the transition to a competency-based curriculum. Effective assessment in a digital epoch offers a comprehensible foundation to the applicability of technology for assessment and feedback. A list of online tools specific for online assessment in different domains is given in Table 2.

Table 2

Online assessment tools with respect to domains of learning for a program of online assessment

Cognitive	Psychomotor	Affective
Multiple choice, True-false, Extended matching questionsHot spot questionsClickersConfidence-based MarkingLabelling questionsOnline animation-based questionsOnline video-based questionsOpen book exam	Virtual PatientsVirtual reality/Haptic devicesDigital portfoliosElectronic logbooksOpen LabyrinthCalibrated Peer ReviewSimulators for specific area of expertise: e.g., Minimally Invasive Surgical Trainer Endoscopic Sinus Surgery Simulator	Video-based Communication AssessmentObjective Structured Video ExamTelemedicine assessment toolVideo patient vignettesVirtual patientsDigital portfoliosElectronic logbooksOpen LabyrinthCalibrated Peer Review

Blended Programmatic Assessment—The Buy-In

As stated earlier, introducing an entire program of online assessment in the CBME curriculum is neither feasible nor practical; nonetheless, it must be made part of the traditional PA of any institute and areas must be identified to develop online modules, thus making the introduction of online assessment feasible in that particular content/expertize area. Since online clinical assessment is a challenge, innovative tools have to be generated like using zoom video communications for virtual clinical examination. However, it is very difficult to replace the tests of clinical, communication, and practical skills with formats that do not allow the examiner to observe a student's performance directly.[16]

Considering all such arguments, the “Blended Programmatic Assessment” will definitely give leverage to the teachers to have a multimodal face-to-face assessment interspersed with online assessment activities [Figure 2].

Figure 2

A framework for blended programmatic assessment

As individual assessments are never used for making final decisions in PA and low-stake assessments are very much the part of PA, the immense potential of online assessment can be used to plan low-stake assessments during the entire course. The whole program needs to be supplemented with opportunities for online feedback and reflections. The key task shall be the timely creation and monitoring of a balanced face-to-face and online assessment program that focuses on resource availability, faculty buy-in, and support of regulatory and licensing bodies, fortified with the provision of continuous mentoring and quality assurance. As such, “Blended Programmatic Assessment” will be a new dimension in the implementation of CBME curriculum for medical training.

Blended Programmatic Assessment—Fundamental Concerns

The statement, “Students can, with difficulty, escape from the effects of poor teaching, they cannot (by definition if they want to graduate) escape the effects of poor assessment” signifies the importance of effective assessment very well.[17] The broad perspectives laid down by van der Vleuten and Schuwirth (2005)[3] to defend different forms of assessment are in congruity with the aforementioned statement including blended programmatic assessment. These perspectives address two key concerns—intellectual and logistics.

Intellectual concerns

Given the importance assigned to students' assessment, it is critical to examine the fundamental concerns of assessment within the blended context, i.e., validity, reliability, acceptability, feasibility, and educational impact, to control the associated risks and realize its desirable outcomes. The model of the utility of assessment given by van der Vleuten implies that it is possible to compensate for any lacunae in assessment tools by their strengths.[18] The same model needs to be kept in mind when using online assessment tools in medical education. When we translate the concept to the entire program of blended assessment, it will be better to look at the utility or for that matter, individual characteristics like the validity and reliability of the blended PA itself, rather than looking at the utility/validity/reliability of individual online assessments and/or traditional assessment tools. This allows us to use a variety of online and face-to-face methods to assess competencies.[19]

Though content and construct validity are important, face validity needs to be accorded equal weightage, particularly when introducing online assessment for the first time to students who may be unfamiliar with its processes and may require reassurance. The content validity of online assessment can be enhanced by introducing additional features: like animations, videos, and sound (with headphones) added to online questions and “Hotspot” questions (which require students to place a mark anywhere on an image or diagram dragging labels directly over an image) etc.[20] The online nature and technological aspects of such assessments can significantly impact the authenticity in comparison to other forms of paper-based assessment media.[21]

The reliability of an assessment refers to the dependability of results. It can be seen in terms of marker consistency, temporal stability, and context specificity.[22] The reliability attributes of blended PA are detailed in Table 3.

Table 3

Reliability attributes of blended programmatic assessment

Providing students with an opportunity to demonstrate their progress and achievements by documenting their evidence of learning. This would further create opportunities for the teachers to monitor the learning process and identify learners’ strengths and weaknesses to enable any remedial action.Incorporating a variety of assessment techniques and activities to give students multiple opportunities to demonstrate their learning.Providing clarity of learning goals with explicitly defined, interpreted, and shared rubrics to allow students to assess their learning and guide them towards the expected level of performance.

Acceptability of an online assessment method by all stakeholders, vis-a-vis teachers, students, institutions, and society is a key attribute of blended PA that influences the faculty motivation to use the method and enhances the student's trust of the results. Similarly, it is pertinent to recognize the various feasibility issues like affordability that govern the use of an assessment method. The acceptability of blended PA will definitely vary at the level of different stakeholders—both interclass and intraclass variations will be seen. Similarly, feasibility will be an issue across institutes, depending upon infrastructure.

For blended PA, the educational impact must be vouched for. During online learning and assessment, the ease of reverting to previous steps while using and being assessed on virtual platforms, thus providing an opportunity for corrections and skill enhancement must be weighed against the possibility of development of careless behavior and actual translation while dealing with real-life situations. The best way to ensure the utility of blended programmatic assessment is to have multiple online tests on multiple content areas by multiple examiners using multiple online tools [as given in Table 2] in multiple settings.

Logistic concerns

Just as the intellectual concerns warrant due consideration there are many legal, technical, and economic issues that require rigorous and comprehensive planning to cater to the design and implementation of quality assessment in e-learning.[23] thus having an impact on the blended programmatic assessment [Table 4].

Table 4

Legal, technical, and economic concerns having impact on blended programmatic assessment

Issues	Examples
Legal Issues	Copyright for graphics, video/audio or use of questions from other institutions need due consideration as it mandates permission from concerned authorities
Technical Issues	Online assessment “system” infrastructural requirements; sub-systems such as routers, switches, and network load-balancers, operating system and application software, and trained personnel to manage software and hardware
Economic Issues	Online assessment involve considerable costs; For purchase of computers, dedicated assessment servers, assessment software, programmers’, and assessment software IT support technicians’ salaries

Tips for implementation

Though COVID-19 pandemic has provided opportunities to the medical teachers to innovate and use hitherto underutilized online assessment methods and entwines them with existing assessment methods, and the opportunities thus provided can be utilized to develop a framework of “Blended Programmatic Assessment” particularly when in many countries competency-based medical education and programmatic assessment are newer concepts and are still in their introductory phase and are amenable to experimentation; the paradigm shift is never easy.

For implementing such a program, immense mobilization of resources will be required, besides investing time and efforts for faculty development and capacity building. Faculty development for implementing blended programmatic assessment will have a dual focus—”training in teaching technology” and “training in information technology.” We feel that initially, this will certainly add to the cost of the assessment but a sustained program will eventually lead to better allocation and utilization of resources, thus nullifying the enhanced cost factor. The tips as outlined in Table 5 will help in introducing the “Blended Programmatic Assessment.”

Table 5

Tips for implementing Blended Programmatic Assessment in Competency-Based Undergraduate Curriculum

Align program of learning and program of assessment to the level of each learning and data pointIdentify institutional support system for e-learning and e-assessmentIdentify curricular contents feasible to be covered by online meansIdentify e-assessment opportunities in-alignment with e-learning curricular experiencesPlan e-learning, e-assessment, online feedback, and e-reflections interspersed throughout the courseEnsure quality of your Blended Programmatic Assessment

Tools focusing more on designing realistic digital simulations should also be developed. Still further, with the advent of technology, augmented and virtual reality may be incorporated with online assessment for highly engaging experiences. Efforts must be made to make online assessment suitable for effective as well as skills domain, and using it along with traditional means so as to compute a blended program of assessment in the competency-based medical curriculum.

Conclusion

COVID pandemic has forced the use of online modes for teaching and assessment. Lockdown and closure of institutes even in the post-lockdown phase have made the use of online tools for teaching, learning, and assessment activities inevitable. Teaching–learning and assessment through this mode is likely to continue, though with lesser encounters, even during the postpandemic phase. A framework of “Blended Programmatic Assessment” will provide ways and means to the medical teachers and stakeholders for the seamless incorporation of online assessment with face-to-face traditional assessment in competency-based curricula.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.