Peer assessment platform of clinical skills in undergraduate medical education.

Introduction

Medical diagnosis is one of the most important interdisciplinary courses during early-stage education and training in clinical practice among medical students.[1] Mastery of physical examination skills, an the essential clinical skill in medical diagnosis courses, requires a large amount of training and practice.[1,2] Nevertheless, concerns have been raised on how students can receive adequate training with limited instructor resources in this time-intensive course.

At Nanjing Medical University, there are approximately 1000 medical students in each class. The number of instructors and tutors is insufficient, so they must carry a heavy workload to fulfill the clinical mission of the teaching hospital. In the development of formative evaluation,[3,4] a major drawback in the current system of assessing physical examination skills is that tutors are required to observe students one at a time, as students demonstrate physical examination procedures to the examiners. This system is very impractical and can impose an excessive workload on tutors when a large number of students are ready to be assessed at the same time. Another flaw is that testing items are randomly selected from a question bank, to save time during testing; however, these questions only cover a small portion of the information that medical students must acquire. Testing items can vary among different students, making it even more difficult to assess them in a fair manner, given the underlying biases of tutors. Furthermore, traditional assessment is conducted at the end of the semester, and tutors are unable to provide proper feedback as they are not allowed to clarify errors made by students, to maintain fairness. In this way, the assessment system benefits students who take the exam later. Scores obtained using this testing system provide little benefit as far as students’ subsequent levels of clinical skills and practice.

In this study, we present a clinical skills peer assessment platform (CSPAP)[5-7] designed to resolve the above problems. We developed this platform for the purpose of students being able to benefit from its dexterity in conducting physical examinations, to enable them to attain deep insight into basic clinical principles. We also expect this platform to represent an improvement over the current training and assessment methods in terms of fairness and objectivity.

Materials and methods

Physical examination

The physical examination consisted of blood pressure measurement, the head and neck system (superficial lymph node palpation of the head and neck), chest system (respiratory auscultation), cardiac system (cardiac auscultation), abdominal system (liver palpation), and nervous system (Babinski reflex).

Paper exam

A paper exam was designed to assess students’ understanding of technique and the clinical significance of physical examination. The following questions were proposed.

If an enlarged lymph node is found during palpation of the superficial lymph nodes, how should we describe the enlarged lymph node?

Answer: We should note the number, size, location, hardness, tenderness, texture, mobility, redness, adhesion, and presence of fistula when we describe enlarged lymph nodes.

2. What is the content and sequence of cardiac auscultation and the mechanisms of cardiac murmur?

Answer: Auscultation encompasses the heart rate, heart rhythm, heart sounds, extra heart sounds, murmur, and pericardial friction rub.

Auscultation sequence: mitral valve region, pulmonary valve region, aortic valve region, second aortic valve auscultation region, tricuspid valve region.

Mechanisms: (1) Blood flow acceleration, (2) Stenosis of valve opening caliber or large vascular channel, (3) Incomplete valve closure, (4) Abnormal blood flow channel, (5) Foreign body or abnormal structure in the cardiac cavity.

CSPAP

The CSPAP was broken down into different testing compartments, with each assessment made against a designated criterion. After finishing the physical examination assignments, students (peer authors) were required to submit video recordings of their assignments in a timely manner. The workflow of the CSPAP is illustrated in Figure 1. These video recordings were then assessed, with reference to the assessment criteria and standard physical examination videos produced by the Department of Medical Diagnosis. The following assessment processes involved all tutors and students. An assignment was first selected and assessed by a tutor, the results of which were taken as the gold standard. This assignment was subsequently delivered to students for peer assessment, without informing them that the tutors had assessed it. The peer assessment results were then investigated with reference to the gold standard, to determine the assessment capability of the reviewer (student). Reviewers were considered in terms of their weighted skill in assessing others. In addition to this gold standard assignment, a reviewer (student) randomly reviewed another three peer assignments, such that each student’s assignment was assessed by three reviewers. The consequent peer assessment of this assignment was a comprehensive combination of these three peer-reviewed results, taking into account each reviewer’s weight, as adjusted using Bayesian methods.[8][9] The CSPAP was used to evaluate the assessment capability of peer authors in obtaining the final results of the assignment completed by each student. The peer assessment result revealed the performance of each peer author in physical examination practice, and their assessment capability demonstrated their competence in identifying errors made in practice. Together, these aspects form the cornerstone of the CSPAP in evaluating a student’s mastery of clinical skills (Figure 2).

Figure 1.

Flowchart of the clinical skills peer assessment platform (CSPAP).

Figure 2.

Clinical skills peer assessment platform (CSPAP) user interface.

Traditional assessment

The traditional assessment approach refers to an assessment exam administered at the end of each semester. Tutors, rather than peers, are responsible for evaluating the performance of clinical skills among their students by administering several tests and exams.

CSPAP components and training process

The CSPAP functions through cooperation among the following three major components: the Instructor Portal allows the instructor to create training curricula for students and to set initial grading standards (the gold standard); the Student Portal allows students to register their customized training sessions and to conduct peer assessment based on the gold standard; and the Recording Portal is readily set up in training rooms to digitally record all training sessions and upload the corresponding videos to the platform server so that these can be retrieved for multi-person assessment.

The overall training process based on the CSPAP comprises two sessions: the operation session and the assessment session. In the operation session, the instructor is responsible for creating training curricula on the Instructor Portal, including designing the practice activity, booking the training rooms, and informing students about time limits for training. Students are responsible for making appointments on the Student Portal and finishing their registered training sessions in the correct room and at the correct time. The Recording Portal is set up in training rooms to record students’ performance and training videos are uploaded to the platform server, such that students can later retrieve their own videos using the Student Portal.

The assessment session can be unlocked by the Instructor at any time once the training session begins. The assessment session can be further divided into three steps: a) the instructor sets the gold standard; b) students conduct peer assessment; c) the system calculates a weighted grade.

Instructor sets the gold standard video

The gold standard video is initially graded by the instructor and is distributed to every student for grading, to test their assessment ability. When the instructor unlocks the assessment session, that instructor must set at least one video as the gold standard. The system then automatically distributes the video(s) to students, together with other unrated videos.

b. Students conduct peer assessment

After the instructor sets the gold standard video(s), students are able to conduct peer assessment. Each student must evaluate at least three videos, and at least one of these videos should be the gold standard video. To ensure the validity of students' assessment ability, the students are not informed which is the gold standard video.

c. System calculates a weighted grade

Along with conducting peer assessment, the system collects students’ grade results, calculates students’ assessment ability, weights their grades, and presents a weighted grade for each student's training. The weighted grade is dynamic and can fluctuate when students are newly added to the training.

Participant groups

Year 2012 (second year) students majoring in clinical medicine were selected as the control group, and their Year 2013 (third year) counterparts were assigned to the study group. Written informed consent was obtained from all participants, and the personal information of each enrolled student was strictly protected. All study procedures were approved by the ethics committee of Nanjing Medical University.

Teaching time, practice time, teaching method, and assessment criteria were the same between the control and study groups. Students were informed in advance regarding the standard physical examination video and evaluation criteria. In the control group, medical students were assessed using the traditional assessment approach. In the study group, enrolled participants were evaluated using the traditional assessment method combined with the CSPAP. Maximum scores for the physical examination, paper exam, and peer assessment were each 20 points.

Statistical analysis

All data were analyzed using SPSS 19.0 (IBM Corp., Armonk, NY, USA). Scores for the physical examination and paper exam were expressed as mean±standard deviation (SD). Comparison between two groups was statistically assessed using the t-test. Linear regression was performed to analyze the correlation among different categories of scores using GraphPad Prism version 5.0 (GraphPad, San Diego, CA, USA). One-way analysis of variance was carried out to investigate the association between obtained scores and sex. A P value  < 0.05 was considered statistically significant.

Results

Participants

A total 126 Year 2012 students were included in the control group, and 126 Year 2013 students were assigned to the study group.. Among the medical students in the control and study groups, no differences were observed in terms of participants’ age, college entrance examination scores, and current academic ranking.

Comparison of physical examination score and paper exam score

In terms of the physical examination, the mean score obtained by students in the control group was significantly lower than that of their counterparts in the study group (P < 0.05). Similarly, the average paper exam score of the study group was considerably higher than that obtained by the control group (P < 0.05). The peer assessment score among Year 2013 medical students was also significantly higher than that among their Year 2012 counterparts (P < 0.05), as illustrated in Figure 3.

Figure 3.

Comparison of physical examination scores and paper exam scores.

Correlation between two paired categories of scores among Year 2013 medical students.

Correlation among different categories of scores for Year 2013 medical students

A correlation coefficient of 0.4907 was obtained between the peer assessment scores and physical examination scores of Year 2013 medical students (P < 0.01). For this group, the peer assessment score was significantly correlated with the paper exam score (r = 0.5513, P < 0.01). These results demonstrated a significant positive association between peer assessment scores and physical examination scores, as well as between peer assessment scores and paper exam scores among Year 2013 medical students.

Correlation between different categories of scores and sex

As illustrated in Table 1, no significant correlation was observed between the control and study groups in terms of the physical examination score (F =  1.135), paper exam score (F = 0.106), and peer assessment score (F = 0.732). There were no significant differences among the different categories of scores or by sex.

Table 1.

Correlation analysis according to different categories of scores and sex among Year 2013 medical students.

Score	Group	Sum of squares	df	Mean square	F	P
Physical examination scores of Year 2013 students	Between-group	3.724	1	3.724	1.135	0.289
	Within-group	406.816	124	3.281
	Total	410.540	125
Paper exam scores of Year 2013 students	Between-group	0.761	1	0.761	0.106	0.745
	Within-group	888.971	124	7.169
	Total	889.732	125
Peer assessment scores of Year 2013 students	Between-group	1.672	1	1.672	0.732	0.394
	Within-group	283.296	124	2.285
	Total	284.968	125

Discussion

Medical and health care professionals use peer assessment as a valuable method to share knowledge, evaluate performance, and foster professional development. Peer assessment nurtures professional growth in medical students before they enter clinical practice and can contribute to developing the self-assessment skills students need to judge their own abilities when working as independent health care practitioners. As a formative evaluation tool, peer assessment has been widely studied, with beneficial results in medical education.

Physical examination has always been the cornerstone of medical diagnosis education, as it ranks among one of the most essential skills students must master as they enter clinical practice.[10] In this article, we demonstrated that the CSPAP can not only improve the physical examination skills of medical students but can also enhance their knowledge of basic clinical principles. In addition, we found no significant differences between male and female students in terms of these beneficial outcomes.

More importantly, the CSPAP has multiple advantages. First, the platform can provide medical students with adequate resources for studying. Multi-media resources are essential to becoming proficient in conducting physical examination, especially for items requiring the performance of complex maneuvers.[11,12] Highly-experienced clinicians are involved in the production of standard physical examination videos, in which step-wise examination techniques are demonstrated. These videos are available for all students to study and be used in practice and to evaluate their peers’ performance.

Second, the CSPAP introduces a pattern of active learning. In contrast to traditional lecture-based learning, active learning highlights the active involvement and participation of students during the learning process.[13,14] Medical students can achieve all aspects of skill improvement in terms of gathering information and management, learning new knowledge, contemplating and solving problems, and communication and cooperation.[15,16] The CSPAP empowers medical students who are motivated by active learning, as there is unlimited opportunity for them to upload assignments within the given timeframe. This essentially means that such students will practice until they are satisfied with their performance, so as to achieve high scores. The study profiles of students are cloud-based and trackable by each student, allowing them to assess and learn from each other.

As a form of teamwork, peer assessment is believed to enhance self-awareness, facilitate personality development, and promote more active roles among students during the learning process. The CSPAP can easily address testing items that are difficult to assess using traditional standardized tests; this process is facilitated by implementing a Bayesian statistical model and taking into account the sensitivity and specificity of each rater. The CSPAP can not only assess the performance of students in a relatively fair manner, the platform can also reveal many problems in their learning, which provides tutors with first-hand materials that can be used to improve teaching designs.

Third, introducing formative assessment provides assessment feedback to students and tutors. A great deal of information can be obtained from peer assessment, including attitudes toward studying, training times and corresponding scores, existing problems and methods to be improved, and differences in teaching outcomes among tutors. These data are valuable for tutors to devise more precise teaching methods for different students.[17,18] Feedback on students’ learning is available online, which makes it possible for students to conveniently access information about their own learning status.

Study limitations

Our servers are incapable of handling a large number of students who are simultaneously performing online assessments. In addition, the assessment criteria checklist should include greater detail. Challenges coexist with opportunities when it comes to reforming the teaching of clinical skills to medical students in China.[19,20] Future work will concentrate on establishing a convenient and productive CSPAP based on our objective structured clinical exam animation facilities, to meet the needs of contemporary clinical skills education. We hope that further application of this platform will result in training more professional and skilled clinicians.