Assessment of structured physical examination skills training using a retro-pre-questionnaire.

PURPOSE: The effectiveness of physical examination skills (PES) training is very rarely assessed using the "post-then-pre" approach. In this study, a retro-pre-questionnaire was used to study the effect of structured physical examination skills training (SPEST) imparted to second-year undergraduate medical students.
METHODS: KIST Medical College (KISTMC) affiliated to Tribhuvan University Nepal admitted its first batch of MBBS students in November 2008. The university curriculum recommends the involvement of Medicine and Surgery Departments in PES training, but the methods for teaching and assessment are not well defined. KISTMC has made training more structured and involved the Medicine, Surgery, Gynaecology and Obstetrics, Orthopaedics, ENT, Ophthalmology, Paediatrics, and Family Medicine Departments. SPEST includes the teaching/learning of basic PES for 210 minutes once a week for 28 weeks. Self-assessment is done by using a retro-pre-questionnaire at the end of the last session of training, and these data are analysed using SPSS.
RESULTS: Out of 100 students, 98 participated in the objective structured clinical examination (OSCE); 82 completed the retro-pre-questionnaire. Forty-six skills representing various systems were selected for inclusion in the retro-pre-questionnaire from among the many skills taught in different departments. The average perceived skills score (maximum score, 46×4=184) before training was 15.9 and increased to 116.5 after training. The increase was statistically significant upon the application of a paired t-test.
CONCLUSION: The students perceived that their level of skills improved after the training. The retro-pre- instrument seems to be useful for assessing the learners' self-reported changes in PES after training if a large number of skills need to be assessed. However, it should be noted that although a retro-pre-questionnaire may reveal valuable information, it is not a substitute for an objective measure or gold standard.

The retrospective post-then-pre design is widely used for assessing learners’ self-reported changes in knowledge, awareness, skills, confidence, attitudes, or behaviors. It takes less time, is less intrusive, and for self-reported change, avoids the pre-test sensitivity and the response shift bias that result from a pre-test overestimation or underestimation [1]. The effectiveness of physical examination skills (PES) training is very rarely assessed using a self-assessment instrument. In this study, a self-assessment instrument, a retro-pre-questionnaire, was used to study the perceived effect of structured physical examination skills training (SPEST) imparted to second-year undergraduate medical students as a part of early clinical exposure [2].

METHODS

This study was conducted in KIST Medical College (KISTMC), a newly established medical school in the private sector located in Lalitpur district of Nepal. The first batch of MBBS students was admitted in November 2008. The college is affiliated with Tribhuvan University Nepal. The university curriculum recommends the involvement of Medicine and Surgery Departments for PES training, but the methods for teaching and assessment are not well defined. KISTMC has made training more structured and involved the Departments of Internal Medicine, General Surgery, Gynaecology and Obstetrics, Orthopaedics, Otorhinolaryngology, Ophthalmology, Paediatrics, and Family Medicine for imparting training to secondyear MBBS students as a part of early clinical exposure. SPEST includes the teaching/learning of basic PES for 210 minutes (3.5 hours) once in a week for 28 weeks. Each SPEST session is divided into three sub-sessions: Tutorial (T), Demonstration (D), and Practice (P). The Tutorial element encompasses an overview by a faculty preceptor on the skills to be taught; the Demonstration involves an actual demonstration of each of the skills taught with a step-by-step description of the procedure; and Practice involves the performance/practice of each demonstrated skill by the students using a sequential description to be observed by the preceptor. At the end of the last session of training, a retro-pre-questionnaire was used for assessing the learners’ self-reported changes.

Forty-six skills were selected from many skills taught during training and included in the retro-pre-questionnaire (Table 1). The skills included in the questionnaire represented various systems taught in the different departments during the training. The retro-pre-questionnaire was modelled in line with the Dreyfus model of skill aquisition. Before use, the retro- pre-questionnaire was piloted with 10 students of a senior cohort. It was found to be reliable and valid. The objective item scores were added to obtain the overall score and the descriptive statistics for before and after the training. A paired t-test was used for evaluating the difference in the overall scores. The difference between the overall scores before and after was found to follow the normal distribution, as confirmed by the Shapiro-Wilk test. The data were analysed using SPSS ver. 15 (SPSS Inc., Chicago, IL, USA).

Table 1.

Students’ responses to the retro-pre-questionnaire before and after structured physical examination skills training in KIST Medical College (KISTMC) affiliated to Tribhuvan University Nepal

Skill (item)		Response (%)
		Not confident		Somewhat confident		Very confident		Able to do independently
		Before training	After training	Before training	After training	Before training	After training	Before training	After training
1	Measuring body temperature	12.5	1.2	48.8	13.4	28.8	25.6	10.0	59.8
2	Measuring respiratory rate	55.1	2.4	38.8	18.3	06.1	37.8	0.0	41.5
3	Recording pulse	58.2	1.2	32.8	6.2	07.5	46.3	0 1.5	46.3
4	Measuring blood pressure	56.1	2.4	33.3	18.4	09.1	39.0	0 1.5	40.2
5	Detecting pedal edema	70.4	2.4	25.9	25.6	3.7	35.4	0.0	36.6
6	Detecting cyanosis	60.9	2.4	34.8	18.3	4.3	53.7	0.0	25.6
7	Detecting clubbing	80.0	4.9	13.3	18.3	6.7	43.9	0.0	32.9
8	Palpating cervical lymph nodes	75.0	7.3	25.0	48.8	0.0	28.0	0.0	15.9
9	Demonstrating Range of Motion at Elbow Joint	73.9	14.6	26.1	46.4	0.0	24.4	0.0	14.6
10	Demonstrating range of motion at hip joint	83.3	14.6	16.7	45.1	0.0	24.4	0.0	15.9
11	Palpating peripheral pulses	75.0	7.3	25.0	46.3	0.0	22.0	0.0	24.4
12	Assessing chest expansion	88.9	11.0	11.1	37.8	0.0	32.9	0.0	18.3
13	Determining position of trachea or palpation for trachea	100.0	14.6	0.0	42.7	0.0	24.4	0.0	18.3
14	Percussion of anterior chest	91.7	11.0	8.3	30.5	0.0	35.4	0.0	23.2
15	Auscultation of posterior chest	93.8	15.9	6.2	46.3	0.0	26.8	0.0	11.0
16	Eliciting vocal resonance on anterior chest	91.7	23.2	8.3	39.0	0.0	24.4	0.0	13.4
17	Measuring jugular venous pressure	100.0	41.5	0.0	39.0	0.0	13.4	0.0	6.1
18	Localizing apex beat	80.0	9.8	20.0	42.7	0.0	32.9	0.0	14.6
19	Auscultation of precordium	100.0	40.2	0.0	34.1	0.0	14.6	0.0	11.0
20	Using otoscope for examining ears	100.0	23.2	0.0	41.5	0.0	23.2	0.0	12.1
21	Using nasal speculum for examining ears	100.0	43.9	0.0	31.7	0.0	19.5	0.0	4.9
22	Using head mirror with light for examining ears	100.0	24.4	0.0	54.9	0.0	12.2	0.0	8.5
23	Measuring visual acuity	69.6	11.0	30.4	48.9	0.0	24.4	0.0	15.9
24	Measuring visual fields by confrontation method	80.0	20.7	20.0	37.8	0.0	30.5	0.0	11.0
25	Examining uniocular eye movement (muscles)	83.3	24.4	16.7	43.9	0.0	18.3	0.0	13.4
26	Examining pupillary light reaction	89.3	4.9	10.7	35.4	0.0	39.0	0.0	20.7
27	Palpation of liver	91.7	11.0	8.3	32.9	0.0	34.1	0.0	22.0
28	Palpation of kidneys	100.0	11.0	0.0	31.7	0.0	36.6	0.0	20.7
29	Palpation of spleen	100.0	12.2	0.0	31.7	0.0	34.1	0.0	22.0
30	Eliciting shifting dullness	100.0	20.7	0.0	34.1	0.0	30.5	0.0	14.6
31	Measuring liver span	100.0	43.9	0.0	32.9	0.0	13.4	0.0	9.8
32	Examining facial nerves	100.0	22.0	0.0	45.1	0.0	22.0	0.0	11.0
33	Examining accessory nerves	100.0	32.9	0.0	37.8	0.0	20.7	0.0	8.6
34	Eliciting tone of muscles in upper limbs	100.0	22.0	0.0	50.0	0.0	18.2	0.0	9.8
35	Eliciting reflexes in lower limbs	100.0	12.2	0.0	50.0	0.0	25.6	0.0	12.2
36	Eliciting planter response	100.0	19.5	0.0	37.8	0.0	34.2	0.0	8.5
37	Eliciting position sense in lower limbs	100.0	26.8	0.0	42.7	0.0	23.2	0.0	7.3
38	Assessing orientation to time, place, and person	100.0	3.7	0.0	17.0	0.0	47.6	0.0	31.7
39	Measuring height in children	61.9	1.2	38.1	19.5	0.0	46.3	0.0	32.9
40	Measuring mid-arm circumference in children	60.5	2.4	39.5	23.2	0.0	42.7	0.0	31.7
41	Demonstration of dehydration in infants	100.0	7.3	0.0	29.3	0.0	42.7	0.0	20.7
42	Counting respiratory rate in children under five years of age	100.0	13.4	0.0	46.4	0.0	25.6	0.0	14.6
43	Demonstration of pallor in children	100.0	6.1	0.0	29.3	0.0	50.0	0.0	14.6
44	Demonstration of jaundice in children	100.0	6.1	0.0	29.2	0.0	48.8	0.0	15.9
45	Measuring height of fundus	100.0	22.0	0.0	35.4	0.0	30.4	0.0	12.2
46	Eliciting obstetrical grips or Leopold’s manoeuvre	100.0	48.8	0.0	24.4	0.0	15.8	0.0	11.0

RESULTS

The students’ responses to the retro-pre-questionnaire before and after the structured physical examination skills training are given in Table 1. Out of the 100 students trained, 82 completed the retro-pre-questionnaires. The average skill score (maximum, 46×4=184) for before the SPEST module was 15.9 (median, 13.5) and that after was 116.5 (median, 116). The increase in scores was statistically significant (mean±SD, 100.5±23; 95% confidence interval, 95.45 to 105.59), which implied that the students perceived that they learned most of the skills after the SPEST module and that the course was effective. Further, the paired correlation was not very high (r= 0.240), but it was in the positive direction and was statistically significant (P=0.03), implying that the questionnaire before score had a low impact on the questionnaire after score.

DISCUSSION

The most widely used evaluation tool is a traditional “prethen- post” test, where participants are asked a series of questions at both the start of a program (pre-test) and then again at the end of program (post-test). This tool is believed to measure changes in participant knowledge, attitudes, or behaviours regarding whatever the program content is [3,4]. The criticism of the traditional pre/post tool has led to the use of a “retrospective pre-test” tool. This type of tool is well accepted because it is implemented at only one point in time. The theory behind designing this type of tool is that by testing what participants believe about the program content after program completion, their standard of assessing the changes in knowledge, skills, or attitudes is consistent and thus, not subject to a response shift bias [3,5,6]. Others note that the post-then-pre design reduces incomplete data sets and is convenient to administer given the time constraints that many program providers face, and is also easier for the program participants to complete [3,7,8].

If the objective is to identify how participants perceive the changes that they have made in knowledge, skills, attitudes, or behaviour, then a retro-pre method may be adequate to capture information on this type of data [3].

Our objective was to study the perceived impact of SPEST imparted to second-year undergraduate medical students. The results from the paired t-test showed that the difference between before and after the SPEST was statistically significant, which implied that the students did learn most of the skills after the implementation of the SPEST module and that the training was effective. Further, the paired correlation was not very high, but it was in the positive direction and was statistically significant, implying that the retro-pre before score had a low impact on the retro-pre after score. It is important to acknowledge that all self-reported information could be considered somewhat subjective.

In summary, because the students perceived that their level of skills improved after the training, we concluded that the retro-pre instrument is useful for assessing learners’ self-reported changes in PES after training if a large number of skills are to be assessed. However, it should be noted that although a retro-pre instrument may reveal valuable information, it is not a substitute for an objective measure or gold standard.