Impact of a focused review course in cardiovascular and thoracic surgery on test performance.

Background: The Core Curriculum Review Course in Cardiovascular and Thoracic Surgery is a 4-day educational program consisting of 77 didactic lectures that provide a comprehensive review of the material required for surgeons preparing for the American Board of Thoracic Surgery competency written examination. The lectures are supplemented with a written syllabus and interactive audience participation system. We sought to determine whether participation in this course could improve participants' performance on a cardiothoracic subject-based test.
Methods: Sixty-five participants attended the 2018 course. Before beginning the course lectures, a multiple-choice pretest consisting of 77 questions was administered via mobile application to gauge the participants' baseline knowledge. A second multiple-choice posttest was made available beginning 7 weeks after the course, also by mobile application.
Results: Twenty-nine participants completed both the pretest and the posttest. The median pretest score was 47% (36 of 77 correct answers). The median posttest score was 61% (47 of 77 correct answers), representing an increase of 14%. The Wilcoxon signed-rank test indicated a significant difference between the pretest and posttest scores (z = -4.36; P = .00). Overall, 25 participants (86%) improved their posttest score. Conclusions: The core curriculum review course was successful in improving participants' performance on the course tests, indicating that the participants' fund of knowledge was likely increased by attendance at the program. Additional strategies should be considered to address particular areas of study both for individual participants and for residents currently in training.

Participant scores improved across all content areas.

Participation in the Core Curriculum Review Course can improve participant scores on an exam that focuses on cardiothoracic subject matter to prepare participants for board certification.

This study shows that a formal review course combined with online materials and a structured study schedule can identify areas of knowledge deficiency for focused learning. These data could be useful for residency programs and the board for improving residency education.

See Commentaries on pages 286 and 287.

The need for structured education during surgical training was recognized at the organization of the first residency programs, underscoring the importance of formalized acquisition of medical knowledge. As the scientific field expanded, more attention was paid to establishing national curricula and tools for both faculty and residents to address knowledge gaps during training.2, 3, 4, 5 After completion of residency, however, surgeons are typically responsible for their own individual education and preparation to meet the educational standards set by national societies and certifying boards.

The Core Curriculum Review Course in Cardiovascular and Thoracic Surgery was instituted in 1992 as part of a concerted effort to improve knowledge for individuals seeking to pass the American Board of Thoracic Surgery (ABTS) qualifying examination. The course is held over a 4-day period in Salt Lake City, Utah and consists of a series of condensed, didactic lectures that cover the fundamental “core” concepts in the field. Each lecture has associated interactive questions and a printed syllabus to help participants identify those areas needing more focused study. The materials are then captured in a digital format and hosted in a Web-based mobile application that is accessible on demand for additional review. Finally, daily text message reminders are sent to each participant to encourage adherence to a regular study schedule over the ensuing weeks before the ABTS examination.

We sought to evaluate whether attendance at the Core Curriculum Review Course with its accompanying online materials could improve participants' overall fund of knowledge and therefore assist in preparation for the ABTS qualifying examination. We assessed core knowledge before and after participation in the review course by utilizing a pretest and posttest consisting of multiple choice questions directly linked to each lecture topic.

Methods

Participants

A total of 65 participants attended the Core Curriculum Review Course in 2018. Of these attendees, 29 completed both the pretest administered prior before the start of the lectures and the posttest administered 50 days after the conclusion of the course. Of these participants, 18 had completed their residency in June 2018, 3 were fellows, 1 was a resident, and 3 had been practicing surgeons for over a year; 4 participants did not disclose their level of experience. Participants were not randomly assigned to control and “treatment” groups given the retroactive, quasi-experimental nature of the analysis. The Institutional Review Board of Intermountain Healthcare considered the study to be exempt.

Instrument and Design

At the beginning of the Core Curriculum Review Course and before any lectures, participants were instructed to complete a pretest consisting of 77 multiple choice questions to establish their baseline understanding. Each question was directly correlated with an associated topic in cardiovascular and thoracic surgery under 7 general content areas: acquired cardiac disease, cardiovascular disease, congenital heart disease, esophageal disease, heart failure/transplantation, pulmonary disease, and mediastinal disease. The test is compiled and administered within a Web-based testing application designed specifically for the course by a third-party vendor. Course attendees were not required to participate in the pretest, and attendance at each lecture was not mandatory.

During the next several weeks, participants were encouraged to adhere to a regular study schedule using the written syllabus, the online recorded lecture materials, and their own individual notes from course attendance. In addition, daily text notifications were sent to each participant's mobile phone as reminders to study a particular topic. Completion of a focused study and review was solely at the discretion of each individual and was not mandatory.

At 7 weeks after the educational review course, participants received an email prompting them to complete a posttest consisting of 77 multiple choice questions to assess their current level of understanding. As with the pretest, each question was directly correlated with one of the lecture topics and were designed to be similar, but not identical, to the questions included in the pretest. Again, participation in the posttest was not mandatory.

Analysis

The data were compiled in Excel (Microsoft, Redmond, Wash) to generate descriptive statistics for participant information as well as for item responses. SPSS Version 26 (IBM, Armonk, NY) was used to create histograms and Q-Q plots as a visual check for normality of data. SPSS was also used to assess test retest reliability. Owing to the presence of nonnormal data, a Wilcoxon signed-rank test was also performed in SPSS.

Results

Response Rates

Of the 65 individuals who took part in the Core program, 59 completed the pretest, for a completion rate of 91%. The overall completion rate for the posttest was 48%. For the purpose of this study, only data from those who completed both the pretest and posttest were considered. These 29 participants represented a completion rate of 45% for both the pretest and posttest.

Descriptive Statistics

Although performance on the pretest and posttest individually was not a focus of this study, descriptive statistics for the overall item responses for both the pretest and posttest are provided in Table 1. These data point to the relative difficulty of the exams, with participants scoring poorly on the pretest (mean ± SD, 47 ± 13%). Scores on the pretest were nonnormally distributed, with a skewness of 1.51 (standard error [SE], 0.43) and kurtosis of 6.93 (SE, 0.85). Scores on the posttest were higher (mean, 62 ± 13%) and were also nonnormally distributed (but not to the same degree as with the pretest data), with a skewness of 0.64 (SE, 0.43) and kurtosis of -0.11 (SE, 0.85). Q-Q plots, shown in Figures 1 and 2, provided a visual check to confirm the issues of nonnormality and the presence of outliers and to support the decision to use a nonparametric test to assess the differences between the pretest data and posttest data.

Table 1

Descriptive statistics for item responses in the 29 participants who completed both the pretest and posttest

Variable	Number correct		Number incorrect		% correct		% incorrect
	Pretest	Posttest	Pretest	Posttest	Pretest	Posttest	Pretest	Posttest
Minimum	14	30	1	7	18	39	3	10
Maximum	75	69	55	47	97	90	82	61
Average	36	48	39	29	47	62	53	38
Median	36	47	41	30	47	61	53	39
SD	9.97	10.02	9.52	10.08	13	13	13	13

Figure 1

Q-Q plot for pretest. The presence of nonnormal data and outliers in the dataset supports the use of a nonparametric test to assess differences between the pretest and posttest data. PREPER, Pretest performance.

Figure 2

Q-Q plot for posttest. There are no outliers in this dataset, but there are nonnormal data, supporting the use of a nonparametric test to assess the differences between the pretest and posttest data. POSTPER, Posttest performance.

Content Validity and Reliability

Content validity was established in the development and refinement of the exam questions by the planning committee members. Each physician member of the planning committee is a currently practicing ABTS-certified cardiovascular and thoracic surgeon. All 77 questions in the respective exams are linked to a corresponding topic (acquired cardiac disease, cardiovascular disease, congenital heart disease, esophageal disease, heart failure/transplantation, pulmonary disease, and mediastinal disease), with multiple questions per topic.

Because the content was the same in the pretest and posttest same, the present study focused on the sample of participants who completed both the pretest and posttest, which allowed for the use of the Pearson correlation coefficient as a measure of test–retest reliability. There was a marginal, positive correlation between the pretest and posttest that was not statistically significant [r (27) = 0.33; P = .80].

Nonparametric Statistic

Both pretest and posttest data were skewed and kurtotic, which pointed to issues of normality. To account for nonnormality, Wilcoxon signed-rank test was used. Results indicated a statistically significant difference between pretest scores (median, 47%) and posttest scores (median, 61%) (z = -4.36; P = .00).

The test–retest reliability correlation was undeniably low (r = 0.33), with the small sample size as a likely contributing factor. Another factor could be the effect of bias or practice. With the items on the tests being identical for both the pretest and posttest, the increase in test scores between the pretest and posttest could be attributed to participants being more familiar with the exam materials.

Because the data were not normally distributed and the effective sample size was small, the Wilcoxon signed-rank test was used to assess for a difference in the location of the center of the data (the median) between the pretest and posttest scores. A statistically significant test statistic was obtained, supporting the idea of a meaningful change in median scores that was not due to chance alone. In terms of null hypothesis testing, the null hypothesis that there were no changes to the median between the pretest and posttest can be rejected.

Individual Content Area and Topic Analysis

The small sample size limited our ability to provide meaningful statistical comparisons between the different content areas; nonetheless, the differences in raw mean percentage scores from pretest to posttest for those who completed both tests indicated improvement in all content areas. Esophageal disease saw the greatest improvement (percent difference, 29%), and congenital heart disease saw the least improvement (percent difference, 13%). Mean score improvements in the remaining content areas are shown in Figure 3.

Figure 3

Comparison of pretest and posttest scores by subject category. All categories demonstrate improvement in posttest scores, with the most marked improvements seen in the cardiovascular disease and esophageal disease categories.

In-depth analysis of specific content topics indicated poor baseline knowledge in 16 topics, with 20% or less of participants answering the associated questions correctly on the pretest. For 6 topics, mean percentage scores were <10%: developmental anatomy (5%), cardiac transplantation (7%), thoracic trauma (8%), atrioventricular septal defect (8%), and combined coronary/carotid/valvular disease (10%). Results for each individual topic are provided in Appendix 1.

Discussion

This study demonstrates that a focused review course in cardiovascular and thoracic surgery can improve test score performance from pretest to posttest. Figure 4 is a graphical abstract representing the methods, key results, and implications of this analysis. The quasi-experimental design precluded the ability to randomly assign participants to a treatment and control group and ultimately limited our ability to assign causation. Nonetheless, we can conclude with some level of certainty that the series of lectures and study of the printed and online review materials likely resulted in the 14% increase in median scores for the participants who completed both the pretest and posttest. Although the combination of variables certainly was a cause, it probably was not the sole cause owing to other external factors, such as the use of accompanying online materials or practical experience in the field (Video 1).

Figure 4

Methods, results, and implications of the study. Methods include administration of the pretest, attendance at the review course, administration of the posttest, and score analysis. Key results are an overall 14% improvement in mean score, a statistically significant change. Implications are that attendance at the review course increases the overall fund of knowledge, improves standardized test scores, and can help identify areas of focus for residency training programs.

Summary of study findings. Video available at: https://www.jtcvs.org/article/S2666-2736(21)00180-7/fulltext.

One important limitation of this study is the small sample size inherent to this particular educational activity. A select number of individuals are preparing for the ABTS examination each year, and it will take many years to obtain large datasets for more accurate analysis. In addition, participation in the pretest and posttest is voluntary, and only roughly one-half of the course attendees completed the posttest, further reducing the dataset. One solution would be to initiate mandatory participation in both tests similar to residency in-training examinations.

A second limitation is the inability to establish a causal relationship between course attendance and test performance. Participation and engagement in the individual lectures and online materials are voluntary. Self-study habits vary widely, and individuals may use other sources for additional review. Other studies have shown that a regular study schedule and practice examinations during residency can improve in-training examination score and national board pass rates, and even relatively short periods of intense study can result in improved test scores.6, 7, 8, 9 Web-based learning has been shown to have a positive effect in medical education and has been adopted by the thoracic surgery community., These self-directed online curricula demonstrate improvement for in-training examination scores and allow residents to structure their own individual study methods. Although it is difficult to separate performance improvement due to the review course versus self-study afterward, a positive effect from the review course is supported from these data given the limited time that the majority of these particular participants have for study and preparation.

Two important issues are raised from this analysis, both of which revolve around the central concept of improving resident education. First, although the study data show improvement between 2 separate assessments, whether a similar outcome can be shown for participant performance on the ABTS qualifying examination is unknown. Passing this exam is the ultimate goal of the review course, but this would require a larger effort and cooperation to analyze outcomes from the board.

Second, specific content analysis revealed an overall paucity of core knowledge among the participants, as evidenced by the mean pretest score of 47%. Particularly worrisome are the several content areas with mean scores <20%, suggesting a lack of fundamental education in these topics. These findings represent an important opportunity to focus teaching efforts not only at the review course and online materials, but also during residency training. These data could be useful to individual residency programs for refining their curricula.

Conflict of Interest Statement

The authors reported no conflicts of interest.

The Journal policy requires editors and reviewers to disclose conflicts of interest and to decline handling or reviewing manuscripts for which they may have a conflict of interest. The editors and reviewers of this article have no conflicts of interest.

Q	Topic	N	FFEOOR	PRENOOR	PPETOTAL	FFE%	POSTOOR	POSTINOOR	POSTTOTAL	POST%	DFF
1	Esophageal diagnostic procedures	94	25	38	63	40	28.00	3.00	31.00	90	51
2	Pulmonary diagnostic testing	94	33	30	63	52	20.00	11.00	31.00	65	12
3	Thoracic trauma	93	5	57	62	8	14.00	17.00	31.00	45	37
4	Bullous emphysema and pneumothorax	94	34	29	63	54	21.00	10.00	31.00	68	14
5	Pulmonary infections	93	28	34	62	46	14.00	17.00	31.00	45	0
6	Pulmonary fungus, tuberculosis	93	21	41	62	34	10.00	21.00	31.00	32	−2
7	Mediastinal tumors	93	8	54	62	13	14.00	17.00	31.00	45	32
8	Pulmonary anomies	93	31	31	62	50	24.00	7.00	31.00	77	27
9	Lung cancer–diagnosis	93	34	28	62	55	16.00	15.00	31.00	52	−3
10	Lung cancer–staging	93	31	31	62	50	14.00	17.00	31.00	45	−5
11	Lung cancer–surgery	93	15	47	62	24	23.00	8.00	31.00	74	50
12	Lung cancer–metastatic disease and adjuvant therapy	93	34	28	62	55	16.00	15.00	31.00	52	−3
13	Benign lung tumors	93	46	16	62	74	29.00	2.00	31.00	94	19
14	Thoracic outlet syndrome	93	31	31	62	50	25.00	6.00	31.00	81	31
15	SVC syndrome and pulmonary embolism	93	17	45	62	27	12.00	19.00	31.00	39	11
16	Chest wall congenital deformities and tumors	93	23	39	62	37	10.00	21.00	31.00	32	−6
17	The diaphragm	93	20	42	62	32	17.00	14.00	31.00	55	23
18	The pleura	93	18	44	62	29	22.00	9.00	31.00	71	42
19	The trachea	93	48	14	62	77	18.00	13.00	31.00	58	−19
20	Esophageal congenital disorders, burns/structure	93	59	3	62	96	31.00	0.00	31.00	100	5
21	Esophageal motility disorders	93	12	50	62	19	24.00	7.00	31.00	77	58
22	Gastroesophageal reflux disease and benign esophageal tumors	93	10	52	62	16	11.00	20.00	31.00	35	19
23	Esophageal cancer	92	21	40	61	34	21.00	10.00	31.00	68	33
24	Cardiac anatomy	92	33	28	61	54	25.00	6.00	31.00	81	27
25	CV physiology	92	45	16	61	74	28.00	3.00	31.00	90	17
26	CV pharmacology/coagulation	92	48	13	61	79	27.00	4.00	31.00	87	8
27	Critical care physiology	92	21	40	61	34	17.00	14.00	31.00	55	20
28	Acute heart failure	92	45	16	61	74	23.00	8.00	31.00	74	0
29	Cardiopulmonary bypass/myocardial protection	92	25	36	61	41	22.00	9.00	31.00	71	30
30	Coronary artery disease	92	13	48	61	21	28.00	3.00	31.00	90	69
31	Myocardial infarction	92	22	39	61	36	17.00	14.00	31.00	55	19
32	Myocardial revascularization	92	40	21	61	66	21.00	10.00	31.00	68	2
33	Postinfarction ventricular aneurysm and VSD	92	29	32	61	48	25.00	6.00	31.00	81	33
34	Ischemic mitral disease	92	49	12	61	80	30.00	1.00	31.00	97	16
35	Combined coronary/carotid/valvular disease	92	6	56	61	10	9.00	22.00	31.00	29	19
36	Aortic valve disease	92	24	37	61	39	18.00	13.00	31.00	58	19
37	Mitral valve disease	92	8	53	61	13	12.00	19.00	31.00	39	26
38	Tricuspid and multiple valve disease	92	47	14	61	77	11.00	20.00	31.00	35	−42
39	Prosthetic valve selection	92	29	32	61	48	21.00	10.00	31.00	68	20
40	Endocarditis	91	12	48	60	20	11.00	20.00	31.00	35	15
41	Clinical trials in coronary artery disease	91	8	52	60	13	29.00	2.00	31.00	94	80
42	Thoracic radiography	91	45	15	60	75	27.00	4.00	31.00	87	12
43	Anesthesia and echocardiography	91	15	45	60	25	13.00	18.00	31.00	42	17
44	Cardiac tumors	91	30	30	60	72	24.00	7.00	31.00	77	6
45	Hypertrophic cardiomyopathy	91	30	30	60	50	22.00	9.00	31.00	71	21
46	The pericardium	91	37	23	60	62	8.00	23.00	31.00	26	−36
47	Cardiac transplantation	91	4	56	60	7	4.00	27.00	31.00	13	6
48	Lung and heart-lung transplantation	91	27	33	60	46	20.00	11.00	31.00	66	20
49	Transplant immunology	91	51	9	60	85	28.00	3.00	31.00	90	5
50	Assisted circulation	91	9	51	60	15	10.00	21.00	31.00	32	17
51	Surgical treatment of heart failure	91	35	25	60	58	3.00	28.00	31.00	10	−49
52	Cardiac arrhythmias	91	28	32	60	47	29.00	2.00	31.00	94	47
53	Aortic aneurysm	91	15	45	60	25	13.00	18.00	31.00	42	17
54	Aortic dissection	91	37	23	60	62	26.00	5.00	31.00	84	22
55	Complications and less invasive operations	91	45	15	60	75	26.00	5.00	31.00	84	9
56	Developmental anatomy	91	3	57	60	5	6.00	25.00	31.00	19	14
57	Palliative operations	91	25	35	60	42	16.00	15.00	31.00	52	10
58	Patent ductus arteriosus and aortopulmonary window	91	20	40	60	33	11.00	20.00	31.00	35	2
59	Coarctation of the aorta–interrupted aortic arch	91	26	34	60	43	26.00	5.00	31.00	84	41
60	Atrial septal defect and partial anomalous venous connection	91	7	53	60	12	9.00	22.00	31.00	29	17
61	Ventricular septal defect	91	34	26	60	57	18.00	13.00	31.00	58	1
62	Atrioventricular septal defect	91	5	55	60	8	22.00	9.00	31.00	71	63
63	Tetralogy of Fallot	91	30	30	60	50	10.00	21.00	31.00	32	−18
64	Coronary anomalies	91	15	45	60	25	14.00	17.00	31.00	40	20
65	Truncus arteriosus	91	37	23	60	62	27.00	4.00	31.00	87	25
66	Pulmonary stenosis– pulmonary atresia	90	40	19	59	68	29.00	2.00	31.00	94	26
67	Vascular rings and sling	90	22	37	59	37	9.00	22.00	31.00	29	−8
68	Congenital aortic stenosis	90	26	33	59	44	21.00	10.00	31.00	68	24
69	Sinus of Valsava aneurysm/fistula	90	47	12	59	80	28.00	3.00	31.00	90	11
70	Ebstein anomaly	90	20	39	59	34	15.00	16.00	31.00	48	14
71	Single ventricle anomalies	90	40	19	59	68	28.00	3.00	31.00	90	23
72	Hypoplastic left heart syndrome	90	44	15	59	75	2.00	29.00	31.00	6	−68
73	Transposition of the great arteries	90	21	38	59	36	26.00	5.00	31.00	84	48
74	Corrected TGA	90	23	36	59	39	27.00	4.00	31.00	87	48
75	DORV and atrial isomerism	90	8	51	59	14	9.00	22.00	31.00	29	15
76	Total anomalous pulmonary venous connection–cor triatriatum	90	11	48	59	19	14.00	17.00	31.00	45	27
77	Antireflux operation in the setting of gastroesophageal reflux disease	70	23	19	42	56	18.00	10.00	28.00	64	10

Q, Question; FFEOR, pretest correct answers; PRENOOR, prestest incorrect answers; PPETOTAL, pretest total answers; FFE%, pretest percent correct; POSTOOR, posttest correct answers; POSTINOOR, posttest incorrect answers; POSTTOTAL, posttest total answers; POST%, posttest percent correct; DFF, percentage difference between pretest and posttest; SVC, superior vena cava; CV, cardiovascular; VSD, ventricular septal defect; TGA, transposition of the great arteries; DORV, double outlet right ventricle.