Assessment of Resident Knowledge in Pediatric Rheumatology.

Pediatric Rheumatic illnesses are complex, chronic, and often multi-systemic. Our goal was to assess the efficacy of 2 standardized pediatric rheumatology lectures, administered to pediatric residents, in improving the pediatric residents' knowledge and confidence in pediatric rheumatology. Two lectures, 1 hour each, were delivered by 2 residents and given 1 week apart to 28 pediatric residents. Pre- and post-tests assessed knowledge and residents' self-rated competence. Change in knowledge was assessed using paired t-tests. Twenty-eight residents participated in the study. Sixty-three percent (17/27) had an improvement in score from pre-test to post-test. Thirty-seven percent (10/27) reported increased competence from pre-test to post-test. Ninety-six percent (22/23) of the residents found the lectures to be beneficial. Residents in the third post-graduate year (PGY3) cohort had the most significant improvement in scores. The lectures given to the pediatric residents increased pediatric knowledge and the pediatric residents found these lectures to be beneficial.

Introduction

Musculoskeletal pain is a common complaint amongst children. According to The Rheumatologist, there are currently only an estimated 280 to 300 Pediatric Rheumatologists in the US and 8 states that do not have any. Children in the US travel an average of 57 miles to the nearest pediatric rheumatologist, whereas other pediatric specialists typically require less than 25 miles of travel. Multiple studies have demonstrated that recent residency graduates and general pediatricians report a lack of confidence in their clinical skills when evaluating and treating children with musculoskeletal complaints.[3-7] At the time of our study, our institution had 2 pediatric rheumatologists who taught residents through bedside rounds, formal lecture series, case discussions during morning report, at case conferences, and during clinic rotations. Resources made available to the residents each year include the AAP membership and pediatric Prep questions. We believe that additional annual standardized lectures covering pediatric rheumatology would reinforce the information that residents receive throughout the year.

The objective of this study was to investigate whether two 1-hour lectures led by residents would lead to improvement in rheumatology knowledge. Using a pre-test post-test design, we evaluated resident knowledge and self-rated competence in pediatric rheumatology. The study design was based on previously published studies on improving resident education and skills using pre-test post-test designs.[8,9]

Methods

Setting and Participants

This study was performed at the Rutgers Robert Wood Johnson Medical School. Participants included the group of 32 pediatrics residents in the residency program.

Intervention

Two lectures (each 1 hour long) were modeled after the comprehensive AAP online course “Pediatric Musculoskeletal Medicine: Arthritis and Rheumatic Diseases” and presented to pediatric residents at Rutgers Robert Wood Johnson Medical School. We discussed with the AAP prior to creating the presentations and were given permission to use the module. Both lectures were reviewed prior to their presentation by our pediatric rheumatologists (Drs. Moorthy and Boneparth) to ensure that the content was accurate. Residents on vacation or leave of absence during the time that the lectures were given were excluded. The first lecture, titled “Systemic Rheumatic Diseases” included a review of Systemic Lupus Erythematosus (SLE), Dermatomyositis (DM), Kawasaki’s Disease (KD), Henoch-Schönlein Purpura (HSP), Takayasu arteritis, Wegner’s granulomatosis, and Polyarteritis nodosa. The second lecture, titled “Juvenile Idiopathic Arthritis” included Systemic JIA, Oligoarticular JIA, Polyarticular JIA, Psoriatic JIA, and Enthesitis Related JIA.

Outcomes Measured

Prior to the first lecture, residents were administered a pre-test. After each lecture was completed, residents completed a post-test. The questions used in the pre- and post-tests were gathered from the AAP Rheumatology Module. Each test had 9 rheumatology-related questions, plus an additional identifier of post-graduate year. The maximum possible score for both pre- and post-tests was 18 points. Questions 1 to 5 and 7 had only 1 correct answer and were worth 1 point each. For questions 6, 8, and 9, residents had to mark multiple items to get the correct answer and receive credit. Question 6 had 3 correct answers and scored 3 points, question 8 had 4 correct answers and scored 4 points, and question 9 had 5 correct answers and was worth 5 points.

In both the pre- and post-tests, the residents were asked to rate their level of competence in recognizing and diagnosing a rheumatologic disease using a Likert scale ranging from 1 to 5 (1 = not at all competent, 5 = very competent). The post-test had 2 additional questions asking how many sessions were attended and if they found the lectures beneficial.

Analysis of Outcomes

SPSS (SPSS Inc., Chicago, IL, USA) was used for statistical analysis. Initial descriptive analyses were performed on all variables. Individual variables were examined for ceiling and floor effects. All variables were examined to ascertain the data distributions and assess for normality. Based on data distribution we selected statistical tests. Change in knowledge was assessed via paired t-tests and change in competence was assessed via Wilcoxon Signed Rank test.

Ethical Approval and Informed Consent

This research study was approved by the Rutgers Health Sciences Institutional Review Board, #Pro2014000150. Written consent was not indicated for the pre and post-tests as there were no identifiers except for post-graduate year of training and the participants’ favorite store.

Results

Out of 32 residents, 28 participated (Tables 1 and 2). Of the 28 participating residents, 10 were post-graduate year (PGY) 1, 10 were PGY2 and 8 were PGY3. Forty-six percent (13/28) attended both lectures, 21% (6/28) attended only the first lecture and 14% (4/28) attended only the second lecture. Ninety-six percent (22/23) of the residents who attended at least 1 lecture found the lectures to be beneficial. A larger proportion of PGY2 residents (90%) attended at least 1 lecture and reported that the lectures were beneficial to them (100%), relative to PGY1 (80% attended and 89% found it beneficial), and PGY3 (75% attended and 83% found it beneficial).

Table 1.

Participant Characteristics.

	All residents	PGY1	PGY2	PGY3
Total number of residents who participated	28	10	10	8
Number of residents who completed pre-test	28/28 (100%)	10/10 (100%)	10/10 (100%)	8/8 (100%)
Number of residents who completed post-test	27/28 (96%)	9/10 (90%)	10/10 (100%)	8/8 (100%)
Number who attended both lectures	13/28 (46%)	5/10 (50%)	4/10 (40%)	4/8 (50%)
Number who attended only first lecture	6/28 (21%)	2/10 (20%)	3/10 (30%)	1/8 (13%)
Number who attended only second lecture	4/28 (14%)	1/10 (20%)	2/10 (20%)	1/8 (13%)
Number who attended at least 1 lecture	23/28 (82%)	8/10 (80%)	9/10 (90%)	6/8 (75%)
Number who did not attend any lectures	4/28 (14%)	1/10 (10%)	1/10 (10%)	2/8 (25%)
Number who reported increased competence (after second lecture)	10/27 (37%)	4/10 (40%)	4/10 (40%)	2/8 (25%)
Number who found lectures beneficial (after second lecture)	22/23 (96%)	8/9 (89%)	9/9 (100%)	5/6 (83%)
Number with improvement in score from pre-test to post-test (1 point or greater improvement in the score from pre-test to post-test)	17/27 (63%)	5/9 (56%)	5/10 (50%)	7/8 (88%)

1 Resident (PGY1) failed to complete post-test so unable to ascertain if that resident attended any lectures.

Abbreviation: PGY, post-graduate year.

Table 2.

Scores of the Pre- and Post-Tests.

All residents (PGY1, 2, and 3)
	Pre-test	Post-test	P value
Number of residents who completed test	28	27
Mean score ± SD	7.5 ± 3.4 (range 0-13)	8.8 ± 3.4 (range 2-15)	.03
Median score	8 (range 0-13)	9 (range 2-15)
Mode score	8	9
Mean reported competence level ± SD	2.5 ± 0.7	2.9 ± 0.6	.002
Number of correctly answered questions (mean ± SD)	2.9 ± 1.0 (range 1-5)	3.6 ± 1.7 (range 1-7)	.04
Most common correctly answered question (mode)	Question #7—24/28 (85.7%)	Question #7—25/28 (89.3%)
Most common incorrectly answered question (mode)	Question #8—28/28 (100%)	Question #9—27/28 (96.4%)
	Question #9—28/28 (100%)
Resident cohorts
Mean PGY 1 score ± SD	7.4 ± 3.5	8.3 ± 3.6	NS
Mean PGY 2 score ± SD	8.3 ± 1.5	8.3 ± 2.0	NS
Mean PGY 3 score ± SD	6.8 ± 4.8	9.9 ± 4.6	.003
Mean PGY1 reported competence level ± SD	2.0 ± 0.7	2.6 ± 0.7	.046
Mean PGY2 reported competence level ± SD	2.9 ± 0.7	3.3 ± 0.4	.046
Mean PGY3 reported competence level ± SD	2.8 ± 0.5	3.0 ± 0.5	NS

Paired sample t-tests were used to compare differences in scores. Wilcoxon Signed Rank Test was used to compare differences in competence levels.

Abbreviations: NS, not statistically significant; PGY, post-graduate year; SD, standard deviation.

First Lecture

Sixty-eight percent (19/28) of residents attended the first lecture. Seven residents were PGY1, 7 were PGY2, and 5 were PGY3. The mean pre-test score of all the residents prior to the first lecture was 7.5 ± 3.4 (range 0-13). The mean pre-test scores by year were: 7.4 ± 3.5 for PGY1, 8.3 ± 1.5 for PGY2, and 6.9 ± 4.8 for PGY3. Pre-test self-reported competence level for all of the residents was reported with a mean of 2.5 ± 0.7, with the means of PGY1, PGY2, and PGY3 individual cohorts reported as 2.0 ± 0.7, 2.9 ± 0.7, and 2.8 ± 0.5 respectively.

Second Lecture

Sixty-one percent (17/28) of residents attended the second lecture. Six residents were PGY1, 6 were PGY2, and 5 were PGY3. The mean post-test score completed after lecture 2 of all the residents was 8.8 ± 3.4 (range 2-15). The mean post-test scores by year were: 8.3 ± 3.6 for PGY1, 8.3 ± 2.0 for PGY2, and 9.8 ± 4.6 for PGY3. Post-test competence ratings for all of the residents was reported with a mean of 2.9 ± 0.6. The individual post-test self-reported competence means of cohorts PGY1, PGY2, and PGY3 were reported as 2.6 ± 0.7, 3.3 ± 0.4, and 3.0 ± 0.5 respectively.

Pre- to Post-Test Improvement

Overall, there was improvement in scores from pre-test to post-test (paired sample t-test P = .03). Sixty-three percent (17/27) of residents had an improvement in score from pre-test to post-test. One resident (PGY1) did not complete a post-test and was not included in the post-test results. PGY3 residents had the most significant improvement in score from pre- to post-tests (6.9 ± 4.8 pre-test mean to 9.9 ± 4.6 post-test mean, P = .003). Overall, the pre-test mean of questions answered correctly was 2.9 ± 1.0 (range of 1-5) compared to the post-test mean of 3.6 ± 1.7 (range of 1-7) (P = .04).

Most Frequently Answered Correct and Incorrect Questions

In the pre-test, all the residents answered the questions 8 and 9 incorrectly. Question 8 related to DM and question 9 related to KD. Question 7, related to SLE, and was the most correctly (85.7%) answered question followed by question 4 (75%), which related to JIA. Similarly, in the post-test, most residents answered question 9 incorrectly (96.4%). Most residents answered the post-test questions 7 (89.3% and 4 (71.4%) correctly.

Self-Reported Competence Level

Self-reported competence data is summarized in Table 3. In the pre-test 7.1% (2/28) of residents marked that they were “not at all competent” (1) when asked about competence in pediatric rheumatology and none of the residents marked “very competent” (5). Thirty-nine percent (11/28) residents reported they were “less competent” (2), 42.9% (12/28) reported they were “competent” (3), 7.1% (2/28) reported they were between “competent” and “more competent” (3.5), and 3.6% (1/28) reported they were “more competent” (4). On the post-test, none of the residents reported their competence level as “not at all competent” (1) or “very competent” (5). Twenty-one percent (6/27) reported they were “less competent” (2), 57.1% (16/27) reported they were “competent” (3), 3.6% (1/27) reported they were between “competent” and “more competent” (3.5), and 14.3% (4/27) reported they were “more competent” (4). Thirty-seven percent of residents reported increased competence from pre-test to post-test. Using the Wilcoxon Signed Rank Test for analysis, this improvement in pre- to post-test competence level was statistically significant (P = .002). Forty percent (4/10) of the PGY1 and PGY2 cohorts reported an increase in competence (P = .046 for both PGY 1 and 2 cohorts) compared to 25% (2/8) in the PGY3 cohort (P = NS).

Table 3.

Self-Reported Competence Percentages (All Residents).

Competence level	1 “Not at all competent”	2 “Less competent”	3 “Competent”	3.5*	4 “More competent”	5 “Very competent”
Pre-test	2/28 (7.1%)	11/28 (39.3%)	12/28 (42.9%)	2/28 (7.1%)	1/28 (3.6%)	0/28 (0%)
Post-test	0/28 (0%)	6/27 (21.4%)	16/27 (57.1%)	1/27 (3.6%)	4/27 (14.3%)	0/28 (0%)

3.5 was not an offered choice and was written in by 2 residents on the pre-test, and 1 resident on the post-test.

Non-Attendance of Lectures

There were 4 residents who did not attend any of the lectures, but did complete the pre- and post-tests. One of these residents had no change in score, 2 residents had improved scores, and 1 resident had a decrease in score from pre- to post-tests. Three of the residents did not report any change in competence, while 1 resident reported increased competence.

Discussion

Most residents (82%) attended at least 1 lecture. From pre- to post-tests, there was a significant improvement in test score and an increase in reported competence levels. Most residents (96%) reported that the lectures were beneficial. Review of the results and pre- and post-test questions suggests that the residents need to review the topics on DM and KD. Also, these questions had multiple correct answer choices and required that the residents choose all of the correct answer choices for the question to be scored as correct. The residents performed well on the topics of SLE and JIA answering these questions correctly most often on both the pre- and post-tests.

The PGY3 cohort of residents had the highest number of residents to improve from pre- to post-tests (88%) as well as the greatest improvement in score (average 3 point increase). This could be due to the fact that they have received the most training of all the other residents in the pediatric class. Interestingly 25% of PGY3 residents reported an increase in competence after receiving the lectures compared to 40% reported increased competence from the PGY1 and PGY2 residents.

Our study had a number of limitations, 1 being the small sample size. Due to the small sample size and study design we were unable to have a control group. Without a control group, we may have lost some internal validity and cannot account for confounding variables and experimental noise. In addition, the results of our study may reflect some bias, as the post-test was given right after the second lecture, which may have led to higher scores due to immediate recall. Additionally, we did not address in our pre- and post-test if residents had previously taken a rheumatology elective as a medical student or during residency, which could have confounded performance on the pre- and post-tests.

Although our study did have a small sample size, other studies have used similar sample size as well as pre- and post-test study design with educational intervention and reported success. One study utilized a musculoskeletal exam video-based curriculum to assess improvement in rheumatology knowledge in medical students and pediatric residents. Their subject group was 11 to 28 participants for each session, and included 8 sessions with a pre- and post-test given at each session. The primary outcome was improvement in score from pre- to post-test. The secondary outcome was learner satisfaction. The results of the study showed statistically significant improvement in pre- to post-test scores after each lecture. The participants also noted overall satisfaction with the lectures.

Similarly, a recent study from Brazil assessed the impact of a virtual learning environment in pediatric rheumatology on pediatric residents’ knowledge using a pre-test and post-test. The researchers found that those who completed 6 virtual pediatric rheumatology modules had a higher improvement in test scores compared to residents who did not attend the modules. Residents also rated the virtual learning environment favorably.

Another study to use the pre- and post-test design to measure the effectiveness and usefulness of a specific teaching modality was published by Wayne et al which used simulator training to improve ACLS knowledge in internal medicine residents. The sample size in this study was larger, however the study did lack a control group. There were 41 internal medicine residents included in the study. Pre- and post-tests were completed before and after four 2-hour simulator training sessions. The results of the study showed a significant improvement in scores after simulator training and all residents met or exceeded the mastery competency standard.

Drs. Megan Curran and David Sherry have been involved in developing and testing the objective structured clinical exams (OSCE) for pediatric rheumatology fellows to evaluate their communication and professionalism skills. A pre-test and post-test design was used to evaluate the fellows’ change in skills. The study results showed that there was improvement in post-test scores as well as professionalism after training using the OSCE.

Despite the limitations of this study, the results indicate that the lectures in pediatric rheumatology were found to be beneficial to pediatrics residents and there was an improvement of knowledge after each lecture. Pediatric residency programs should evaluate the quantity and quality of pediatric rheumatology education, and should consider supplementing the curriculum with additional focused lectures. Further research is required on pediatric rheumatology education, knowledge retention, and the ideal design of educational interventions.

Conclusions

Musculoskeletal complaints are common amongst children, rheumatic diseases are complex and difficult to diagnose, and pediatric rheumatology is still a relatively small field. Therefore, all pediatric residents must acquire an understanding of common diseases in this specialty in order to provide general pediatric care. The present study found a significant increase in resident knowledge and self-rated competency in pediatric rheumatology after attending lectures focused on this specialty. Pediatrics residents also found these lectures to be beneficial.