Teaching Critical Appraisal Through the Lens of Study Design in Journal Club.

Training in endocrinology presents a major challenge: How to both develop a fellow's medical knowledge and teach that fellow to independently increase that fund of knowledge as a future endocrinologist. Journal clubs hold a traditional role in teaching the most recent modern advances in medicine but frequently struggle to provide trainees with the skills needed to assess the quality of current literature. Using journal clubs to instill the skills needed for critical appraisal overcomes this difficulty and substantially enhances the value of journal clubs to trainees. Here, our journal club is reviewed, in which we teach critical appraisal by focusing on understanding study design as the key to assessing an article's strengths and weaknesses. This journal club starts the year by reviewing study designs in a structured sequence, with sessions built to foster group participation and involvement of both fellows and faculty. Later, fellows put this learning into practice as they independently assess articles alongside their faculty. Generalizing this practice allows for implementation in other programs, including even those without faculty with a background in research methodology.

A Challenge

Helping an endocrinology fellow grow his or her fund of medical knowledge poses a multidimensional challenge. Not only does a young physician need to learn the medical literature as it is at the time of training, but that trainee must also recognize that much of what is learned will be outdated within a few years. So, a fellow must learn the current state of endocrinology and also how to stay up to date with the endocrine literature in order to maintain the requisite knowledge base. As it is a skill expected of all physicians, especially those in a cognitive specialty like endocrinology, endocrinology fellowship programs must accept the daunting challenge of imparting this ability. However, with the many demands on fellow and faculty time (seeing patients, writing notes, preparing for presentations, working on research projects, etc.), teaching this skill can be neglected.

Journal clubs represent one of the historical approaches to teaching trainees of all levels and in all disciplines, dating back at least to William Osler nearly 150 years ago [1]. Journal clubs stand out in medical training for a host of unique features, including their use of peer-to-peer approaches to teaching and creating a relatively informal learning environment [1]. They help meet requirements for continuing medical education, keep attendees aware of the most recent advances in medicine, and encourage social contact within programs [2]. While still usually held in a manner consistent with this longstanding approach, journal clubs have evolved far beyond their traditional in-person, small-group setting. Today, many specialties host journal clubs over social media [1], with some offering recorded versions as podcasts, such as the Endocrine Society’s Endocrine Feedback Loop.

Despite being a time-honored feature of training programs, journal clubs come with many challenges and may not meet trainees’ needs. Fellows usually fill the role of presenter in journal club sessions but vary in the skills needed to assess the medical literature. The sessions themselves may suffer from poor organization, low attendance, and minimal interaction [2]. Over the years, investigators and expert educators have identified many techniques to improve journal clubs, including incentivizing attendance, carefully choosing clinically relevant articles, and tasking a leader with running the sessions [3]. Though necessary for a successful journal club, these features alone are insufficient to accomplish the goal of teaching trainees how to independently appraise—and so truly stay up to date with—the medical literature.

A Solution

For journal clubs to serve our endocrine trainees well, their purpose must be to teach a critical-appraisal approach that seeks to put findings into a clinical context [3]. Critical appraisal couples an inquisitive and skeptical approach to interpretation of scientific literature with a basic understanding of scientific and statistical methods [4]. Learning critical appraisal empowers future endocrinologists to independently assess the endocrine literature, deal with “information overload,” and hopefully improve the patient care they provide [4]. When critical appraisal is not emphasized, journal clubs become merely a method of reviewing current literature. Critically appraising an article keeps a physician from being a passive recipient of someone else’s conclusions, which may be a biased interpretation of the data. Instead, such an approach helps endocrinologists deal with our field’s rapidly expanding literature, determining what studies warrant a change in clinical practice. Teaching this skill ensures that trainees not only learn but also learn how to learn. The medical literature generally supports that journal clubs can increase trainees’ skills in critical appraisal [3], though we lack high-quality randomized controlled trials proving such a benefit [5].

An Example

Once teaching critical appraisal is seen as the ultimate goal of a fellowship’s journal club, a comprehensive curriculum can be created and implemented. These ideas can seem esoteric and difficult to practically apply to one’s own journal club, particularly if one lacks formal education or practical experience with these approaches. Below is an example of how this author implemented such research into practice, though it is only one of many approaches that could be used.

We implemented a journal club focused on teaching critical appraisal through the lens of understanding study design as its overall purpose, with a secondary purpose of helping fellows and faculty stay current with the medical literature. The journal club runs on an annual cycle with the academic calendar, meeting monthly for an hour. The first month features a didactic lecture introducing fellows to key study designs used in endocrinology, highlighting the “pros and cons” of each approach, in addition to a basic verbal and graphical depiction of each design (see ref. [6] for examples). The discussion of the “pros” of each study designs helps the fellows understand why investigators might chose a certain approach and what types of questions they address. The “cons” component unpacks the inherent weaknesses of each design, including the biases (treating or assessing subjects differently based on which study group they are in) and confounders (other exposures related to the studied exposures actually causing the outcome) to which each one is susceptible. Linking biases to the study design that creates them fosters an understanding of why the biases exist and so prevents them from simply being a confusing list that is easily forgotten. By way of example, knowing that a prospective cohort study defines groups based on the presence or absence of an exposure explains why such a design works well for rare exposures and not rare outcomes. Understanding that such a study subsequently follows this cohort over time reveals why it can both provide an incidence (proportion of new cases over time) and avoid a recall bias (because investigators collected data on exposure before the outcome occurred). Clinical trials, on the other hand, prevent most biases by blinding subjects, study personnel, and investigators, and randomizing helps to balance confounders between groups.

The subsequent 5 months of the journal club rotate through each of these study designs: a case-control study, a cross-sectional study, prospective and retrospective cohort studies, and a clinical trial. Each month, the assigned fellow and the leader of the journal club identify an article from the endocrine literature that utilizes that month’s assigned study design and was published within the previous year. Fellows use a structured review instrument (SRI) to prepare each session. An SRI provides a checklist of items a presenter needs to assess in an article and include in a presentation (see ref. [6] for SRIs created by our program). Educational researchers in fields as diverse as orthopedic surgery and clinical pharmacology have demonstrated the effectiveness of SRIs in improving the quality of presentations at journal clubs [7, 8]. Unfortunately, many SRIs suffer from a focus on statistical methodology, an area usually outside the expertise of fellows (and often their faculty) and with limited benefit to understanding the clinical applicability and impact of a study. Instead, fellows use SRIs based upon the relevant study design, an approach far more accessible to trainees and even to programs without a journal club leader well versed in research methodology. These SRIs start by reviewing the key components of the study design, highlighting the biases the fellow needs to evaluate for. The SRI then leads them through the questions they need to ask and answer of each article they prepare to present, some standard and some unique to the study design of the article (eg, Is the outcome defined in a clinically meaningful way? Does the studied population resemble the population you care for? Does the exposure clearly predate the outcome?). Each SRI ends by asking for their assessment of whether this study should change their clinical practice.

The fellow and journal club leader then meet prior to the session to allow for clarification of confusing aspects of the study and for identification of key learning points. The leader coaches the fellow on how to present the study so that critical appraisal happens in real time by the participants, focusing the presentation on the article itself and not a literature review of the topic. Inclusion/exclusion criteria, tables, and figures are reviewed to determine what questions will be asked of the audience to help them analyze the article themselves, before hearing the presenter’s assessment. The goal is for the attendees to summarize the strengths and weakness of the methods, analyze the results, and critique the authors’ conclusions before the presenter gives his or her analysis. The article is circulated in advance, though prereading is not requested, in order to remove any deterrent to attendance.

Each session starts with the leader providing a basic review of the study design used. The leader then briefly presents an additional topic relevant to the paper being reviewed (eg, P values vs 95% CIs, sensitivity analysis, intention-to-treat analysis). The topic presented primes the attendees to look for specific strengths and weakness germane to the article being reviewed. For example, reviewing the principles of evaluating observational studies (a.k.a. the Bradford Hill criteria) prepares attendees to assess a study for its identification of a dose–response relationship, explanation of biologic plausibility, confirmation of a temporal relationship between the exposure and outcome, etc. The presentation is then turned over to the fellow. In the introduction, the fellow lists only key points made by the authors. The methods section focuses on the study design, inclusion/exclusion criteria, and definitions of exposure and outcome. Statistical methods are only briefly summarized, avoiding listing techniques understood by neither the presenter nor the audience. The presenter asks attendees if the inclusion/exclusion criteria are too narrow, what biases the methodology has and has not accounted for, and what confounders have not been considered. The review of the results focuses on tabular and graphical data. The attendees provide their own assessment of the results, including the clinical meaningfulness and any unexpected/inconsistent findings. Finally, the discussion starts with a listing of the authors’ conclusions but segues into the group analyzing whether the results support those conclusions, based on concerns identified in the previous 2 sections. Attendees are asked how previously identified biases, unaddressed confounders, inclusion/exclusion criteria, definitions of outcomes, etc., could affect the results. The session ends with a question posed first to the group and then the presenter: Will this change your clinical practice? Fellows and junior faculty provide their thoughts first, followed by senior faculty and the presenting fellow.

The second half of the academic year pivots to additionally focus on keeping up to date with the medical literature, with each session featuring 2 presentations (1 by a fellow and 1 by a faculty) of recently published endocrine articles in which other study designs are explored or previously reviewed ones are revisited. Fellows and faculty each prepare their sessions independently (unless input is requested of the leader). The leader allows the presenters to run the entire session, participating along with the group in the real-time analysis of the paper. This format allows the fellows and faculty to put learned skills into practice while still receiving guidance. Additionally, it simulates a “real-world” experience of them independently evaluating a study and presenting their assessment to peers. Reviewing only the current literature motivates faculty to join, helping everyone in the program stay up to date with the medical literature.

Application and Conclusion

By teaching critical appraisal through a better understanding of study design, endocrinology fellowship programs can better equip their trainees for their future careers. Using journal clubs to teach critical appraisal transforms this traditional and sometimes tedious teaching activity into a dynamic conference that meets the needs of trainees. Though such an approach presents challenges, they can be overcome with planning and using available resources. Fellowship program directors without a background in research methodology may need to request assistance, such as asking an endocrinology colleague to lead the journal club or establishing a relationship with faculty elsewhere in the institution who can advise fellows in session preparation. Alternatively, a program director can learn to present the basic concepts of study design and ask fellows to use SRIs such as those created by our program (see ref. [6] to access) to help in analyzing an article independently [3]. The benefits make such efforts more than worthwhile, teaching fellows (and often faculty) valuable skills that enhance their reading of the endocrine literature and so improve the care they provide to their patients now and in the future as independent endocrinologists.