Tips for teachers of evidence-based medicine: clinical prediction rules (CPRs) and estimating pretest probability.

BACKGROUND: Clinical prediction rules (CPR) are tools that clinicians can use to predict the most likely diagnosis, prognosis, or response to treatment in a patient based on individual characteristics. CPRs attempt to standardize, simplify, and increase the accuracy of clinicians' diagnostic and prognostic assessments. The teaching tips series is designed to give teachers advice and materials they can use to attain specific educational objectives. EDUCATIONAL
OBJECTIVES: In this article, we present 3 teaching tips aimed at helping clinical learners use clinical prediction rules and to more accurately assess pretest probability in every day practice. The first tip is designed to demonstrate variability in physician estimation of pretest probability. The second tip demonstrates how the estimate of pretest probability influences the interpretation of diagnostic tests and patient management. The third tip exposes learners to various examples and different types of Clinical Prediction Rules (CPR) and how to apply them in practice. PILOT TESTING: We field tested all 3 tips with 16 learners, a mix of interns and senior residents. Teacher preparatory time was approximately 2 hours. The field test utilized a board and a data projector; 3 handouts were prepared. The tips were felt to be clear and the educational objectives reached. Potential teaching pitfalls were identified.
CONCLUSION: Teaching with these tips will help physicians appreciate the importance of applying evidence to their every day decisions. In 2 or 3 short teaching sessions, clinicians can also become familiar with the use of CPRs in applying evidence consistently in everyday practice.

INTRODUCTION

Accurately determining the probability of disease from history and physical exam is extremely important in the day-to-day management of patients and impacts greatly on how clinicians interpret subsequent diagnostic tests. A clinical prediction rule (CPR) is a clinical tool that quantifies the individual contributions that various components of the history, physical examination, and basic laboratory results make toward the diagnosis, prognosis, or likely response to treatment in a patient.1 CPRs attempt to standardize, simplify, and increase the accuracy of clinicians’ diagnostic and prognostic assessments.

In this article, we present tips aimed at helping clinical learners use clinical prediction rules and to more accurately assess pretest probability in every day practice.

As with other articles in this series, educators experienced in teaching evidence-based medicine developed these tips and use them extensively as brief and efficient ways of clarifying statistical concepts to learners in the course of clinical problem solving or critical appraisal of studies and reviews. They are designed for use by teachers skilled in EBM in time-pressed settings where key learning points must be interpolated into clinically oriented discussions. They assume a high degree of interactivity between learners and teachers, characteristic of small group settings.

An article from the Canadian Medical Association Journal described the development of this series and pertinent background information.2 For each of the tips, we provide guidance on timing of use, a teaching script, a “bottom line,” and a summary card. We also identify the appropriate target audience and provide time estimates for performing the exercise for each tip. These tips conform to short teaching segments that the authors characteristically use as adjuncts to clinically focused exercises. We do not attempt to deal comprehensively with the topics pertaining to these tips, either in practice or in this manuscript. For example, a discussion of more general issues, such as how prediction rules are derived and validated, is beyond the scope of this article.1

The first teaching tip demonstrates the wide variability among individual practitioner’s estimates of the probability of disease. The next tip builds on the first, and illustrates that this wide variation in pretest probability can also lead to a wide variation in clinical care. The last teaching tip demonstrates how using objective criteria in a Clinical Prediction Rule can narrow this variability among practitioner’s individual estimates and clinical care. The tips follow a sequence that builds on prior knowledge. However, each tip can stand alone depending on the circumstances and the learners’ knowledge base. In selecting where to begin or which tip to use, the teacher needs to assess learners’ current level of understanding of these concepts.

TEACHING TIP 1: DEMONSTRATING THE VARIABILITY IN PHYSICIAN ESTIMATION OF PRETEST PROBABILITY

When to Use This Tip?

Physicians are often unaware of how their estimations of probability of disease may vary from their colleagues’ and that there is tremendous variability in how each provider weighs certain clinical and laboratory findings. Why 1 physician considers 1 aspect of the history to be important and another physician does not may be based on individual experience, on the literature, or both. Some physicians claim clinical “intuition” guides their estimations of risk. The objective of this exercise is to help clinicians realize that other good clinicians may have dramatically different estimations of probability of disease or risk from their own. This is best done in a setting with approximately 8–10 learners, but can be done in smaller groups or even large group lecture settings. This tip is appropriate for learners with a beginning level of prior EBM knowledge but can be done and is interesting to perform with all levels of learners. This exercise takes about 20 minutes.

The learning objective is:

Understand that physicians vary in how they interpret the importance of different aspects of the history, physical exam and basic laboratory results.

Demonstrate the variability in physician estimation of pretest probability.

This tip addresses a frequently encountered stumbling block: learner reluctance to commit to an independent estimation of probability of disease. Asking learners to write their numbers down privately allows them to risk disagreement with their peers.

The Script

To demonstrate variability in assessments of pretest probability presents a simple case to the group. Either a scripted case as presented or a real patient being seen by the learners can be used.

Example scenario:

A 45-year-old female on estrogen replacement therapy who has no significant past medical history presents to the emergency room at 4 am with the sudden onset of chest pain and shortness of breath. The chest pain lasted seconds, but her feeling of shortness of breath has persisted for several hours. Her physical exam is significant for an O2 saturation of 95%, pulse rate of 98 beats per minute, and an otherwise normal physical exam. Her chest x-ray is clear.

After presenting the case to the group ask them to write down their estimated probability that the patient has experienced a pulmonary embolus based on the information provided in the script. Try to avoid providing any further data with regards to the case if the learners request. For example, saying that the patient has empirically been started on heparin may bias the learners’ pretest probability. Make sure before proceeding that each learner privately writes down their own estimation of the probability of disease on a piece of paper. It is very important to have learners commit to a number and arrive at their conclusion independently.

Once they finish this first task, have them each tell you their assessment of the risk of pulmonary embolism. Alternatively, the teacher can collect the numbers in folded pieces of paper to preserve independence. Write their estimates on the board as illustrated in Figure 1, listing the numbers along a vertical line from 0 to 100%. Group the estimates to the nearest 10% increment. Once everyone has given their estimates, highlight the extremes and ask the learners to describe what aspects of the case most influenced their estimation and why. For example, some learners may focus on the patient’s use of hormone replacement as increasing their pretest probability, whereas others may focus on the patient’s atypical presentation of a pulmonary embolism, thus reducing their pretest probability. Writing learner responses on the board may facilitate this discussion.

Figure 1

Learners are given a scenario involving a patient with suspicion of pulmonary embolism. They independently record their estimate of probability that the patient actually has PE. The vertical lines to the right of the scale correspond to estimates of individual learners, grouped by nearest decile.

Other clinical examples where variation in pretest probability is frequently encountered include the management of patients with chest pain or suspected deep vein thrombosis. Focus on an example or case that is the most relevant to the learner.

Bottom Line

There is considerable variability in the estimation of pretest probability between clinicians for everyday clinical problems.

See Appendix 1 for the summary card on this tip.

TIP 2: ESTIMATES OF PRETEST PROBABILITY INFLUENCE OUR INTERPRETATION OF DIAGNOSTIC TESTS AND HOW WE MANAGE OUR PATIENTS.

When to Use This Tip

Learners need to make a connection between estimating a patient’s pretest probability and making an action plan: discharge, further testing, or initiating treatment. The content of tip 1 is prerequisite to tip 2, and when tip 1 has just been taught, tip 2 naturally flows from it. The tip is suitable for an audience of 8–10. Learners without prior exposure to EBM may have difficulty with action thresholds. The tip is therefore best utilized in learners with some EBM experience. Motivated beginners and intermediate learners would be appropriate. No more than 15–20 minutes are needed to complete this exercise.

The learning objective is:

To demonstrate how the variability in estimations of pretest probability directly impact on patient management.

If in the specific case of a patient with possible pulmonary embolism there is a spread between low, moderate, and high pretest probability, ask the learners to again write down what they would do next if given the result of a low probability ventilation/perfusion (V/Q) scan. For example they can order another test, initiate therapy, or discharge the patient home. As noted in the presentation of the previous tip, it is critical that each learner commits to an action and physically writes down their decision before this point is open to discussion. Publicly asking each learner in front of the rest of the group will lead to a similar group of answers and negate the value of these teaching tips.

Learners who perceive the patient to have a low pretest probability will typically discharge the patient home with follow-up, whereas those with intermediate or high pretest probability will typically opt for further testing such as d-dimer, lower extremity dopplers, spiral CT, or angiogram. Using a figure such as Figure 1 from the previous exercise, write these potential actions down on the board and categorize them in 2 groups: further diagnostic tests or no further work up. Figure 2 demonstrates how to display these results on the board to highlight the interplay of pretest probability and the result of a diagnostic test.

Figure 2

Learners who have provided estimates of probability of pulmonary embolism in a patient based on a hypothetical scenario are now led to consider the impact of possible results of selected tests on their choice of actions. How high or low the horizontal action threshold is drawn will influence clinical action.

A stumbling block may occur if some learners with the same pretest probability choose different clinical actions. For example, for 2 learners with moderate pretest probability given a low probability VQ scan, 1 may feel comfortable discharging the patient, but another may want to do 1 more test “just to be sure”.

This stumbling block presents an excellent opportunity to discuss the influences of one’s action threshold where each learner balances his or her perception of the risk of failing to diagnose a pulmonary embolism with the risk of treatment with anticoagulants, or the risk of further diagnostic tests such as an angiogram. Ask the learners how confident they want to be regarding the exclusion of a pulmonary embolus before discharging the patient from the emergency department, and try to break it down to low, medium, or high threshold. Draw a horizontal line through the corresponding percentages and name this their diagnostic or action threshold as in Figure 2. The learner who wants to get more tests despite a low pretest probability will have a very low diagnostic threshold.

Variability in the estimation of pretest probability can lead to different clinical actions.

See Appendix 1 for the summary card for this tip.

TEACHING TIP 3: EXPOSING LEARNERS TO VARIOUS CLINICAL PREDICTION RULES.

This tip exposes learners to various examples and different types of Clinical Prediction Rules (CPR) and how to apply them in practice. This tip is best used with a similar audience of 8–10 learners, as in the prior 2 tips. The tip works best with learners who have an intermediate or better skill level and have a good grasp of the concepts of pretest probability covered in the first 2 tips. In addition learners familiar with likelihood ratios (LR) and those who have been exposed to the teaching tip on LR by Richardson et al.3 will be more likely to benefit from this tip. When these conditions are met and time permits, all 3 tips work well together.

Stumbling blocks Learners frequently misinterpret a component of a CPR; they may not score it properly, or they may apply it incorrectly. For example, the Ottawa ankle rule4 was developed to help clinicians decide when an x-ray is not indicated for patients presenting to an emergency department with an ankle injury. However, the rule does not predict who among patients not so excluded actually has a fracture. Learners frequently confuse this point and assume the rule can help stratify patients according to their risk of fracture.

Similarly, the rule defined by the Well’s criteria in Figure 3,5, which will be used in this next teaching tip, has its own pitfalls. The Well’s criteria help stratify patients into low, medium, or high risk for pulmonary embolus. Learners (and even seasoned clinicians) may be confused over what constitutes a risk factor for thromboembolism as defined by the rule. For example, a history of uterine cancer that has been surgically resected in the distant past (as opposed to a current history of cancer) would not be considered a risk factor as defined in the original study of Wells et al.6 Familiarity with the correct use of the CPR is therefore critical to this tip, and may require the teacher to review of the original articles.

Figure 3

The Wells’ criteria are a clinical prediction rule to assist in predicting pulmonary embolism. In the case example in tip 3 there are a total of 0 points making the patient low probability with a pretest probability of approximately 2.5%.

As with the other 2 scripts, this exercise is ideally performed with 8–10 learners. Expect it to take 20 to 30 minutes.

The learning objectives is:

To understand how to apply a clinical prediction rule in everyday practice.

To understand how clinical prediction rules can decrease variability among physicians in clinical practice.

Preparing to Teach

Some preparation is required before instituting this tip. Familiarize yourself with the CPR you want to use, and review the original article if necessary to review what the CPR is meant to predict, in what setting it has been validated and other important characteristics (TEXT BOX).1,7,8 Make sure that the predictive variables are clear. For example, what constitutes history of malignancy, immobilization and what can be called an alternative diagnosis? The basic outline of the Wells CPR can be written (or copied from the original article) as in Figure 3 and photocopied and distributed to each learner during the exercise.

Present a case history such as used in connection with tip 1. If the learners have not been exposed to either of the previous tips, ask them to write down their assessment of pretest probability without the assistance of the CPR. Distribute the handout and ask the learners to complete the CPR with the data from the case and have them privately write down a pretest probability based on this CPR. Once everyone has committed to an answer go over the CPR stepwise.

In this example, clinicians may assess the patient in question as having atypical symptoms of pulmonary embolism, with heart rate less than 100 and other explanations such as muscular skeletal pain or anxiety at least as likely as a diagnosis of pulmonary embolism. She has no further risk factors as assessed by Wells’ criteria. This clinical information adds up to 0 point using this CPR, yielding a low pretest probability. After each learner commits to pretest probability based on their Wells score, write their answers on the board as in Figure 1, grouping answers as low, medium, or high probability. If time permits have the learners commit to a clinical action given a low probability V/Q scan as in tip 2.

Once again, it is important to have the learners each commit to a score or action and write it down to avoid group consensus. Discuss any differences of interpretation of the CPR among the group, including differences in individual components, and how they each may have reached a different score. In applying the CPR to this case, some learners may argue that the patient’s symptoms are typical of pulmonary embolism, and score the patient higher on the chest pain criteria; other learners may disagree about valid alternative diagnoses. Disagreement between learners regarding any rule element may or may not impact the patient’s final pretest probability, or the recommended clinical action. For example, a clinician who feels the patient’s symptoms are classic for pulmonary embolism will reach a CPR score of 3, or a moderate pretest probability, but the learner’s action threshold for further testing may also be different, such that subsequent testing demonstrating a low probability V/Q scan may lead to the same clinical action, namely a decision to not pursue further testing for pulmonary embolism. In most cases, learners will appreciate that despite some disagreements, consistent use of a CPR can decrease variability among clinicians.

Countless other clinical prediction models exist, which may better fit your clinical practice and can be taught in a similar fashion. Table 1 lists a small sampling of such CPRs with their references.

Table 1

Examples of Various CPRs

CPR	Criteria	Comments
Ottawa Ankle Rule4	Ankle radiographic series required only if there is pain in the malleolar zone and one or more of the following:	– Prospectively validated in multiple settings
		– Reliably ‘rules out’ fracture (sensitivity) but cannot reliably ‘rule in’ (specificity)
	– Bone tenderness at posterior edge (distal 6 cm) or tip of lateral malleolus·Bone tenderness at posterior edge (distal 6 cm) or tip of medial malleolus
	– Inability to bear weight both immediately after the injury and in the emergency department
Alcohol screening 10	– Have you ever felt you should cut down on your drinking?	– Use in screening, not in known alcoholics
	– Have people annoyed you by criticizing your drinking?	– Rule less accurate when used immediately after direct questioning regarding alcohol use. Must ask the questions in a nonjudgmental way.
	– Have you ever felt guilty or bad about drinking?
	– Have you ever had a drink first thing in the morning to steady your nerves or to get rid of a hangover?
Clinical Evaluation for Predicting DVT11	– Active Cancer (1 point)	Suspected DVTs in Emergency setting stratified into High, Medium, and Low risk based on the sum of the point system:
	– Paralysis (1 point)	– >3 high probability
	– Recent immobilization (1 point)	– 1–2 moderate probability
	– Local tenderness over the Deep Venous system (1 point)	– 0 low probability
	– Entire Leg Swollen (1 point)
	– Calf circumference> 3 cm than other leg (1 point)
	– Pitting edema confined to symptomatic leg (1 point)
	– Collateral veins (1 point)
	– Alternative diagnosis as least as likely (−2 points)

Use of a CPR narrows variability between clinicians in estimating pretest probabilities and in decision making regarding testing and treatment

See Appendix 1 for the summary card for this tip.

Report on Field Testing

We use field testing of the tips for the purpose of verifying the clarity and practicality of the descriptions. Field testing also frequently generates examples of the kinds of variations in approach that occur when an experienced teacher of evidence-based medicine adapts and modifies the approaches to fit their own style, context, and learner level. One of the contributors, (S.K.), conducted a field test of all 3 tips at her home institution with 16 learners, a mix of interns and senior residents. Overall, these tips were rated highly by the both learners and by S.K. The objectives were felt to be clear and the learners were able to articulate the major teaching points from the exercises. Preparatory time was approximately 2 hours. The field test utilized a board and a data projector; 3 handouts were prepared.

Trainees from prior test drives have consistently voiced the need to have concepts being taught grounded in an article or specific case example to best appreciate the relevance of the content. Thus, S.K. opted to include a case example of a recent patient on the wards with possible pulmonary embolism. She prepared a handout with the case written out and provided space for the learners to write down their pretest probability to assist in tips 1 and 2. For tip 3, a handout with the Wells article was prepared along with a handout of an article reviewing the rational clinical exam for pulmonary embolism.9

S.K. rated her learners as naïve to the concept of Clinical Prediction Rules but fairly comfortable with the concepts of EBM, therefore S.K. suggests that these teaching tips may be more difficult to use in learners with little prior EBM exposure. S.K. highlighted the importance for individual learners to write down their pretest probability and clinical actions privately, e.g., on an individual file card, for these tips to work as intended. The learners had difficulty applying the rule directly from the original Wells article as they had not reviewed the article previously, suggesting that the teacher should be prepared to walk the learners through the CPR, or prepare a simplified handout for the criteria as in Figure 3. Advanced learners in the group also questioned the omission of D-dimers from the Wells criteria used in tip 3, which is why S.K. felt it would be useful to add the article referenced above where the role of D-dimers is described.

The strength of the tips was the “aha” moment at the end of the last tip when learners could appreciate how their group estimates moved from highly variable in tip 1 to very precise in tip 3. Learners were impressed with the difference and appreciated the power of CPR’s to decrease variability.

CONCLUSION

Clinical decisions made during everyday clinical practice are frequently based on physician estimates of pretest probability or of prognosis. These estimates are, in turn, often based on shaky grounds or what is better known as “clinical intuition”. Clinical Intuition however is not always based on the evidence and may vary greatly from 1 physician to the next. These teaching tips highlight this variability in pretest probability and demonstrate how it impacts on clinical decision making. Teaching with these tips will help physicians appreciate the importance of applying evidence to their every day decisions. In 2 or 3 short teaching sessions, clinicians can also become familiar with the use of CPRs in applying evidence consistently in everyday practice.

Methodological standards for CPRs: when reviewing the prediction rule prior to your teaching session use these points as a guide7,8,10

1. Outcome

○ Should be clearly defined and clinically important. If a surrogate outcome is used, it must have a definite relationship with a clinically important outcome.

○ Blind Assessment—The presence or absence of an outcome should ideally be determined without knowledge of the status of the predictor.

2. Predictive Variables

○ Clear, clinically sensible, and reproducible definition of the variables.

○ List of all variables considered, but not included in the rule.

○ Blind Assessment—assessment of the predictors without knowledge of the outcome.

3. Patient Population

○ Patient characteristics that are likely to affect the performance of the rule should be described.

4. Description of Study Site

○ Office, clinic, ER, hospital

5. Prospective Validation: Level of Evidence

○ Prospectively validate the rule in a group of patients different from the group in which it was derived.

6. Effects of Clinical Use Prospectively Measured (Impact Analysis)

○ Are physicians actually willing to use the rule and does it affect clinical outcomes.

7. Mathematical Techniques Described

○ multivariate analysis

8. Describing the Results of a Clinical Prediction Rule

○ sensitivity, specificity, likelihood ratio, positive and negative predictive value

9. Reproducibility

○ Interobserver reliability of the clinical predictors

10. Sensibility

○ Does the rule have face validity?