Venous thromboembolism risk assessments on trauma patients has suboptimal interobserver reliability among inexperienced clinicians (fourth-year medical students).

BACKGROUND: Venous thromboembolic disease is a major cause of morbidity and mortality in hospitalized patients worldwide. The objective of this study is to compare interobserver reliability for qualitative and quantitative venous thromboembolism (VTE) risk assessments in hospitalized trauma patients.
METHODS: We conducted a retrospective medical record review of 40 randomly selected trauma patients admitted to a 448-bed urban level-I trauma center from January 2013 to January 2014. Interclass correlation coefficient (ICC) was calculated based on a two-way mixed model. The sample was equally distributed between patients admitted to the floor and the intensive care unit (ICU). Eight fourth-year medical students performed the risk assessments by the medical record. Two forms for risk assessment were used: a qualitative screening and a quantitative screening. The composite of intraobserver and interobserver variabilities was determined.
RESULTS: The ICC for qualitative VTE risk assessments was 0.845 and for quantitative VTE risk assessment was 0.628.
CONCLUSION: To optimize accuracy of VTE risk stratification and appropriate prophylaxis, medical students and first-year residents should be formally trained to perform quantitative assessments.

Introduction

Venous thromboembolism (VTE) is a significant cause of mortality and morbidity in hospitalized patients worldwide. Fatal pulmonary embolism (PE) is the most common preventable cause of hospital death. The incidence of clinically symptomatic VTE in the USA is ~250,000–300,000 new cases each year.1,2 PE occurs in 0.2%–0.4% of hospitalized patients and causes ~200,000 deaths annually.3 It is estimated that the USA spends ~US$7–10 billion in direct medical costs on VTE each year.4 Each hospital-acquired deep vein thrombosis (DVT) represents an incremental inpatient cost of US$10,000, while each PE represents ~US$20,000 in additional cost. Without prophylaxis, the incidence of DVT in general surgery patients is about 15%–40%, while it is 40%–80% in major trauma.5 Despite guidelines being regularly published by the American College of Chest Physicians (ACCP), VTE prophylaxis remains underutilized. Current estimates suggest that only 30%–60% of patients at risk receive appropriate VTE prophylaxis.6

Established clinical risk factors include presence of inherited conditions, such as factor V Leiden mutation, prothrombin gene mutation, and protein S or C deficiency, or acquired risk factors, such as major surgery involving the abdomen, pelvis, and lower extremities; active malignancy and its treatment; immobilization; trauma, especially involving the spinal cord injuries or spinal fractures; fracture of pelvis, hip, and leg; presence of central venous catheter; pregnancy; drugs; congestive heart failure; chronic renal disease; the antiphospholipid syndrome; obesity; smoking; older age; and history of thromboembolism.

Trauma is the leading cause of death in people of age <40 years, and the care of patients with trauma accounts for more hospital days per year than the care of patients with heart disease or cancer. DVT and PE are common complications of major trauma.7 The 2016 Annual Report of the National Trauma Data Bank (NTDB) reports DVT/thrombophlebitis as one of the top 10 posttrauma complications, with incidence of 0.60%. The incidence of PE is reported as 0.27% (NTDB).

The Eastern Association for the Surgery of Trauma has developed evidence-based practice management guidelines for the prevention of VTE in trauma patients.8

The process of providing VTE prophylaxis begins with assessment of each patient’s VTE risk. Assessment forms are often completed by first-year residents on admission or arrival to the floor; based on the relative inexperience of the interns, we hypothesize that these VTE risk assessments may be inconsistent. The objective of this study is to compare interobserver reliability for qualitative and quantitative VTE risk assessments in hospitalized trauma patients.

Methods

Study design

We conducted a retrospective medical record review of 40 randomly selected trauma patients (data only) admitted to a 448-bed urban level-I American College of Surgeons-verified adult trauma center from January 2013 to January 2014. Random selection was by the closed-envelope draw technique. Accordingly, 500 envelopes were made for patients seen and randomly selected. The sample was equally distributed between patients admitted to the floor and the intensive care unit. As this study had deidentified data that were used for study/review, the institutional review board (IRB) at SUNY Downstate exempted this study from full IRB review. Because of this, the IRB of SUNY Downstate Medical Center deemed patient consent also unnecessary.

Data analysis

Eight fourth-year medical students on surgical rotation performed the risk assessments using the deidentified medical records. Two forms for risk assessment were used, a qualitative screening and a quantitative screening. Students were instructed to carry out the following: 1) risk score assessment; 2) risk stratification – low, medium, and high risk; 3) identification of contraindications to pharmacological prophylaxis, whether absolute or relative; and 4) VTE prophylaxis plan. The qualitative screening comprised the point-based VTE protocol currently followed in our hospital (not an established risk assessment), wherein the students had to identify the medical conditions associated with risk for VTE, contraindications to pharmacological prophylaxis, and, based on their assessment, choose the appropriate prophylaxis from the order set according to the risk (Figure 1). The quantitative method was a modified version of the Caprini score; students had to identify the risk factors, which in turn were given a particular score to be added to estimate the cumulative score. Patients were then classified to be at low (<2), medium (2–5), and high (>5) risk (Figure 2). Prophylaxis included early ambulation, enoxaparin, unfractionated heparin, and warfarin. Students were instructed on how to fill the assessment forms. Statistical analysis was performed by calculation of interclass correlation coefficient (ICC) based on a two-way mixed model. The composite of intraobserver and interobserver variabilities was determined.

Figure 1

VTE prophylaxis screening and ordering form.

Abbreviations: VTE, venous thromboembolism; SCr, serum creatinine; GFR, glomerular filtration rate; GI/GU, gastrointestinal/urogenital; PLT, platelet; HIT, heparin-induced thrombocytopenia; OR, operating room; wt, weight INR, international normalized ratio; SQ, subcutaneous; PO, per os; IPC, intermittent pneumatic compression; GCS, graduated compression stockings; UFH, unfractionated heparin;q, every.

Figure 2

VTE risk assessment form.

Abbreviations: VTE, venous thromboembolism; CRP, C-reactive protein; COPD, chronic obstructive pulmonary disease; OC, oral contraceptive; HRT, hormone replacement therapy; DVT/PE, deep vein thrombosis/pulmonary embolism; BMI, body mass index; CHF, congestive heart failure.

Results

We excluded incomplete charts from the final sample. We also excluded one observer due to multiple incomplete assessments. The final sample thereby consisted of 35 patients assessed by seven observers for qualitative assessment and 37 charts assessed by eight observers for quantitative assessment.

Qualitative risk assessment

The final sample had 35 charts assessed by seven observers. Cronbach’s alpha was measured to be 0.644. The ICC for this group was 0.845 (95% CI 0.753–0.911) (Figure 3).

Figure 3

Qualitative VTE risk assessment.

Abbreviations: VTE, venous thromboembolism; df, degrees of freedom.

Quantitative risk assessment

The final sample had 37 patients assessed by eight observers. Cronbach’s alpha was determined to be 0.869 for this group, and ICC was determined to be 0.628 (95% CI 0.413–0.787) (Figure 4).

Figure 4

Quantitative VTE risk assessment.

Abbreviation: VTE, venous thromboembolism.

Discussion

PE resulting from DVT, collectively referred to as VTE, is the most common preventable cause of hospital death.9 Despite the reality that hospitalized medical and surgical patients routinely have multiple risk factors for VTE, making the risk for VTE nearly universal among inpatients, large prospective studies continue to demonstrate that these preventive methods are significantly underutilized.6,10,11–13 The Agency for Healthcare Research and Quality9,15 calls thromboprophylaxis the “number one patient safety practice’’ against VTE. Current estimates suggest that only 30%–60% of patients at risk receive appropriate VTE prophylaxis. Prevention requires a reliable tool for risk stratification for the development of VTE, screening strategies, and effective prophylaxis to significantly reduce mortality in trauma patients. A study has shown that implementation of a mandatory computerized provider order entry-based clinical decision support tool significantly improved compliance with VTE prophylaxis guidelines in hospitalized adult trauma patients.14 Many risk assessment tools are available for estimating thromboembolism risk.15 Examples are the Caprini score,16,17 Padua score,18 Wells score,19,20 the risk assessment profile developed by Greenfield et al,21 the revised Geneva score, and the simplified revised Geneva score.22 No single risk assessment has been prospectively validated as being superior to others. Using an individualized, point-based protocol in the assessment process is a complex task and might contribute to variability in VTE prophylaxis-prescribing behavior. There are no published data on how reliably medical residents can perform risk assessment and prophylaxis using a point-based VTE risk assessment tool.23 The relative inexperience of interns might lead to inconsistency and underutilization of VTE prevention strategies. It is necessary that a risk assessment tool that would reduce inconsistency among residents be utilized, thereby enabling them to order an effective prophylactic measure. Studies have suggested that even with repetitive use of a VTE risk assessment tool, the aggregate resident interrater reliability is moderate for risk stratification and contraindications, with reliability being only fair for VTE plan.24 Some have suggested that risk-appropriate VTE prophylaxis prescription varied dramatically among residents and that a substantial number of residents may habitually provide suboptimal care. Residents may be a more reliable target for quality improvement efforts.25 Providing personal clinical effectiveness feedback, including data and peer-to-peer coaching, improves resident performance and results in significant reduction in harm for patients.26

In our study, we sought to compare interobserver reliability for qualitative and quantitative assessment in trauma patients and to determine whether qualitative or quantitative risk assessment is a better tool for residents to evaluate VTE risk, as these forms are often completed by first-year residents.

Limitations of this study include that we presumed that fourth-year medical students are equivalent to incoming first-year residents in terms of clinical experience, and risk assessment forms were completed by them. It would have been ideal if the observer pool consisted of incoming residents. Another limitation is that performance of the residents should have been compared to that of senior residents to determine the effect of formal training on resident performance. As most of the history and physical examinations are done by junior residents, and hence the DVT risk assessment, ideally, the study should track incoming resident performance and evaluate which type of risk assessment has better patient outcomes. However, our aim was to see whether quantitative assessment gave better assessment of VTE risk and whether the interobserver reliability was optimal.

Conclusion

Quantitative VTE risk assessment has been shown to be more reliable than qualitative assessment, yet our findings suggest that quantitative interobserver reliability is suboptimal among fourth-year medical students to first-year residents. To optimize the accuracy of VTE risk stratification and appropriate prophylaxis, inexperienced clinicians should be formally trained to perform quantitative assessments. Qualitative VTE risk assessment is more challenging to use, and institutes should consider using the quantitative method for VTE risk assessment.