Selecting the best clinical vignettes for academic meetings: should the scoring tool criteria be modified?

BACKGROUND: The performance of scoring tools to select clinical vignettes for presentation at academic meetings has never been assessed.
OBJECTIVE: To measure the psychometric properties of two scoring tools used to select clinical vignettes and to determine which elements are most helpful.
DESIGN: Prospective observational study. PARTICIPANTS: Participants submitting clinical vignette abstracts, Society of General Internal Medicine annual meetings (2006-2007). MAIN MEASURES: The 2006 scoring tool had three criteria (clarity, significance, and relevance) with brief general descriptors. The 2007 modified tool had five criteria (clarity, significance, relevance, teaching value, and overall assessment) with more detailed descriptors. KEY
RESULTS: A total of 938 clinical vignette abstracts were submitted (484 in 2006; 454 in 2007); 59.5% (n=288) were accepted for presentation. Cronbach's alpha was 0.81 for the 2006 three-item tool and 0.95 for the 2007 modified five-item tool. Simplifying the five-item 2007 tool to three items (relevance, teaching value, overall assessment) yielded a Cronbach's alpha of 0.95. The agreement between the number of clinical vignettes accepted for presentation (2007) using the average score of the five items with the number that would have been accepted using the simplified three items (relevance, teaching value, overall assessment) was almost perfect, with kappa 0.89 (95% confidence interval, 0.85 to 0.93).
CONCLUSIONS: Both scoring tools performed well, but a simplified tool with three items (relevance, teaching value, and overall assessment) and detailed descriptors was optimal; the simplified tool could improve the reviewer efficiency and quality of clinical vignettes presented at national meetings.