AIM: Compare how different researchers performed in screening for informative evidence about medical education. METHOD: Six researchers with three different levels of involvement in a systematic literature review screened articles by title and (where available) abstract, and then by reading articles they had selected in full text. The reference standard was a consensus decision to include or exclude the article in the final analysis, whose results are published elsewhere. RESULTS: The single screener most involved in the literature search, who was also the most junior member of the topic review group, achieved a sensitivity approaching 100% and a specificity of 98-100% for informative articles. She far outperformed the other researchers, all of whom had as much or more topic knowledge and greater research experience. CONCLUSION: It was not possible to improve on the performance of the single motivated and capable primary screener and trying to do so increased the number of uninformative articles retrieved. One interpretation is that the primary searcher was more practiced and focused on the task than her more senior colleagues, yet they tended to become worse rather than better with practice. The fact that a well informed but relatively naïve person consistently outperformed her more "expert" colleagues might suggest an alternative explanation: Given the patchy standard and qualitative nature of the evidence, perhaps experts found it harder than a novice to make reliable choices, in which case their unreliable performance reflects the nature of present day education evidence. This case study illustrates the value of quality assuring the article selection process. Given the amount of disagreement uncovered by the study, we suggest that consensus between reviewers is an important reference standard against which the performance of any single primary screener should be checked.
AIM: Compare how different researchers performed in screening for informative evidence about medical education. METHOD: Six researchers with three different levels of involvement in a systematic literature review screened articles by title and (where available) abstract, and then by reading articles they had selected in full text. The reference standard was a consensus decision to include or exclude the article in the final analysis, whose results are published elsewhere. RESULTS: The single screener most involved in the literature search, who was also the most junior member of the topic review group, achieved a sensitivity approaching 100% and a specificity of 98-100% for informative articles. She far outperformed the other researchers, all of whom had as much or more topic knowledge and greater research experience. CONCLUSION: It was not possible to improve on the performance of the single motivated and capable primary screener and trying to do so increased the number of uninformative articles retrieved. One interpretation is that the primary searcher was more practiced and focused on the task than her more senior colleagues, yet they tended to become worse rather than better with practice. The fact that a well informed but relatively naïve person consistently outperformed her more "expert" colleagues might suggest an alternative explanation: Given the patchy standard and qualitative nature of the evidence, perhaps experts found it harder than a novice to make reliable choices, in which case their unreliable performance reflects the nature of present day education evidence. This case study illustrates the value of quality assuring the article selection process. Given the amount of disagreement uncovered by the study, we suggest that consensus between reviewers is an important reference standard against which the performance of any single primary screener should be checked.