INTRODUCTION: Assessment decisions increasingly rely on synthesis of information from a variety of sources. It is known that aggregation of information to make decisions is open to a number of biases. The aim of this research was to investigate bias, accuracy and confidence of assessment decision making. METHODS: The participants were consultation skills assessors. A model for incremental information was developed with participants being shown results from purposefully selected, but authentic, data from the University's final summative 10-station Objective Structured Clinical Examination (OSCE). After each piece of information, participants gave a pass-fail decision and their confidence in that choice. Following the information from 10 OSCE stations the participants were given a discordant fictional anecdote and again participants gave a pass-fail decision and their confidence. RESULTS: When there is overwhelming evidence to support a pass or fail, participants were not as confident as the data would support. Participants were less confident to make a fail decision than a pass. Despite considerable evidence from multiple results some participants altered decisions based on isolated contradictory information from an anecdote. DISCUSSION: These findings are significant in understanding decision-making. Given equivalent levels of evidence, decision makers are less confident to fail than pass and less robust information can undermine more robust information.
INTRODUCTION: Assessment decisions increasingly rely on synthesis of information from a variety of sources. It is known that aggregation of information to make decisions is open to a number of biases. The aim of this research was to investigate bias, accuracy and confidence of assessment decision making. METHODS: The participants were consultation skills assessors. A model for incremental information was developed with participants being shown results from purposefully selected, but authentic, data from the University's final summative 10-station Objective Structured Clinical Examination (OSCE). After each piece of information, participants gave a pass-fail decision and their confidence in that choice. Following the information from 10 OSCE stations the participants were given a discordant fictional anecdote and again participants gave a pass-fail decision and their confidence. RESULTS: When there is overwhelming evidence to support a pass or fail, participants were not as confident as the data would support. Participants were less confident to make a fail decision than a pass. Despite considerable evidence from multiple results some participants altered decisions based on isolated contradictory information from an anecdote. DISCUSSION: These findings are significant in understanding decision-making. Given equivalent levels of evidence, decision makers are less confident to fail than pass and less robust information can undermine more robust information.
Authors: Boaz Shulruf; Arvin Damodaran; Phil Jones; Sean Kennedy; George Mangos; Anthony J O'Sullivan; Joel Rhee; Silas Taylor; Gary Velan; Peter Harris Journal: BMC Med Educ Date: 2018-01-06 Impact factor: 2.463
Authors: Boaz Shulruf; Barbara-Ann Adelstein; Arvin Damodaran; Peter Harris; Sean Kennedy; Anthony O'Sullivan; Silas Taylor Journal: BMC Med Educ Date: 2018-11-20 Impact factor: 2.463