OBJECTIVE: To assess whether a computer program using a variant of Angoff's method can detect anomalous behaviour indicative of cheating in multiple choice medical examinations. DESIGN: Statistical analysis of 11 examinations held by the Royal College of Paediatrics and Child Health. SETTING: UK postgraduate medical examination. PARTICIPANTS: Examination candidates. MAIN OUTCOME MEASURES: Detection of anomalous candidate pairs by regression of similarity of correct answers in all possible pairs of candidates on the overall proportion of correct answers. Anomalous pairs were subsequently assessed in terms of examination centres and the seating plan of candidates, to assess adjacency. RESULTS: The 11 examinations were taken by a total of 11,518 candidates, and Acinonyx examined 6,178,628 pairs of candidates. Two examinations showed no anomalies, and one examination found an anomaly resulting from a scanning error. The other eight examinations showed 13 anomalies compatible with cheating, and in each pair the two candidates had sat the examination at the same centre, and for six examinations with seating plans, the candidates in the anomalous pairs had been seated side by side. The raw probabilities of the anomalies varied from 3.9x10(-11) to 9.3x10(-30) (median = 1.1x10(-17)), with Bonferroni-corrected probabilities in the range 2.4x10(-5) to 4.1x10(-24) (median = 1.6x10(-11)). This suggests that one anomalous pair is found for every 1000 or so candidates taking this postgraduate examination. CONCLUSIONS: This statistical technique identified a small proportion of candidates who had very similar patterns of correctly answered questions. The likelihood is that one candidate has copied from the other, or that there was collusion, or that a technical error occurred in the exams department (as happened in a single case). Analysis of similarities can be used to identify cheating and as part of the quality assurance process of postgraduate medical examinations.
OBJECTIVE: To assess whether a computer program using a variant of Angoff's method can detect anomalous behaviour indicative of cheating in multiple choice medical examinations. DESIGN: Statistical analysis of 11 examinations held by the Royal College of Paediatrics and Child Health. SETTING: UK postgraduate medical examination. PARTICIPANTS: Examination candidates. MAIN OUTCOME MEASURES: Detection of anomalous candidate pairs by regression of similarity of correct answers in all possible pairs of candidates on the overall proportion of correct answers. Anomalous pairs were subsequently assessed in terms of examination centres and the seating plan of candidates, to assess adjacency. RESULTS: The 11 examinations were taken by a total of 11,518 candidates, and Acinonyx examined 6,178,628 pairs of candidates. Two examinations showed no anomalies, and one examination found an anomaly resulting from a scanning error. The other eight examinations showed 13 anomalies compatible with cheating, and in each pair the two candidates had sat the examination at the same centre, and for six examinations with seating plans, the candidates in the anomalous pairs had been seated side by side. The raw probabilities of the anomalies varied from 3.9x10(-11) to 9.3x10(-30) (median = 1.1x10(-17)), with Bonferroni-corrected probabilities in the range 2.4x10(-5) to 4.1x10(-24) (median = 1.6x10(-11)). This suggests that one anomalous pair is found for every 1000 or so candidates taking this postgraduate examination. CONCLUSIONS: This statistical technique identified a small proportion of candidates who had very similar patterns of correctly answered questions. The likelihood is that one candidate has copied from the other, or that there was collusion, or that a technical error occurred in the exams department (as happened in a single case). Analysis of similarities can be used to identify cheating and as part of the quality assurance process of postgraduate medical examinations.