Developing a protocol for an educational software competition.

This project developed a protocol for the inaugural Instructional Software Competition of the American Dental Education Association (ADEA). The evaluation instrument was derived from the Guidelines for the Design of Educational Software developed by the ANSI-accredited Standards Committee for Dental Informatics. Eleven judges were calibrated in a conference call and rated a total of 30 submissions using a 66-question instrument. The maximum score was 204 points. The mean score of WWW-based programs was 106.7 points, and of CD-ROM-based programs 109.5 points. The summative review of the judging process identified several potential improvements, such as distinguishing between standalone programs and educational support material; increasing the number of answer choices on rating scales; differential weighting of criteria; and a more discriminative approach to judging formative and summative evaluations. We plan to improve the protocol by supporting the process through a Web-based application; calibrating judges with an online handbook; improving and adapting the rating instrument itself; using at least three judges for each program; and conducting a measurement study.