Evaluating informatics applications--some alternative approaches: theory, social interactionism, and call for methodological pluralism.

A review of evaluation literature concerning CDSSs indicates that randomized controlled clinical trials (RCTs) are the 'gold standard' for evaluation. While this approach is excellent for studying system or clinical performance, it is not well suited to answering questions concerning whether systems will be used or how they will be used. Because lack of use of CDSS has been of concern for some years, other evaluation research designs are needed to address those issues. This paper critiques RCT and experimental evaluation approaches and presents alternative approaches to evaluation that address questions outside the scope of the usual RCT and experimental designs. A wide range of literature is summarized to illustrate the value of evaluations that take into account social, organizational, professional, and other contextual considerations. Many of these studies go beyond the usual measures of systems performance or physicians' behavior by focusing on 'fit' of the system with other aspects of professional and organizational life. Because there is little explicit theory that informs many evaluations, the paper then reviews CDSS evaluations informed by social science theories. Lastly, it proposes a theoretical social science base of social interactionism. An example of such an approach is given. It involves a CDSS in psychiatry and is based on Kaplan's 4Cs, which focus on communication, control, care, and context. Although the example is a CDSS, the evaluation approach also is useful for clinical guideline implementation and other medical informatics applications. Similarly, although the discussion is about social interactionism, the more important point is the need to broaden evaluation through a variety of methods and approaches that investigate social, cultural, organizational, cognitive, and other contextual concerns. Methodological pluralism and a variety of research questions can increase understanding of many influences concerning informatics applications development and deployment.