Cognitive impact assessment of electronic knowledge resources: a mixed methods evaluation study of a handheld prototype.

RATIONALE: We recently proposed a new method to systematically assess the cognitive impact of knowledge resources on health professionals.
OBJECTIVE: To describe promises and shortcomings of a handheld computer prototype of this method.
BACKGROUND: We developed an impact scale, and combined this scale with a Computerized Ecological Momentary Assessment technique.
METHOD: We conducted a mixed methods evaluation study using a 7-item scale within a questionnaire linked to a commercial knowledge resource. Over two months of Family Medicine training, 17 residents assessed the impact of 1,981 information hits retrieved on handheld computer. From observations, log-reports, archives of hits and interviews, we examined issues associated with hardware, software and the questionnaire.
FINDINGS: Fifteen residents found the questionnaire clearly written, and only one pointed to the questionnaire as a major reason for their low level of use of the resource. Residents reported technical problems (e.g. screen trouble) or limitations (e.g. limited tracking function) and socio-technical issues (e.g. software dependency).
CONCLUSION: Lessons from this study suggest improvements to guide future implementation of our method for assessing the cognitive impact of knowledge resources on health professionals.