Cross-platform digital assessment forms for evaluating surgical skills.

A variety of structured assessment tools for use in surgical training have been reported, but extant assessment tools often employ paper-based rating forms. Digital assessment forms for evaluating surgical skills could potentially offer advantages over paper-based forms, especially in complex assessment situations. In this paper, we report on the development of cross-platform digital assessment forms for use with multiple raters in order to facilitate the automatic processing of surgical skills assessments that include structured ratings. The FileMaker 13 platform was used to create a database containing the digital assessment forms, because this software has cross-platform functionality on both desktop computers and handheld devices. The database is hosted online, and the rating forms can therefore also be accessed through most modern web browsers. Cross-platform digital assessment forms were developed for the rating of surgical skills. The database platform used in this study was reasonably priced, intuitive for the user, and flexible. The forms have been provided online as free downloads that may serve as the basis for further development or as inspiration for future efforts. In conclusion, digital assessment forms can be used for the structured rating of surgical skills and have the potential to be especially useful in complex assessment situations with multiple raters, repeated assessments in various times and locations, and situations requiring substantial subsequent data processing or complex score calculations.

Numerous structured assessment tools have been introduced for the evaluation of surgical skills, applying a multitude of different scales and rating techniques. The data collection forms used in such tools provide a structured framework for the evaluator and are commonly paper-based. The traditional paper-based rating forms are tried and tested, allow fast assessment, and are intuitive for raters regardless of their digital readiness. However, paper-based rating forms can involve a time-consuming process of manual score calculation. This difficulty can be addressed by using image-scanning techniques such as optical mark recognition, although such techniques require dedicated equipment.

Only a few reports have described digital rating forms [1-3], with the exception of reports dealing with computer-based testing as such. Digital assessment forms for evaluating surgical performance might be considered more inconvenient, insecure, and vulnerable in comparison with paper-based forms, and moreover, the evaluator must be familiar with the necessary electronic devices and platforms. Nevertheless, digital forms could have advantages over paper-based forms, especially in complex assessment situations where multiple performances and raters need to be managed, assessments occur at a range of times and/or locations, or when substantial subsequent data processing is needed. Digital assessment forms offer the possibility of connecting to an online database, which could support simultaneous assessment using multiple devices and evaluators as well as providing immediate scoring.

A range of software platforms and technologies could potentially be used for digital assessment forms, and the advantages and disadvantages of currently available electronic form management tools have recently been described [1]. In contrast to other reports on the development of digital assessment forms that required the integration of multiple types of tools and surveys for complex workplace-based or practice-based assessments [1,2], in this case, it was only necessary to assess saved final-product performances from a virtual-reality otology simulator using one assessment form that allowed multiple raters on different platforms. Therefore, we developed a digital assessment form for this use. This led to the development of several other examples of digital assessment forms for evaluating skills in the context of surgical training, including both technical and non-technical skills. In the following sections, the development and testing of these forms will be described, and their application in the assessment of surgical skills will then be discussed.

First, it was necessary to choose a database platform for the digital assessment forms. We chose FileMaker Pro 13 (FileMaker Inc., Santa Clara, CA, USA), because it supports different desktop platforms (Windows 7/8, OSX 10.7 or newer) as well as the iPad and iPhone via the FileMaker Go app. The database can either be stored on the device itself or be hosted on a local or external server, allowing for web-based access through most modern web browsers. The estimated costs (online prices, June 2014) are US$329 for one license for FileMaker Pro 13 (a free trial is available), as well as US$132 or more annually for external database hosting or US$348 for the software necessary to establish a local server. The FileMaker Go app for iPhone/iPad is currently free. We found that the FileMaker platform provided excellent flexibility in designing the database, displaying the fields, and designing the visual layout of the forms, and therefore it was easy to develop new forms. In our experience, developing a new basic rating form from scratch took from two to eight hours. Good online documentation exists for this software, as well as excellent beginner’s guides and external consulting can be hired if more complex solutions are needed. These factors facilitate the development of digital assessment forms that meet local requirements.

Next, we developed working examples of several rating forms—including a range of rating structures—for some of the assessment tools that have been reported in the fields of surgery and otorhinolaryngology. An overview of the newly developed forms is given in Table 1, and an example of one of the developed forms can be seen in Fig. 1. The forms have been made public for free download [4] and they can be used as is, modified as needed, or serve as inspiration for novel work. In each form, the time of assessment and the evaluator identification (ID) is automatically saved with each assessment. The evaluator needs to enter the participant ID and the case ID if relevant. The forms have checkboxes, drop-down menus, radio buttons, and/or free text fields for rating and feedback. Cumulative scores and sub-scores are automatically calculated and updated as the evaluator performs the assessment, and these scores are stored along with the entered data. The data can later be exported to a range of different formats, including Excel® (Microsoft, Redmond, WA, USA) and comma-separated files, for further processing.

Table 1.

Examples of digital assessment forms for some structured assessment tools for the evaluation of surgical skill

Assessment tool	Assessment type	Structure/scales	Special features of the digital assessment form
Modified Welling Scale (WS) [5]	Final-product assessment of mastoidectomy performance	Dichotomous ratina of 25 items (1 = adeguate/0 = inadeguate)	Automatic calculation of total score
Non-Technical Skills for Surgeons in Denmark (NOTSSdk) [6]	Video-based assessment of non-technical performance of general surgeons	5-ooint Likert-like ratina scale (very poor to very good) for 4 main categories and 13 sub-elements.	None
		7-point Likert-like rating scale (very ooor-verv aood) for global ratina score.
		Feedback notes for each of the 13 sub-elements and for global feedback
Objective Structured Assessment of Ultrasound Skills (OSAUS) [7]	Video-based assessment of point-of-care ultrasonography performance	5 elements rated from 1-5 using Objective Structured Assessment of Technical Skills-like scales with descriptions of scores	Automatic calculation of total score
Standardized Patient History Taking and Physical Examination Checklist in Hoarseness [8]	Standardized patient assessment of history taking and physical examination	Checklist with 18 items on history taking and 12 items on physical examination	Automatic calculation of sub-scores (sum and percentage) and total score (sum and percentage)

Fig. 1.

The digital Objective Structured Assessment of Ultrasound Skills (OSAUS) assessment form in FileMaker Go on iPad Mini.

Finally, the digital assessment forms were tested with FileMaker Pro 13 on Windows 7 and OSX 10.7, and FileMaker 13 Go on iPhone and iPad running iOS 7. The forms were tested using both device-stored and externally hosted databases. Using WebDirect access to the external server, testing was also performed with a range of common web browsers (Firefox, Chrome, Safari, and Internet Explorer) on the abovementioned devices. Overall, the digital assessment forms proved stable and consistent across platforms and devices. However, if web-based access is likely to be the primary mode of use, then it would be preferable to optimize the forms accordingly, as the layout can change slightly in the web browser compared to the desktop software and mobile app. As well, some problems and crashes occurred using web-based access from handheld devices. Currently, the only handheld devices on which these forms are accessible are the iPhone and iPad, for which the FileMaker 13 Go app is available.

Some general issues should be considered when adopting digital assessment forms. First, each rating form needs to be tailored to the specific assessment situation. Second, evaluators need to be introduced to the digital rating forms and the device(s) that will be used. Moreover, combining data from more than one device or rater will need to be done manually unless a server-hosted database with online access is used. Furthermore, regular backup and device management should also be considered. Finally, as technology develops and new devices and platforms emerge, the digital assessment forms will need to be updated, resulting in continued development and maintenance costs.

The digital assessment forms described in this report have been developed and tested in a local, controlled, and no-stakes setting for evaluating recorded performances. The next step would be applying these digital assessment forms to the live evaluation of surgical performance, in a real-life setting and in high-stakes assessments. Doing so would require significant experience with the digital platform in the local institution, and it would be necessary for the local setup to have been thoroughly tested and proved stable; for example, if a server-based deployment is chosen, Internet connectivity must be assured. We therefore recommend implementing digital assessment forms for performance rating after the consideration of institutional needs, equipment, expertise, and resources, and after the advantages and disadvantages have been weighed in comparison with paper-based rating forms. Further research into the use of digital assessment forms for evaluating surgical skills compared to paper-based rating forms will help determine whether digital assessment forms prove to be timesaving, reliable, and feasible.