The assessment of usability of pain medical device by physiatrists and physiotherapists: A Delphi survey.

ABSTRACT: When developing a new medical device, it is essential to assess the usability of such a device through various stakeholders.This study assessed the usability of pain medical devices through a Delphi survey administered to physiatrists and physiotherapists.A Delphi survey was conducted on the problems and improvements in hardware and software for a panel consisting of 10 physiatrists and 10 physiotherapists. A total of 3 rounds of surveys were conducted, and the third round of survey was confirmed through a Likert scale (1 = strongly agree to 5 = strongly disagree).The 2 groups generally had a common perception of the problems and improvements in pain medical devices. However, the physiatrist group mostly identified problems such as linking patient information, whereas the physiotherapist group deemed hardware problems such as device weight or connection cables as being more important (mean [standard deviation]; physiatrist, hardware 2.90 [0.93], software 2.28 [0.91] / physiotherapist, hardware 3.04 [0.84], software 3.03 [1.13]).To date, analysis has not been conducted by dividing the focus of various stakeholders using pain medical devices. The difference in view of the usability of these 2 stakeholder groups should be considered when improving the hardware or software of pain medical devices in the future. Further research is warranted to investigate other stakeholders such as patients and device developers to improve the devices.

Introduction

In the development of a medical device, an engineering process should be performed with usability standards based on human capabilities (physical, sensory, emotional, and intellectual).[ Usability allows the device to be used efficiently and effectively, improves safety, reduces fatigue and stress, and can provide satisfaction and comfort.[ From the seller's point of view, appropriate usability also reduces the need to adjust the design, lowering the cost of updates after launch and enhancing competitiveness.[

A pain medical device is a type of medical equipment that can effectively control pain.[ It is widely used to reduce pain in various diseases.[ However, inadequate usability of pain medical devices causes complications in 26% of people using such devices, and in the United States, approximately 1500 patients are treated with electric heating pads-related burns annually.[

Usability in medical devices has been overlooked because they are used by experts, and there is a lack of research on the relationship between users and medical devices.[ Most innovations in medical device design without proper understanding of the user's needs fail, and such devices are sometimes recalled owing to errors.[ The participation of various stakeholders helps to integrate diverse needs and requirements and accelerate problem-solving of usability.[ More research is needed on the interaction between the stakeholders and the pain medical device, as in the human-device interaction field.

Usability is defined as improving medical devices through user interface design; medical devices include hardware and software.[ International standards for medical device design continue to be developed, and studies have been performed on software and physical products such as hardware. Efforts have been made to improve the usability of software for medical devices.[ As software and hardware are essentially different fields, each improvement requires different skills and technicians.

Delphi may not derive adequate results if the panelists are unprofessional or biased. However, Delphi is flexible and suitable when there is incomplete knowledge regarding a phenomenon, particularly when the goal is to develop predictions or improve the understanding of a problem, opportunity, or solution.[

The purpose of this study is to identify the needs and requirements of various stakeholders (physiatrists and Physiotherapists), and to analyze the similarities and differences of the stakeholders in order to improve the usability of pain medical devices.

Methods

The study was conducted using the 3-round Delphi method.

Panelists consisted of a physiatrist and rehabilitation specialist and a physiotherapist, who were certificated in South Korea. The physiatrist has more than 5 years of experience as a rehabilitation specialist, and the physiotherapist has more than 3 years of experience in pain treatment centers; both are experts in pain medical devices. Furthermore, both the physiatrist and physiotherapist are specialists who provide treatment for more than 50 patients complaining of pain per week. Ten physiatrists and ten physiotherapists participated in the study.

The Delphi panel had 2 rounds of meetings before conducting the survey. Through the meetings, it was decided to assess the problems and improvements of pain medical devices by dividing them into hardware and software.

The method of Delphi survey was decided through a meeting. To analyze the difference between the assessment by physiatrists and physiotherapists, the survey was divided into 2 groups in Round 1 and 2. In Round 1, problem of hardware and software and the improvement were collected as open questions. The suggested items were categorized, and the 5 most suggested items were selected. In Round 2, the priority of the items selected in Round 1 were derived. Each panel selected 3 items based on priority. Five items with high priority were selected. In Round 3, all the items selected by the 2 groups were merged, and the Likert scale was used for scoring each item (Fig. 1). The survey response was obtained via emails. This study is a survey study conducted on medical devices, and ethical approval was not necessary.

Figure 1

Survey Methods. HW = hardware, IMP = improvement, SW = software.

Round 1

In Round 1, the demographics of the panel were investigated, 3 problems associated with pain medical devices were described in hardware and software, and 5 improvements to be proposed for the development of pain medical devices were described as open questions (Supplement Table S1). The items suggested in Round 1 were separated into the physiatrist group and the physiotherapist group and then categorized into the questions for Round 2 survey. A total of 5 items were selected in the order of the most suggested items for each category of hardware and software. If there are multiple items belonging to the fifth-highest priority, all the items were suggested. Each question was separately categorized into the physiatrist group and the physiotherapist group.

Round 2

In Round 2, closed-ended questions were asked for the items categorized in Round 1. The questions categorized for physiatrists were posed to the physiatrist group, and those categorized for physiotherapists were posed to the physiotherapist group. The panel was asked to select up to 3 factors they deemed most important in order of priority:

factors related to hardware problems in pain medical devices,

factors related to software problems in pain medical devices, and

factors related to improvements in pain medical devices (Supplement Table S2).

The 5 items with the highest number of accumulated responses were selected for Round 3 survey. If there is the same number of accumulated responses, the items with more responses and a higher priority were selected.

Round 3

In Round 3, the questions selected in Round 2 from the 2 groups were combined to be posed to both groups. Each item was evaluated based on the 5-point Likert scale. The Likert scale ranged from 1 to 5 points, with 1 being extremely problematic, 2 very problematic, 3 somewhat problematic, 4 slightly problematic, and 5 not at all problematic.[ (Supplement Table S3).

One moderator organized and analyzed the responses. All the respondents were kept anonymous, except the moderator. To ensure the anonymity of the responses, all the respondents shared 1 e-mail address through which they sent their responses back to the same e-mail address. The order of questions in the survey was randomly assigned in Round 2 and Round 3 to avoid presentation bias.

Statistical analysis

The Mann–Whitney test was used to compare the differences between the physiatrist and physiotherapist groups in the demographic characteristics and identified with the Likert scale in Round 3 questionnaire using SPSS version 21.0 (SPSS, Inc., Chicago, IL). The statistical significance was set at P < .05.

Results

This study was conducted from January 1 to December 31, 2019; a total of 3 rounds of the Delphi survey were conducted. The general characteristics of the Delphi panel are summarized in Table 1, and there was no statistically significant difference between the 2 groups, except for the number of beds in the institution. E-mails were sent to all participants of the panel in each round, and all 20 (100%) participants responded.

Table 1

Characteristics of the Delphi panel.

Characteristic	Physiatrist (n = 10)	Physiotherapist (n = 10)	P value
Age, year	37.2 ± 1.40	34.1 ± 8.73	.075
Gender, n (%)
Male	5 (50)	5 (50)
Female	5 (50)	5 (50)
Career period, yr	12.4 ± 1.51	10 ± 9.71	.089
Number of beds in the institution, n (%)
≥ 500	4 (40)	10 (100)
300–499	3 (30)
100–299	2 (20)
<100	1 (10)
Characteristic of pain medical device, Likert scale
There are a lot of brand-new pain medical devices in working institution	3.6 ± 0.53	3.3 ± 0.82	.400
There are a lot of expensive pain medical devices in working institution	3.6 ± 1.01	3.3 ± 0.82	.604
There are a lot of brand-new pain medical devices in working institution compared with other institutions	3.9 ± 0.78	3.8 ± 0.63	.842
There are a lot of expensive pain medical devices in working institution compared with other institutions	3.7 ± 1.12	3.5 ± 0.70	.604

The items selected in Round 1 from each group are presented in Table 2. For the factors related to hardware problems of pain medical devices, both the physiatrist and physiotherapist groups most frequently pointed out that the devices are too large and heavy.

Table 2

Results of round 2.

Table 2-1. Analysis of factors related to hardware problems of pain medical devices
Group	Factors related to hardware problems, n	Total	1st priority	2nd priority	3rd priority
Physiatrist	The device is too large, n	7	5	2	0
	The connection cable between the device body and probe is complicated, n	6	2	3	1
	It is difficult to operate the device, n	5	2	2	1
	The device is too heavy, n	4	1	0	3
	The device does not have various functions, n	4	0	2	2
	Only one patient can be treated with one device, n	3	0	1	2
	There is no device to monitor vital signs, n	1	0	0	1
	The device screen is too small, n	0	0	0	0
Physiotherapist	The connection cable between the device body and probe is complicated, n	7	1	4	2
	The device is too large, n	6	4	2	0
	The operator needs to continuously hold the probe when applying the device to patients, n	3	3	0	0
	The device does not have various functions, n	3	2	0	1
	The device is too heavy, n	3	0	0	3
	It is difficult to move the device, n	3	0	0	3
	Cleaning the device is difficult, n	2	0	2	0
	The device makes loud noises, n	2	0	1	1
	There is no display screen or it is too small, n	1	0	1	0
	All devices have different specifications, n	0	0	0	0
	The device makes noise, n	0	0	0	0

For the factors related to software problems of pain medical devices, the 2 groups commonly pointed out that alarm setting is not supported or is too simple, as well as the lack of a preset function and an indicator for device malfunction.

For improvements in pain medical devices, the 2 groups commonly suggested that the devices should be made smaller and lighter, and software operation should be easier. Furthermore, both groups suggested improving the ease of movement of the device (device holder).

On hardware problems of pain medical devices, both groups pointed out that the devices were too large and the connection cable between the device body and probe was complicated. In addition, the devices were heavy and did not have various functions. In terms of the difference between the 2 groups, physiatrists pointed out difficulty in operation as a problem, whereas physiotherapists pointed out that the operator had to continuously hold the probe while using it on patients, and the devices were difficult to move (Table 2–1).

On software problems of pain medical devices, both groups pointed out that the devices had no malfunction indicator and preset function. In terms of the difference between the 2 groups, physiatrists pointed out that the patient information was not linked, the device screen was unintuitive, and the device was too manually operated for precise operation, whereas physiotherapists pointed out that the programs were too simple and not diverse enough to be applied widely, the patient information was not being saved, and the alarm setting was absent or too simple (Table 2–2).

For improvements in pain medical devices, both groups suggested that the devices should be smaller and should have a minimum weight. The physiatrist group suggested linking devices and medical systems, simplifying software operation, providing specific guidelines for different treatment regimes, and using 1 device for various treatments as possible improvements, whereas the physiotherapist group suggested making it convenient to apply the anchoring part to patients, saving individual patient data on the devices, ease of cleaning, enhancing the external material and design, and convenient device movement as possible improvements (Tables 2–3).

Table 3

Factors related to problems and improvements of pain medical devices; Round 3 of Delphi method.

The average Likert score for the hardware problems of both groups was 3.00, indicating that a problem exists for the selected item. The average score of the physiatrist group was 2.90, whereas that of the physiotherapist group was 3.04, showing limited statistically significant difference between the2 groups. With respect to software problems, the average score of both groups was 2.66, which indicates that a problem exists for all selected items. The average score of the physiatrist group (2.28) was lower than that of the physiotherapist group (3.03). The physiatrist group had lower scores for all items compared with those of the physiotherapist group, except for items with manually operated devices. A statistically significant difference was observed for 2 items (malfunction indicator not being displayed and patient information not being linked) (P < .05). The average score for improvements in pain medical devices in both groups was 2.58, indicating that the selected items need to be improved. The average score of the physiatrist group (2.22) was lower than that of the physiotherapist group (2.86) (Table 3). The physiatrist group had lower scores for all items compared with those of the physiotherapist group, except for items stating that the devices should be smaller, weigh less, and easy to move. A statistically significant difference was observed for items stating that specific guide manuals should be provided for different treatment regions, data for individual patients should be saved on the devices, and the devices should be easy to clean (P < .05).

Discussion

There are many studies on usability in the field of medical devices, but only few studies on pain medical devices.[ This study aimed to examine the needs and requirements of different users based on usability with respect to the problems and improvements in pain medical devices currently in use. Moreover, differences in perception between physiatrists and physiotherapists were identified.

Because there are diverse user demands, the opinions of an array of users were accepted and organized through open-ended questions in Round 1 for each group and then prioritized in Round 2. After combining the priorities determined for each group, the extent to which each item was perceived as a problem was deduced using the Likert scale in Round 3. The average Likert score for all items in Round 3 was between 2 and 3 points, indicating that both physiatrists and physiotherapists were aware of the problems and considered improvements necessary. No or limited statistically significant difference was observed between the 2 groups for most of the items. Physiatrists and physiotherapists have different occupations and working environments, but their experiences of using pain medical devices and perceived problems were generally similar.

With respect to hardware problems in Round 1, however, the physiatrist group pointed out the difficulty of device operation and small screen size as problems, whereas the physiotherapist group pointed out various problems related to the actual use and inconvenience in the management of the devices because they are familiar with the operation of the devices.

With respect to software problems, the physiatrist group pointed out the patient information not being linked, absence of a preset function, and lack of specific guidelines for different treatment regions as problems. It can be inferred that physiatrists consider patient information as important for determining the intensity, method, and location of pain treatment, and hence should be promptly available. Since physiatrists frequently use the devices on patients, they suggested that the devices can be remotely connected through a mobile device or a computer. On the other hand, the physiotherapist group pointed out the inability to save patient information and problems with alarm setting, indicating higher demands for shortening the time required for setting the devices and the actual treatment time.

In Round 2, the 2 groups had similar opinions on hardware problems but different perceptions of software problems. The physiatrist group mentioned that patient information was not linked, whereas the physiotherapist group pointed out the lack of variety of programs. In terms of improvements, the physiatrist group placed a higher priority on software-related items such as linking patient information, whereas the physiotherapist group placed a higher priority on hardware-related items such as the size and weight of the devices.

In Round 3, both groups recognized the problems in hardware- and software-related factors as well as improvements, and there was no significant difference in their perception of the problems. However, the overall average score for software problems of the physiatrist group was 0.8 points lower than that of the physiotherapist group, indicating that physiatrists noted that there are more problems. In terms of improvements, both groups had similar scores for hardware problems, but the physiatrist group had a lower score for software problems. A statistically significant difference was observed among the 3 items. In particular, the physiotherapist group pointed out that device cleaning should be easier, whereas the physiatrist group selected it as a problem that requires further improvement. The convenience of device cleaning was not noted as an item requiring improvement by the physiatrist group because they rarely manage the devices themselves, although they still suggested it as one of the important improvement factors. Since roles differ depending on the group, it is confirmed that there could be a difference between problem recognition and improvement points.

Although there are similarities in the needs and requirements of the 2 groups, differences in their experiences resulting from different working environments were observed. Because there are differences in the areas of focus of the 2 groups, it is important to review the needs of physiatrists and physiotherapists according to the areas that require usability improvement.

In terms of the limitations of this study, the results cannot be generalized because there were only 10 participants in each group of the Delphi panel. For the physiotherapist group, only those who work at large hospitals with at least 500 beds were recruited because of the difficulty of finding qualified participants. However, since there was no significant difference in the characteristics of the pain medical devices being used by the 2 groups, bias due to differences in devices are considered negligible. The survey did not consider devices in which the latest technology, such as digital technology, is applied. In the characteristic survey of pain medical devices, the Likert scale is between 3 and 4, indicating that the devices included in this study are average or slightly older. Recently, digital technology has been applied to medical device.[ Devices with new digital technologies are being developed, but evidence of clinical benefits and effectiveness is still lacking.[ Cooperation with various stakeholders is important when applying new digital technology to medical devices to improve usability.[ This study would be helpful in improving digital technology based on the user experience of the 2 stakeholder groups. If improvements in devices reflect the priorities of improvements presented in this study, more efficient, effective, and cost-effective devices can be developed by reducing the cost and effort on less priority items.

Conclusions

This is a novel study to identify the needs and requirements of physiatrists and physiotherapist as different users of pain medical devices. The results of this study can be used to improve the usability of pain medical devices. Physiatrists and physiotherapists, the main stakeholders of pain medical devices, had similar responses; however, there were other responses that were more focused in each group. Focusing on areas where each group deems important will efficiently improve usability. This is a survey study that investigated the user experience of physiatrists and physiotherapists. Further research is needed on whether the pain medical device becomes efficient and safe when the user experience derived from this study is applied to devices. This study surveyed physiatrists and physiotherapists among various stakeholders; it is necessary to investigate other stakeholders such as patients who are treated using pain medical devices or device developers who will determine if it is possible to implement improvements in future pain medical devices.

Acknowledgments

The authors would like to thank all panelists of this study for their efforts and enthusiastic collaboration throughout the study.

Author contributions

Conceptualization: Jong Moon Kim.

Data curation: Hyunseok Kwak, Hyunjung Oh, Byoungwoo Cha, Jong Moon Kim.

Formal analysis: Hyunseok Kwak, Jong Moon Kim.

Funding acquisition: Jong Moon Kim.

Investigation: Hyunjung Oh, Byoungwoo Cha, Jong Moon Kim.

Methodology: Hyunseok Kwak, Jong Moon Kim.

Project administration: Hyunseok Kwak, Hyunjung Oh, Byoungwoo Cha.

Supervision: Jong Moon Kim.

Writing – original draft: Hyunseok Kwak.

Writing – review & editing: Hyunseok Kwak, Jong Moon Kim.