Evaluation of Usability and Acceptance of a New Autoinjector Intended for Methotrexate Subcutaneous Self-Administration in the Management of Rheumatoid Arthritis.

INTRODUCTION: Rheumatoid arthritis (RA) is a chronic autoimmune inflammatory disease, for which the introduction of injectable treatments has had a major impact on quality of life directly related to the disease. The purpose of this descriptive study was to evaluate the usability of a new autoinjector, intended for methotrexate self-administration, based on the device's design and instructions for use (IFU).
METHODS: This multicenter trial included three user groups: a group of patients with established RA subdivided into two groups according to their hand disability, and a group of caregivers or nurses. Each subject performed three simulated injections with a water-filled device on a foam pad. The first injection was made just after reading the IFU without further instructions (first phase). The second phase consisted of two injections made after explanations provided upon request of the subject in an optimum environment and in a "worst-case" home environment. The usability of the autoinjector was assessed by a questionnaire (success: ≥75% of positive responses) and by a score card reflecting injection performances (success: execution of ≥75% of handling steps).
RESULTS: Forty-two subjects were enrolled in the study. During the first phase, the great majority of subjects succeeded in the usability questionnaire (90.5%) and in the injection performance (95.2%) with no major differences between the user groups. In the Second phase, all subjects from all three user groups succeeded in the usability questionnaire and had a positive rate of device handling, regardless of the environment and of the user group. No safety concerns were raised during the study.
CONCLUSIONS: This study found a very high level of usability and subject acceptance of the autoinjector, intended for methotrexate self-administration, regardless of the hand disability and environmental conditions. FUNDING: Nordic Group. TRIAL REGISTRATION: EudraCT reference number: 2014-A0141245.

Introduction

Rheumatoid arthritis (RA) is a chronic autoimmune inflammatory disease for which the introduction of injectable treatments has had a major impact on quality of life directly related to the disease. Lack of patient adherence is considered one of the major issues in contemporary medicine. It is thought that improving patients’ adherence to long-term therapies is more efficient than any biomedical progress [1]. Several studies have demonstrated that self-injection versus injection by healthcare workers can increase patients’ treatment adherence, and reduce costs for society by decreasing the frequency of healthcare professionals’ visits. Moreover, it also benefits patients in terms of costs, time, ease of use, improved self-esteem, and greater quality of life [2, 3].

Patients with RA suffer from reduced manual dexterity, meaning that administering self-injections correctly can be physically problematic. Different injection technologies, such as prefilled syringes or autoinjectors, have been introduced. Autoinjectors automatically insert the needle and deliver a controlled and fixed dose of the required drug. These devices have been shown to provide numerous benefits, including a reduced risk of injection site reactions, reduced discomfort and greater ease of use compared with classic syringes [4, 5].

This study aimed to evaluate the usability of a newly developed autoinjector intended for methotrexate (MTX) administration, NORDiMET® (methotrexate), Nordic Group BV, Hoofddorp, The Netherlands, recently approved by the European Medicines Agency for the management of adult patients with active RA [6]. This labeling occurred after the end of the study. The trial was conducted in 42 subjects, divided into three user groups: a group of patients with established RA subdivided into two groups according to their hand disability, and a group of caregivers or nurses with experience in treating patients directly. The study population is a reflection of the foreseen user population of the device.

Methods

Investigators and Patients

This was a multicenter usability study, performed in France in compliance with Good Clinical Practice according to European directives and French laws (EudraCT reference number: 2014-A0141245). The investigators were three rheumatologists with mixed private/hospital activity. All study assessments were performed during a single consultation.

The trial was carried out with 42 subjects allocated to three user groups. The first two groups were composed of RA patients split by their hand disability measured by Cochin scale [7, 8] (Table 1). Patients with high hand disability (Cochin score ≥20) and low hand disability (Cochin score <20) were enrolled respectively in groups 1 and 2. The third group was composed of professional nurses or caregivers already involved in RA patients’ treatment. For RA patients, any previous treatment by any auto-injector device was an exclusion criterion.

Table 1

The Cochin scale

In the kitchen
1.	Can you hold a bowl?†
2.	Can you grasp a full bottle and raise it?‡
3.	Can you hold a plate full of food?‡
4.	Can you pour liquid from a bottle into a glass?†
5.	Can you unscrew the lid from a jar that has been opened before?‡
6.	Can you cut meet with a knife?‡
7.	Can you prick things well with a fork?†
8.	Can you peel fruit?†
Dressing
9.	Can you button your shirt?†
10.	Can you open and close a zipper?†
Hygiene
11.	Can you squeeze a tube of toothpaste?‡
12.	Can you hold a toothbrush effectively?‡
At the office
13.	Can you write a short sentence with an ordinary pen?§
14.	Can you write a letter with an ordinary pen?§
Other
15.	Can you turn a round door knob?‡
16.	Can you cut a piece of paper with scissors?§
17.	Can you pick up coins from a table top?§
18.	Can you turn a key in a lock?‡

Answers to the questions: 0 = Yes, without difficulty; 1 = Yes, with a little difficulty; 2 = Yes, with some difficulty; 3 = Yes, with much difficulty; 4 = Nearly impossible to do; 5 = Impossible to do

†Question in factor 2

‡Question in factor 1

§Question in factor 3 after factor analysis

Study Product and Simulated Injections

The study product was an autoinjector from SHL Group aimed to MTX subcutaneous (SC) self-administration. It is a single-use, disposable, fixed-dose needle-based injection system with automatic functionality (NIS-AUTO) device (Fig. 1).

Fig. 1

Description of the autoinjector

After the removal of the cap, the button-free device is activated simply by pressing the needle shield directly against the injection site and holding it during the MTX distribution (maximum duration of 10 s). The beginning and the end of the injections are signaled to the subject by sounds, called “clicks” and by slight vibrations.

For this study, the autoinjector has been assembled with a pre-filled glass syringe containing 0.6 ml water for injection. No active MTX was used in the study and only simulated-use of the autoinjector was performed on foam pads mimicking skin behavior (Fig. 2).

Fig. 2

Description of the foam pad

Study Design

All the subjects included in the study were ≥18 years old and were able to understand the study procedures, which was materialized by signing the informed consent. Additional inclusion criteria for the first two groups were: RA, according to 2010 ACR/EULAR classification [9]; Cochin score ≥20 for the first group and Cochin score <20 for the second group; absence of previous treatment by any autoinjector device.

The study consisted of two phases. During the first phase, the instructor (a trained person from the investigator’s staff or the investigator himself/herself) gave the autoinjector and its instructions for use (IFU) to the subject without further explanations. Immediately after reading the IFU, the subject performed the first simulated injection, his/her performances were evaluated by an independent observer using a detailed questionnaire, called “score card” (Table 2).

Table 2

The score card

Before use	Did the subject check through the viewing window if he/she sees a fluid?
	Did the subject check if the tamper-evident seal is intact?
During the simulated injection	Did the subject hold the autoinjector in one hand?
	Did the subject pull off the green cap?
	Did the subject make a “skin” fold on the pad?
	Did he/she make the “skin” fold with the forefinger and thumb?
	Did the yellow needle shield point towards the injection site?
	Did the subject place the autoinjector on the pad in a (approximately) 90° angle?
	Did the subject place the autoinjector on the pad so the entire rim of the yellow needle shield touched the pad?
	Did the subject apply downward pressure on the autoinjector (needle shield fully pushed in) so that a click was heard?
	Did the subject hold the autoinjector with the needle shield fully pushed in until a second click was heard before removing the autoinjector?
	Did the subject wait 2–3 s after the second click before removing the autoinjector?
	Did the subject hold the “skin” fold from before the injection until after he/she removed the autoinjector from the pad?
After use	Did the subject look through the viewing window of the autoinjector after the injection?
	Did the subject put the green cap back on the autoinjector after the injection?

All the questions were answered by: “Yes, immediately”, “Yes, but after some hesitations or problems”, or “No, not at all”. The first two answers were considered as “correct handling”

At the end of the simulated injection, the instructor interviewed the subject and completed the usability questionnaire: part 1 (understanding the IFU and general impression) and part 2 (usability questionnaire) (Table 3).

Table 3

The usability questionnaire

Part	Question	Measure
Part 1: Understanding the IFU (after phase 1)	Did you check the viewing window?	Yes/neutral/no
	If yes, did you see a fluid in the viewing window?
	Did you check the tamper-evident seal of the autoinjector?
	If yes, was the tamper-evident seal intact?
	Could you see on the autoinjector that the injection was made?
	Did you see green plastic after using the autoinjector?
	How useful where the pictures in the IFU?	Adjusted 5-point Likert scale*
	How clear was it how to move the autoinjector towards the site of injection?
	How would you score the setup on the IFU in terms of ability to follow the handlings steps?
Part 2: Usability (after phase 1 and 2)—subject device acceptance and system ease of use	How clear was it to check that the autoinjector was ready for injection?	Adjusted 5-point Likert scale*
	How easy or difficult was it to hold the autoinjector?
	How easy or difficult was it to take the cap off?
	How would you score the force needed to press on the autoinjector in order to inject?
	How would you score the simplicity of the injection process?
	How would you rate the audibility of the first click?
	How well could you feel the first click?
	How would you rate the audibility of the second click?
	How well could you feel the second click?
	After removing the autoinjector, how clear was it to check that the injection was fully given?
Part 3: Satisfaction (after phase 2)—overall satisfaction and willing to accept the device	Overall, how satisfied were you with the autoinjector?	Adjusted 5-point Likert scale*
	How willing are you to use the autoinjector for your own medication?
	How is your overall impression of the autoinjector?
	How would you rate the overall use of the autoinjector?
	How would you rate the safety feature of the autoinjector?
	Would you recommend this device to (other) RA-patients?	Yes/neutral/no

* For example: 1/very useful; 2/useful; 3/no opinion; 4/useless; 5/very useless. Positive answers: “Yes” and the two first points of the adjusted 5-point Likert scale. Neutral answers: “Neutral” or “No opinion” (not taken into account in the total number of answers). Negative answers: “No” and the two last points of the adjusted 5-point Likert scale. Positive response rate: number of positive answers out of the sum of positive and negative answers

At the beginning of the second phase, the instructor could provide explanations concerning the IFU upon request of the subject. Afterwards, the subject performed two simulated injections: in an optimal environment (full light, no noise) and in a “worst-case” home environment (dimmed light, TV or radio on, phone ringing). During both simulations, the independent observer assessed the performance in compliance with the IFU using the same “score card” as during the first phase. At the end of the second phase, the instructor interviewed the subject and completed part 2 of the usability questionnaire and part 3 (satisfaction) (Table 4).

Table 4

Study flowchart

		Evaluation of injection performances by the independent observer with the “Score card”	Usability questionnaire completed by the instructor
			Parts 1 and 2	Parts 2 and 3
First phase: After IFU reading without further explanations		During the simulation	At the end of the first phase	/
Second phase: After explanations provided by the instructor upon request of the subject	Optimal conditions (full light, no noise)	During the simulation	/	At the end of the second phase
	“Worst case” home environment (dimmed light, TV or radio on, phone ringing)	During the simulation	/

The protocol, the informed consent form and all other documents related to the study were submitted for review to Independent Ethics Committee and to ANSM (Agence Nationale de Sécurité du Médicament et des produits de santé). These two regulatory authorities confirmed that this study didn’t meet the biomedical research criteria as defined in Article R.L1121-1 of the French Public Health Code. Nevertheless, the Ethics Committee stated that there was no ethical obstacle to do this study (notification number: MD-092014).

Study Objectives

The primary objective of this study was to evaluate the usability of the new MTX autoinjector based on the rate of correct handling during the second phase “worst-case” injection according to the “score card” and on the rate of positive responses of the usability questionnaire.

The secondary objectives were to assess the understanding of IFU (usability questionnaire part 1), the usability (usability questionnaire part 2), the subject satisfaction (usability questionnaire part 3), and the correct device handling during the first phase and the optimal environment simulation during the second phase.

Determination of the Sample Size and Statistics

According to the recommendations by the FDA, usability studies should include 15 subjects from each major user group [10]. Therefore, initially, three groups of 15 subjects were scheduled to be enrolled in the study. As the investigators had recruitment difficulties in completing the first user group (RA patients with Cochin score ≥20), they decided to stop the recruitment process when 12 subjects were enrolled in this group.

Statistical analyses were performed using SAS® version 9.1. Considering the sample sizes, only descriptive analyses were performed.

The handling steps according to the “score card” were considered correct if the observer filled “Yes, immediately” or “Yes, but after some hesitations or problems”, and incorrect if “Not at all” was filled (Table 2). The rate of correct handling steps for each subject was the number of correct steps out of total number of answers. A rate of 75% was considered as a “correct handling of the device”.

For the usability questionnaires were considered as positive answers “Yes”, when the choice was “Yes/No opinion/No” and the two first points of the adjusted 5-point Likert scale when applicable (Table 3). The neutral answers were “No opinion” and the 3rd point of the Likert scale; the neutral answers have not been taken into account in the total number of answers. Answers “No” and the two last points of the Likert scale were considered as negative. The positive response rate was calculated for each subject as follows: number of positive answers out of the sum of positive and negative answers. A rate of 75% was considered as “positive usability”. Safety analysis was carried out on the full analysis set.

Results

Demographic Data

Forty-two subjects were enrolled in the study between November 2014 and March 2015 by three rheumatologists. A total of 126 simulated injections (three simulations per subject) were performed and scored by the independent observer. The demographic data in the full analysis set (FAS) and in the different user groups are detailed in the Table 5. The mean age in the three user groups displayed no statistically significant differences. Disease duration was different across the groups: in the first user group (Cochin score ≥20) the disease duration was around 20 years, whereas in the second user group (Cochin score <20), it was only 9 years.

Table 5

Demographic data

Characteristic	Group 1RA patients(Cochin score ≥20) N = 12	Group 2RA patients(Cochin score <20) N = 15	Group 3nurses or caregivers N = 15	Full set N = 42
Female, n (%)	9 (75%)	12 (80%)	11 (73%)	32 (76%)
Age (years), mean ± SD	61 ± 12	60 ± 11	54 ± 18	58 ± 14
Age of RA diagnosis (years), mean ± SD	40 ± 16	51 ± 12	NA	NA
Cochin score, mean ± SD	34 ± 12	5 ± 6	NA	NA

Positive Response Rate According to the Usability Questionnaire

After the first injection and without any additional instruction, the great majority of subjects (90.5%) displayed a positive response rate ≥75% for the first two parts of the usability questionnaire (92% of RA patients with high hand disability, 80% of RA patients with mild hand disability, and 100% and caregivers) (Fig. 3). After the second phase, all the subjects of all the three user groups displayed a positive response rate ≥75% concerning the second and the third part of the usability questionnaire, thus meeting one of the primary objectives of the study.

Fig. 3

Percentage of positive responses for the usability questionnaire for the full analysis set

Overall, the usability questionnaire, scored >80% of positive answers for each question, except for the question “Did you check the tamper-evident seal of the autoinjector?” in the Part 1, which scored 64.3% (Fig. 4). Although sample sizes did not allow performing robust statistical investigation, no major differences were descriptively observed among the three user groups.

Fig. 4

Percentage of positive responses for the usability questionnaire for the full analysis set

In the Part 1, 64.3–100% of users indicated that they checked properly the device’s quality, and this before any additional explanation potentially made by the instructor (questions 1–6). The IFU seemed clear and useful for 95–100% of subjects (questions 7–9). In Part 2, concerning the subject device acceptance and ease of use, completed twice (before and after potential additional explanations), five patients in the first group, one in the second group, and one nurse indicated that they met some difficulties with removing the cap from the autoinjector. For all the other questions of this part, the rate of positive answers was >80% after the first phase and improved up to 100% after the second phase. In Part 3, concerning satisfaction, all the questions met 100% of positive responses, with the exception of the last question, which scored 97.5%.

Correct Performance Rate According to the “Score Card”

The great majority of subjects displayed a correct performance rate ≥75% during the first phase (100% of the RA patients with high hand disability, 93% of the RA patients with mild or no hand disability and 93% of the caregivers). During the second phase, 100% of subjects in each of the three user groups displayed a correct performance rate ≥75% in both optimal and “worst-case” environment, thus attaining the co-primary objective of the study (Fig. 5).

Fig. 5

Percentage of subjects achieving over 75% of correct handling steps according to the score card

Overall, the handling steps were conducted successfully; more than 78% success for each step, irrespective of the user group, previous explanations, or environment conditions (Fig. 6). Some difficulties could have appeared during the first phase, especially concerning the viewing window checking and the removal of the device after the injection. All of these issues seemed to disappear during the second phase, after the instructor’s explanations. The injection itself steps were performed correctly for 100% of patients during all the phases. Notably, the environmental conditions during the second phase did not seem to influence the injection performances of the patients. The rate of success in the second phase is over 97% for all the handling steps.

Fig. 6

Percentage of subjects executing correct handling steps for the full analysis set according to the score card

Safety

No adverse events or deaths were reported during the study.

Discussion

The purpose of this multicenter study was to assess the usability and the IFU understanding of a newly developed MTX SC autoinjector in subjects with variable manual dexterity and in different environmental conditions. The different user groups enrolled were comparable in terms of age and sex ratio. The study met the design requirements for the number of subjects per user group [10].

The results demonstrate the high acceptance, usability, and intuitive handling of this new device regardless of the hand disability or environmental conditions. From 9 to 100% of RA patients and caregivers found the IFU clear and useful and were plenty satisfied with the usability and handling of this new MTX autoinjector. In the first phase, which consisted of injection straight after IFU reading without any further explanation, the great majority of subjects displayed a correct performance rate ≥75% according to the “score card” (93–100% of users) and a positive response rate ≥75% for the usability questionnaire (80–100% of users), revealing a very intuitive device handling. These scores grew up to 100% for all the subjects during the second phase, performed after potential instructor’s additional explanations, highlighting the importance of experience and of therapeutic education. The points that needed to be particularly addressed during this training seemed to be device quality and readiness for injection checking and the importance of hearing the “second click”, which notifies the end of the injection. This trial did not raise any safety concerns, showing reliability and robustness of the device tested on 126 simulated injections.

Since few years autoinjector pens have been introduced in the field of rheumatology for biologic and conventional disease-modifying anti-rheumatic drugs. Several studies showed that patients with RA and other chronic disorders, such as diabetes, have been found to prefer autoinjectors when compared to more “classic” devices (vials or syringes), in particular because they seem to be more convenient and easy to use and less painful than prefilled syringes [4, 5, 11–13]. Some patients even preferred these devices over oral treatment and seemed to have improved quality of life [14-16]. All of these data confirm the interest of introducing for RA patients new MTX autoinjectors, such as the one assessed in this trial.

Limitations of this study consisted mainly of the limited number of users, the simulated water injections, the descriptive, more than analytical character of the study, and the lack of comparator. Such a randomized open-label trial is currently ongoing comparing the MTX autoinjector presented here to MTX prefilled syringes in a cohort of 280 patients with active RA (ClinicalTrials.gov Identifier: NCT02553018).

Conclusions

This study found a very high level of usability, acceptance, and satisfaction of a new MTX autoinjector, regardless of the level of manual dexterity or environmental conditions. The high performance rate and compliance with the usability questionnaire during the first phase, consisting of an injection immediately after reading the IFU without further explanations, showed an intuitive device handling and clarity of the IFU. These findings show the interest expressed by RA patients, nurses and caregivers for this new MTX autoinjector.