Usability and Acceptability of the Abatacept Pre-Filled Autoinjector for the Subcutaneous Treatment of Rheumatoid Arthritis.

INTRODUCTION: The purpose of the present study was to determine whether the abatacept autoinjector can be used by the intended population without patterns of preventable use errors, and is acceptable when assessed against key user needs.
METHODS: Two independently conducted simulated-use studies, with no active drug administered, quantified use errors and evaluated the abatacept autoinjector and competitor devices on key attributes (comfort, control, ease of use, confidence of dose) and overall acceptability. Autoinjector preference was also assessed. Participants were patients with rheumatoid arthritis, caregivers, and healthcare professionals (HCPs). Participants were informed that a new rheumatoid arthritis autoinjector was being tested but were blinded to the intended drug and sponsor identity.
RESULTS: In the formative (pre-validation) study (n = 54), two high-priority use errors occurred, both of which resulted from protocol non-compliance rather than mental confusion or physical limitations. In the summative (validation) study (n = 99), one high-priority use error occurred; this was deemed a simulated-use study artifact as participant behavior was guided by actual experience associated with the feel of drug delivery into the skin rather than by protocol, so no mitigation steps were considered necessary. Across user groups, average scores were consistently high for the pre-defined key attributes. Overall acceptability scores (7-point scale) were significantly higher for the abatacept versus competitor autoinjectors-formative study: patients 6.7 vs 5.2 (P = 0.0001), caregivers 7.0 vs 4.6 (P = 0.0093), HCPs 6.8 vs 5.1 (P = 0.0020); summative study: patients 6.5 vs 5.9 (P = 0.0404), caregivers 6.8 vs 5.8 (P = 0.0047), HCPs 6.8 vs 5.1 (P = 0.0002). The abatacept autoinjector was preferred to competitor devices: patients 85.7% vs 14.3% (P = 0.00002), caregivers 84.2% vs 15.8% (P = 0.00443), HCPs 95.0% vs 5.0% (P = 0.00004). Positive experiences with the abatacept autoinjector were attributed to the rubberized grip, device size, visualization of dose progression, button ergonomics, and ease of use.
CONCLUSION: The abatacept autoinjector demonstrated usability without patterns of preventable use errors, and with high acceptability ratings across all key attributes assessed. Preference over competitor autoinjectors was due to device ergonomics, visualization of dose progression, confidence of dose delivery, and overall ease of use. FUNDING: Bristol-Myers Squibb.

Introduction

Abatacept, a fully human fusion protein, is the only biologic for the treatment of rheumatoid arthritis (RA) that selectively modulates the CD80/CD86:CD28 co-stimulatory signal required for full T cell activation and is available in both intravenous and subcutaneous (SC) formulations. The intravenous formulation of abatacept has demonstrated efficacy in several patient populations, including methotrexate-naive patients with early RA [1] and patients with an inadequate response to methotrexate [2-5] or to anti-tumor necrosis factor therapy [6, 7]. SC abatacept has been shown to be non-inferior to intravenous abatacept [8, 9]. The SC formulation, available as a pre-filled syringe, was first approved in the US in July 2011 for the treatment of moderate-to-severe RA in adults. Since then, SC abatacept has received marketing approval for the treatment of adult RA in numerous regions, including the EU, Japan, Canada, and Australia.

Although the SC delivery method affords users the benefit of self-injection at home, the pre-filled syringe requires several hand manipulations. This can be difficult for individuals with RA, as the disease often affects the small joints of the hand and impairs dexterity. To address this limitation and increase options for patients, a pre-filled, single-use autoinjector for abatacept (ClickJect®; Bristol-Myers Squibb, Princeton, NJ, USA) has been developed, with the aims of increasing the ease of the injection process and minimizing use error risk.

The abatacept autoinjector was designed and developed using human factors engineering (HFE). The autoinjector has a large rubberized grip and a hidden needle, and uses an automated injection process including automated delivery of the entire syringe contents. It also incorporates visual feedback to indicate the end of the injection, and shields the needle after injection. These features were designed to facilitate the injection process by improving handling ergonomics and reducing the number of hand manipulations, as well as to help patients who are new to self-injection and to overcome barriers to self-injection such as needle phobia.

This report describes the results of two studies that were conducted to assess the usability and acceptability of the abatacept autoinjector. A formative (pre-validation) study was performed to identify aspects of the product design and instructions for use (IFU) that could be further optimized to reduce the risk of user errors, while a summative study was performed as a final validation of the usability of the product and its labeling.

Methods

Two usability studies—formative and summative—were performed to measure the success of the HFE approach for device development (for details of the HFE, see Supplementary File 1). The goals of these studies were to determine: (1) whether the abatacept autoinjector, when provided with IFU, can be used without patterns of preventable use errors that would cause user or patient harm; and (2) if the autoinjector is acceptable for real-world use based on an assessment of key user needs (i.e., comfort, control, ease of use, confidence of dose) and overall acceptability.

The summative and formative studies were conducted by an independent company, Ipsos-Insight, LLC, separate from the study sponsor (Bristol-Myers Squibb). Participants were informed that the studies were testing a new autoinjector for an RA therapy, but no other specifics were shared. Throughout the studies, participants were blinded to the drug for which the autoinjector was intended, as well as to any sponsor involvement. Informed consent was obtained from all patients for being included in the study; study risks were included in the informed consent.

Study Population

To replicate actual use, the study population included individuals from three defined user groups who met the following criteria: (1) patients with RA: both injection naive and experienced, with a minimum disease duration of 6 months and a diagnosis of moderate-to-severe RA (as evidenced by current treatment with a biologic or non-biologic disease-modifying anti-rheumatic drug); (2) caregivers: both injection naive and experienced, with a family member who has been diagnosed with RA; and (3) healthcare professionals (HCPs): injection experienced, who routinely work with and train patients with RA, and with a minimum of 2 years of experience. For the summative study, a minimum of 15 respondents per user group were recruited in accordance with the US Food and Drug Administration human factors guidance [10].

In each study, users were divided into two groups: trained and untrained (Fig. 1). To reduce the potential for use error, users should be trained, by an HCP, on the correct technique for preparing and performing injections when a therapy is prescribed, prior to the first injection. However, because there is no guarantee that all users will be adequately trained before their first use, the device was also validated with untrained participants. For the formative study (Fig. 1a), the trained arm comprised only patients; for the summative study (Fig. 1b), the trained arm included both patients and caregivers. Because most HCPs are not expected to receive any training on how to use the autoinjector, this group was not represented in the trained arm of either study. Prior to the studies, representatives from Bristol-Myers Squibb trained a nurse educator on the correct use of the autoinjector. The nurse educator in turn trained patients and caregivers using only the IFU and a device, to ensure the training was representative of that available in a routine clinical setting.

Fig. 1

Trained and untrained user groups by study type. HCPs healthcare professionals

Use Environment

As these were simulated-use studies, efforts were made to simulate an end user’s actual use environment, and the study materials were designed to replicate the ergonomics and steps of actual administration. Injections are typically performed in low-traffic, low-distraction, low-noise settings, with normal office environment lighting conditions, and this environment was therefore replicated at the market research facilities where the studies were conducted. Study materials included an injection pad, which was strapped to the desired injection location (abdomen or thigh) in place of the injection site, a refrigerator (unplugged) for device storage, a hand sanitizer to mimic hand washing, and production-grade devices filled with medical-grade silicone oil to simulate the viscosity of abatacept and, hence, the injection time. The devices did not contain active therapy.

Study Procedures and Assessments

During the formative and summative studies, each participant was provided with the IFU and a device, and was asked to perform the necessary steps per the IFU to simulate an injection. The investigators evaluated each step (based on a risk analysis conducted prior to the study; see Supplementary File 1) using an assessment checklist, based on the perceptual, cognitive, and physical requirements of each step. High-priority tasks were defined as those that required correct completion for successful dose delivery.

The investigators then rated the observed performance as ‘correct performance’ (A), ‘performed with difficulty’ (D), or ‘use error’ (UE). A ‘performed with difficulty’ (D) classification included all close calls and near misses in which the user experienced confusion or difficulty, misinterpreted the IFU, or made an error that would result in mistreatment or harm, but then recovered and was able to continue so that no actual performance failure occurred. If a performance error occurred, but the user self-identified that the error had been made without prompting, it was also considered a ‘performed with difficulty’ (D) error, since the individual acknowledged the incorrect step and would self-correct with subsequent usage. A ‘use error’ (UE) classification included instances where the participant did not complete the step as appropriate. Following the usability assessment, the investigator and participant discussed the underlying causes of any difficulties (D) or use errors (UE) encountered during the simulated-use assessments, to ascertain if modifications could be incorporated into the product design or IFU to further optimize ease of use.

Participants were also asked to rate the acceptability of the autoinjector against a series of key attributes (i.e., comfort, control, ease of use, confidence of dose) and overall acceptability using a 7-point scale (1 = very unacceptable, 4 = neutral, and 7 = very acceptable). If time allowed, those participants who were experienced with an RA autoinjector were provided with their current or most recent autoinjector, allowed to re-familiarize themselves with it, and asked to provide competitive ratings. The participants did not perform a simulated injection with the competitor autoinjector. Participants experienced with an RA autoinjector were also asked for their preference of autoinjector based on their experiences during the simulated-use assessment.

A paired t test was used to identify statistical significance (P < 0.05) for the competitive rating analysis, while an exact binomial test was used for the user preference analysis. Finally, respondents were asked to name key positive features of the abatacept autoinjector, based on their simulated-use experience.

Results

The formative (pre-validation) study was conducted from February to March 2014 at market research centers in Fort Lee, NJ, USA; Fort Lauderdale, FL, USA; and San Francisco, CA, USA. A total of 34 patients with RA, 10 caregivers, and 10 HCPs were recruited (Fig. 1a). The summative (validation) study was conducted from August to September 2014 at market research centers in Fort Lee, NJ, USA; Baltimore, MD, USA; Stamford, CT, USA; and Dallas, TX, USA. A total of 51 patients with RA, 33 caregivers, and 15 HCPs were recruited (Fig. 1b). Respondents from Fort Lee, NJ, USA who had participated in the formative study were not eligible to participate in the summative study. The participant demographics are shown in Table 1.

Table 1

Participant demographics of (A) patients with RA, (B) caregivers, and (C) healthcare professionals

Parameter, %a		Formative study		Summative study
		Untrained (n = 24)	Trained (n = 10)	Untrained (n = 35)	Trained (n = 16)
(A) Patients with RA
Injection experience	Naive	33	30	20	25
	Experienced	67	70	80	75
Injection experience device typeb	Autoinjector	56	43	68	75
	Pre-filled syringe	44	57	32	25
Age, years	18–39	4	0	6	6
	40–49	38	10	23	25
	50–59	33	60	34	50
	60–69	21	30	29	19
	70–75	4	0	9	0
Sex	Female	71	80	74	81
	Male	29	20	26	19
Disease duration, yearsc	–	6	11	8	5
Education	High-school graduate	21	10	11	19
	Some college	29	60	40	31
	College graduate/post-graduate	50	30	48	50
Dexterity	No difficulty	21	0	11	6
	Slight/moderate difficulty	50	60	57	75
	Considerable difficulty	29	40	31	19
Current medication	MTX	54	50	43	44
	Prednisone	8	0	0	6

MTX methotrexate, RA rheumatoid arthritis

aPercentage values are presented, unless otherwise noted

bOf injection-experienced participants: formative untrained, n = 16; formative trained, n = 7; summative untrained, n = 28; summative trained, n = 12

cMedian values are presented

Simulated-Use Usability Assessment

Most participants completed each individual step of the simulation task without use errors (Table 2). In the formative study, two use errors were observed with the high-priority injection steps, both related to the respondent [one untrained patient and one HCP (i.e., untrained)] not holding the device in place long enough to complete the injection. Although these errors were observed, post-study actions were not taken to modify the device design or labeling, since the root-cause analysis identified the outcome to be a result of the respondents’ non-compliance to the protocol and study methodology, rather than a result of mental confusion or physical limitations. However, minor layout changes were made to the IFU to improve compliance with two (low-priority risk) use steps: ‘pinch skin’ and ‘check the autoinjector for damage’.

Table 2

Frequency of completion of individual use steps during the formative and summative studies

Task, %	Formative study				Summative study
	Patients		Caregivers	Healthcare professionals	Patients		Caregivers		Healthcare professionals
	Untrained (n = 24)	Trained (n = 10)	Untrained (n = 10)	Untrained (n = 10)	Untrained (n = 35)	Trained (n = 16)	Untrained (n = 17)	Trained (n = 16)	Untrained (n = 15)
Take autoinjector out of refrigerator; wait 30 min	96	100	90	100	97	100	88	100	93
Wash hands	92	90	100	100	94	100	100	100	93
Check expiration date	96	100	100	100	91	100	94	100	93
Check autoinjector for damage	79	80	100	100	89	100	94	100	87
Check the liquid	92	100	100	100	97	100	94	100	100
Choose an injection site	96	100	100	100	100	100	100	100	100
Clean the site	96	100	100	100	97	100	82	100	93
Remove needle covera	100	100	100	100	100	100	100	100	100
Position the autoinjector	100	100	100	100	100	100	100	100	100
Pinch skin	88	100	80	100	91	100	100	94	93
Push down to unlock the autoinjectora	100	100	100	100	100	100	100	100	100
Press buttona	100	100	100	100	100	100	100	100	100
Hold for count and/or wait until blue indicator stops movinga	96	100	100	90	97	100	100	100	100
Dispose used autoinjector into sharps container	100	100	90	100	94	100	100	100	100

Steps classified as ‘A—correct completion’ plus ‘D—performed with difficulty’

aTasks highlighted in bold are high-priority tasks, defined as those that required correct completion for successful dose delivery

In the summative study, all participants except one (an untrained patient) completed all of the high-priority injection steps (Table 2). The exception related to the ‘hold for count and/or wait until blue indicator stops moving’ step. The use error was deemed an artifact of the study, as the patient was currently using a competitor autoinjector and was applying their expectations of the ‘feel’ in the skin of an injection to indicate the appropriate hold time. Since this tactile end-of-dose indication does not apply to a simulated-use setting, no mitigation steps were deemed necessary.

Simulated-Use Acceptability Assessment

Average acceptability scores for the device were consistently high for each user group for all measures (comfort, control, ease of use, confidence of dose, overall acceptability) (Table 3). In the formative study, all 54 participants responded (34 patients, 10 caregivers, and 10 HCPs); mean overall acceptability scores (out of 7) for patients, caregivers, and HCPs were 6.6, 7.0, and 6.8, respectively. In the summative study, 94 out of 99 participants responded (48 patients, 31 caregivers, and 15 HCPs); mean overall acceptability scores for patients, caregivers, and HCPs were 6.6, 6.8, and 6.8, respectively.

Table 3

Participant user experience data

Rating	Formative study			Summative study
	Patients (n = 34)	Caregivers (n = 10)	Healthcare professionals (n = 10)	Patients (n = 48)	Caregivers (n = 31)	Healthcare professionals (n = 15)
Comfort	6.5 (0.8)	6.6 (1.0)	6.8 (0.6)	6.4 (1.0)	6.3 (1.0)	6.9 (0.4)
Control	6.6 (0.9)	6.8 (0.4)	6.7 (1.0)	6.9 (0.4)	6.8 (0.5)	6.8 (0.4)
Ease of use	6.7 (0.8)	6.9 (0.3)	6.8 (0.4)	6.6 (0.8)	6.9 (0.3)	6.7 (0.6)
Confidence of dose	6.8 (0.5)	7.0 (0.0)	6.4 (1.6)	6.6 (0.7)	6.9 (0.4)	6.7 (0.6)
Overall acceptability	6.6 (0.8)	7.0 (0.0)	6.8 (0.4)	6.6 (0.6)	6.8 (0.5)	6.8 (0.4)

All values are expressed as mean (standard deviation)

Each score was quantified using a 7-point scale, where 1 = very unacceptable, 4 = neutral, and 7 = very acceptable

Current Autoinjector User Subgroup Analysis

Among patients who had previously administered injections with a competitor autoinjector, overall mean acceptability ratings were significantly greater for the abatacept autoinjector versus competitor autoinjectors across all user groups in both studies [formative study: patients 6.7 vs 5.2 (P = 0.0001), caregivers 7.0 vs 4.6 (P = 0.0093), HCPs 6.8 vs 5.1 (P = 0.0020); summative study: patients 6.5 vs 5.9 (P = 0.0404), caregivers 6.8 vs 5.8 (P = 0.0047), HCPs 6.8 vs 5.1 (P = 0.0002); Table 4]. In the individual categories of comfort, control, ease of use, and confidence of dose, all groups rated the abatacept autoinjector at least the same or superior to competitor autoinjectors; this difference was significant for almost all parameters, across all user groups, and in both studies (Table 4).

Table 4

Comparative user experience data following abatacept autoinjector use in (A) the formative study or (B) the summative study, from participants who had previously administered injections with an alternative autoinjector

Rating	Patients (n = 12)			Caregivers (n = 5)			Healthcare professionals (n = 10)
	Mean abatacept autoinjector rating	Mean competitor autoinjector rating	Mean paired difference	Mean abatacept autoinjector rating	Mean competitor autoinjector rating	Mean paired difference	Mean abatacept autoinjector rating	Mean competitor autoinjector rating	Mean paired difference
(A) Formative study
Comfort	6.6	4.6	P = 0.0033	7.0	4.2	P = 0.0135	6.8	4.5	P < 0.0001
Control	6.8	4.6	P = 0.0025	7.0	4.0	P = 0.0132	6.7	4.6	P = 0.0095
Ease of use	7.0	5.9	P = 0.0006	7.0	5.2	P = 0.0533	6.8	5.2	P = 0.0046
Confidence of dose	6.8	6.3	P = 0.0819	7.0	5.8	P = 0.2080	6.4	5.6	P = 0.2999
Overall acceptability	6.7	5.2	P = 0.0001	7.0	4.6	P = 0.0093	6.8	5.1	P = 0.0020

Each score was quantified using a 7-point scale, where 1 = very unacceptable, 4 = neutral, and 7 = very acceptable

Analyses were restricted to those participants who provided both abatacept and competitor autoinjector ratings

Participants demonstrated a strong preference for the abatacept autoinjector over competitor autoinjectors (Table 5). When preference data from the two studies were combined, a significantly greater number of participants in each user group preferred the abatacept autoinjector over their current or most recent autoinjector [patients 85.7% vs 14.3% (P = 0.00002), caregivers 84.2% vs 15.8% (P = 0.00443), HCPs 95.0% vs 5.0% (P = 0.00004)]. For the individual studies, this preference remained significant for patients in the formative (P = 0.00049) and summative (P = 0.01062) studies, and for HCPs in the summative study (P = 0.00049). Although the other comparisons were not significant, all preference data showed at least a trend in favor of the abatacept autoinjector in both studies.

Table 5

Autoinjector preferences from participants who had previously administered injections with an alternative autoinjector

Respondents, n (%)	Formative study			Summative study			Total of formative and summative studies
	Abatacept autoinjector preference	Competitor autoinjector preference	Test preference of abatacept autoinjectora	Abatacept autoinjector preference	Competitor autoinjector preference	Test preference of abatacept autoinjectora	Abatacept autoinjector preference	Competitor autoinjector preference	Test preference of abatacept autoinjectora
Patients	12 (100)	0 (0.0)	(73.54, 100.00); P = 0.00049	18 (78.3)	5 (21.7)	(56.30, 92.54); P = 0.01062	30 (85.7)	5 (14.3)	(69.74, 95.19); P = 0.00002
Caregivers	5 (100)	0 (0.0)	(47.82, 100.00); P = 0.06250	11 (78.6)	3 (21.4)	(49.20, 95.34); P = 0.05737	16 (84.2)	3 (15.8)	(60.42, 96.62); P = 0.00443
HCPs	7 (87.5)	1 (12.5)	(47.35, 99.68); P = 0.07031	12 (100)	0 (0.0)	(73.54, 100.00); P = 0.00049	19 (95.0)	1 (5.0)	(75.13, 99.87); P = 0.00004

aDisplayed as (95% confidence interval); P value

Key Positive Features

In total, 31 formative study participants and 87 summative study participants reported on the positive features of the abatacept autoinjector. The most frequently noted positive features were: rubberized grip (formative study, noted by 58% of respondents; summative study, 56%), device size (formative, 55%; summative, 52%), visualization of dose progression (window size and location, colored plunger rod; formative, 45%; summative, 57%), button ergonomics (shape, reach, activation force; formative, 19%; summative, 37%), and ease of use (number of steps, simplicity; formative, 29%; summative, 30%) (Fig. 2).

Fig. 2

Key positive factors of the abatacept autoinjector reported by respondents

Discussion

The independently conducted studies reported here found that the newly designed autoinjector for the SC delivery of abatacept in patients with RA was easy to use, with low residual risk to users in a real-world setting. Most participants, whether patients, caregivers, or HCPs, trained or untrained, performed all high-priority use steps correctly. Participants rated the abatacept autoinjector highly for user experience in terms of comfort, control, ease of use, confidence of dose, and overall acceptability, with scores nearing the maximum of the 7-point scale used. All user groups across both studies rated the abatacept autoinjector significantly higher than competitor devices for overall acceptability, and when preference data from both studies were combined, significantly more participants in each user group preferred the abatacept autoinjector over their current or most recent device. The drivers for the positive ratings and preference could be grouped into three main categories: device ergonomics (size, rubberized grip, button reach, force to activate), which provides comfort and security during injection; visualization of dose progression (size and location of window, colored plunger rod), which provides confidence of dose delivery; and simplicity of the process, which contributes to the overall ease of use of the device.

Because RA can affect the small joints of the hands, many patients with RA suffer from compromised dexterity. In addition to interfering with activities of daily living, poor dexterity can affect the ability to perform the steps required for self-injection in patients who would otherwise be eligible for an SC therapy. Autoinjectors are available for the administration of many SC therapies for chronic conditions, including those in which patients have impaired dexterity [11, 12]. The single-use, pre-filled autoinjector described here was developed following the principles of HFE, to help users to inject one dose of SC abatacept in a home environment. As evidenced by the results of the current studies, the new autoinjector is an important advance for patients receiving, or eligible to receive, SC abatacept, as it increases the ease of the injection process without introducing usability challenges.

In both the formative and summative studies, the vast majority of respondents completed each individual step of the simulated injection without use errors. These results demonstrate that, overall, both trained and untrained users were able to operate the autoinjector safely and effectively to deliver the SC injection by following the IFU.

Furthermore, the overall preference in favor of the abatacept autoinjector over other available injection devices across a range of attributes indicates that the device’s features were favored by users with experience of such devices. The fact that similar results were obtained across the two studies provides further credence to these findings. These data also indicate that the device features designed in early development (i.e., shape, rubberized grip, placement/size of window, colored plunger rod, button ergonomics) contributed to the favorable comfort, control, confidence of dose, ease of use, and acceptability ratings for the device, indicating value by the end users. Furthermore, as no patterns of error were observed in the studies, use risk was mitigated.

A limitation of these evaluations is the simulated-use design of the studies, rather than autoinjector use in a clinical setting. However, according to US Food and Drug Administration guidance, simulation testing, as conducted here, is an acceptable method for assessing the safe and effective use of an autoinjector device. Greater support for the ease of use of the autoinjector may have been obtained by restricting the study population to patients with more severe hand deformity, for whom any improvement would be most beneficial. In addition, respondent numbers were low in the subgroup analyses. A further limitation is the lack of a simulated injection using the competitor autoinjector device; permitting a simulated injection with a competitor device may have allowed for a more accurate comparison with the abatacept autoinjector versus relying on participant memory. The time span between using and rating the competitor device may have influenced the comparative data; therefore, not capturing and adjusting for this time difference may affect the interpretation of these results. Lastly, the autoinjector was branded in order for participants to evaluate its colored design features. As such, HCPs may have ascertained which product the autoinjector was designed to deliver.

Conclusions

The abatacept autoinjector was found to be highly acceptable against key measures of comfort, control, ease of use, confidence of dose, and overall acceptability in two independently conducted simulated-use studies. High overall acceptability ratings were achieved, and these ratings were significantly greater than those for competitor devices. In addition, a significantly greater number of participants in each user group preferred the abatacept autoinjector over their current or most recent autoinjector.

Participants’ positive experiences with the abatacept autoinjector can be attributed to the following key features: device ergonomics (size, shape, rubberized grip, button reach, and activation force), which provides comfort and security during injection; visualization of dose progression (size and location of window, colored plunger rod), which provides confidence of dose delivery; and simplicity of the process, which contributes to the overall ease of use of the device. These results show that HFE optimized the device design and IFU of the abatacept autoinjector to ensure its effective use by patients, caregivers, and HCPs without patterns of preventable use errors.

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 313 kb)

Supplementary material 2 (DOCX 1338 kb)