Development of a measure of prescriber satisfaction with academic detailing: the PSAD.

INTRODUCTION: Academic detailing (AD) is an educational outreach strategy to provide clinicians with current evidence-based information, which has been shown to change prescribing behaviours. The overall effectiveness of AD interventions is associated with prescriber satisfaction; however, most approaches use single items or non-validated measures. This study aims to develop and validate an instrument to assess prescriber satisfaction with AD interventions.
METHODS: A group of candidate items was generated and refined based on constructs identified through a literature review and in consultation with an expert panel. The initial instrument was piloted with 183 primary care providers who participated in an AD intervention on opioid-related pain management. To support the validity and reliability of the measure, psychometric properties were examined.
RESULTS: Ten candidate items were developed based on the following themes: acceptability, feasibility of implementation, usefulness, perception of efficacy, overall satisfaction, willingness to repeat and willingness to change. One item related to willingness to change did not contribute to assessing an individual's ability and lowered the measure's internal consistency and was therefore dropped.
CONCLUSION: Results supported the validity and reliability of a refined 9-item measure of Provider Satisfaction with Academic Detailing (the PSAD). This measure should be considered for broad use across educational outreach programmes as a standardized measure to assess provider satisfaction and provide continuous quality improvement.

Introduction

Academic detailing (AD) is an educational outreach strategy used to provide healthcare professionals with up-to-date, unbiased, evidence-based information that can improve patient care and health outcomes.1,2 AD interventions have been shown to facilitate the dissemination and implementation of clinical protocols and therapeutic guidelines, positively impact the quality of care and successfully change prescribing behaviour across several clinical areas.3–7 AD interventions are commonly delivered in one-on-one interactions between a trained ‘academic detailer’ (usually, a healthcare professional) and a healthcare provider, which are typically referred to as ‘visits’.3 AD visits usually take place in the setting where the participants conduct their practices (e.g. hospital, clinic, office) and a single AD programme can be implemented in one or more visits, contingent on its purposes, resources and other logistic considerations.3

Arguably, healthcare providers’ receptiveness to AD is key to the prospects of successful implementation of AD programmes. The Implementation Research Outcomes Model proposed by Proctor et al.8 identifies several outcomes critical to evaluating successful implementation: acceptability, adoption, appropriateness, costs, feasibility, fidelity, penetration and sustainability. This conceptual model proposes that the level of analysis for several of these outcomes is the individual provider being targeted by the programme, namely for the outcomes of acceptability, adoption, appropriateness, feasibility and fidelity. These same constructs appear in the implementation research literature under the canopy of the ‘satisfaction’ construct.9,10 The effectiveness of AD interventions has also been shown to be associated with overall prescriber satisfaction but typically single item measures or non-psychometrically validated surveys are used.11–13 Several studies also reported having used prescriber satisfaction to monitor the implementation of AD programmes and inform continuous quality improvement.10,12 Given the importance of satisfaction as a process outcome, as well as a potential predictor of the programme’s effectiveness, there is a need for a valid and reliable measure of prescriber satisfaction with AD. This study aimed to develop a new measure to assess prescriber satisfaction with AD programmes and test its psychometric properties.

Methods

The measure of Prescriber Satisfaction with Academic Detailing (PSAD) was developed under the umbrella of a Centers for Disease Control and Prevention (CDC)-funded AD programme targeting opioid prescribers in the state of Illinois. Briefly, this programme entailed assessing the impact of providing AD to primary care providers (PCPs) on opioid prescribing patterns. Further details of the original study were published elsewhere.14 Although this measure was first developed for an opioid-specific AD programme, our goal was to develop a measure that could be used to assess the satisfaction of recipients of AD across a wide range of topics and clinical specialties. The development of the PSAD instrument involved 6 stages. The first three stages of the process pertain to the development of the initial group of candidate items, stages 4 and 5 relate to the assessment of the psychometric properties of the instrument and, finally, stage 6 pertains to the refinement of the final measure. The study protocol was reviewed and approved by the UIC Institutional Review Board. Participants provided written informed consent at the beginning of the first visit.

Development of the initial group of candidate items

The need for the development of this new measure arose organically in the first stages of the development of the protocol for the overarching study. According to the conceptual model for the assessment of implementation research outcomes, outcomes of feasibility, acceptability, adoption, appropriateness and fidelity are relevant to the measurement of the success of the implementation of AD programmes such as ours. After an extensive literature search failed to identify any validated instruments suitable to the assessment of providers’ satisfaction with AD programmes, we decided to develop a novel instrument for this purpose.

Stage 1. Literature review

The first step of this endeavour consisted of a literature review to identify and describe themes related to prescribers’ satisfaction with AD and educational outreach interventions in the literature. A structured literature search was developed to retrieve manuscripts published until April 2018 that reported having assessed physician’s satisfaction with AD and educational outreach interventions. The search strategy included a combination of the following terms, as text words and MeSH terms: “academic detailing”, “educational outreach”, “experience”, “satisfaction” and “acceptability”. We included qualitative, quantitative or mixed-methods studies. Reports that included the assessment of constructs related to satisfaction (e.g. acceptability, experience) were also eligible. In addition, the references of articles that met the inclusion criteria were scanned to identify additional sources. Two independent reviewers (ST and AR) screened the titles identified, and discrepancies were resolved by a third researcher (AM).

Stage 2. Development of an initial set of candidate items

The development of candidate items was based on the following elements: (1) instruments or questions used to measure the same construct in other AD studies; (2) themes identified in the literature review. An initial set of candidate items was developed in a core-group meeting based on collective input and consensus. The core group of researchers includes members with expertise in educational measurement, psychometrics and survey development. At this stage, an ordinal rating response scale, ranging from ‘not at all’ = 1, ‘slightly’ = 2, ‘moderately’ = 3, ‘very’ = 4 and ‘extremely’ = 5, was selected to accompany these items.

Stage 3. Content-experts consultation (face and content validity)

Face and content validity of the items as well as response scales included in the initial instrument were evaluated by an expert panel consisting of one pharmacist, two physicians and one advanced nurse practitioner with extensive clinical expertise in chronic pain management. The content experts were asked to comment on whether the items were adequately worded and covered the content of interest. During this stage, the experts consulted were also invited to propose new themes and items, if they thought it was relevant.

Psychometric assessment

Stage 4. Study sample and data collection

At this stage, a set of 10 candidate items was field tested. Data were collected from a sample of PCPs within a large healthcare system located in the greater Chicagoland area. Participants received an AD intervention on safe and appropriate opioid prescribing (implemented in a two-visit format) administered by trained detailers with a pharmacy background. Detailers were trained to deliver key messages from the CDC’s Guidelines for Prescribing Opioids for Chronic Pain.15 Respondents were invited to complete the survey by themselves at the end of each visit and instructed to seal it in a manila envelope before returning it to the detailer.

Stage 5. Analysis

Reliability

A scale’s reliability refers to the proportion of the variance in the scale scores that can be attributed to the true score in the latent variable rather than to random error.16 In this paper, we report the examination of the PSAD’s internal consistency reliability. The internal consistency reliability assesses the homogeneity of the items that comprise a scale; scales are said to be internally consistent if the items are highly correlated with each other. To investigate the PSAD internal consistency reliability, we looked at inter-item correlations, item–total correlations (crude and corrected) and at Cronbach’s α. Due to the categorical nature of the items’ response scale, inter-item correlation matrix and item–total correlations were estimated using Spearman rank correlation coefficient.

Validity

An instrument is considered valid if it measures what it proposes to measure, or in other words, if the subject’s position in the latent trait is the underlying cause of the observed item variance.

Construct validity

The examination of construct validity assesses how well an instrument measures the underlying construct of interest. It involves the specification of factors/constructs that account for the variance of the item’s responses as well as the hypothesized relationships amongst them.16–18

In this paper, we report the assessment of three aspects of construct validity: factor analysis, convergent validity and criterion validity. However, it should be noted that, to date, there is no published gold standard for the assessment of the satisfaction of subjects involved in AD interventions. Accordingly, for our instrument, the psychometric validity can arguably only be implied rather than proved.

Factor analysis

Exploratory factor analysis (EFA) was used to explore the factor structure of the measure. In preparation for the EFA, the Kaiser–Meyer–Olkin measure of sampling adequacy test (KMO) was used to evaluate the suitability of the respondent data for factor analysis and the extent of the interrelation of various variables.19 Briefly, the KMO index ranges from 0 to 1, with values above 0.50 being considered acceptable for factor analysis.19 According to our literature search, we hypothesized that all items of the questionnaire would load in a single underlining dimension. The factor analysis also served as a basis to corroborate the unidimensionality of the scale. A polychoric correlation matrix was used to account for the categorical nature of the item data.20 The final model was developed using principal axis factoring, rather than maximum likelihood, due to the non-parametric distribution of our data. Regarding the rotation method applied to the correlation matrix, we first attempted to use an oblique rotation (Promax) that allows for correlations between factors, which is expected when dealing with data of this nature.21,22 After obtaining a single factor, we switched to an orthogonal rotation (Varimax) to allow for a simpler interpretation.21

Convergent validity

Convergent validity measures the degree of overlap between measures of theoretically related constructs.23 For this purpose, we estimated the correlation between our instrument and a measure of detailers’ perceived efficacy of the visit (DAVE).24 Whilst the constructs assessed by these two instruments do not perfectly overlap, both instruments measure aspects related to the acceptability and feasibility of AD interventions. The convergence between the overall score of the PSAD and the overall score of the DAVE was estimated using Pearson product-moment correlation coefficient. The overall scores were calculated by summing the scores obtained in each item. Correlation coefficients were considered strong if ≥0.7, moderate if <0.7 and ≥0.3, or weak if <0.3.25 High and moderate correlations indicate the measures are assessing similar constructs.

Criterion-related validity (predictive validity)

Criterion validity assesses whether measures have an empirical association with an external criterion or gold standard.16 In this study, we hypothesized that high reported satisfaction with the intervention would be positively associated with individual’s likelihood to participate in a second visit. To test this hypothesis, we used independent samples t-test to compare the mean PSAD baseline scores of individuals that accepted participating in a second visit with those who did not.

Item response theory

Item response theory models were applied to further examine the dimensionality of the measure and to explore the performance of the items in terms of the amount of information that each item yields for the estimation of the individuals’ ability levels. To achieve this purpose, a Graded Response Model was fitted to the data. Item and test information functions were estimated and graphically plotted to assess the amount of information yielded by each item for the estimation of the latent trait (Theta). In item response theory, the term ‘information’ is used to designate the precision with which the latent trait is measured across different levels of ability when using a given item. Therefore, a high information value is associated with a standard error of measurement.26 The test information function is given by the sum of item information functions.

Results

Literature review

The search strategy retrieved 66 unique titles. After applying the inclusion and exclusion criteria defined in the review protocol, seven titles were retained for further inspection. The following six constructs were identified in this stage: acceptability, feasibility, usefulness, perception of efficacy, overall satisfaction with the quality of the interviews, and willingness to repeat the experience.

Item development

During the first stage of item development, eight candidate items were proposed based on the constructs identified in the literature review (Figure 1). The content experts enrolled in the consultation stage confirmed the face and content validity of the candidate items. Two additional items related to the prescribers’ willingness to change their practice and the consistency of the message with their practice were generated after experts’ consultation. Figure 1 shows the list of candidate items that were carried out to the field-testing stage.

Figure 1

Final pilot instrument for AD programme.

Sample population and survey administration

Data were collected between June and August 2018. In total, 183 PCPs participated and completed the PSAD at the first visit, whilst 157 participated and completed the PSAD at both the first and second visits (84.9% retention rate). Table 1 shows that the study sample, at the time of the first visit, had a mean time of practice experience of 14.9 (SD, 11.9) years. Approximately 55% of the healthcare providers recruited were women; in terms of their type of practice, most respondents were Doctors of Medicine (52%) and Doctor of Osteopathic Medicine (33.3%). First visits had an average length of 14.3 (SD, 0.2) minutes, whilst second visits lasted for 11.6 (SD, 0.2) minutes on average. More information about our study sample can be found in Table 1.

Table 1

Provider characteristics for visits 1 and 2.

Provider characteristics	Visit 1 (n=183)	Visit 2 (n=157)
Gender, n (%)
Women	101 (55.19)	85 (54.14)
Men	82 (44.81)	72 (45.86)
Provider type, n (%)
DO	61 (33.33)	51 (32.48)
MD	96 (52.46)	83 (52.87)
NP	18 (9.84)	16 (10.19)
PA	8 (4.37)	7 (4.46)
Resident, n (%)	31 (16.94)	19 (12.10)
Years of practice, mean (SD)	14.92 (11.88)	15.31 (11.59)

DO, Doctor of Osteopathic Medicine; MD, Doctor of Medicine; NP, Nurse Practitioner; PA, Physician Assistant; SD, standard deviation.

Psychometric analysis

Table 2 shows the distribution of responses to each of the candidate items for visits 1 and 2. All items, except for item 9 (likelihood to change), present a left-skewed distribution with more than half of participants selecting the option ‘extremely’, the most positive extreme of the response scale. This pattern of response could be indicative of the presence of a ceiling effect. Another aspect of response patterns that were inspected in this analysis was the missingness, given its theoretical association with the acceptability of items. Notably, all the candidate items displayed a low proportion of missing values (≤1.64%). The mean total score was 43.8 (SD, 6.1) for visit 1 and 44.2 (SD, 6.0) for visit 2.

Table 2

Descriptive statistics for visits 1 and 2.

Visit 1 (n=182)	1: Not at all	2: Slightly	3: Moderately	4: Very	5: Extremely	Missing	Item mean score (SD)
	n (%)	n (%)	n (%)	n (%)	n (%)	n (%)
1: The detailer was knowledgeable	0 (0)	1 (0.6)	11 (6.0)	67 (36.6)	103 (56.3)	1 (0.6)	43.8 (6.13)
2: The detailer was an effective communicator	0 (0)	1 (0.6)	5 (2.7)	58 (31.7)	118 (64.5)	1 (0.6)
3: Academic detailing is an effective way to stay updated on important topic(s)	1 (0.6)	5 (2.7)	26 (14.2)	52 (28.4)	98 (53.6)	1 (0.6)
4: The printed material was useful	1 (0.6)	7 (3.8)	26 (14.2)	52 (28.4)	95 (51.9)	2 (1.1)
5: I would be receptive to future visits	4 (2.2)	4 (2.2)	30 (16.4)	59 (32.2)	84 (45.9)	2 (1.1)
6: This topic was relevant to my practice	0 (0)	3 (1.64	16 (8.7)	55 (30.1)	107 (58.5)	2 (1.1)
7: This is an important topic	0 (0)	0 (0)	4 (2.2)	48 (26.2)	130 (71.0)	1 (0.6)
8: The key messages are feasible to implement in my practice	1 (0.6)	2 (1.2)	14 (7.7)	53 (28.9)	110 (60.1)	3 (1.6)
9: My practice is likely to change as a result of this visit	11 (6.0)	27 (14.8)	52 (28.42	35 (19.1)	55 (30.1)	3 (1.6)
10: The key messages were consistent with my practice	0 (0)	0 (0)	14 (7.7)	53 (28.9)	115 (62.8)	1 (0.6)
Visit 2 ( n =158)
1: The detailer was knowledgeable	0 (0)	0 (0)	8 (5.1)	44 (28.0)	104 (66.2)	1 (0.6)	44.2 (6.01)
2: The detailer was an effective communicator	0 (0)	1 (0.6)	6 (3.8)	42 (26.8)	108 (68.8)	1 (0.6)
3: Academic detailing is an effective way to stay updated on important topic(s)	1 (0.6)	1 (0.6)	17 (10.8)	43 (27.4)	94 (59.9)	1 (0.6)
4: The printed material was useful	0 (0)	4 (2.6)	19 (12.1)	42 (26.8)	92 (58.6)	0 (0)
5: I would be receptive to future visits	6 (3.8)	8 (5.1)	32 (20.4)	37 (23.6)	73 (46.5)	1 (0.6)
6: This topic was relevant to my practice	1 (0.6)	2 (1.3)	17 (10.8)	47 (29.9)	89 (56.7)	1 (0.6)
7: This is an important topic	0 (0)	0 (0)	5 (3.2)	42 (26.8)	110 (70.1)	0 (0)
8: The key messages are feasible to implement in my practice	0 (0)	3 (1.9)	17 (10.8)	47 (29.9)	90 (57.3)	0 (0)
9: My practice is likely to change as a result of this visit	2 (1.3)	24 (15.3)	39 (24.8)	31 (19.8)	61 (38.9)	0 (0)
10: The key messages were consistent with my practice	0 (0)	3 (1.9)	15 (9.6)	45 (28.7)	92 (58.6)	2 (1.3)

m= mean, SD = Standard Deviation

Crude and corrected item–total correlations as well as Cronbach’s α estimates for visits 1 and 2 are presented in Table 3. In this instance, high positive values for the item–total correlation indicate high internal consistency suggesting that the items are discriminating well between highly and meagrely satisfied participants. Results for both visits consistently show that all items display strong correlations with the scale total, even when the variance of the item is removed from the total (corrected item–total correlations). The differences between crude and corrected item–total correlations are small for all items except for item 9 (likelihood to change). In respect to Cronbach’s α calculated for the full set of candidate items, high values were obtained in both visit 1 (α=0.9265) and visit 2 (α=0.9316), indicating the instrument’s internal consistency reliability. Having the values obtained for the full set of items as reference, we verified that Cronbach’s α values decrease slightly when each one of the items was omitted, with the exception of the omission of item 9 (α=0.9285 for visit 1; α=0.9282 for visit 2), suggesting that it may be more appropriate to report it separately from the other items.

Table 3

Item–total correlation, corrected item–total correlation and Cronbach’s α for visits 1 and 2.

	Item–total	Item–total omitting item	Cronbach’s α
	Spearman’s rho (ρ)	Spearman’s rho (ρ)	Total=0.927
Visit 1 ( n =182)			Omitting each item
1: The detailer was knowledgeable	0.75	0.70	0.917
2: The detailer was an effective communicator	0.73	0.69	0.919
3: Academic detailing is an effective way to stay updated on important topic(s)	0.82	0.77	0.916
4: The printed material was useful	0.82	0.76	0.917
5: I would be receptive to future visits	0.83	0.76	0.918
6: This topic was relevant to my practice	0.78	0.73	0.919
7: This is an important topic	0.68	0.65	0.918
8: The key messages are feasible to implement in my practice	0.76	0.73	0.915
9: My practice is likely to change as a result of this visit	0.74	0.54	0.929
10: The key messages were consistent with my practice	0.73	0.69	0.922
Visit 2 ( n =158)			Total=0.932
Satisfaction			Omitting each item
1: The detailer was knowledgeable	0.70	0.67	0.925
2: The detailer was an effective communicator	0.70	0.66	0.926
3: Academic detailing is an effective way to stay updated on important topic(s)	0.78	0.74	0.923
4: The printed material was useful	0.78	0.72	0.923
5: I would be receptive to future visits	0.84	0.77	0.924
6: This topic was relevant to my practice	0.82	0.79	0.923
7: This is an important topic	0.66	0.63	0.929
8: The key messages are feasible to implement in my practice	0.85	0.82	0.920
9: My practice is likely to change as a result of this visit	0.81	0.67	0.928
10: The key messages were consistent with my practice	0.79	0.76	0.923

Table 4 displays item–item correlations for visits 1 and 2. The correlation matrix shows that, in both visits, correlations between items were of moderate or strong strength, which reinforces the notion that these items are measuring a single underlying latent construct. Once again, item 9 (likelihood to change) is the single exception, having displayed week correlations with items 7 (ρ=0.36) and 10 (ρ=0.38) in the first visit.

Table 4

Item–item correlation for visits 1 and 2.

Visit 1 (n=182)	Spearman’s rho (ρ)
1: The detailer was knowledgeable	1
2: The detailer was an effective communicator	0.79	1
3: Academic detailing is an effective way to stay updated on important topic(s)	0.67	0.67	1
4: The printed material was useful	0.57	0.53	0.73	1
5: I would be receptive to future visits	0.54	0.57	0.68	0.65	1
6: This topic was relevant to my practice	0.54	0.57	0.64	0.63	0.70	1
7: This is an important topic	0.65	0.53	0.60	0.60	0.53	0.65	1
8: The key messages are feasible to implement in my practice	0.65	0.59	0.66	0.64	0.58	0.62	0.72	1
9: My practice is likely to change as a result of this visit	0.40	0.41	0.43	0.45	0.53	0.46	0.36	0.44	1
10: The key messages were consistent with my practice	0.65	0.63	0.61	0.58	0.58	0.59	0.61	0.66	0.38	1
Visit 2 ( n =158)
1: The detailer was knowledgeable	1
2: The detailer was an effective communicator	0.84	1
3: Academic detailing is an effective way to stay updated on important topic(s)	0.71	0.75	1
4: The printed material was useful	0.61	0.65	0.70	1
5: I would be receptive to future visits	0.56	0.57	0.65	0.64	1
6: This topic was relevant to my practice	0.59	0.58	0.61	0.61	0.72	1
7: This is an important topic	0.53	0.44	0.52	0.50	0.55	0.73	1
8: The key messages are feasible to implement in my practice	0.58	0.59	0.69	0.70	0.69	0.74	0.69	1
9: My practice is likely to change as a result of this visit	0.44	0.43	0.51	0.55	0.60	0.59	0.46	0.65	1
10: The key messages were consistent with my practice	0.60	0.56	0.66	0.62	0.65	0.70	0.67	0.78	0.55	1

ρ, Spearman’s rank correlation coefficient

All reported values were statistically significant (p<0.1)

In both visits, KMO values obtained for the full set of items as well as for each individual item are all well above 0.8, suggesting that the data are adequate to perform factor analysis (Table 5). The principal axis factoring model identified a single factor with an eigenvalue greater than 1.00 that accounted for most of the underlining variance in the data. These findings were consistent in both visits 1 and 2. The factor loadings (FL) observed for all items were high, with item 9 presenting the lowest FL and highest uniqueness (U) in visit 1 (FL=0.54; U=0.71) being replaced in this by item 7 in visit 2 (FL=0.65; U=0.57) (Table 5).

Table 5

Item factor loading and uniqueness for visits 1 and 2.

Visit 1 (n=182)	Eigenvalue	Factor loadings	Uniqueness	KMO
‘Satisfaction’ factor	5.80			Total=0.92
1: The detailer was knowledgeable		0.82	0.33	0.86
2: The detailer was an effective communicator		0.75	0.43	0.86
3: Academic detailing is an effective way to stay updated on important topic(s)		0.82	0.32	0.95
4: The printed material was useful		0.81	0.34	0.95
5: I would be receptive to future visits		0.79	0.37	0.91
6: This topic was relevant to my practice		0.76	0.42	0.93
7: This is an important topic		0.77	0.40	0.90
8: The key messages are feasible to implement in my practice		0.82	0.33	0.93
9: My practice is likely to change as a result of this visit		0.54	0.71	0.96
10: The key messages were consistent with my practice		0.68	0.53	0.96
Visit 2 ( n =158)
‘Satisfaction’ factor	5.69			Total=0.91
1: The detailer was knowledgeable		0.73	0.47	0.85
2: The detailer was an effective communicator		0.70	0.51	0.83
3: Academic detailing is an effective way to stay updated on important topic(s)		0.79	0.37	0.94
4: The printed material was useful		0.77	0.41	0.95
5: I would be receptive to future visits		0.77	0.40	0.94
6: This topic was relevant to my practice		0.79	0.37	0.93
7: This is an important topic		0.65	0.57	0.90
8: The key messages are feasible to implement in my practice		0.86	0.26	0.90
9: My practice is likely to change as a result of this visit		0.67	0.56	0.95
10: The key messages were consistent with my practice		0.78	0.39	0.91

KMO, Kaiser–Meyer–Olkin measure of sampling adequacy test.

An attempt to explore construct validity was made using the available data. Convergent validity was tested by correlating the scores resulting from the sum of the full set of the PSAD measure candidate items with the DAVE. The results obtained showed that these measures are weakly correlated (r=0.2115, p<0.005 for visit 1; r=0.3110, p<0.000 for visit 2).

The criterion-based validity of this measure was not supported by our findings. Based on the data available, we failed to reject the null hypothesis that satisfaction scores in the baseline visit were not associated with a higher likelihood of agreeing to participate in the follow-up visit (43.41 for those that accepted visit 2 versus 46.1 for those that rejected visit 2; p=0.037).

Considering the results obtained in the item–item correlation matrix and the factor analysis, we concluded that the items have a sufficiently high correlation to justify the estimate of a single latent trait score. This is relevant as, under the item response theory, each item of the test measures the underlying latent trait; thus, most of the models applied assume the unidimensionality of the group of items being tested. Figure 2 shows the amount of information plotted against ability (item information function), for each of the ten candidate items evaluated in this analysis, for visits 1 and 2. Due to the categorical nature of the data, the item information functions are neither unimodal nor symmetric. This happens because each category contributes its own amount of information, which may peak over a different ability range. The results show that Item 9 (likelihood to change) yields much less information than the other candidate items. The graphic representation of the item information function also shows that all the curves begin to drop after the mean (ability=0), indicating the presence of a ceiling effect. Figure 3 shows the test information function for visits 1 and 2. No significant differences were found when comparing the test information functions estimated with and without the contribution of Item 9. The results were consistent between visits 1 and 2.

Figure 2

Item information function for visits 1 and 2.

Figure 3

Test information function graphs with and without item 9 (visit 2).

Discussion

This paper reports the development of a new measure to assess prescriber’s satisfaction with AD and educational outreach interventions – the PSAD. A series of carefully conceived steps were undertaken to ensure that the items included in this instrument cover a substantial range of constructs relevant for the assessment of satisfaction with an AD intervention, whilst being comprehensive enough to apply to a broad range of themes and settings. This systematic process included literature reviews, content-expert review and psychometric testing in over 180 PCPs. This process narrowed down an initial list of 10 candidate items to a 9-item unidimensional measure. In the end, we obtained a short measure, amenable to be applied in only a couple of minutes, that seemed to be well received and well understood by the target population as it had a very low rate of missing data.

The literature review stage identified several publications in the field of AD and educational outreach that collected and discuss data on prescriber’s satisfaction with interventions. However, a validated measure was not used in any of the identified literature. Rather, single items or non-psychometric lists of questions were applied.11–13 We believe that the development of a measure with sound psychometric properties will allow for a standardized and systematic collection of data on prescriber satisfaction, thus contributing information about the process of implementation that can help to monitor and improve AD programmes. Items related to a detailer’s effectiveness of communication and knowledge may be useful in monitoring team performance and helping to inform training and messaging decisions. Likewise, the item related to the usefulness of printed materials may support decisions regarding the need to refine the content and format of materials used during the visit.

Classic test theory methods were applied to assess the measure’s internal consistency reliability, criterion validity and convergent validity and to explore its factor structure. The results obtained in this first application of the measure support its internal consistency reliability, but future research is needed to further establish other forms of reliability (e.g. inter-rater and test-retest reliability). As noted in the results section, the slight increase in the Cronbach’s α, when excluding item 9, suggests it might be separately reported. Our findings did not support this measure’s criterion-based validity. However, it seems relevant to highlight that the choice of the criterion was made in an opportunistic fashion, contingent on the data that was available to us at this stage. Future research applying the PSAD measure should revisit this issue. Regarding the attempt to establish the PSAD measure’s convergent validity, it is important to note that there is no gold standard to measure the prescribers’ satisfaction currently available in the literature; therefore, we are limited in our ability to interpret the weak correlation coefficient obtained in our analysis. Evidence obtained in the EFA strongly suggests that the PSAD measure is a unidimensional measure of satisfaction, which is compatible with the factor structure hypothesized a priori and supports the construct validity.

Item response theory methods were applied to examine the amount of information yielded by each of the candidate items. Once again, the item about ‘likelihood to change’ (item 9) displayed a weak performance when compared with the remaining item pool. The distribution of the responses to this instrument as well as the graphical representation of the item and test information functions suggest that there is an apparent ceiling effect in this measure. Literature in the field of satisfaction measurement proposes that items measuring basic features of the intervention will differentiate amongst individuals at the lower end of the satisfaction continuum, and items that tap specific features or services that exceed expectations will do the same at the upper end.27–29

Whilst the item related to the prescribers’ likelihood to change their prescribing behaviour because of the AD visit (item 9) covers an important desirable outcome of AD interventions, our results suggest that it is not a good fit in a satisfaction measure. The version of the PSAD developed in this study is composed of nine items covering the constructs knowledge, effectiveness of communication, effectiveness, usefulness, willingness to repeat experience, acceptability, acceptability, feasibility and consistency.

Finally, there are several limitations to this study that we aim to address in future endeavours. The present study was not strictly designed for measure validation but was rather conducted opportunistically, leveraging an ongoing CDC-funded AD intervention. As a result, some of the information relevant to test the construct validity of the newly developed instrument was not collected. Furthermore, this study may not be powered to test hypothesized relationships (i.e. criterion-based and convergent validity hypothesis testing); therefore, those results should be interpreted carefully. Future research should assess its generalizability as an evaluative tool for AD and the extent to which the measure correlates with changes in prescribing and patient outcomes.

Conclusion

The PSAD measure is a new 9-item instrument intended to assess healthcare professional’s satisfaction with AD. Considering the broad nature of its items, this measure has the potential of being applied in educational outreach programmes beyond the scope of opioid prescribing. The results obtained in this first field test support the reliability and validity of this new instrument, and it is hoped that the final version of this measure will generalize to broad use as an evaluative tool for AD.