Development and validation of the Patient Opioid Education Measure.

BACKGROUND: Although there are screening tools to aid clinicians in assessing the risk of opioid misuse, an instrument to assess opioid-related knowledge is not currently available. The purpose of this study was to develop a content-valid, understandable, readable, and reliable Patient Opioid Education Measure (POEM).
METHODS: Using concept mapping, clinicians caring for patients with chronic pain participated in brainstorming, sorting, and rating need-to-know information for patients prescribed opioids. Concept mapping analyses identified seven clusters addressing knowledge and expectations associated with opioid use, including medicolegal issues, prescribing policies, safe use and handling, expected outcomes, side effects, pharmacology, and warnings.
RESULTS: The 49-item POEM was verbally administered to 83 patients (average age 51.3 ± 9.8 years, 77.1% female, 47.1% African American) taking opioids for chronic nonmalignant pain. Patients averaged in total 63.9% ± 14.3% (range 23%-91%) correct responses on the POEM. The POEM demonstrated substantial test-retest reliability (interclass correlation coefficient 0.87). The POEM had a mean readability Lexile (L) score of 805.9 ± 257.3 L (equivalent to approximately a US fifth grade reading level), with individual items ranging from 280 L to 1370 L.
CONCLUSION: The POEM shows promise for rapidly identifying patients' opioid-related knowledge gaps and expectations. Correcting misunderstandings and gaps could result in safer use of opioids in a clinical care setting.

Introduction

Over the past two decades, prescribing of opioid analgesics for the management of chronic nonmalignant pain has steadily risen.1,2 As a result of increased prescribing, rates of opioid-related addiction, overdose, and death have also escalated.3–5 To combat potential misuse and diversion, strategies have been developed, such as use of pain management agreements cosigned by the patient and physician to establish rules and expectations concerning chronic pain management and prescription of opioid analgesics.6 Such contracts may provide authority for physicians to request routine urine drug tests and mandate regular office visits to evaluate patient response to medication. A description of significant withdrawal signs and symptoms is usually included in the agreement to inform patients about possible health consequences if the contract is broken because of patient noncompliance.

Understanding and acting upon medical-related information and instructions pose significant obstacles7 for the more than one third of American adults with basic or below-basic health literacy skills.8 Patients with limited health literacy skills often lack medical-related knowledge9 and experience difficulty comprehending instructions and demonstrating intended medication administration and dosing.10–12 Moreover, while an abundance of patient education materials, eg, prescription and over-the-counter medication labeling, pharmacy-generated information sheets, and medication guides approved by the US Food and Drug Administration are widely available, most are of minimal use due to poor visual presentation and exceedingly high reading demands.13–16 Additionally, the readability and formatting characteristics of typical patient-physician opioid pain management contracts far exceed the actual health literacy skills of American adults.17

Improper opioid dosing related to patients not understanding the prescription directions results in a significantly increased risk of overdose and death.3–5 Arguably, a patient must possess adequate opioid-related knowledge (eg, side effects, contraindications, proper handling, and storage) not only to control chronic pain, but also to manage opioid medication as prescribed by the physician. While a wide collection of patient-centered opioid-related measurement tools (eg, self-efficacy18 and bowel function index19) are available, a tool designed to assess an individual’s knowledge about opioid analgesics does not currently exist. To address this gap, the present study developed a content-valid, understandable, readable, and reliable Patient Opioid Education Measure (POEM). It was anticipated that patient responses to POEM items would enable clinicians to identify knowledge gaps related to chronic opioid therapy more rapidly and tailor their counseling accordingly.

Materials and methods

Concept mapping overview

Concept mapping, a structured quantitative and qualitative process, was used to guide development of the POEM. Concept mapping enables researchers to develop a conceptual framework regarding a specific issue. Concept mapping consists of three phases, including: brainstorming to identify key items related to the specific issue; sorting and rating each item identified during the brainstorming phase; and grouping items into conceptual clusters.20 While concept mapping has been primarily used to plan and evaluate program activities, it is also appropriate for instrument development.21,22 With respect to chronic pain, concept mapping has been used to guide the development of the Screener and Opioid Assessment for Patients with Pain23 and to explore the opioid-related educational needs of primary care physicians.24

Expert content panel for item generation through brainstorming

Fourteen individuals participated in the brainstorming phase of the concept-mapping procedures. The content expert panel was composed of the following individuals currently treating and/or involved in the care of patients with chronic pain: primary care physicians (n = 8), pain/addiction specialists (n = 4), a PhD level clinical psychologist (n = 1), and a PhD level pain expert/researcher (n = 1).

The 14 content experts were sent detailed instructions for the brainstorming phase of the project electronically. Specifically, each content expert was asked to respond to the following focus prompt: “Thinking as broadly as possible, please list specific need-to-know information for patients prescribed opioids.” Content experts were asked to brainstorm and provide at least 15 statements based on this focus prompt.

Content experts returned their completed statement list to the principal investigator electronically. Once data from the brainstorming phase had been collected, the principal investigator reviewed all submitted statements and removed duplicates. Next, members of the research team individually reviewed the revised statement list and were asked to identify duplicate statements/concepts and make suggestions as to the rewording and/or combining of statements. Lastly, a final statement list was generated by the principal investigator to be used in the sorting and rating process.

Expert content panel for item sorting and rating

Thirty-seven individuals participated in the sorting and rating phases of the concept-mapping process. In addition to the 14 individuals who took part in the brainstorming phase, an additional 23 individuals were recruited to complete the sorting and rating phases. The content expert panel for the sorting and rating phases was composed of the following individuals currently treating and/or involved in the care of patients with chronic pain: primary care physicians (n = 24), pain/addiction specialists (n = 8), PharmD level pharmacists (n = 2), a PhD level clinical psychologist (n = 1), a PhD level pain expert/researcher (n = 1), and a masters level medical librarian with expertise in the development of patient education materials (n = 1).

Content experts were provided with a username and password to access the secure Concept Systems® website (http://www.conceptsystemsglobal.com). First, content experts were instructed to complete a “free” sorting of the statements based on similarity “in any way that makes sense to you.” Using the instructions employed by Rosas and Camphausen,22 there were four restrictions to the sorting process, ie, all statements could not be put into a single pile, all statements could not be put into their own separate piles (although some items could be grouped by themselves), statements could not be placed in two piles simultaneously, and “miscellaneous” piles were not permitted. Once sorted into piles, content experts were asked to rate the importance of each item on a five-point Likert-type scale (1, relatively unimportant; 2, somewhat important; 3, moderately important; 4, very important; 5, extremely important), as related to patient knowledge regarding opioid use.

Concept mapping analyses

Data were analyzed using Concept Systems (Concept Systems Inc., Ithaca, NY, USA) software (version 4.0.175). Similar to factor analysis, the software uses multidimensional scaling and cluster analyses to analyze sorted and rated items and organize the statements into a “map” of clusters.20 With the guidance of Concept Systems support staff, the principal investigator generated several different cluster map solutions. In cluster map solutions, the size of a cluster is a visual representation of the extent to which the statements in that cluster are grouped together. Therefore, the smaller the area of the cluster, the more often content experts sort these statements together. Conceptually, a larger area suggests a broader, less well defined concept. These cluster maps were then shared and reviewed by the research team members. The research team was then convened as a group to select the most representative grouping of items and “name” each of the clusters/content areas in the map.

Drafting of alpha versions of the POEM

A priori, statements with an average importance rating ≥4.00 (equivalent to “very important”) were considered for possible inclusion in POEM alpha version one (POEM_alphaV1). The principal investigator drafted POEM_alphaV1. Four doctoral level instrumentation experts (three in public health and one in psychology) reviewed POEM_alphaV1 and provided detailed qualitative feedback regarding the following: overall layout, individual POEM statements, and scoring options. Next, the principal investigator reviewed all submitted comments and created POEM alpha version two (POEM_alphaV2) which was then distributed to other members of the research team for their review and critique.

Lastly, POEM alpha version three (POEM_alphaV3) was created by the principal investigator based on all comments submitted by the research team. POEM_alphaV3 included a total of 49 items and was designed to be verbally administered to the patient. Forty-four items assessed opioid-related knowledge content, while the remaining five items addressed issues pertaining to expected outcomes stemming from the use of opioids for chronic pain management.

Pilot testing overview of POEM_alphaV3

All study procedures were approved by the institutional review board at The Ohio State University (2012H0147). In June 2012, the principal investigator submitted an electronic data request to The Ohio State University Wexner Medical Center Information Warehouse to identify patients meeting all of the following criteria: seen in one of two clinics within The Ohio State University primary care network during the past 6 months, age 30–65 years, and receiving long-term opioid medication for management of chronic nonmalignant pain. Trained research assistants reviewed the resulting list of electronic patient medical records to confirm preliminary study eligibility. A total of 324 patients met all the aforementioned criteria.

The principal investigator verified preliminary eligibility criteria with each patient’s primary care physician. Based on primary care physician recommendations, 20 patients were ineligible to participate in the study (eg, dismissed from the practice). Therefore, a total of 304 patients were recruited to participate in the study via individualized letters and telephone calls.

Two strategies were used to recruit potential patients to participate in this study. First, patients were sent an individualized letter via the US postal service describing the purposes of the study. Second, approximately one week following the initial recruitment letter mailing, a trained research assistant called eligible patients to invite them to participate in the study.

The research assistants attempted to contact each preliminarily eligible patient by telephone at varying times of the day, for a maximum of four calls. The research assistants were unable to contact 43 (14.1%) of the 304 eligible patients (eg, nonworking telephone numbers, no answer). Of 261 patients contacted, 195 (74.7%) initially agreed to participate in the study. During September–December, 2012, research assistants were able to schedule a baseline interview with 111 (56.9%) of the 195 patients initially agreeing to participate in the study. A total of 83 (74.8%, or 27.3% of the 304 eligible) patients scheduled to complete the 40-minute baseline one-on-one interview with a trained research assistant, at one of two The Ohio State University primary care clinics, did so.

Baseline data collection to assess psychometric properties of POEM_alphaV3

All one-on-one interviews took place in a private room. Final eligibility to participate was determined as the first step in the interview. Patients were ineligible to participate in the study if they were unable to speak and/or understand English, appeared acutely ill, or had visual acuity worse than 20/50 with or without corrective lenses (using the Rosenbaum Pocket Vision Screener). All patients keeping the interview appointment met the final eligibility criteria. Informed consent was obtained from each participant and documented in written form by the research assistant prior to verbal administration of study materials.

In random order, the research assistant verbally administered a collection of sociodemographic items, a health literacy screening item, and the POEM_alphaV3 to each patient. Sociodemographic items (age, race/ethnicity, educational attainment, health insurance coverage type, and self-reported general health status) were from the 2011 Behavioral Risk Factor Surveillance Survey.25 Similar Behavioral Risk Factor Surveillance Survey items have shown to have both high validity and reliability.26 An established health literacy item, “How often do you have problems learning about your medical condition because of difficulty understanding written information?” (response options: always, often, sometimes, occasionally, or never) was also administered to patients.27 This one-sentence screening item has been shown to be highly predictive of patients’ objectively measured health literacy skills using robust tools such as the Rapid Estimate of Adult Literacy in Medicine and Short-Test of Functional Health Literacy in Adults.27–29 At the conclusion of the baseline one-on-one interview, each patient received a $40 supermarket gift card as compensation for time spent during the study.

Follow-up data collection to assess psychometric properties of POEM_alphaV3

Patients completing the baseline interview were invited by the research assistant to return to the clinic in 7–10 days for a follow-up interview. Patients were told that the follow-up interview would take approximately 20 minutes to complete. Of the 83 patients completing the baseline one-on-one interview, 68 (81.9%) also completed the follow-up interview. The purpose of the follow-up interview was to assess stability (test-retest reliability) of POEM_alphaV3 over time. Therefore, the research assistant only administered POEM_alphaV3 to patients during the follow-up interview. At the conclusion of the follow-up interview, each patient received a $30 supermarket gift card as compensation for their time.

Preliminary evaluation of POEM_alphaV3

Upon completion of both individual baseline and follow-up interviews, patients’ responses to individual POEM_alphaV3 items were entered into an Excel spreadsheet verbatim. Patients’ responses were reviewed in detail by the principal investigator. With the exception of one multi-item question (three items in total), all other POEM_alphaV3 items were not revised.

As part of POEM_alphaV3, each patient was asked to identify how his/her opioid was currently administered (pill, tablet, capsule, liquid, and/or patch) and to describe how he/she took each dose and/or placed the patch on his/her skin. Upon careful inspection of patient responses to these items, our research team determined that this question should have included additional prompts to assess and evaluate all specific multiple steps relevant to opioid administration (eg, dosing intervals, how medication was taken [swallowed whole or crushed]). For scoring purposes in this study, this multi-item question was removed due to patients’ inconsistent and nonspecific response patterns. As described below, we extensively revised the original formatting of this multi-item question to comprehensively assess patient step-by-step administration practices.

Assessing language complexity of POEM_alphaV3

Language complexity of each item included within POEM_ alphaV3 was assessed using the Lexile® analyzer (MetaMetrics Inc., Durham, NC, USA).30,31 Lexile (L) scores, based on sentence length and familiarity of word choice, range from 0 L to 2000 L. A score of 900 L corresponds to an approximate US sixth grade reading level.

Scoring of POEM_alphaV3

For scoring purposes in this study, POEM_alphaV3 included a total of 46 items and was composed of three sections. The first section (POEM_alphaV3_Section I) included 25 open-ended, opioid-related knowledge items that were administered to and completed by all study participants. The second section (POEM_alphaV3_Section II) included 16 open-ended opioid-related knowledge items. Five (n = 5) of the 16 items were probing items with no correct or incorrect response (eg, “Did your doctor give you a pain agreement to read and sign?”). The remaining 11 items included within POEM_alphaV3_Section II were administered to patients based upon their responses to the five probing items. For this reason, total number of POEM_alphaV3_Section II items administered to and answered by each patient varied.

Patients’ verbal responses to individual POEM_alphaV3_ Section I and POEM_alphaV3_Section II items were considered correct if they matched acceptable responses agreed upon a priori by the research team. Responses to POEM_ alphaV3_Section I and POEM_alphaV3_Section II items were scored as either correct (1 point) or incorrect (0 points), because credit was not awarded for partially correct responses. Total opioid-related knowledge scores were calculated for POEM_alphaV3_Section I and POEM_alphaV3_Section II separately and overall. An overall POEM_alphaV3_ Section I and POEM_alphaV3_Section II percentage score was computed based on total number of correct responses divided by total number of items answered.

POEM_alphaV3_Section III comprised five items addressing issues pertaining to patients’ expected outcomes stemming from opioid use for chronic pain management. Four (n = 4) items were measured using a multiple-choice format (eg, “How do you feel about taking pain medicine?”). A 10-point (1, not important; 10, very important) Likert-type scale was used for the remaining expected outcome item (“How important is it that pain medicine takes away most of your pain?”).

Analyses

Data analyses were conducted using the Statistical Package for the Social Sciences version 20.0 software (SPSS Inc, Chicago, IL, USA). Descriptive statistics (frequencies, percentages, means, and standard deviations) were performed on all variables. Bivariate analyses using Spearman’s rank correlation coefficients (p) were performed to assess the relationship between baseline POEM_alphaV3 (POEM_ alphaV3_Section I and POEM_alphaV3_Section II combined) correct percentage scores (first and second sections combined) with patients’ educational attainment (categorized as

The Kuder-Richardson 20, a coefficient of reliability, was calculated to measure the internal consistency of POEM_ alphaV3_Section I. The Kuder-Richardson 20 is a special version of Cronbach’s coefficient alpha (a) for items that are dichotomous (“correct” versus “incorrect”).32 An internal consistency of ≥0.70 is considered desirable.33

Coefficient of stability (test-retest) of individual POEM_ alphaV3_Section I and II items was assessed using Cohen’s kappa. Test-retest reliability of total POEM_alphaV3_ Section I and II scores (individually and combined) and the outcome expectation measured on a continuous scale (“How important is it that pain medicine takes away most of your pain?”) were assessed using an intraclass correlation coefficient. Cohen’s kappas and intraclass correlation coefficients were used specifically to assess strength of agreement between baseline and follow-up POEM_alphaV3 (first and second sections) survey administrations. Using the metrics proposed by Landis and Koch,34 Cohen’s kappas were classified as follows: 0–0.20, slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and 0.81–1.0, almost perfect agreement. An intraclass correlation coefficient ≥0.75 is considered excellent reproducibility.35

Construction and evaluation of the final version of POEM

The final version of the POEM included all POEM_alphaV3 items in addition to the revised multi-item administration questions. Language complexity of these additional items was assessed using the Lexile analyzer.30,31

Results

The number of statements generated per content expert ranged from 17 to 42, with a mean of 24.1 ± 7.3. Combined, a total of 338 statements were generated by the 14 content experts. Upon removal of duplicates and/or combining statements, the final list comprised 131 unique statements.

The multidimensional scaling analysis of the perceived similarity of statements produced a stress value (a goodness of fit statistic) of 0.20 after 21 iterations, lower (ie, better) as compared with the mean stress value of 0.28 based on the pooled analysis of 69 concept mapping studies.36 The point map (Figure 1) presents the results of the multidimensional scaling analysis in raw format that is further refined in the next step.

Figure 1

Point map of multidimensional scaling of the 133 statements.

A seven-cluster concept map solution was then generated (Figure 2). The seven clusters identified through the concept mapping process were labeled as medicolegal issues, prescribing policies, safe use and handling, expected outcomes, side effects, pharmacology, and warnings. The concept map presents each cluster as having one to five layers representing the average rating importance of statements included in the cluster. The cluster legend presents the value range included within each cluster. Thus, single-layered clusters contain statements, with averages from 3.51 to 3.64 (least important), while five-layered clusters contain statements with averages from 4.04 to 4.17 (most important).

Figure 2

Seven-cluster concept map with importance ratings.

The most important concept, with five layers, was warnings (average importance rating, 4.17). The second and third most important concepts, with four layers, were prescribing policies (average importance rating, 4.03) and safe use and handling (average importance rating, 4.01). Pharmacology (average importance rating, 3.79), with three layers, was the fourth most important concept. The fifth and sixth most important concepts (two layers) were medicolegal issues (average importance rating, 3.67) and side effects (average importance rating, 3.64). Expected outcomes (average importance rating, 3.51) was a single-layer cluster.

Figure 3 represents a ladder graph comparing importance ratings, by clusters, of pain specialists (n = 8) and primary care physicians (n = 24). The ladder graph shows that the two groups tended to rate the clusters in a reasonably similar manner with respect to importance, with a strong positive correlation of 0.84.

Figure 3

Pattern match comparing pain specialists (n = 8) versus primary care physicians (n = 24).

A total of 47 statements (35.9%) had an importance rating of ≥4.00 and were included in POEM_alphaV3 (see Appendix 1). Statements per cluster were as follows: medicolegal issues (n = 3 of 19 total statements [15.8%]), prescribing policies (n = 9 of 15 total statements [60.0%]), safe use and handling (n = 9 of 18 total statements [50.0%]), expected outcomes (n = 5 of 29 statements [17.2%]), side effects (n = 5 of 21 statements [23.8%]), pharmacology (n = 8 of 18 statements [44.4%]), and warnings (n = 8 of 11 statements [72.7%]).

The sociodemographic characteristics of patients completing both baseline (n = 83) and follow-up (n = 68) interviews are presented in Table 1. Overall, the patients were of mean age 51.3 ± 9.8 (range 31–65) years, with females (n = 64, 77.1%) comprising the majority of study participants. Significant differences, across sociodemographic characteristics, did not emerge between those completing POEM_alphaV3 baseline and follow-up survey administrations.

Table 1

Summary statistics for POEM_alphaV3 baseline and follow-up survey samples

Variable	POEM_alphaV3 baseline survey sample (n = 83) Mean ± SD or n (%)	POEM_alphaV3 follow-up survey sample (n = 68) Mean ± SD or n (%)	P-value
Age (years)	51.3 ± 9.8 (range 31–65)	51.3 ± 9.7 (range 31–65)	0.97
Gender			0.93
Male	19 (22.9)	16 (23.5)
Female	64 (77.1)	52 (76.5)
Ethnicity
Multiethnic	6 (7.2)	6 (8.8)	0.98
African American	38 (45.8)	32 (47.1)
American Indian	2 (2.4)	2 (2.9)
Asian	1 (1.2)	1 (1.5)
Caucasian	35 (42.2)	26 (38.2)
Latino	1 (1.2)	1 (1.5)
Educational attainment
Elementary	1 (1.2)	1 (1.5)	0.95
Some high school	9 (10.8)	8 (11.8)
High school or GED	26 (31.3)	23 (33.8)
Some college	34 (41.0)	26 (38.2)
College graduate	13 (15.7)	10 (14.7)
Health insurance type
Medicaid	29 (34.9)	24 (35.3)	0.70
Medicare	22 (36.5)	20 (29.4)
Private/commercial	20 (24.1)	18 (26.5)
Self-pay	12 (14.4)	6 (8.9)
Self-reported health
Fair/poor	51 (61.5)	43 (63.2)	0.98
Good	27 (32.5)	22 (32.4)
Very good/excellent	5 (6.0)	3 (4.4)

Abbreviations: GED, general equivalency diploma; POEM_alphaV3, Patient Opioid education Measure alpha version three; SD, standard deviation

Distribution of patients’ baseline POEM_alphaV3_ Section I scores are displayed in Figure 4. The mean score on POEM_alphaV3_Section I was 15.9 ± 3.9 (range 5–23), converting to an average correct percentage score of 63.9% ± 15.6% (range 20%–92%). The coefficient of internal consistency of POEM_alphaV3_Section I was adequate (Kuder-Richardson 20, 0.73).

Figure 4

Distribution of POEM alpha version three (V3) section I (POEM_ alphaV3_section I) baseline scores (maximum score 25).

Abbreviation: POEM_alphaV3, Patient Opioid education Measure alpha version three.

Pairwise correlations, using Spearman’s correlation coefficients, between POEM_alphaV3_Section I percentage correct score and educational attainment (ρ = 0.23, P ≤ 0.04) and the health literacy screening item (ρ = 0.28, P ≤ 0.01) were both statistically significant. POEM_alphaV3_Section I had substantial test-retest reliability (interclass correlation coefficient 0.87).

Patients averaged 62.6% ± 23.6% (range 0%–100%) correct responses on POEM_alphaV3_Section II. On POEM_ alphaV3_Sections_I and II combined patients averaged 63.9% ± 14.3% (range 23%–91%) correct responses.

Table 2 presents an overview of individual POEM_ alphaV3 opioid-related knowledge items (Sections I and II) in relation to the percentage of patients answering correctly, Cohen’s kappa index, and Lexile score. For POEM_ alphaV3_Section I, the percentage of those answering these items (n = 25) correctly ranged from 14.5% to 89.2%. Cohen’s kappa indices for POEM_alphaV3_Section I ranged from 0.12 to 0.90. Combined POEM_alphaV3_ Section I items had a mean readability score of 844.6 ± 249.0 L, with individual items ranging from 280 L to 1,370 L.

Table 2

Overview of POEM_alphaV3 individual section I and II items

	% answering correctly	Test-retest reliability	Lexile score
Section I POEM knowledge items
Where do you keep or store your pain medicine at home?	14.5	K = 0.62	700
Why does your doctor need to know about all the other medicines you are taking?	24.1	K = 0.29	840
Pain medicine can cause serious and life-threatening breathing problems. What is it about taking pain medicine that causes these breathing problems to happen?	25.3	K = 0.56	900
Some people become addicted to pain medicine.	25.3	K = 0.15	450
Name at least one change that can be seen in someone who is addicted?
What would happen if you stopped your pain medicine suddenly or cold turkey?	45.8	K = 0.36	800
Why will your doctor want to see you on a regular basis?	48.1	K = 0.34	700
What will happen to your body after taking pain medicine for a long time?	50.6	K = 0.50	830
Why would you have side effects if you stopped your pain medicine suddenly or cold turkey?	51.8	K = 0.38	960
Why must you take your pain medicine exactly the way your doctor told you to?	54.2	K = 0.38	860
Pretend it is the weekend. You have run out of your pain medicine. You call your doctor’s office and ask for more pain medicine. What will happen? Why?	62.7	K = 0.25	340
Pretend that you want to stop taking your pain medicine. What should you do?	65.1	K = 0.53	330
You have tried many different pain medicines and you still have pain. What could happen if you take different pain medicines at the same time to relieve your pain?	67.5	K = 0.22	900
While taking pain medicine, what is the safe amount of medicines that can make you sleepy (eg, sleeping pills, antihistamines, anxiety pills) to take?	68.7	K = 0.45	1,370
What could happen if you took more pain medicine than was prescribed at any one time?	71.1	K = 0.60	980
Why must those close to and trusted by you (eg, your family, friends) know that you are taking pain medicine?	77.1	K = 0.31	1,120
Why do you need to know how pain medicine affects you before driving or using heavy equipment (eg, power tools)?	78.3	K = 0.40	1,260
What would happen if you changed the written pain medicine prescription form (“script”) your doctor gave you?	79.5	K = 0.49	1,140
Pretend that your pain is much worse. You think you need to take more pain medicine. What should you do?	83.1	K = 0.46	280
Pretend that one of your family members or a friend has the same pain symptoms and/or diagnosis as you.	84.3	K = 0.65	780
He is also taking the same pain medicine as you.
He has run out of his pain medicine and doesn’t see his doctor until next week. He asks for some of your pain medicine to get him through the next few days.
What should you do? Why?
What can you do to keep your bowels moving?	84.3	K = 0.90	470
Name at least one thing you can do.
What happens to the bowel movements of people who take pain medicine?	85.5	K = 0.58	840
What would happen if you tried to sell your pain medicine to someone?	86.7	K = 0.38	790
While taking pain medicine, what is the safe amount of beer to drink?	86.7	K = 0.45	860
While taking pain medicine, what is the safe amount of wine to drink?	86.7	K = 0.52	860
While taking pain medicine, what is the safe amount of hard liquor/spirits/booze to drink?	89.2	K = 0.65	1,150
Section II POEM knowledge items
At what pharmacy do you get your prescription(s) filled?	74.7	K = 0.93	710
Is this the only pharmacy where you get your prescriptions filled?
Why should all of your prescriptions be filled at only one pharmacy?	26.5	K = 0.41	820
Has your doctor asked you for a urine (pee) or blood sample?	–	–	870
Why does your doctor ask you for a urine (pee) or blood sample while you are taking pain medicine?	54.5	K = 0.43	1,220
Did your doctor give you a pain agreement to read and sign?	–	–	720
Why did your doctor have you sign a pain agreement?	25.0	K = 0.12	640
What should you do if you don’t understand what the pain agreement says?	53.0	K = 1.00	740
What will your doctor do if you don’t do what is stated in the pain agreement?	80.9	K = 0.62	890
Right now, how many doctors prescribe any medicine for you?	–	–	750
Have you told your other doctors that you have been prescribed pain medicine by Dr __________?	97.4	K = 1.00	950
Why must you tell all of your other doctors that you are taking pain medicine prescribed by Dr __________?	97.1	K = 1.00	1,090
Do you have sleep apnea (snoring, blocked breathing during sleep, periods of not breathing)?	–	–	1,060
Does Dr __________ know that you have sleep apnea (snoring, blocked breathing, periods of not breathing)?	93.1	K = 1.00	1,070
If you are taking pain medicine, why must Dr __________ know that you have sleep apnea?	14.8	K = 0.33	930
Have you ever abused alcohol or drugs?	–	–	570
Why must your doctor know about your alcohol and drug abuse before giving you pain medicine?	63.6	K = 0.66	1,090

Notes: – designates probing question without a correct or incorrect response.

Abbreviation: POEM_alphaV3, Patient Opioid education Measure alpha version three.

For POEM_alphaV3_Section II, the percentage of those answering these items (n = 11) correctly ranged from 14.8% to 97.4%. Cohen’s kappa indices for POEM_alphaV3_Section II ranged from 0.12 to 1.00. Combined, POEM_alphaV3_ Section II items had a mean readability score of 882.5 ± 187.6 L, with individual items ranging from 570 L to 1,220 L.

POEM_alphaV3_Section III comprised the five outcome expectation items. Cohen’s kappa indices for POEM_alphaV3_Section III items (n = 4) with multiple-choice formatting response options ranged from 0.24 to 0.49 (Table 3). The remaining outcome expectation item (“How important is it that pain medicine takes away most of your pain?”) had a Cohen’s kappa of 0.69. Combined, POEM_alphaV3_ Section III items had a mean readability score of 596.0 ± 271.3 L, with individual items ranging from 330 L to 970 L.

Table 3

Overview of POEM_alphaV3 individual section III items

Section III POEM outcome expectation items										Test-retest reliability	Lexile score
How do you feel about taking pain medicine?										K = 0.24	510
a. I am sure it will make me pain-free
b. I am sure it will help reduce the pain, but not take it all away
c. I am sure it will help, but I am not sure how much
d. I am sure it will not reduce my pain
I expect pain medicine to take away:										K = 0.49	390
a. All of my pain
b. Most of my pain
c. Some of my pain
d. None of my pain
Pain medicine has taken away:										K = 0.49	330
a. All of my pain
b. Most of my pain
c. Some of my pain
d. None of my pain
Has your functioning (ability to move around) changed since taking pain medicine?										K = 0.39	970
a. Yes, it has gotten much worse
b. Yes, it has gotten somewhat worse
c. No, it has stayed the same
d. Yes, it has gotten somewhat better
e. Yes, it has gotten much better
How important is it that pain medicine takes away most of your pain?										ICC 0.69	780
On a scale of 1 to 10:
Not important at all					Important			Very important
1	2	3	4	5	6	7	8	9	10

Abbreviations: ICC, interclass correlation coefficient; POEM_alphaV3, Patient Opioid Education Measure alpha version three.

The final version of the POEM included 49 items (see Appendix 2). Combined, the revised (items 6a and 6b) items had a mean readability score of 626.7 ± 110.6 L, with individual items ranging from 510 L to 730 L. Overall, the final version of the POEM had a mean readability score of 805.9 ± 257.3 L, with individual items ranging from 280 L to 1,370 L.

Discussion

The POEM was designed to assess patients’ comprehensive understanding of opioid medication instructions/counseling and outcome expectations related to long-term chronic pain management with opioid analgesics. By design, the POEM includes a collection of open-ended knowledge items encompassing six domains, ie, medicolegal issues, prescribing policies, safe use and handling, side effects, pharmacology, and warnings, designed to be nonthreatening and/or leading. For these particular POEM items, an open-ended assessment approach was employed over true-false and nonmultiple choice formats to reduce guessing and allow the opportunity to gauge patients’ degree of understanding of these concepts. Initial psychometric analysis of the POEM suggests that this tool has good validity and reliability in measuring patients’ knowledge and outcome expectations regarding the prescription of long-term opioids.

Overall reading demand of the POEM was below the sixth grade level. While the POEM was designed to be verbally administered, assessment of reading level remains pertinent because patients’ aural literacy and/or listening comprehension capacity are critical in assuring that items are understandable and interpreted as intended.37,38

Even while very few patients answered two of the individual POEM items correctly, these items were retained because of their importance to proper opioid use. For example, the POEM item with the fewest number of patients answering correctly addresses a critical concept, ie, “Where do you keep or store your pain medicine at home?”, regarding proper handling. Importantly, our results mirror a recent study where knowledge of proper home storage was minimal among a sample of emergency department patients prescribed opioids.39

The main strength of the study was the application of concept mapping to generate candidate POEM items. Additionally, a diverse mix of individuals, including pain specialists, primary care physicians, and nonclinicians, submitted and assessed POEM statements for importance and relevance. Another strong point was the way in which the POEM was designed and subsequently qualitatively evaluated by a team of instrumentation specialists prior to being pilot-tested with patients. Formatting of POEM knowledge items is also a study strength. Specifically, open-ended versus close-ended (true/false/do not know and/or multiple choice) response options were used to gauge patients’ degree of understanding by having them describe in their own words their understanding of medicolegal issues, prescribing policies, safe use and handling, side effects, pharmacology, and warnings. Lastly, test-retest reliability of the POEM was very strong.

While preliminary psychometric testing of the POEM was promising, several limitations must be acknowledged. First, response bias and social desirability may have come into play. However, this potential limitation was most likely ameliorated as a result of the open-ended assessment approach of the POEM. Second, the cross-sectional nature of this study does not support inferences about causal directions. Third, because many POEM_alphaV3_Section II items used a skip-response format, we were unable to assess internal consistency (using the Kuder-Richardson 20) of this section accurately because of a large number of nonresponders to these individual items. Fourth, the newly created multi-question opioid administration items were not formally tested in this study. However, the clinical utility of these questions will be investigated in a future follow-up study.

Conclusion

Overall, the POEM shows great promise as a measure of patients’ knowledge and expectations regarding chronic opioid use. It is anticipated that the POEM can aid clinicians in identifying patients at greatest risk of limited knowledge regarding opioids. This could be accomplished, for example, by administration of the POEM by telephone or in person following initial prescription of an opioid analgesic. Any knowledge deficits identified would then become the focus of patient or caregiver counseling. The POEM should be tested and cross-validated in diverse patient populations. Future areas of interest concerning use of the POEM could include examining the reciprocal relationship between knowledge and outcome expectations and adequacy of pain control.

Appendix 1

Statements (n = 47), by cluster, with average importance ratings of ≥4.0

Clusters and statements	Mean
Cluster 1: medicolegal issues (3/19 total statements)
1. Altering an opioid prescription is a felony	4.42
2. You may not sell your opioids to any other person under any circumstances	4.39
3. It is illegal to share prescribed opioid medication with people for whom it was not prescribed	4.33
Cluster 2: prescribing policies (9/15 total statements)
4. You should inform your physician of all medications you are taking	4.64
5. The presence of nonprescribed drug(s) or illicit drug(s) in the urine may be grounds for termination of the doctor/patient relationship	4.47
6. If you violate your pain contract, your prescribing doctor and/or clinic will prescribe no more opioids for you	4.39
7. Make sure you carefully follow the pain contract or opioid agreement you were given	4.33
8. If you were given a pain contract or an opioid agreement be sure you understand what it says	4.25
9. Physicians may require drug screening while taking opioids	4.19
10. You will be seen on a regular basis and given prescriptions for enough medication to last from appointment to appointment	4.17
11. It is important that patients use one pharmacy if they will be on opioids long term	4.14
12. Prescriptions for opioids will only be given during an office visit or during regular office hours	4.00
Cluster 3: safe use and handling (9/18 total statements)
13. Tell your doctor if you have had trouble with substance abuse/alcohol abuse in the past	4.64
14. Illicit substances, such as cocaine and marijuana, should not be used while taking opioids	4.56
15. If one physician prescribes an opioid for you, you should inform all your other physicians who prescribe medication for you of that fact	4.50
16. It is important not to cut or split a long-acting opioid	4.42
17. Take your opioids exactly as instructed	4.42
18. You are responsible for keeping your pain medication in a safe and secure place, such as a locked cabinet or safe	4.28
19. The success of treatment depends on full agreement and understanding of the risks and benefits of using opioids to treat pain	4.06
20. Talk with your doctor if you feel you need to stop taking opioids	4.06
21. Handle opioids like valuable property	4.03
Cluster 4: expected outcomes (5/29 total statements)
22. For most people, opioids reduce pain, but do not completely eliminate pain	4.36
23. The best indication that opioids are helping you is that you can increase your activity and can do things that you could not do before taking opioids	4.33
24. Opioids do not relieve all types of pain	4.11
25. Opioids can be one aspect of a comprehensive pain treatment plan	4.11
26. Improving how you function is more important than reducing your pain	4.00
Cluster 5: side effects (5/21 total statements)
27. Opioids may cause respiratory depression (problem breathing or slow down breathing)	4.25
28. Opioids may cause constipation	4.22
29. Opioids may cause sedation	4.11
30. While taking opioid medication, you incur risk of serious complications if you suffer from uncorrected sleep apnea (snoring, blocked breathing during sleep, periods of not breathing)	4.08
31. Opioids may affect cognition (mental status)	4.06
Cluster 6: pharmacology (8/18 total statements)
32. Opioids are safe if taken correctly, but can be deadly if taken incorrectly	4.64
33. Physical dependence can occur with the use of opioid medications	4.44
34. Opioids can be addicting	4.19
35. Overdose can occur when a person takes too much medication	4.11
36. Some opioid tablets should not be broken	4.11
37. Tolerance to opioids will develop with regular use	4.03
38. Some opioid tablets should not be chewed	4.00
39. Opioid patches do not work well if they are cut	4.00
Cluster 7: warnings (8/11 total statements)
40. Taking a large quantity of opioids at once can be fatal	4.69
41. Taking sedatives in combination with opioids can be dangerous or fatal	4.61
42. Drinking alcohol in combination with opioids can be dangerous or fatal	4.56
43. Taking other pain medication in combination with opioids can be dangerous or fatal	4.31
44. Taking opioids too often or at a higher than prescribed dose results in potentially dangerous over-medication	4.31
45. Sudden stoppage of opioids can be very uncomfortable (withdrawal)	4.22
46. It may not be safe to drive while taking opioids	4.14
47. Your family should know and watch for the signs of dangerous drug overdose	4.06

Note: The full list of 133 statements and importance ratings is available from the authors.

Not important at all					Important			Very important
1	2	3	4	5	6	7	8	9	10