Validity of medication adherence self-reports in adults with type 2 diabetes.

OBJECTIVE: To assess the validity of self-report measures of diabetes medication adherence and evaluate the effect of depression on the validity of these reports. RESEARCH DESIGN AND METHODS: Adults with type 2 diabetes, treated with oral medications, completed a set of medication adherence self-reports that varied response scales and time frames, were administered structured clinical interviews for depression, and provided blood samples for HbA(1c) as part of a screening for an intervention study. A subsample of participants with HbA(1c) ≥7.0% and clinically significant depression received Medication Event Monitoring System (MEMS) bottle caps to record adherence. Analyses examined relationships between adherence measures and HbA(1c) and, in the subsample, MEMS. Moderated linear regression evaluated whether depression severity modified relationships with HbA(1c).
RESULTS: Participant (n = 170, 57% men, 81% white, mean HbA(1c) 8.3% [SD, 1.7]) adherence self-reports were significantly (r = -0.18 to -0.28; P < 0.03) associated with lower HbA(1c). In the subsample (n = 88), all self-reports were significantly (r = 0.35 to 0.55; P ≤ 0.001) associated with MEMS-measured adherence. Depression significantly moderated the relationship between three of six self-reports and HbA(1c); at high levels of depression, associations with HbA(1c) became nonsignificant.
CONCLUSIONS: Results support the validity of easily administered self-reports for diabetes medication adherence. One-month, percentage-based ratings of adherence had the strongest associations with MEMS and HbA(1c); those requiring the report of missed doses had weaker associations. One-week self-ratings and measures that require respondents to record the number of missed doses appear to be vulnerable to bias from depression severity.

Treatment nonadherence is a common and important problem in diabetes care that negatively impacts treatment outcomes (1). Many studies have focused on the identification of factors that contribute to adherence and on the evaluation of interventions to improve adherence. However, the value of these studies is often limited by measurement challenges in the assessment of medication adherence. Although objective measures, such as electronic monitoring caps that record the timing of pill bottle openings, are often thought to be preferable to subjective self-reports, they are costly, often not feasible for certain study designs, and are impractical for clinical practice. Given the continued need for valid adherence self-report measures, understanding the factors that influence the validity of self-reports remains important (2). However, information on the validity of self-reported medication adherence often goes unreported in studies investigating the psychometric properties of widely used diabetes treatment adherence scales (3), and few studies focus directly on the issue of validity in the evaluation of self-reported medication adherence measures.

For self-report measures of diabetes medication adherence to be valid, there should be evidence of a robust correlation with other adherence measures, preferably those not subject to the biases associated with self-report (concurrent validity). In addition, because diabetes medication adherence has a causal impact on glycemic control, participants’ adherence self-reports should correlate with HbA1c levels (criterion validity). Relationships with HbA1c levels are expected to be relatively weaker than with other measures of adherence because medication adherence is not the only factor that impacts glycemic control (2).

Nevertheless, studies often rely exclusively on HbA1c to demonstrate the validity of adherence measures in diabetes. For example, studies have documented significant relationships between self-reported adherence and HbA1c in adults with type 2 diabetes, using versions of the Morisky Medication Adherence Scale (4–8) and other self-reports of medication adherence (9,10). Other large studies, however, have failed to find relationships between similarly measured self-reported adherence and HbA1c (11). Relatively few studies have examined the relationship between self-reported adherence and objectively monitored medication adherence; fewer still have examined how the characteristics of self-report questions affect this relationship.

Significantly closer concordance has been observed between single-item global self-ratings of adherence and concurrently assessed adherence measured by electronic monitoring cap, compared with the concordance for self-report measures that focused on frequency of missed doses in HIV/AIDS (12). The recalled time frame of these measures also related to concordance: 1-month recall periods were more accurate than 3- and 7-day recall periods (12). Another study found that electronically-monitored adherence to cholesterol-lowering medication and a modified version of the Morisky scale were each associated with cholesterol lowering, whereas other adherence methods—such as pill counts, the original Morisky scale, and recall questions about missed doses—did not predict cholesterol lowering (13). These studies are rare in their examination of the influence of measurement characteristics on the validity of medication adherence self-reports.

The current study sought to evaluate the validity of four self-report measures of medication adherence in a sample of adult patients with type 2 diabetes treated with oral medications for diabetes and related conditions (i.e., hypercholesterolemia, hypertension) who were recruited for a depression treatment trial. Because depression has been consistently associated with treatment nonadherence in diabetes, including self-reported and objectively monitored adherence (14), and in light of the literature linking mood and biases in memory and recall (15–17), we also used a detailed assessment of depression to evaluate whether depression severity influenced the validity of self-reports for medication adherence.

RESEARCH DESIGN AND METHODS

Study samples and procedures

Participants were recruited from the Diabetes Center and primary care clinics at Massachusetts General Hospital or were self-referred via hospital e-mail mailing lists and radio advertisements for a larger intervention study on the efficacy of cognitive behavioral therapy for adherence and depression (18) in patients with type 2 diabetes. Entry criteria for the study included the ability to read and write in English, a diagnosis of type 2 diabetes, a prescription for medication to treat hyperglycemia (either insulin or oral medication) and taking oral medication for diabetes or a diabetes-related condition (i.e., hypercholesterolemia or hypertension), and age between 18 and 70 years. Analyses indicated no significant differences in the relationships of self-reported adherence to HbA1c levels for patients not taking an oral antihyperglycemic medication (because they were taking insulin only; n = 25) versus those who were taking an oral antihyperglycemic medication, so both groups of participants were included.

To qualify for the intervention study and to receive an electronic bottle cap, participants were required to meet DSM-IV criteria for major depressive disorder (MDD) or dysthymia or to exhibit clinically significant depressive symptoms despite the prescription of an antidepressant. They were also required to have an HbA1c ≥7.0%. Prescriptions for diabetes and depression medications were required to have been stable for 2 months. Those with severe mental illness requiring immediate treatment or precluding ability to participate were excluded. Participants provided informed consent, and the Partners HealthCare Institutional Review Board approved the study.

All participants were evaluated for depression severity, completed all self-report measures, and provided a blood sample to measure HbA1c levels at the initial baseline evaluation. Self-report questions for medication adherence asked participants about their adherence to their overall diabetes-related oral medication regimen (i.e., prescribed for hyperglycemia, hypercholesterolemia, or hypertension). The subsample of participants who qualified for the intervention were given an electronic pill bottle cap at the baseline visit and were asked to continue using it until their next study visit. Thus, the electronically monitored adherence assessment followed the self-reports and captured only pre-intervention data.

Measures

Depression severity.

The Montgomery Åsberg Depression Rating Scale (19), a structured clinical assessment of 10 commonly occurring symptoms of depression over the past week, was used to measure depression symptom severity. Scores range from 0 to 60: 0–6 indicates no depression; 7–19 indicates mild depression; 20–34 indicates moderate depression; and 35–60 indicates severe depression. Internal reliability was high (α = 0.85).

Self-ratings for medication adherence.

Questions developed by Lu et al. (12) to assess adherence to antiretroviral medications over the past month were adapted to assess adherence to oral diabetes-related medications. The instructions for all self-ratings were as follows: “The following questions ask about your diabetes medications. This includes oral medications you take for diabetes, high blood pressure, and high cholesterol.” Participants were asked to quantitatively evaluate their adherence using 11 response categories (0, 10, 20… 100%) with the question, “What percent of the time did you take all your diabetes medications as your doctor prescribed?” Participants were also asked to qualitatively rate their adherence (rating scale: very poor, poor, fair, good, very good, excellent) with, “On average, how would you rate your ability to take all your diabetes medications as your doctor prescribed?” As in Lu et al. (12), participants were asked to respond considering the past month. One-week time frames were added to facilitate comparison with other measures.

Other adherence self-reports.

On the basis of the approach of the AIDS Clinical Trials Group (ACTG) Adherence Questionnaire (20), we asked participants to report the number of prescribed doses for each diabetes-related medication and record the number of missed doses, per medication, during the past 7 days. We refer to this measure as “missed doses.” A calendar was used to facilitate recall, and adherence was calculated as the percentage of prescribed doses taken for the past week. For participants taking multiple medications, a mean percentage of doses taken across medications was calculated. Participants also completed an item from the Summary of Diabetes Self Care Activities (SDSCA), which asked participants, “On how many of the last seven days did you take your recommended diabetes medication” (21). We refer to this measure as “adherent days.”

Electronically monitored medication adherence.

Medication Event Monitoring System (MEMS) bottle caps were used to track one medication per participant. For participants taking an antihyperglycemic medication, MEMS caps tracked the antihyperglycemic medication that the participant took the most frequently or found most difficult to remember. For the 25 participants taking insulin and no oral antihyperglycemic medication, hypertension- or hypercholesterolemia-related medications were monitored. MEMS adherence was calculated to provide the percentage of doses taken by dividing the number of times a bottle was opened by the number of openings prescribed during the time frame. Trained assessors followed the approach used by Safren et al. (22) to correct MEMS data only if participants could identify a day on the calendar when they took their medication in a way that it would not be recorded by MEMS (e.g., took out two pills in the morning and pocketed one for later that day). Participants were required to have at least 5 days of MEMS data for the current analyses.

Other measures.

Participants completed a structured interview of psychiatric diagnoses (23) and provided a blood sample for HbA1c at the baseline visit, which was analyzed at the same hospital laboratory. They also completed study-generated questionnaires regarding their treatment regimen, comorbid illnesses, presence of diabetes complications, time since diagnosis, and demographic and socioeconomic factors.

Data analysis

All analyses were done in SPSS 18.0 software. All variables were examined for normality and were analyzed as continuous. The missed doses measure was not normally distributed; therefore, correlations and regression analyses for this variable are based on a base 10 logarithm of the original data. Bivariate correlations were used to determine correlations between HbA1c, MEMs data, and self-report adherence data. We used Williams’ t test (Wt) (24) to compare the strength of correlations across adherence measures. Computations based on those of Lu et al. (12), transformed the 6-point Likert scales of the qualitative self-rating items to 100-point scales (0, 20, 40, 60, 80, and 100). We then calculated means and SDs for all indices of adherence on a 0–100 scale and computed discrepancies between self-reported adherence and MEMS adherence.

Multivariate linear regression analyses focused on HbA1c (rather than MEMS) because it was available for the full sample. Sex, age, education, and the prescription of insulin were evaluated as potential covariates to be included in multivariate analyses if significantly correlated with HbA1c or MEMS. Because depression was a focus of the study, we evaluated evidence for whether depression severity moderated the relationship between self-reported adherence and HbA1c through interaction terms, with appropriate post hoc probing of significant moderation effects. This procedure involved the computation of two conditional variables for depression, one in which lower depression severity was coded as 1 SD below the centered mean and one in which higher depression severity was coded as 1 SD above. The slopes of these analyses were then used to plot regression lines for the relationship between self-reports and HbA1c at high and low levels of depression severity (25). We did not examine MEMS in these moderation analyses because of the smaller subsample with available data. This subsample also had a restricted range of depression severity and glycemic control as a result of our design.

RESULTS

We limited our analyses to the 170 screened participants who provided a baseline HbA1c and answered the self-reported adherence questions. Of these, 88 qualified for the intervention study and provided MEMS cap data before intervention. The average number of days of adherence collected by MEMS caps was 20.59 (SD, 11.81). Number of days of monitoring was not significantly associated with MEMS adherence (P > 0.10).

Sociodemographic and background characteristics

Of the 170 participants included in the first set of analyses, 56.5% were men, 81% were white, mean age was 56 years (SD, 10) and mean HbA1c was 8.3% (SD, 1.7). Participants had an average of 14 years (SD, 3) of education, and 59% qualified for a diagnosis of MDD. Of the 88 people given an MEMS cap, 52.3% were men and their mean age was 57 years (SD, 8). These participants had an average of 14 years (SD, 3) of education, 84% were white, mean HbA1c was 8.8% (SD, 1.6), and 74% met criteria for MDD. Table 1 presents descriptive statistics for those in the overall sample and for the subsample included in the MEMS analyses. Consistent with the expected effects of the intervention’s selection criteria, MEMS participants had significantly more severe Montgomery Åsberg Depression Rating Scale scores (t [166] = –3.77, P < 0.001) and were more likely to meet diagnostic criteria for MDD (χ2 [1, n = 168] = 21.87, P < 0.001) than those who did not qualify for the intervention. They also had a significantly longer duration of diabetes (t [164] = −2.71, P = 0.008) and were more likely to be on insulin (χ2 [1, n = 168] = 8.81, P = 0.003).

Table 1

Participant characteristics

Correlations between adherence and HbA1c

All adherence self-report items were significantly and inversely correlated with HbA1c. MEMS adherence and HbA1c were similarly significantly correlated (r = −0.25, P = 0.021; see first two data columns in Table 2). Comparing correlation coefficients revealed no significant differences in correlations between any of the self-report measures and HbA1c levels (all P > 0.17).

Table 2

Validity and descriptive data for adherence measures

Correlations between self-reported and MEMS adherence

All correlations between self-reported adherence and MEMS were positive and significant (see third and fourth data columns in Table 2). Comparisons by Wt test indicated that the quantitative 1-week adherence self-rating was more strongly correlated to MEMS adherence than the qualitative 1-week self-rating (Wt = −0.29, P = 0.005) and missed doses during the past week (Wt = −3.49, P < 0.001). In addition, the 1-month quantitative adherence self-rating was more strongly correlated to MEMS than the 1-month qualitative self-rating (Wt = −3.22, P = 0.002). There was no difference between the 1-week quantitative self-rating and the 1-month quantitative self-rating (Wt = −1.02, P = 0.31). The remaining comparisons were not significant (P > 0.06).

Self-reported adherence and MEMS discrepancies

Reported adherence, expressed as a percentage, was highest for missed doses of medication and lowest for 1-month qualitative self-ratings (see fifth data column in Table 2). Discrepancies between self-reported adherence and subsequent MEMS adherence are presented in the last column of Table 2 and show that 1-month qualitative self-ratings and missed doses tended to have the greatest discrepancy with MEMS data, whereas quantitative self-ratings showed closest concordance. Mean differences were smallest, and near zero, for the 1-month percentage-based rating. Age, depression severity, and education were not significantly related to these discrepancies (data not shown).

Multiple regression and moderation analyses

Of potential covariates, only insulin use was significantly associated with HbA1c (r = 0.32, P = 0.003) and was included in the multivariate models. Results from separate regression models for each measure of self-reported adherence are presented in Table 3. The inclusion of depression severity as a main effect did not meaningfully attenuate relationships between self-reports and HbA1c. However, several significant moderation effects were found. Specifically, depression severity was a significant moderator of the relationship between 1-week qualitative adherence self-ratings (P = 0.025) and 1-week quantitative self-ratings (P = 0.046) and HbA1c. A significant interaction (P < 0.001) was also found between depression and reports of missed doses. Post hoc probing of these significant interaction effects showed that the relationship between self-reported adherence and HbA1c was negative and significant at low levels of depression severity but near zero and nonsignificant at high levels (Fig. 1).

Table 3

Hierarchical linear regressions assessing interaction effects between self-reported adherence and depression severity in predicting HbA1c

Figure 1

Regression lines for relationships between self-reported medication adherence and HbA1c levels as moderated by level of depression. For each of these measures, the moderation shows that self-report was not associated with HbA1c at high levels of depression but was significantly associated with HbA1c at low levels of depression. Depression significantly moderates relationship between qualitative 1-week adherence self-ratings and HbA1c (A), quantitative 1-week adherence self-ratings and HbA1c (B), and self-reported missed doses and HbA1c (C). *P = 0.001; **P < 0.001.

CONCLUSIONS

The findings from this study support the validity of easily administered self-report measures to assess medication adherence in adults with type 2 diabetes. We compared self-reported and MEMS-measured adherence and found significant correlations for all self-report items, suggesting that self-reported adherence shares a significant amount (7–30%) of variance with objectively monitored adherence, our proximal indicator of validity. This was found despite the lack of overlap in time frames and even though MEMS only monitored one medication whereas the self-report items required participants to consider their diabetes-related oral medications in aggregate. These associations, especially those for the percentage-based global self-ratings, were considerably stronger than those found by a previous study of more than 500 patients with type 2 diabetes between the 4-item self-report Morisky Medication Adherence Scale or the adherent days item from the SDSCA and pharmacy refill records (4% shared variance each) (11). A previous study of more than 800 patients with heart failure or hypertension also found a weaker correlation between self-reported medication adherence based on a modified version of the Morisky scale, and MEMS adherence (9.6% shared variance) (26). Our results suggest that part of the variation in the amount of the shared variance between self-reports and MEMS adherence can be attributed to measurement characteristics (discussed in detail below).

The relatively more modest relationships between adherence and HbA1c were expected, because glycemic control is affected by a variety of factors beyond medication adherence, including diet and exercise, degree of insulin deficiency, and adequacy of the prescribed treatment regimen. Although modest relationships between adherence self-reports and HbA1c are often interpreted as reflecting the limited validity of these measures, this may not be an appropriate conclusion. For example, a recent meta-analysis of 21 studies in pediatric type 1 diabetes indicated that adherence and glycemic control share less than 8% of their variance. No differences in the strength of this association were found for studies that used self-reported adherence versus those that used objective meter downloads (27). Similarly, in a large sample of adults with type 2 diabetes, objective pharmacy refill data were significantly associated with current HbA1c and change in HbA1c over time; however, the amount of variance explained was only 4% and 1.7%, respectively (28). Thus, it is important to recognize that validating adherence self-reports based only on relationships with clinical outcomes such as HbA1c may lead to less accurate conclusions because of the relatively modest impact of adherence on this outcome (2,29). The strength of the association between adherence and HbA1c should also inform clinicians working with patients with suboptimal glycemic control who report being adherent to their treatment regimen.

Our findings provide novel information on the influence of characteristics of self-report measures on the validity of adherence estimates. Measures asking respondents to report missed doses tended to over-predict subsequently monitored MEMS adherence, whereas global quantitative self-ratings were more concordant with subsequent MEMS data. It may be surprising that less specific, single-item global measures resulted in more valid adherence estimates than more intensive reviews of missed doses. However, this is consistent with the literature on the validity of self-reported adherence, which suggests that patients can provide general estimates of adherence more accurately than specific missed doses (13,30,31). In contrast to the findings of Lu et al. (12), which showed that qualitative self-ratings were more accurate than other self-reports in predicting MEMS adherence, we found that percentage-based self-ratings were most strongly associated with MEMS adherence. Because these studies differed substantially in the characteristics of their samples, replication of these results is needed. However, both studies suggest that global ratings provide more valid estimates than self-reports that rely on a recall of missed doses.

Our findings also suggest an important role for depression severity in the measurement of self-reported medication adherence in individuals living with diabetes. Analyses revealed significant interaction effects between depression severity and several adherence self-reports, the 1-week qualitative and quantitative self-ratings, and the measure of missed doses during 1 week, in predicting HbA1c. In each case, adherence reports were not associated with HbA1c levels at higher levels of depression severity but were significantly related at lower levels. The moderation effect was not significant for the SDSCA item regarding adherent days or the 1-month adherence self-ratings. The moderating effect of depression is consistent with research demonstrating that mood can affect memory and recall abilities. Specifically, studies have consistently found that clinically depressed individuals demonstrate a recall bias that favors remembering negative stimuli rather than positive stimuli (32). Therefore, participants with greater depression severity possibly remembered themselves as less adherent than they actually were, whereas nondepressed participants more accurately remembered their adherence. We can only speculate about why the depression effect was not found for the days-adherent or 1-month self-rating items. It is plausible that the SDSCA item was less vulnerable to this effect due to its focus on days of adherence rather than nonadherence. It is also plausible that participants were more likely to attempt to remember missed doses during the past week when responding to the 1-week self-ratings but more likely to rely on a gestalt for their adherence when providing ratings for 1 month.

We are aware of only one other study that has examined the role of depression severity in the concordance between self-reports for adherence and objectively measured adherence (26). This study of patients with hypertension or heart failure found no significant differences in correlations between self-reported adherence and pharmacy refill adherence for depressed versus nondepressed patients. Although not significant (P = 0.07), this study found that the correlation between self-reported adherence and MEMS adherence was slightly higher for depressed individuals. However, this study used two different self-report measures to screen for depression, treated depression as a categorical variable, and did not conduct formal tests of moderation. Thus, results should be compared with caution. Given the increased prevalence of depression symptoms in patients with diabetes (33) and the consistent relationship between depression and diabetes treatment nonadherence (14), the role of depression in adherence self-reports deserves further investigation.

Our results should be considered in the context of our study design. First, our sample was recruited for a depression intervention study. Although this limits the generalizability of our findings, it allowed us to examine the effect of relatively severe symptoms of depression.

Second, we were limited in our ability to examine the concordance between self-reports and MEMS data because these measures did not cover the same time frame. On the other hand, this allowed us to evaluate predictive validity.

Third, MEMS only tracked one diabetes-related medication, whereas three self-reports (qualitative, quantitative, and missed doses) asked about all diabetes-related medications the participant was taking. In addition, the SDSCA adherent-days item asked about “diabetes medication,” though we expect that given the explicit instructions for the other items, participants likely answered thinking about the same class of medications used for the other self-report measures. No self-reports asked specifically about adherence to insulin. Furthermore, only a subset of the original sample, selected for depression severity and suboptimal glycemic control, was monitored with MEMS caps. Because this subsample had a restricted range of depression severity, we focused our depression moderation analyses on HbA1c only.

Finally, we note that whenever adherence was measured, we were obtaining information on more than the ingestion of medication. Research shows that adherence to a placebo is consistently associated with mortality risk in various patient populations (34,35). The relationship between measures of medication adherence and clinical outcomes likely reflects other behaviors and respondent characteristics that directly influence health; this is often referred to as the “healthy adherer effect.” Given this context, we were less concerned with specifically measuring the link between antihyperglycemic medication and HbA1c and instead focused on adherence to the broad class of medications necessary to manage the risk of diabetes complications. Future studies should build on this work by adapting these self-report measures to refer to specific medications and differentiate between insulin and oral medications.

The accurate measurement of medication adherence in diabetes is important for progress in the development of interventions to improve diabetes self-management and for appropriate treatment decisions in the delivery of care. Clear evidence demonstrates important negative health consequences of nonadherence in patients with diabetes (1). Objective measures of adherence are often impractical for various research designs and do not address the problem of how treatment providers should inquire about treatment adherence with their patients. Developing self-report measures of medication adherence with strong validity would therefore have important benefits for research and practice. Results of the current study contribute to our progress toward this goal and should encourage further investigation into the best practices for the assessment of diabetes treatment adherence.