Validating Health Literacy and Numeracy Measures in Minority Groups.

BACKGROUND: Validation studies of existing health literacy or numeracy tools among racial/ethnic minorities are limited.
OBJECTIVE: This study assessed the validity of the Subjective Numeracy Scale (SNS), the Diabetes Numeracy Test (DNT-5), the Brief Health Literacy Screen (BHLS), and the Short Test of Functional Health Literacy in Adults (S-TOFHLA) by trait (health literacy or numeracy) and by method (subjective or objective) among non-Hispanic white (NHW), non-Hispanic black (NHB), and Hispanic patients with type 2 diabetes mellitus (T2DM).
METHODS: We conducted a secondary analysis of baseline data from the Partnering to Improve Diabetes Education (PRIDE) study, a clustered randomized controlled trial testing the efficacy of a health communication intervention on T2DM outcomes at state Department of Health clinics in middle Tennessee. PRIDE participants with race/ethnicity data available (n = 398) were included in this study. Most patients identified as NHW (59%), 18% identified as NHB, and 23% identified as Hispanic. Pearson correlations among the 4 measures were compared for each racial/ethnic group by trait and method. The convergent validity of each measure with education was also assessed using Pearson correlation analyses. KEY
RESULTS: Significant correlations were observed across all 3 subgroups for the numeracy measures (SNS and DNT-5) and the objective measures (DNT-5 and S-TOFHLA). Nonsignificant correlations were observed among Hispanic participants for the health literacy measures (BHLS and S-TOFHLA, correlation coefficient = 0.13) and among NHB and Hispanic participants for the subjective measures (SNS and BHLS, correlations coefficients = 0.15 and 0.09, respectively). A significant positive correlation was noted between education and each measure across all 3 subgroups.
CONCLUSIONS: Subjective and health literacy measures demonstrate weaker correlations than objective and numeracy measures, respectively, among minority patients in this study. Our findings highlight the need to further evaluate the appropriateness of these tools for use with minority populations, particularly the BHLS for Hispanic patients.

Health literacy is defined as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” (Institute of Medicine, 2004). Inadequate health literacy is a critical barrier to disease management among patients diagnosed with type 2 diabetes mellitus (T2DM). Lower health literacy has been associated with worse diabetes knowledge, self-management, and clinical outcomes (Osborn, Bains, & Egede, 2010; Osborn, Cavanaugh, Wallston, & Rothman, 2010; Rothman, DeWalt, et al., 2004; Rothman, Malone, et al., 2004; Rothman et al., 2005; Sarkar et al., 2010; Schillinger et al., 2002; Williams, Baker, Parker, & Nurss, 1998; Yamashita & Kart, 2011). Numeracy, defined as “the ability to use and understand numbers in daily life,” (Rothman et al., 2006) is a key component of health literacy (Huizinga et al., 2008; Osborn et al., 2013) that is independently associated with worse self-management skills, worse perceived self-efficacy, and worse glycemic control among patients with diabetes (Cavanaugh et al., 2008). Interventions designed to address the literacy or numeracy deficits of patients can improve diabetes outcomes (Bailey et al., 2014; Bowen et al., 2016; Cavanaugh et al., 2009; Rothman, DeWalt, et al., 2004). The use of valid and reliable measures of health literacy is critical to efforts that seek to identify populations most in need of such interventions and most sensitive to their effects.

Although several numeracy and health literacy measures have been developed in recent years, these scales vary considerably in the conceptual dimensions they assess and in their appropriateness for a given context or population (Bailey et al., 2014; Haun, Valerio, McCormack, Sorensen, & Paasche-Orlow, 2014). One important source of variation is in the use of subjective versus objective approaches to assessment. Subjective measures such as the Subjective Numeracy Scale (SNS) developed by Fagerlin (Zikmund-Fisher, Smith, Ubel, & Fagerlin, 2007) and the Brief Health Literacy Screen (BHLS) developed by Chew (Chew, Bradley, & Boyko, 2004; Chew et al., 2008) measure participants' own perceptions of their competencies or preferences. Objective measures such as the Diabetes Numeracy Test (DNT-5) developed by Rothman (Huizinga et al., 2008) and the Short Test of Functional Health Literacy in Adults (S-TOFHLA) developed by Baker (Baker, Williams, Parker, Gazmararian, & Nurss, 1999) use an a priori scoring system determined by experts in which responses to skill-based questions are scored as either “correct” or “incorrect.” Few studies have critically evaluated the concurrent validity of existing numeracy or health literacy tools by trait (health literacy vs. numeracy) or approach (subjective vs. objective). Even fewer have examined if race/ethnicity differentially affects the concordance of these measures according to trait or method (Nguyen et al., 2015) despite the fact that minority patients are at higher risk for having inadequate numeracy or health literacy (Paasche-Orlow, Parker, Gazmararian, Nielsen-Bohlman, & Rudd, 2005) and experience higher rates of T2DM and its complications (Spanakis & Golden, 2013) relative to non-Hispanic white patients.

The purpose of this study was to assess the validity of the SNS, DNT-5, BHLS, and S-TOFHLA by trait and by method for each of three racial/ethnic groups in the study: non-Hispanic white (NHW), non-Hispanic black (NHB), and Hispanic. We hoped to better understand the performance of these measures among different populations to better understand the potential role of these scales in future research efforts as well as in routine clinical practice. We expected that measures of the same trait (numeracy or health literacy) would be more highly correlated with one another than with measures of the other trait, and that measures sharing a common method (eg, subjective or objective) would also be correlated with one another due to shared method variance, but not as highly correlated as measures of the same trait assessed by different methods. Additionally, we assessed the convergent validity of these measures with education level (as a referent standard) across and within the three groups because prior studies have consistently identified moderately positive correlations between education level and measures of numeracy and health literacy (Hanchate, Ash, Gazmararian, Wolf, & Paasche-Orlow, 2008; Miller et al., 2007; Paasche-Orlow et al., 2005). We expected the correlation of each measure with education to be consistent across racial/ethnic groups.

Methods

Study Setting

We conducted a cross-sectional secondary analysis of data from the Partnering to Improve Diabetes Education (PRIDE) study, (Wolff et al., 2016) a cluster randomized controlled trial that tested the impact of an effective health communication program on T2DM outcomes for patients receiving care at 10 state Department of Health clinics in the mid-Cumberland region of Tennessee. The PRIDE study design has been described in detail previously (Heerman et al., 2016; White et al., 2015). Patients at participating clinics were eligible to participate if they were between ages 18 and 85 years, diagnosed with T2DM with their most recent hemoglobin A1C being >7.5%, spoke English and/or Spanish, and agreed to remain enrolled for a duration of 2.5 years. Participants were excluded for poor visual acuity (>20/50 on a pocket screener), a history of significant dementia or psychosis, or if they were diagnosed with a terminal illness with a life expectancy of less than 2 years. Eligible participants were approached by bilingual research staff during regular clinic hours and by phone referral from clinic staff. Informed consent was obtained in the patient's language of preference (English or Spanish). Approval for the PRIDE study was obtained from the Institutional Review Boards of Vanderbilt University and the State of Tennessee Department of Health.

Main Measures

Baseline demographic data included age, gender, race/ethnicity, language preference, highest level of education completed, and income. Numeracy and health literacy were both assessed via two methods (subjective and objective) for each participant upon study enrollment. Participants could choose to complete the assessments in either English or Spanish.

The SNS, as validated by Zikmund-Fisher, Smith, Ubel, and Fagerlin (2007) was used as a subjective measure of numeracy. This 8-item scale was read aloud to participants to assess their numerical abilities in various contexts and their preferences for receiving numerical information, with all items being answered on a 6-point response scale ranging from “not at all good/helpful to extremely good/helpful,” “always prefer words to always prefer numbers,” or “never to very often.” Per instructions, one item—“When you hear a weather forecast, do you prefer predictions using percentages (‘there will be a 20% chance of rain today’) or predictions using only words (‘there is a small chance of rain today’)?”—was reverse coded due to a high score originally indicating a preference for words over numbers, and the responses were summed to create a score that could range from 8 to 48, with higher scores indicating higher subjective numeracy. In this study, the internal consistency (Cronbach's alpha) of the Spanish version of the SNS was 0.72 compared to 0.85 for the English version.

A shortened, validated form of DNT-5 was used as an objective measure of numeracy skills. The DNT-5 was developed by choosing the 5 items from the previously validated (in English and Spanish) DNT-15 (Huizinga et al., 2008; White, Osborn, Gebretsadik, Kripalani, & Rothman, 2011) that most strongly correlated with the total mathematics score from the Wide Range Achievement Test (WRAT-4) (Wilkinson & Robertson, 2006). The DNT-5 contains mathematical questions addressing nutrition, exercise, blood glucose monitoring, and medication. The DNT-5 was either self- or orally administered (based on patient preference), and participants were allowed the use of calculators to simulate day-to-day diabetes calculations. Items were scored as correct or incorrect, and scores were reported as the percent correct with a possible range from 0% to 100%.

The BHLS, as validated by Chew et al. (Chew et al., 2004; Chew et al., 2008) in English and Singh, Coyne, & Wallace (2015) in Spanish, was used as a subjective measure of health literacy. The scale, which contains three items answered on a 5-point response scale, was read aloud to each participant. After one item—“How confident are you in filling out medical forms by yourself?” with responses ranging from “extremely” to “not at all”—was reverse scored, the responses were summed to create a score that could range from 3 to 15, with higher scores indicating higher subjective health literacy. In this study, the internal consistency (Cronbach's alpha) of the Spanish version of the BHLS was 0.53 compared to 0.79 for the English version.

The S-TOFHLA was used as an objective measure of verbal health literacy skills in English (Baker et al., 1999) and Spanish (Aguirre, Ebrahim, & Shea, 2005). For the S-TOFHLA, participants read two prose passages and answered comprehension questions within a time limit of 7 minutes. S-TOFHLA scores could range from 0 to 36, with scores 23 or higher indicating adequate health literacy and scores 22 or lower indicating less-than-adequate literacy.

Statistical Analysis

Data analyses were conducted for the study sample as a whole and separately for each of the three racial/ethnic subgroups (NHW, NHB, and Hispanic). Baseline data (including health literacy and numeracy scores) were compared across racial/ethnic groups using one-way analyses of variance for continuous variables and chi-square tests for categorical variables.

We assessed racial/ethnic variation in the concurrent validity of the numeracy and health literacy measures by trait (health literacy or numeracy) and by method (subjective or objective measures). Following methods originally described by Campbell and Fiske (1959), Pearson correlation analyses were performed to determine monotrait-heteromethod and monomethod-heterotrait validation correlations for the sample as a whole and for each of the three racial/ethnic groups. We also assessed racial/ethnic variation in the convergent validity of the numeracy and health literacy measures with highest level of education completed (as a continuous variable measured in years) using Pearson correlation analyses. Participants with missing scores for numeracy and/or health literacy measures were excluded from analyses of those measures. Findings with a p value of < .05 were considered statistically significant.

Results

From July 2011 to April 2013, 410 patients consented to and were enrolled in the PRIDE study. We included only the 398 PRIDE participants with indicated race/ethnicity data in this secondary analysis. All participants who self-identified as NHW (n = 234) or NHB (n = 72) preferred to complete their numeracy and health literacy assessments in English (n = 306); likewise, all participants who self-identified as Hispanic (n = 92) preferred to complete their assessments in Spanish. BHLS and SNS scores were available for all 398 participants. DNT-5 scores were available for 387 participants, and S-TOFHLA scores were available for 391 participants.

Baseline characteristics of the PRIDE participants are presented in Table . NHW participants scored significantly higher on the DNT-5 compared to NHB and Hispanic participants. NHW and NHB participants scored significantly higher on the BHLS and S-TOFHLA compared to Hispanic participants. Hispanic participants scored significantly higher on the SNS compared to NHB participants, whereas NHW participants did not differ in subjective numeracy from either of those two groups.

Assessment of concurrent validity using the monotrait-heteromethod comparisons for the sample as a whole demonstrated statistically significant positive correlations for the numeracy measures and both health literacy measures (Table ). However, analyses stratified by racial/ethnic subgroup found this pattern to be consistent only for NHW and NHB participants. There was a positive and statistically significant correlation between the two numeracy measures (SNS and DNT-5, Pearson correlation coefficient = 0.41, p < .001) among Hispanic participants but the weakly positive correlation between the two health literacy measures (BHLS and S-TOFHLA, Pearson correlation coefficient = 0.13, p = .22) was not significant.

Assessment of concurrent validity using the monomethod-heterotrait comparisons for the sample as a whole demonstrated statistically significant positive correlations for subjective measures and both objective measures (Table ). However, analyses stratified by racial/ethnic subgroup found this pattern to be consistent only for NHW participants. There were positive and statistically significant correlations between the objective measures (DNT-5 and S-TOFHLA) among NHB (Pearson correlation coefficient = 0.25, p < .05) and Hispanic (Pearson correlation coefficient = 0.56, p < .001) participants but the correlation between these measures was lower for NHB participants than for either of the other two subgroups. The correlations between the subjective measures (SNS and BHLS) were nonsignificant and only weakly positive for NHB (Pearson correlation coefficient = 0.15, p = .21) and Hispanic (Pearson correlation coefficient = 0.09, p = .38) participants.

All four measures demonstrated significant positive correlations with education level in the sample as a whole and in stratified analyses by racial/ethnic subgroup (Table ). However, for Hispanic participants, the correlation between the SNS and highest education level achieved was not as strongly positive as for the other two groups, and the significant positive correlation between education level and scores on the DNT-5 was not as strong for NHB participants as it was for NHW participants.

Discussion

In this study, we examined the concurrent and convergent validity of 2 measures of numeracy and 2 measures of health literacy among three racial/ethnic subgroups of patients with T2DM. By using the multitrait-multimethod technique to assess concurrent validity and educational level to assess convergent validity, we found uniform support for the validity of the numeracy measures despite prior reports of inconsistent correlations between subjective and objective measures of numeracy (Fagerlin et al., 2007; Nelson, Moser, & Han, 2013; Schwartz, Woloshin, & Welch, 2005). Our results were less consistent for the measures of health literacy, especially among Hispanic participants, all of whom completed the measures in Spanish. Taken together, these findings support prior research in highlighting the important differences that exist in the performance of several common measures used to assess health literacy and numeracy across racial/ethnic groups (Nguyen et al., 2015).

Although the positive correlation observed between education and BHLS score was similar across all three racial/ethnic subgroups, the nonsignificant correlations between the BHLS and the S-TOFHLA, as well as between the BHLS and the SNS among Hispanic participants, calls into question the validity of the Spanish version of the BHLS. Further testing revealed that the Cronbach's alpha for the Spanish version of the BHLS in our sample (0.53) fell below the acceptable level of 0.70 for internal consistency reliability. These results may be due, in part, to the fact that the BHLS does not specify if the medical forms, hospital materials, or written information referenced in the survey questions would be available in Spanish or English; thus, it is unclear if the BHLS is measuring Hispanic participants' comfort with health materials in English, Spanish, or both. A recent review found that several health literacy measures have undergone validity testing in Spanish-speaking populations (Stonbraker, Schnall, & Larson, 2015) but the BHLS was not one of them. Furthermore, Spanish-speaking patients were not included in several of the original validation studies of the BHLS (Chew et al., 2004; Chew et al., 2008; McNaughton, Wallston, Rothman, Marcovitz, & Storrow, 2011; Wallace et al., 2007; Wallace, Rogers, Roskos, Holiday, & Weiss, 2006; Wallston et al., 2014). Sarkar, Schillinger, Lopez, and Sudore (2011) tested the concurrent validity of the BHLS relative to the S-TOFHLA among English and Spanish-speaking people by calculating area under the receiver operating characteristic (ROC) and found that BHLS scores discriminate between participants with adequate health literacy and those with inadequate and/or inadequate plus marginal health literacy in both groups. Singh et al. (2015) tested the concurrent validity of individual items of the BHLS relative to the Short Assessment of Health Literacy (SAHL) and the Newest Vital Sign (NVS) among Spanish-speaking adults and found that the “confident with forms” question was a much better predictor of NVS and SAHL scores than the other two items. Importantly, neither of these studies examined the convergent validity of the BHLS with educational level among Spanish-speaking participants. The additional data provided by our study on the concurrent and convergent validity of the BHLS summative score highlights the need for further evaluation of the reliability and validity of this instrument among Spanish-speaking patients before using it to routinely assess health literacy in this population.

Our results should be interpreted in the context of the study limitations. The focus of the PRIDE study was on low-income patients with T2DM, so the sample for this analysis was relatively homogeneous in terms of socioeconomic status. To better evaluate the validity of these literacy and numeracy measures, they should be tested in more socioeconomically diverse populations. Future research on this topic should also include a greater proportion of minority participants. This cross-sectional analysis includes only one test of convergent validity; future longitudinal studies should evaluate the predictive validity of these measures by assessing their association with clinical outcomes, such as glycemic control over time, to better inform our understanding of the psychometric properties of these scales in minority groups. Several other numeracy and health literacy measures have been developed, but the PRIDE study only administered two examples of each. We might have observed different results if we had used different measures for each trait and method assessed in this study. Finally, because all of the Hispanic participants in PRIDE were administered the Spanish versions of the measures, we cannot be sure if the results for the Hispanic group are attributable to a feature of this ethnic group or to the Spanish versions of the measures included in this study.

Conclusion

The need to identify measures of health literacy and numeracy that have been validated for use in diverse communities is particularly important now that health literacy assessment is being incorporated into the Joint Commission's Provision of Care Standards (Wallston et al., 2014). The results of our study (and others like it) can inform both research and clinical practice by providing evidence-based recommendations for the selection of tools to best identify patients with limited health literacy and numeracy in minority populations. Our finding that the correlation of subjective measures is weaker for Hispanic and NHB participants than for NHW participants suggests objective measures may provide a better method of assessment when working with Hispanic and NHB patients. Additionally, the concurrent validity for numeracy measures is better than for health literacy measures among Hispanic participants. These results demonstrate the need to further test and/or adapt these measures before they can reliably be used for research or clinical purposes in minority communities.

Table 1

Baseline Characteristics of PRIDE Participants by Race/Ethnicity

Characteristic	All Participants (n = 398)	Non-Hispanic White (n = 234)	Non-Hispanic Black (n = 72)	Hispanic (n = 92)	p Value
Age, mean (SD)a	50.85 (9.44)	52.04 (8.80)	48.21 (10.04)	49.87 (10.08)	.005
BMI (kg/m2), mean (SD)a	35.85 (8.97)	37.77 (9.00)	35.72 (9.16)	30.98 (6.65)	< .001
Female, n (%)a	243 (61)	140 (60)	41 (57)	60 (65)	.53
Educationb
<12 years, n (%)	147 (37.1)	64 (27.5)	19 (26.4)	64 (70.3)	< .001
12 years, n (%)	138 (34.8)	100 (42.9)	27 (37.5)	11 (12.1)
>12 years, n (%)	111 (28.0)	69 (29.6)	26 (37.5)	16 (17.6)
Incomeb
≤$10,000/year, n (%)	214 (54.3)	127 (54.3)	45 (64.3)	42 (46.7)	.09
>$10,000/year, n (%)	180 (45.7)	107 (45.7)	25 (35.7)	48 (53.3)
Hemoglobin A1C, mean (SD)a	9.62 (2.07)	9.42 (1.96)	10.10 (2.34)	9.75 (2.06)	.04
DNT-5, mean (SD)a	0.46 (0.38)	0.60 (0.36)	0.30 (0.29)	0.23 (0.31)	< .001
SNS, mean (SD)a	26.01 (9.56)	25.93 (9.88)	23.99 (9.02)	27.80 (8.83)	< .039
BHLS, mean (SD)a	10.61 (3.46)	11.09 (3.50)	11.06 (3.20)	9.07 (3.15)	< .001
S-TOFHLA, mean (SD)a	29.22 (10.51)	31.75 (8.11)	31.49 (6.42)	21.20 (13.87)	< .001

Note. BHLS = Brief Health Literacy Screen; BMI = body mass index; DNT-5 = Diabetes Numeracy Test; PRIDE, Partnering to Improve Diabetes Education; SD = standard deviation; SNS = Subjective Numeracy Scale; S-TOFHLA = Short Test of Functional Health Literacy in Adults.

One-way analysis of variance was used to compare differences in age, BMI, sex, mean hemoglobin A1C, and scores for DNT-5, SNS, BHLS, and S-TOFHLA across racial/ethnic subgroups.

Chi-square test was used to compare differences in the highest level of education completed and income across racial/ethnic subgroups.

Table 2

Construct Validity of Numeracy and Health Literacy Measures by Race/Ethnicity in the PRIDE Study

Method of Assessment	All Participants	Non-Hispanic White	Non-Hispanic Black	Hispanic
Monotrait-Heteromethod
Trait: Numeracy
Subjective: SNS	0.37a	0.48a	0.41a	0.38a
Objective: DNT-5
Trait: Health literacy
Subjective: BHLS	0.42a	0.49a	0.43a	0.13c
Objective: S-TOFHLA
Monomethod-Heterotrait
Method: Subjective
Numeracy: SNS	0.25a	0.39a	0.15c	0.09c
Health literacy: BHLS
Method: Objective
Numeracy: DNT-5	0.59a	0.50a	0.25b	0.56a
Health literacy: S-TOFHLA

Note. Pearson correlation coefficients were calculated by comparing scores for specified measures. BHLS = Brief Health Literacy Screen; DNT-5 = Diabetes Numeracy Test; PRIDE = Partnering to Improve Diabetes Education; SNS = Subjective Numeracy Scale; S-TOFHLA = Short Test of Functional Health Literacy in Adults.

p < .001.

p < .05.

p > .05 (not significant).

Table 3

Correlation of Numeracy and Health Literacy Measures with Education by Race/Ethnicity in the PRIDE Study

Test/Screen	All Participants	Non-Hispanic White	Non-Hispanic Black	Hispanic
SNS	0.22a	0.44a	0.37b	0.21c
DNT-5	0.43a	0.41a	0.28c	0.35b
BHLS	0.45a	0.48a	0.41a	0.37a
S-TOFHLA	0.57a	0.37a	0.42a	0.57a

Note. Pearson correlation coefficients were calculated by comparing scores for each measure with highest education level attained. BHLS = Brief Health Literacy Screen; DNT-5 = Diabetes Numeracy Test; PRIDE = Partnering to Improve Diabetes Education; SNS = Subjective Numeracy Scale; S-TOFHLA = Short Test of Functional Health Literacy in Adults.

p < .001.

p < .01.

p < .05.