Validation of the Short-Test of Functional Health Literacy in Adults for the Samoan Population.

BACKGROUND: Health literacy encompasses various levels of communication for an individual, provider, and an organization. Validated and reliable tools have been developed to assess health literacy; however, there is a paucity of tools available to assess health literacy in native languages for indigenous and racial/ethnic minority populations.
OBJECTIVE: This article shares the process taken to translate and evaluate validation and reliability of the Short Test of Functional Health Literacy in Adults for use with the Samoan population.
METHODS: Respondent-driven sampling was used to collect data from 1,543 adults age 45 years and older in American Samoa. A confirmatory factor analysis using a two-factor model for validation was conducted. KEY
RESULTS: The validation results indicated a "good fit" in multiple indices and Cronbach's alpha indicated high internal consistency in both the English and Samoan languages.
CONCLUSIONS: Developing culturally validated and reliable health literacy assessment tools is important to help health care professionals decrease health disparities and address inadequate health literacy in all cultures. [HLRP: Health Literacy Research and Practice. 2022;6(4):e247-e256.] Plain Language Summary: The INSPIRE project studied the Short Test of Functional Health Literacy in Adults (STOFHLA) tested on the American Samoan population age 50 years and older. The results would show if the STOFHLA is a valid tool to measure functional health literacy in American Samoa adults.

Health literacy is an emerging issue and was designated a priority area in public health by the United States Department of Health and Human Services (Institute of Medicine, 2004). Health literacy requires a combination of reading, listening, and analytical, skills that can be applied to making decisions in health situations (National Institutes of Health, 2021). Health literacy encompasses various levels of communication for an individual, provider, and an organization. For the purposes of this study, health literacy will be referred to as the individual's functional health literacy defined as “the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions” (Institute of Medicine, 2004).

Sentell and Braun (2012) found that inadequate health literacy levels were associated with poorer self-reported health status in racial and ethnic minority populations. The U.S. population is comprised of 42.2% racial and ethnic minority and indigenous groups (U.S. Census Bureau, 2020). It is conceivable that these populations may not participate in health promotion or follow medication directions because the populations cannot understand them (Tong, 2012). This may result in more health care utilization and contribute to higher costs (Haun et al., 2015). These groups are affected more than others by health disparities, including access to health care and higher rates of mortality, yet it is difficult to find culturally appropriate valid and reliable health literacy assessment tools (Han et al., 2011; Haun et al., 2012; Nguyen et al., 2015; Polite et al., 2005). Additionally, with the current “infodemic” related to the COVID-19 (coronavirus disease 2019) pandemic, health literacy is crucial to navigating the overabundance of valid and invalid health information (Okan et al, 2020; Paakkari & Okan, 2020).

Health-Related Issues in American Samoa

American Samoa is a U.S. territory located in the South Pacific, approximately 2,400 miles southwest of Hawai'i and 1,600 miles east of New Zealand. The territory is comprised of five islands and two atolls measuring approximately 76.8 square miles. The population was reported to be 55,519 individuals, with 92% who self-identify as Samoan (World Bank, 2020). Approximately 57.8% of the population earns lower than the federal poverty level and is a contributing social determine of health to alarmingly high levels of behavioral risk factors for disease (Tofaeono et al., 2020; U.S. Census Bureau, 2010). The result has been a shift in the health burden for the past 30 years from communicable to non-communicable diseases such as diabetes, hypertension, cancer, and Alzheimer's disease and related dementias. The American Samoa Cancer Registry reported a total of 369 cases of cancer diagnosed between 2007 and 2018 (Pacific Regional Central Cancer Registry, 2020). The most prevalent cancer sites are (1) breast, (2) uterine, (3) colorectal, (4) stomach, (5) lung, and (6) prostate (Pacific Regional Central Cancer Registry, 2020). Tobacco (79%) and obesity (62%) were the most related contributing risk factors in adult cancers (Pacific Regional Central Cancer Registry, 2020). Furthermore, in 2018 of the adults age 18 years and older, 93.5% were obese, 21.5% were daily smokers, and 30.9% self-reported fair or poor health (American Samoa Government, 2018, p. 4). Persons who have higher prevalence estimates of risk factors require more health-related information (Knighton et al., 2017). Assessing individual levels of functional health literacy could identify health disparities within the population.

The INdigenous Samoan Partnership to Initiate Research Excellence (INSPIRE) program is a 5-year U24 project funded by the National Institutes of Health's National Institute of Minority Health and Health Disparities (1U24MD011202). The U24 mechanism supports projects contributing to increase the capability of resources for research by building infrastructure and implementing a study. The research study aim of the INPSIRE program was to assess the health literacy levels in adults age 45 years and older living in American Samoa. The results would address the paucity of data for health literacy and create the first validated and reliable tools to assess functional health literacy in American Samoa. Thus, this article describes the validation and reliability procedures for the English and Samoan versions of the Short Test of Functional Health Literacy in Adults (S-TOFHLA). The data used for this article are a combination of three datasets collected by the INSPIRE program.

The Short Test of Functional Health Literacy in Adults

The S-TOFHLA is a timed reading comprehension test that uses a modified Cloze procedure to measure the understanding of written material (Baker et al., 1999). The STOFHLA was selected because of its use in approximately two thirds of published papers assessing health literacy, its ability to determine health literacy as a variable in education programs (Paasche-Orlow, 2005), and previous use in the Samoan population in California (Tong, 2012). Participants read sentences with missing words and circled the letter (either a, b, c, or d) in front of the word the participant feels would fit best (Figure ). The original S-TOFHLA was comprised of 36 reading comprehension questions, 4 numeracy questions, and had a 12-minute time limit (Baker et al., 1999). The 36 questions were based on 2 passages: preparation for an upper gastrointestinal series (4th grade level) and the patient rights and responsibilities section of a Medicaid application (10th grade level). The numeracy questions were later removed based upon the difference in psychometric properties, reducing the administration time to 7 minutes (Paasche-Orlow et al., 2008). This version was used for the purposes of these studies.

Short Test of Functional Health Literacy in Adults question.

The S-TOFHLA has been translated into several different languages for use with diverse populations including Brazilian Portuguese, Chinese, Hebrew, Hungarian, Serbian, and Turkish. These studies calculated reliability but experienced limitations when adapting the instrument due to issues with contextual translations and the need to modify examples to become relevant to local health systems (Connor et al., 2013). For instance, Baron-Epel et al. (2007) created the Hebrew Health Literacy Test using the S-TOFHLA, as items were not valid in Hebrew or not relevant in the Israeli health system. The scoring was changed to a scale of 0 to 12, and therefore the scoring categories were 0 to 2 for low health literacy, 3 to 10 for marginal health literacy, and 11 to 12 for high health literacy. Irrespective of these modifications, Cronbach's alpha was relatively high in the Hebrew version as well as other translated scales (Table ).

Chang et al. (2012) developed and evaluated the Chinese version of the short Test of Functional Health Literacy in Adolescents (c-sTOFHLAd). This is the only study to perform a confirmatory factor analysis (CFA) to assess the underlying factor structure. The initial analysis began with a two-factor model, but less than adequate results changed the approach to a single factor. Chi-square was not within limits; however, 3 out of 5 goodness of fit indices were within acceptable ranges (Table ). The study concluded the c-sTOFHLAd as the first validated tool available to measure functional health literacy in Chinese adolescents (Chang et al., 2012). Other studies have used CFA to successfully evaluate validity in other health literacy assessment tools, such as the Health Literacy in Dentistry Scale (Ju et al., 2018), the Health Literacy Instrument for Adults (Tavousi et al., 2020), and the Oral Health Literacy Assessment in Spanish (Bado et al., 2018). Developing culturally validated and reliable health literacy assessment tools is important to help health care professionals decrease health disparities and address inadequate health literacy in all cultures (Tong, 2012).

Methods

Ethical Considerations

All study procedures were reviewed and approved by the American Samoa Department of Health's Institutional Review Board (#00001249) and Federal Wide Assurance (#00001749). Participants were screened for eligibility and completed an informed consent protocol, including a signed consent form in their preferred language.

The Samoan S-TOFHLA

The S-TOFHLA was translated into the Samoan language using a modification of the process described by Beaton et al. (2002). Forward translation was performed by four research trainees. An independent party with a background in translation services reviewed the document to obtain synthesis in creating a common translation. The common translation was back translated using four separate focus groups, comprising a total of fourteen participants.

Eligible participants were identified as bilingual, non-medical personnel, male or female, at least age 45 years, and recruited through word of mouth or social media posts. During each focus group, the common translation was transferred into a matrix provided to each participant for back translation. Once completed, the matrices were collected, and slides were shown for each question and the corresponding English translation. An audio-recorded open group discussion took place to obtain feedback. The translation matrix was revised and re-administered to the next focus group until a consensus translation was reached after the fourth focus group.

The consensus translation of the Samoan S-TOFHLA was used to conduct 30 cognitive interviews to collect verbal information on the item response characteristics and quality, to ensure equivalency and cultural appropriateness, and to determine if the questions generated the intended information (Willis & Artino, 2013; Willis et al., 2005). INSPIRE staff were rigorously trained in cognitive interviewing by INSPIRE co-investigators, which included training in the following areas: ensuring participants understood the measurement objectives of the items, knowledge of possible survey errors, recognizing adequate think-aloud and probe responses, being sensitive to cultural and linguistic nuances, and recognizing participant confusion and reluctance (Hay et al., 2014; Willis & Artino, 2013; Willis et al., 2005). One-hour semi-structured cognitive interviews were conducted at locations convenient and private to participants. Staff conducted a pre-interview process with participants to discuss the purpose of the study, obtain informed consent, and for the participants to be aware that the interview would be audiotaped and that their information would remain confidential. The interviewer evaluated the participant's comprehension of the question (question intent, meaning), retrieval from memory for relevant information (recall of information and strategy), decision-making processes (motivation and sensitivity/social desirability), and response processes (mapping the response) (Hay et al., 2014; Willis & Artino, 2013; Willis et al., 2005). At the end of the interview, participants received a $20 gift card for their participation.

Both think-aloud and verbal probing techniques were used to take advantage of the strengths of both methods (Hay et al., 2014; Willis & Artino, 2013; Willis et al., 2005). To reduce interviewer bias associated with verbal probing, a list of scripted and spontaneous probe questions to elicit responses and focus on specific issues that emerge during the interview was created. Staff asked the participants to respond to the survey questions and about information related to the question and answer provided (Willis & Artino, 2013). Interview recordings were transcribed and compiled into online qualitative data analysis software to identify contextual themes. The results were shared with the INSPIRE principal investigator, co-investigators, and program director. Revisions were made accordingly, and a final translation was approved.

Validation and Reliability

Validity and reliability are fundamental elements that distinguish whether an instrument is measuring what it is intended to measure and if it can do so consistently (Tavakol & Dennick, 2011). Cronbach's alpha coefficient is the most widely used measure to determine reliability in social and behavioral research studies (Bonett & Wright, 2015). A coefficient of 0.70 is considered the minimum acceptable reliability coefficient (Thorndike, 1995). CFA has been the preferred method to test validity by measuring the degree of discrepancy between predicted and empirical factor structures to determine if the construct is well-structured (Meyer, 2020; Prudon, 2015). The model is evaluated using the relative chi-squared (χ2/df) test statistic and the following indices of “goodness of fit”: goodness of fit index, adjusted goodness of fit index (AGFI), comparative fit index (CFI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR) (Prudon, 2015). Additionally, it is recommended that at least two different indices be used to determine validity (Hu & Bentler, 1999; Tabachnick & Fidell, 2013).

Sample

Respondent-driven sampling was used from September 2018 to December 2020 to test the efficacy of social relationships and networks in American Samoa to recruit for research purposes. Upon completion of the studies, participants were given a primary incentive and coupons to recruit a maximum of three participants. A secondary incentive was provided to the recruit for those who completed the study before the expiration period. This process continued until the study samples were met. Participants were eligible for inclusion in the studies based on the criteria listed in Table . Exclusion criteria included persons who could not speak or read and write in Samoan less than well.

Procedures

The studies were administered in a dedicated space at the INSPIRE program's main office. Located in the village of Nu'uuli, one of the more populated villages on island, the office was conveniently positioned off the main road, making it accessible to those with or without private transportation. The study staff were comprised of six staff members trained and assigned to four specific roles of the survey process. Administration of the English and Samoan S-TOFHLA followed the directions provided by the original publishers. A scripted introduction was read verbatim by an INSPIRE research staff member. The participants were provided with the instrument and were not informed of a time limit. The researcher began timing using a digital clock once the participants circled the first response. After the designated 7 minutes, the researcher notified the participants the completion of the assessment, collected the instrument, and thanked them for participating. The results were entered into an online survey system for analysis. Participants were provided instructions for recruitment and were referred to receive their primary incentive payment.

Statistical Analysis

Descriptive statistics were obtained using IBM SPSS Statistics 27 for descriptive statistics and to calculate Cronbach's alpha. SAS 9.4 was used for evaluating construct validity using CFA to measure if two higher-order and six lower-order latent variables of the S-TOFHLA can make inferences to individual functional health literacy. The CFA is used more when there is a proven structure using a new data set, rather than exploratory factor analysis when there is no knowledge of the factor structure (Orçan, 2018). Correlation of the two higher-order factors was allowed. The maximum-likelihood parameter estimation method assessed the overall goodness-of-fit and estimated the individual parameters. Items 1–16 represented the gastrointestinal series (higher-order factor 1), and items 17–36 represented the patient rights and responsibilities section (higher-order factor 2) of a Medicaid application (the lower-order factors represented subsections). Chi square along with the following fit indices and cutoff values recommended by Hooper et al., (2008) were used to evaluate the goodness of fit based upon sample size, parsimony, and incremental fit indices: RMSEA <.07, AGFI >.90, SRMR <.08, and Bentler CFI >.90. These are among the standard indices that are reported and better suited for large study samples (Prudon, 2015). In addition, diagonally weighted least squares (DWLS) was calculated for SRMR and AGFI as it is more appropriate for dichotomous items and a better fit for maximum likelihood. Reliability was assessed of the two higher-order factors for internal consistency with Cronbach's coefficient alpha. An alpha coefficient of ≥0.70 was considered acceptable.

Results

RDS recruitment yielded 1,543 total participants (693 English, 850 Samoan). Women (66.1% English, 54.7% Samoan) were recruited more than men (33.9% English, 41.6% Samoans).

Descriptive Statistics

No items were missing values. The mean score for English was 23 (standard deviation [SD] = 9.74) and 17.6 (SD = 9.49) for Samoan. Individual item means, standard deviations, and correlations are listed in Figure (English) and Figure (Samoan). Skewness for English (−0.35) indicates a left tail, a right tail in Samoan (0.03), and can be considered asymmetrical. Kurtosis of both languages (English −1.24, Samoan −1.09) indicate a platykurtic curve.

English sample item means, standard deviations, and correlations.

Samoa sample item means, standard deviations, and correlations.

Validity

CFA was performed through structural equation modeling, with maximum likelihood and DWLS. Table reveals the results of the goodness-of-fit indices.

The two-factor CFA indicated that, in the study population, the data did support the specified model. Chi-square testing of both versions (English, χ2 = 2,535, df 587, p < .0001; Samoan, (χ2 = 1913, df 587 p < .0001) showed the model fit was significant, but chi square is known to be inconsistent when having larger sample sizes (Bentler & Bonnett, 1980).

Additionally, the factor loadings of the 36 items of the Samoan S-TOFHLA were calculated as two factors as a measure of an individual's level of health literacy (Figures ).

English sample standardized results.

Samoan sample standardized results.

Reliability

Cronbach's alpha of the English higher order factors was 0.87 (Factor 1) and 0.96 (Factor 2). The Samoan 0.86 (Factor 1) and 0.95 (Factor 2). All results were higher than the suggested minimum value of 0.70 as an indication of acceptable internal consistency reliability (Thorndike, 1995).

Discussion

The present study evaluated the validity and reliability of the English and Samoan version of the S-TOFHLA for use in the American Samoa population in those who are age 45 years and older. CFA provided support for the validity of a two-factor model in both the English and Samoan S-TOFHLA. Consequently, the English and Samoan S-TOFHLA are the first statistically validated tool to measure functional health literacy in adults age 45 years and older living in American Samoa. Additionally, this is only the second study to use CFA to validate the factor structure. This is important for two reasons. First, it decreases the paucity of validated instruments that are appropriate to use for indigenous populations. Second, the Samoan population has been one of the largest and fastest growing Native Hawaiian and Other Pacific Islander groups in the U.S. Social determinants of health that can effect functional health literacy in indigenous Samoans may change for those who are acculturated in the U.S. The validation of the Samoan s-TOFHLA allows for further research in these groups and to compare the results to find innovative approaches that improve public health practices.

The S-TOFHLA has been extensively used with adults and in languages other than English, but the measure has not been used with adults in American Samoa. The results in the present study indicated that both the English and Samoan versions of the S-TOFHLA had high internal consistency reliability. This is consistent with other studies using English and translated versions of the S-TOFHLA. Additionally, there are similarities in the health systems of American Samoa and the U.S. The Samoan S-TOFHLA was able to maintain the same items, point totals, and cut-offs as the English version.

Given these contributions, there are limitations to the study that deserve mentioning. First, it is important to point out that validity is not a property of the instrument, but of the instrument when used on a sample (Anastasi, 1989; Knekta et al., 2019). The sample included only older adults age 45 years and older. Therefore, the results cannot be generalizable to the overall population of American Samoa. Future research should include other age groups and further evaluation of the psychometric properties. Second, an important limitation to consider is that the S-TOFHLA only measures reading ability or print literacy, and not numeracy. Therefore, one could argue that the measure is not a comprehensive test of health literacy (Housten et al., 2018). In addition, there are instruments that assess health literacy in clinical settings, in health care providers, and regarding sociodemographic and geographic factors. These instruments should be considered in addition to determining how health literacy should be assessed in low-resource urban area populations, including indigenous populations (Haun et al., 2012).

Conclusion

Functional health literacy is defined as the ability of an individual to comprehend health information for informed decision-making. Although it has been designated a priority area, there are limited validated tools to assess health literacy for racial and ethnic minorities and indigenous populations. This is the first study to assess the validity and reliability of the English and Samoan version of the S-TOFHLA. The two-factor model demonstrated the best fit, and the measure's reliability was confirmed using Cronbach's alpha. Consequently, both the English and Samoan versions of the S-TOFHLA have utility as a validated instrument to assess functional health literacy in the indigenous adult population of those age 45 years and older living in American Samoa.

Table 1

Reliability Coefficients of Translated Versions of the Short Test of Functional Health Literacy in Adults

Language	Author	Sample Size	Age Group (years)	Cronbach's Alpha
Brazilian Portuguese	Carthery-Goulart et al. (2009)	312	19–81	-
Chinese	Chang et al. (2012)	300	16–17	0.85
Hungarian	Náfrádi et al. (2019)	302	20+	0.95
Hebrew	Baron-Epel et al. (2007)	119	19+	0.98
Serbian	Jovic-Vranes et al. (2013)	120	21–84	0.94
Turkish	Eyüboğlu & Schulz (2016)	302	18+	0.97

Table 2

Goodness-of-Fit Indices for the Chinese Version of Short Test of Functional Health Literacy in Adolescents

Index	Cut-Off for Good Fit	c-STOFHLAd
Chi Square	p value >.05	p < .001
RMSEA	RMSEA <0.08	0.06[a]
GFI	GFI ≥ 0.95	0.92
AGFI	AGFI ≥0.90	0.90[a]
SRMR	SRMR <0.08	0.068[a]
CFI	CFI ≥0.90	0.89

Note.

Denotes the value is within the specific cut-off for good fit. AGFI = adjusted goodness of fit index; CFI = comparative fit index; c-sTOFHLAd = Chinese version of Short Test of Functional Health Literacy in Adolescents; GFI = goodness of fit index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual.

Table 3

Inclusion Criteria

Criteria	Study 1	Study 2	Study 3
Resident of American Samoa	X	X	X
Age 45 years or above	X		X
Age 60 years or above		X
Had not been diagnosed or treated for colon cancer	X
Did not participate in a previous health literacy study		X	X
Could read and speak in Samoan and English	X	X	X
Could provide written informed consent	X	X	X
Have a home address and working phone	X	X	X
Are willing to provide information on their health behaviors and basic demographics	X	X	X

Table 4

Confirmatory Factor Analysis Goodness-of-Fit Indices Results

Index (Cutoff)	English (DWLS)	Samoan (DWLS)
SRMR (<.08)	0.08 (0.07)[a]	0.07 (0.06)[a]
AGFI (>.90)	0.79 (0.99)	0.86 (.99)
RMSEA (<0.08)	0.07[a]	0.05[a]
Bentler CFI (> 0.90)	0.89	0.91[a]

Note.

Denotes the value is within the specific cut-off for good fit. AGFI = adjusted goodness of fit index; CFI = comparative fit index; DWLS = diagonally weighted least squares; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual.