Effects of Education on Differential Item Functioning on the 15-Item Modified Korean Version of the Boston Naming Test.

OBJECTIVE: Education is expected to have an effect on differential item functioning (DIF) on the 15-item Modified Boston Naming Test in the Korean version of the Consortium to Establish a Registry for Alzheimer's Disease Assessment Packet (BNT-KC). However, no study has examined DIF in the BNT-KC.
METHODS: We used the item response theory to investigate the impact of education on the DIF in the BNT-KC among elderly individuals with or without dementia (n=720). A two-parameter item response model was used to determine the difficulty and discrimination parameters of each item. The Benjamini-Hochberg procedure was used to address the risk of Type I errors on multiple testing.
RESULTS: Four items, "mermaid," "acorn," "compass," and "pomegranate" continued to demonstrate DIF after controlling for multiple comparisons. Those with low education levels were more likely to error on "mermaid" and "compass," while those with high education levels were more likely to error on "acorn" and "pomegranate." "Hand" and "red pepper" were too easily identified to be used for detecting dementia patients. "Monk's hat" and "pomegranate" were less discriminating than other items, limiting their usefulness in clinical setting.
CONCLUSION: These findings may provide useful information for the development of a revised version of the BNT-KC to help clinicians make diagnostic decisions more accurately.

INTRODUCTION

The Boston Naming Test (BNT), introduced in 1983 by Kaplan, Goolglass, and Weintraub, may be the most widely used naming test worldwide.1 Impairment in language function is one of the core symptoms of dementia,2 so the BNT has been used in clinical situations to support the diagnosis of dementia. Moreover, BNT performance is useful for predicting the subsequent development of AD in preclinical cases or prognosis in patients with early AD.34

Meanwhile, the validity of tools used to assess cognitive function is usually influenced by a subject's educational attainment.5 Specifically, those with low education levels are often embarrassed and sometimes refuse to complete pencil-and-paper-based assessments. For those who are not accustomed to reading and writing, confrontational naming tests like the BNT may be more comfortable to perform because literacy is not required. From this point of view, the BNT is an especially useful tool for evaluating cognitive function in elderly Korean individuals since many have been deprived of the chance to learn to write during the Japanese colonization period and Koran war.

A 60-item Korean version of the BNT (K-BNT) was standardized by Kim and Na.6 The K-BNT uses a similar administration method to that of the original BNT, but most of the items were changed due to the linguistic and cultural differences between Korean and English speakers.7 When administered to patients with dementia, the length of the 60-item version often detracts from the patient's concentration and can be a barrier to completion. To address this limitation, several abbreviated forms have been proposed.89 The 15-item Modified Boston Naming Test (BNT-KC) in the Korean version of the Consortium to Establish a Registry for Alzheimer's Disease Assessment Packet (CERAD-K) Neuropsychological Assessment Battery (CERAD-K-N) is one of them.10 Fifteen items were chosen from the K-BNT. Three groups of five items each were identified as having high, medium, and low frequencies of occurrence. Phonemic and semantic dissimilarities in arranging the items were also considered.10 BNT-KC is the most widely used short version of the K-BNT because it is included in the CERAD-K-N, the most extensively used neuropsychological test battery in Korea.

Because cognitive test results are influenced by demographic factors such as age, sex, and educational level, normative information on eight tests in the CERAD-K-N were provided to help interpret the results. However, normative data does not give us enough information about the differences between the items. If the conditional probability of a correct response differs between two ability-matched groups, the item demonstrates differential item functioning (DIF).11 An item with DIF measures differently for one subgroup than another, resulting in less subgroup assessment validity. Thus, understanding the DIF that leads to intergroup score discrepancies may have important clinical implications that normative data cannot provide.

To date there have been several reports about DIF in the Mini-Mental State Examination (MMSE),121314 but there has been little research about DIF in the BNT except for race/ethnicity attributes.11 Considering that educational attainment in Korean elderly individuals varies widely from illiteracy to high education levels, DIF in BNT according to educational level is strongly expected. For example, some items of the BNT-KC such as “mermaid” and “dinosaur” are not usually spoken in ordinary life and are encountered only in literature or movies. Thus, individuals with low education levels, especially those who are illiterate, may have more difficulty responding correctly to these items despite not having naming ability impairments.

This study aimed to investigate the impact of education on DIF in BNT-KC in a clinical sample. To address this issue, the current study used the item response theory (IRT), which was used previously to study DIF in various psychiatric assessment tools.151617

METHODS

Sample

The study sample included total 773 elderly individuals, 386 from the National Dementia Epidemiological Study in Korea (NDESK) and 387 from the memory clinic of Kyungpook National University Medical Center (KNUMC). The study protocol for the NDESK was previously described in detail.18 In brief, the NDESK study investigated the prevalence and associated risk factors of dementia from a nationwide sample of Korean elders. A multi-stage cluster sampling design was adopted. Fifteen catchment areas were selected across six major administrative districts, including metropolitan Seoul and five provinces. Approximately 500 participants (65 years or older) were allotted to each catchment area. A total of 8,199 elderly individuals were invited to participate in the Phase I screening assessment using the Korean version of the MMSE (MMSE-KC) by door-to-door home visits; among them, 6,141 subjects responded (response rate=74.9%). Of those subjects, 2,336 were invited to participate in the Phase II diagnostic assessment for dementia and 1,673 subjects responded (response rate=71.6%). The CERAD-K-N was used to assess cognitive impairment.10 Among the 15 catchment areas, five catchment areas (Yeoncheon-gun, Incheon, Daegu, Gwangju, and Jungnang-gu in Seoul) were selected to investigate the prevalence of sleep problems and depression with executive dysfunction (DED). A total of 3,074 participants were selected from the five catchment areas, of which 2,007 subjects completed Phase I interviews (response rate=65.3%). Of those subjects, 806 were invited to participate in the Phase II diagnostic assessment for dementia, of which 557 subjects responded (response rate=69.1%). Data from 386 participants from five research centers who completed the BNT-KC were used in the current study. The study sample from the memory clinic in KNUMC included 387 elderly individuals aged ≥60 years who were evaluated for the presence of dementia between April 2013 and February 2015. The Institutional Review Board of KNUMC (Daegu, Korea) approved this study. From both samples, the individuals who were unable to complete the BNT-KC due to a visual disturbance, hearing difficulty, poor general condition, or other psychiatric condition were excluded. Among the 773 elderly who completed the BNT-KC, the participants who scored <10 on the MMSE-KC (n=53; 22 from NDESK and 31 from KNUMC) were also excluded because severe cognitive impairments can prevent proper test performance. Ultimately, the data of 720 individuals were analyzed (364 from NDESK and 356 from KNUMC).

Assessments

The CERAD-K-N was administered to every participant by neuropsychologists or trained research nurses.18 The CERAD-K-N consists of nine neuropsychological tests as follows: Verbal Fluency Test, 15-item Modified Boston Naming Test, MMSE-KC, Word List Memory Test, Constructional Praxis Test, Word List Recall Test, Word List Recognition Test, Constructional Recall Test, and Trail Making Test.10 All instruments were previously validated in the Korean population.19 The BNT-KC was administered to participants without cueing. All 15 line drawings were presented and the total number of spontaneous correct responses was calculated. In both NDESK and KNUMC samples, the diagnosis of dementia was defined according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition diagnostic criteria after a thorough clinical interview by clinicians, laboratory tests including complete blood cell count, chemistry profile, serological test for syphilis, echocardiography, chest X-ray, and brain computed tomography or magnetic resonance imaging.2

Statistical analysis

IRT model

To evaluate DIF in BNT-KC between the low and high education groups, a two-parameter item response model was used. The basic goal of IRT modeling is to determine item response functioning (IRF) for each item. An IRF is a mathematical function that describes the link between where an individual falls on the continuum of a given construct such as naming ability and the probability that he or she will give a particular response to a scale item designed to measure that construct, and it can be used to evaluate item quality.20 Difficulty is the ability value that is associated with a 50% probability of responding correctly on an individual item.21 Discrimination is an index of how well an item can differentiate between patients of varying severity levels.21 Item difficulty and discrimination are easily visualized using an item characteristic curve (ICC) for each item (Figure 1). Item difficulty is the location along the latent trait axis (x-axis) where the IRF curve changes direction (its inflection point; β in Figure 1).20 Item discrimination is the steepness of the IRF at the curve's inflection point (i.e., at the item's difficulty level; α in Figure 1).20 A two-parameter IRT model estimates a difficulty parameter and a discrimination parameter of the item.

Figure 1

Item characteristic curve. α: discrimination parameter, β: difficulty parameter.

Unidimensionality assumption

The important assumption of IRT is unidimensionality, which means that a single latent trait is sufficient to account for the observed pattern of item responses. The BNT-KC theoretically assumes that responses to each item are linked to a single construct of naming ability. Statistically, we tested the unidimensionality of the BNT-KC with confirmatory factor analysis using AMOS 21 (2012; IBM, Armonk, NY, USA). Model fit was evaluated with the comparative fit index (CFI), Tucker-Lewis Index (TLI), and the root-mean-square error of approximation (RMSEA).

Differential item functioning

We used IRTLRDIF version 2.0b (D. Thissen, IRTLRDIF v. 2.0b: Software for the Computation of the Statistics Involved in Item Response Theory Likelihood-Ratio Tests for Differential Item Functioning, 2001, unpublished manuscript) to carry out the DIF analyses.22 IRTLRDIF uses the likelihood-ratio test statistic to provide a significance test for the null hypothesis that the parameters of an item's response function do not differ between groups, and the significance of that test statistic is the detection of DIF.22

In the current study, the null hypothesis was that the item parameters do not differ between the low and high education groups. Analyses proceeded by initially constraining both difficulty and discrimination estimates were equal for the two groups across all 15 items. This step produces llAllEqual, the log likelihood for all item parameters constrained equal (no DIF for any item). For each of the 15 items, a model was then fit that constrains all of the remaining items' difficulty and discrimination estimates to be equal but allows the estimates for one item (Item I) to differ between the two groups (llItemInotEqual). The value of G2 (d.f.)=−2 (llAllEqual-llItemInotEqual) provides an omnibus test (df=2) of whether there is DIF for the difficulty and/or discrimination estimate for this item. The difference between the log-likelihood statistics of the two models is distributed as a χ2 statistic (G2) with degrees of freedom equal to the difference in parameter estimates between the two models.22 A significant G2 statistic at the p<0.05 level indicates that at least one of the parameters differs between the groups and is assumed to demonstrate DIF. Given that we conducted DIF analyses across multiple items, it is important to account for the risk of Type I errors. We employed the Benjamini-Hochberg procedure because it has been shown to demonstrate greater power than Bonferroni correction.2324

RESULTS

Sample characteristics

Characteristics of the final sample are summarized in Table 1. Participants were grouped into high (≥7 years) and low (≤6 years) education groups. The proportion of women in the low education group was higher than that in the high education group (71.2% vs. 34.8%, χ2=88.294, p<0.001). The low education group was significantly older (74.7±6.6 vs. 72.5±6.1, t=4.306, df=718, p<0.001) and had lower MMSE-KC scores (19.7±4.8 vs. 23.8±4.4, t=−11.480, df=487.327, p<0.001). The proportion of individuals living in rural areas (Yeoncheon-gun) was higher in the low education group compared to the high education group (23.7% vs. 14.8%, χ2=7.502, p=0.006). However, there was no significant difference in the proportion of individuals who were diagnosed with dementia between the low and high education groups (38.6% vs. 34.3%, χ2=1.195, p=0.274).

Table 1

Characteristics of the study participants (N=720)

*chi-square tests were used for categorical variables and t-tests were used for continuous variables. NDESK: National Dementia Epidemiological Study in Korea, KNUMC: Kyungpook National University Medical Center, SD: standard deviation, MMSE-KC: Korean version of the Mini-Mental State Examination

Unidimensionality assumption

Results from a confirmatory factor analyses showed χ2= 185.972, CFI=0.912, TLI=0.897, and RMSEA=0.047. These findings indicated a reasonable data fit. Results from a confirmatory factor analyses showed χ2=107.701, CFI=0.936, TLI= 0.924, and RMSEA=0.042. These findings indicated a reasonable data fit. We determined that these fit statistics were sufficient to proceed to fitting IRT models.

Differential item functioning

Table 2 shows the frequency of incorrect answers to each item of the BNT-KC in the low and high education groups as well as corresponding difficulty and discrimination parameter estimates. The ICC for each item is demonstrated in Figure 2. For every item, the frequency of incorrect answers was higher in the low education group than in the high education group, which was reflected in the difference in mean BNT-KC scores (7.5±3.1 vs. 10.0±2.9, t=−10.766, df=718, p<0.001). Seven of 15 items initially demonstrated DIF: “balloon,” “bat,” “mermaid,” “acorn,” “compass,” “pomegranate,” and “monk's hat.” After controlling for multiple comparisons, four items, namely “mermaid,” “acorn,” “compass,” and “pomegranate,” continued to demonstrate DIF, while the other items did not. Among the four items demonstrating DIF, “acorn” and “pomegranate” showed higher β parameters in the low education group, whereas “mermaid” and “compass” showed higher β parameters in the high education group.

Table 2

Item parameters for study participants (N=720)

*p<0.05, **p<0.01, ***p<0.001, †differential item functioning test not statistically significant after Benjamini-Hochberg procedure, G2=chi-square with two degrees of freedom (df)

Figure 2

Item characteristic curves for the items of the 15-item Modified Boston Naming Test in the Korean version of the Consortium to Establish a Registry for Alzheimer's Disease Assessment Packet Neuropsychological Assessment Battery (BNT-KC). Four items, namely “(8) mermaid”, “(9) acorn”, “(10) compass”, and “(12) pomegranate”, demonstrated differential item functioning between the low and high education groups. “(1) Hand” and “(2) red pepper” were too easy for most subjects. “(12) Pomegranate” and “(15) monk's hat” seemed less useful for the diagnosis of dementia due to their low discriminating feature.

The order of the four most difficult items was the same in both groups: “monk's hat” (β=1.54 for the low education group, β=1.65 for the high education group), “wind bell” (β=1.10 for the low education group, β=0.84 for the high education group), “dinosaur” (β=0.54 for the low education group, β=0.66 for the high education group), and “funnel” (β=0.14 for the low education group, β=0.21 for the high education group). “Pomegranate” is usually considered a difficult item because it is a low frequency word in the BNT-KC. However, its difficulty parameter (β=−0.77) was similar to those of middle frequency words (β=−0.87–−0.64) in the high education group. Moreover, it was even lower (β=−1.42) than those of the middle frequency words (β=−1.18–−0.23) in the low education group.

The order of the five easiest items was the same in both education groups: “hand” (β=−4.45 for the low education group, β=−151.95 for the high education group), “red pepper” (β=−3.96 for the low education group, β=−9.16 for the high education group), “baduk (go)” (β=−2.35 for the low education group, β=−2.54 for the high education group), “cobweb” (β=−1.96 for the low education group, β=−2.15 for the high education group), and “balloon” (β=−1.80 for the low education group, β=−1.16 for the high education group), all of which were high frequency words.

In the low education group, the least discriminating item was “monk's hat” (α=0.78), which represented the most difficult item. The second-least discriminating item was “pomegranate” (α=0.87) in this group. In the high education group, the least discriminating item was “hand” (α=0.29) and the second-least discriminating item was “red pepper” (α=0.61), which represented the easiest items, and the error rate was <2% in both groups. Interestingly, the third-least discriminating item in the high education group was “pomegranate” (α=0.66), which was the second least discriminating item in the low education group. The most highly discriminating item in the low education group was “mermaid” (α=2.54), while that in the high education group was “baduk (go)” (α=2.54).

DISCUSSION

Here we explored the influence of education on DIF in BNT-KC using IRT. To the best of our knowledge, this was the first study to investigate DIF of BNT by educational attainment. To date, few studies have examined DIF in the BNT except with respect to race/ethnicity attributes.11 In contrast, several studies have reported the influence of education on DIF in the MMSE.12131425 These studies have reported that naming items (e.g., pencil, watch, etc.) are biased by level of education;1425 however, this finding is not consistent across studies.13

In the current study, we found DIF in several items of the BNT-KC that were attributable to educational attainment. Those with low education levels were more likely to respond incorrectly to “mermaid” and “compass” despite having comparable naming ability to that of individuals with high education levels, whereas those with high education levels experienced greater difficulty correctly responding to “acorn” and “pomegranate” compared to those with low education levels.

Both “mermaid” and “compass” are items that are uncommon in everyday life. “Mermaid” is a character in fairytales of western culture of which most Korean elders with low education levels are ignorant. Wrong answers often include “half girl, half fish” or “fish woman,” and even after they were explained the answer (mermaid), they did not understand it. Considering that aged people in Korea had a lower chance of being familiar with western culture, the cultural factors should be considered in the process of developing a revised naming test to aid the diagnosis of dementia.

“Compass” is not a familiar item in everyday life as well. Most people have a basic understanding of a compass—its magnetic characteristic and the usage for orientation—from their time in school. Thus, the higher difficulty experienced in identifying this item in the low educated group is not surprising. Another possibility is that this result may come from the confounding effect of sex on education since, in the current sample, the high education group showed a higher proportion of male sex than the low education group. Korea has a conscription system for men, who are taught how to read a map using a compass during military life. In the analysis of DIF using IRTLRDIF, we could not control the confounding effect of other variables on the results. To eliminate the confounding effect of sex, we had to match the sample according to sex, but we did not consider it in the study design because the smaller sample size may decrease the detectable effect size at a given power.26 To address this limitation, a larger study based on a sex-matched sample may be required.

Two items, “acorn” and “pomegranate,” demonstrated lower naming difficulty in the low education group than in the high education group. Decades ago, individuals raised in rural areas usually stopped going to school after elementary school to help with the farm work. Thus, those individuals usually had fewer years of education than those raised in urban areas. However, both “acorn” and “pomegranate” are fruits of plants that can be more commonly seen in rural than in urban areas. This may have contributed to the lower naming difficulty of these items in the low education group. The finding in the current study that the proportion of individuals living in rural areas was higher in the low education group compared to the high education group may support this interpretation. However, given that individuals may learn the words “acorn” and “pomegranate” during childhood or adolescence, data about childhood residence may be more critical than data about current residence. Unfortunately, we did not collect data on childhood residence in this study.

It is important to note that this result does not mean that the proportion of individuals who responded correctly to “acorn” and “pomegranate” was higher in the low education group than in the high education group. Rather, it means that if the individuals have the same naming ability (i.e., total score on the BNT-KC), the probability of responding correctly to “acorn” and “pomegranate” is higher in individuals with low education levels than in those with high education levels. Although this simplified explanation assumes that DIF types are uniform,11 all of the items in the current study demonstrated uniform DIF (Figure 2).

It is usually expected that the less frequent the word is encountered, the more difficult it is for one to name it. However, in case of “pomegranate,” even though it is a low frequency word, its naming difficulty was similar to those of middle frequency words. This demonstrates that it is not always true that the frequency of exposure to a word is inversely correlated with difficulty naming it.

“Hand” and “red pepper” were too easy for most subjects, so these items appeared less useful for the early diagnosis of dementia in the clinical setting. Besides, “pomegranate” and “monk's hat” seemed less useful for the diagnosis of dementia due to their low discriminating feature. In other words, many individuals with a normal naming ability failed to correctly name these items, while persons with impaired naming ability frequently named them correctly. The low discriminating feature of these items may in part be attributable to the vagueness of the figure in the BNT-KC. Persons with normal cognitive function often mistake the pomegranate for a flower and the monk's hat for a towel. Further study of the visual perceptual errors of these items is needed.

We must acknowledge several study limitations. First, as mentioned earlier, we did not control for the possible confounding effects of other variables such as age and sex. We did not observe an interaction between education and potential confounding variables such as age, sex, and residence. Nevertheless, differences in sociodemographic variables between low and high education groups may have influenced the findings of this study. Several studies have reported an association between increasing age and lower BNT scores.192728 In the current sample, the low education group was approximately 2 years older than the high education group. However, this difference was not large enough to infer any clinical significance, given that normative values for the BNT are usually presented in at least 5-year intervals. The influence of sex on language remains controversial. Although sex did not affect scores on the original BNT or 60-item K-BNT,727 sex had a significant effect on the total BNT-KC score, necessitating the development of normative values for age, sex, and education level.19 Several investigators have reported that AD has a greater negative impact on language function in female patients,2930 although this finding is not consistent across studies.31 To the best of our knowledge, no previous reports have found an influence of sex on DIF in the BNT. However, one study found an influence of sex on DIF in the Picture Naming in Repeatable Battery Assessment of Neuropsychological Status (RBANS),32 which suggests that sex is a potential confounding factor in the current study. The only method we know to control for confounding in DIF analyses based on IRT is matching; however, we were unable to match samples in the current study due to our limited sample size. To adjust for multiple variables without matching, other statistical methods such as the multiple indicators-multiple causes (MIMIC) model or logistic regression model can be used.1333 However, we employed the IRT model to explore item characteristics (e.g., difficulty, discrimination, etc.), which cannot be measured by other approaches. In the future, a sufficiently large sample matched for age and sex may be warranted to examine the effect of education on DIF. Second, patients with dementia were considered a homogeneous group in the current study. However, there have been reports that there are differences in naming difficulty patterns among dementia types (i.e., AD vs. vascular dementia; AD vs. frontotemporal dementia).3435 Although the proportion of patients with dementia did not differ between the low and high education groups, the different dementia types could be influential. Third, the results of this study cannot be generalized to other countries using different language versions of the BNT since the items in the BNT-KC were altered from the original BNT to compensate for the linguistic and cultural differences between Korean and English speakers.7 Nevertheless, the results of this study suggest that DIF attributable to education could exist in other language versions of the BNT.

In summary, the current study revealed that elderly individuals in Korea with low education levels are more likely to incorrectly name “mermaid” and “compass,” while those with high education levels are more likely to incorrectly name “acorn” and “pomegranate.” “Hand” and “red pepper” are too easily named for use in the early detection of dementia, while “monk's hat” and “pomegranate” are less discriminating than other items, which limits their usefulness in the clinical setting. These findings may provide useful information for the development of a revised version of the BNT-KC to help clinicians make diagnostic decisions more accurately.