HIV/AIDS knowledge among men who have sex with men: applying the item response theory.

OBJECTIVE: To evaluate the level of HIV/AIDS knowledge among men who have sex with men in Brazil using the latent trait model estimated by Item Response Theory.
METHODS: Multicenter, cross-sectional study, carried out in ten Brazilian cities between 2008 and 2009. Adult men who have sex with men were recruited (n = 3,746) through Respondent Driven Sampling. HIV/AIDS knowledge was ascertained through ten statements by face-to-face interview and latent scores were obtained through two-parameter logistic modeling (difficulty and discrimination) using Item Response Theory. Differential item functioning was used to examine each item characteristic curve by age and schooling.
RESULTS: Overall, the HIV/AIDS knowledge scores using Item Response Theory did not exceed 6.0 (scale 0-10), with mean and median values of 5.0 (SD = 0.9) and 5.3, respectively, with 40.7% of the sample with knowledge levels below the average. Some beliefs still exist in this population regarding the transmission of the virus by insect bites, by using public restrooms, and by sharing utensils during meals. With regard to the difficulty and discrimination parameters, eight items were located below the mean of the scale and were considered very easy, and four items presented very low discrimination parameter (< 0.34). The absence of difficult items contributed to the inaccuracy of the measurement of knowledge among those with median level and above.
CONCLUSIONS: Item Response Theory analysis, which focuses on the individual properties of each item, allows measures to be obtained that do not vary or depend on the questionnaire, which provides better ascertainment and accuracy of knowledge scores. Valid and reliable scales are essential for monitoring HIV/AIDS knowledge among the men who have sex with men population over time and in different geographic regions, and this psychometric model brings this advantage.

INTRODUCTION

Brazil has adopted the Declaration of Commitment of the United Nations, which aimed at slowing the HIV/AIDS epidemic by 2015. HIV/AIDS knowledge is one of the indicators proposed by the United Nations General Assembly Special Session for monitoring AIDS among vulnerable subgroups including, sex workers, injection drug users and men who have sex with men (MSM). This indicator requires that adequate HIV/AIDS knowledge is an essential prerequisite for the adoption of behaviors that reduce the risk of infection.[a] Global estimates among low and middle income countries indicate that 56.0% of MSM did not have correct knowledge about HIV/AIDS, 70.0% had never been tested for HIV and 46.0% did not use condoms the last time they practiced anal sex.[2]

In Brazil, the AIDS epidemic is concentrated in population subgroups with a markedly higher risk of acquiring HIV infection among MSM.[4,6] Recent data from a national survey found a worrisome HIV prevalence rate of 14.2% among this population.[10] The promotion of safer sex has been the main policy strategy for HIV prevention and access to information is a key element in reducing vulnerability to HIV/AIDS by providing an opportunity to change individual attitudes and increase safe practices.[b]

Despite its importance, there is no standard instrument used to assess HIV/AIDS knowledge across different populations and many studies vary in their choice of the number and formulation of questions used.[18,19] Moreover, the knowledge score has been expressed as the sum of the correct responses to each item, based on Classical Test Theory (CTT), and an arbitrary cut-off point is established in order to classify good or adequate knowledge.[9,11] Thus, the total score is dependent on the set of items that compose the measuring instrument, which makes it difficult to compare results between different studies.[c]

Recent studies assessed HIV/AIDS knowledge using the Item Response Theory (IRT).[1,13,15] IRT is considered the modern theory of psychometry, focuses on each item of the measuring instrument and assumes that the performance of a given test can be explained by individual characteristics, not directly observable, named latent traits.[3,14] Although IRT does not contradict classical methods, it is based on the assumption that the estimate of the latent trait is independent of sample items, which allows the equalization of scores on evaluations when using different instruments applied to the same population.[14] This is especially important when assessment of HIV/AIDS knowledge is used to monitor effectiveness of public health policies over time among vulnerable populations. More precise and accurate analysis of knowledge may allow proper planning of new prevention intervention strategies.

The objective of this study was to evaluate the level of HIV/AIDS knowledge among men who have sex with men from Brazil using the Item Response Theory.

METHODS

This analysis is part of a cross-sectional study of MSM carried out in 10 Brazilian cities in 2008-2009. The main objectives of the national study were to estimate the prevalence of HIV and syphilis and to assess knowledge, attitudes and sexual practices of MSM.[10]

Participants should be residents of the following cities: Manaus, Recife, Salvador, Campo Grande, Brasília, Curitiba, Itajaí, Santos, Rio de Janeiro and Belo Horizonte. These were a priori chosen by the Department of STD, AIDS and Viral Hepatitis of the Ministry of Health (STD/AIDS/MH). Eligibility criteria included adult MSM (18 years old or over), who reported at least one sexual contact with another man in the 12 months preceding the interview.

The sample size was previously calculated as 250 to 350 participants by city (α = 0.05, power = 0.90, estimated prevalence = 13.6%) [10] to provide independent estimates for each city and it was obtained using the Respondent Driven Sampling technique.[12] This technique is a chain link sampling method used to address hard to reach populations and their social networks, since the lack of a sampling base does not allow the application of traditional probabilistic method. The recruitment is carried out by participants themselves using a dual incentive system, and begins with a convenience sample of members of the target population, named seeds. In this study, these were selected during preliminary formative research, when individuals of different ages and socioeconomic classes were included. In each city, participants received three unique coupons, non-falsifiable, to distribute to their peers. Individuals who came to the study site with a valid coupon and who met the inclusion criteria were considered the first “wave” of the study. Each participant also received three coupons to invite new acquaintances, repeating this process thereafter until the desired sample size was reached in each city.

Data were collected in face-to-face interviews, composed of questions regarding sociodemographic data, behavior, social context, health care and HIV/AIDS knowledge. Participants were also invited for HIV and syphilis testing.

HIV/AIDS knowledge was assessed using 10 statements to which participants should indicate whether each one was true, false or “did not know”. The statements were chosen from previously used instrument by the STD/AIDS/MH at face value. For this analysis, correct responses were categorized as 1 and those which were considered incorrect or to which respondents replied “did not know” were categorized as 0.

Descriptive analysis of the sample was carried out and HIV/AIDS knowledge was assessed by IRT using the two-parameter logistic model, difficulty and discrimination. Because our main interest was to assess the overall result and the quality of the instrument using the methodology of IRT, we conducted the analysis considering the 10 cities simultaneously, since the parameter estimates do not depend on the recruitment sampling method.[8] This model makes the assumption of unidimensionality, i.e., that a given test should measure one single latent trait, which indicates a dominant skill responsible for the performance of a set of items of the test. In this study, HIV/AIDS knowledge was the dominant latent trait measured by the 10 items.

The logistic model represents the probability that the participants correctly responded to a given item as a function of their level of HIV/AIDS knowledge and the difficulty and discrimination parameters of the item. The responses to the different items are independent (local independence). It is described as a non-linear logistic function and expressed by the item characteristic curve (ICC) defined by the parameters of the item. The probability of a correct answer varies from 0 to 1 and the knowledge scale assumes a normal distribution pattern (-∞ to +∞) with mean = 0 and standard deviation = 1, but in practice -3 to +3 is used, corresponding to 99.97% of all individuals in a population. It is assumed that every individual has a specific level of knowledge, score theta, which places him on the scale. The discrimination parameter of each item (a) is expressed by the slope of the ICC at the inflection point when the probability of a correct response is 0.5. In general, the metric for this parameter is 0 to 3, where higher values produce steeper curves and indicate items with high discrimination power.[14] Baker’s scale was used to interpret this parameter.[d] The difficulty parameter of each item (b ) corresponds to the point on the scale of knowledge where the probability of correct response is 0.5. Typically, the values are between -3 to +3 and higher values indicate the most difficult items. For a better understanding of our results, the parameters and the knowledge scores were transformed so that the knowledge scores were presented on a scale of 0 to 10.[14]

The item parameters were estimated by marginal maximum likelihood, proposed by Bock & Aitkin,[5] and the knowledge score (theta) was estimated by the expected a posteriori method based on Bayesian statistical principles. BILOG-MG software for Windows 3.0.2 was used for these analyses and the convergence criterion used was 0.0001.[18]

Full-information factor analysis was carried out to test the unidimensionality of the knowledge scale, i.e., the items must share only one underlying variable if they are to be combined into a scale. Full-information factor analysis takes into account all the empirical data and it is based on the tetrachoric correlation matrix using TESTFACT software, version 4.0.[e] The marginal maximum likelihood was used to estimate the probabilities of all possible response patterns that occurred in the sample according to the different levels of knowledge.

The differential item functioning (DIF) was assessed by the likelihood ratio test for age and years of schooling, using the IRTLRDIF software, version 2.0b.[f] When individuals with the same level of knowledge in different groups do not have the same probability of correctly answering a given item, this is taken as an indicator of DIF. Differences in parameters were tested by chi-squared statistic and the significance level considered was 0.05. The characteristic curves of the items which presented with DIF were analyzed and differentiated between uniform DIF (parallel ICC) and non-uniform (crossover ICC).[g]

This study was approved by the National Research Ethical Committee (CONEP 14,494) and all participants signed the informed consent form.

RESULTS

On average, there were 15 (range 8-20) waves of recruitment in each city, resulting in 3,859 participants, with 3,746 (97.1%) responding to the ten items on HIV/AIDS knowledge. Most of the participants was more than 25 years old (52.1%), had more than eight years of schooling (69.7%), were single or separated (83.4%) and non-white (72.4%). Only 19.0% was living alone, and 28.8% were classified as economic class AB (higher).

Overall, the proportion of correct answers varied from 34.5% to 98.5%, with the lowest proportions shown for items 1 and 2 (35.5% and 34.5%, respectively). Some beliefs still exist in this population regarding the transmission of the virus by insect bites, by using public restrooms, and by sharing utensils during meals (Table).

Table

Proportion of correct HIV/AIDS knowledge responses as reported by the men who have sex with men, the difficulty and discrimination parameters for each item, estimated by Item Response Theory. Brazil, 2008-2009. (N = 3,746)

Item	% Correct response	Difficulty (bi)	Discrimination (ai)
1. The risk of transmitting HIV is small if one follows the treatment correctly.	35.5	14.45	0.04
2. People are using less condoms because of AIDS treatment.	34.5	14.23	0.04
3. A person can get the AIDS virus by using public toilets.	78.1	3.72	0.95
4. A person can get the AIDS virus through insect bites.	75.5	3.70	0.72
5. A person can become infected by sharing eating utensils, cups or food.	85.7	3.28	1.01
6. The risk of HIV + mothers infecting their babies is small if she receives treatment in pregnancy and childbirth.	75.8	2.70	0.32
7. The risk of HIV infection can be reduced if you have relations only with an uninfected partner.	72.6	2.03	0.20
8. A healthy person can be infected with the AIDS virus.	94.1	1.61	0.59
9. A person can get the virus from sharing a syringe or needle.	96.9	1.51	0.78
10. Anyone can get the AIDS virus if condoms are not used.	98.5	1.27	0.95

b: Difficulty parameter of each item; a: Discrimination parameter of each item

The crude mean and median total score of correct items were 7.5 (SD = 1.5) and 8.0, respectively, with 43.2% of the sample falling below the average. On the other hand, the mean and median overall HIV/AIDS knowledge score using IRT were 5.0 (SD = 0.9) and 5.3, respectively, with 40.7% of the sample with knowledge levels below the average.

The parameters for each item, estimated by IRT, can also be seen in the Table, in descending order of the parameter of difficulty. Items 1 and 2 stood out as the most difficult items, with the larger values of difficulty parameter (b = 14) exceeding the scale (0 to 10). This points to potential problems in these items, as indicated by the low percentage of correct responses for these items. The eight remaining items showed low degree of difficulty, with values below the average of the scale, ranging from 1.0 to 4.0. The average difficulty of the set of items, excluding items 1 and 2, was 2.48 (SD = 1.0). The discrimination parameters ranged from 0.04 to 1.01. Item 5 was considered the best item, with the highest discrimination parameter (a5 = 1.01), capable of differentiating individual HIV/AIDS knowledge. According to Baker’s classification, five items (3, 4, 5, 9 and 10) showed moderate discrimination, one item showed low discrimination (8), and four items (1, 2, 6 and 7) had very low discrimination (values lower than 0.34). The overall average of all items was 0.56 (SD = 0.38), and it can be considered of low discriminatory value.

One item of each type of discrimination is shown in Figure 1, represented on the left by the ICC and on the right by the curve of the item information. As observed, the ICC of item 5 is the steepest curve, gently sloping to the right of the knowledge scale. This indicates that a slight increase in the level of knowledge is capable of significantly increasing the probability of a correct response to this item. Because item 5 had the highest degree of discrimination, it is capable of distinguishing individual knowledge at much closer levels, as compared to the remaining items. On the other hand, ICC for item 1 is almost a straight line parallel to the scale of knowledge, indicating that an increase in the theta values does not significantly change the probability of correctly answering the item, i.e., low discriminatory power of this item. The item information curve shows how much the item contributes to the measure of knowledge, i.e., it indicates precisely which levels of theta were better discriminated. Usually the item brings best information about a few theta levels than others, and its representation resembles a normal curve type. The information curve of item 5 brought more information for measuring the levels of knowledge around value 3.0. Outside this amplitude, the item starts producing incorrect information on the level of knowledge, since the standard error curve is inversely proportional to the information curve. On the other hand, the information curve of item 1 showed that this item hardly contributed with any information for measuring HIV/AIDS knowledge.

Figure 1

Graphical representation of the characteristic and information curves of the items selected.

Figure 2 presents a graphical representation of the total information curve, i.e., the sum of all information functions, for all the ten items. Overall, the curve indicates that the 10 items provided better differentiation among individuals who are below the midpoint on the knowledge scale. For individuals with levels above the average, the items have little discrimination, producing most of the information error, as seen by the standard error curve (dotted line). In addition, there was no measurable knowledge score above six on the scale.

Figure 2

Total Information curve (10 items).

A macroscopic view of the item analysis in our sample is shown in Figure 3, which simultaneously provides the total information curve, the curve of HIV/AIDS knowledge scores (IRT), and the total observed scores (CTT). In summary, the estimated latent knowledge among MSM was limited by the parameters of the items, i.e., it more accurately measured individuals with lower knowledge but little differentiation was obtained among those with median or high knowledge levels.

Figure 3

Graphical representation of the items and HIV/AIDS knowledge scores.

Regarding the existence of the unidimensionality, Full-information factor analysis indicated four factors with the following variance partition: 31.0%, 10.0%, 5.2% and 3.8%. The first component is predominant in relation to the other factors, which confirms that the set of analyzed items presents a unidimensional structure, explained by a variance of 31.0%.

The DIF by age and schooling was identified in six items (1, 2, 3, 5, 6 and 7) whose ICC are shown in Figure 4. Items 3 and 5 presented uniform DIF in the two variables, producing parallel curves that differ between the two groups only in the difficulty parameter. This indicates that individuals who are younger than 25 years old and those with more than eight years of schooling have a greater probability of correctly responding to these items compared to those 25 years old or more and with eight or fewer years of education, respectively. On the other hand, crossover DIF has been found for items 2 and 6 with respect to age, and for items 1 and 7 with respect to schooling. This indicates that the ICC of these items differ with respect to both parameters difficulty and discrimination.

Figure 4

Items characteristic curves with differential item functioning.

DISCUSSION

Currently, there is no validated instrument capable of measuring HIV/AIDS knowledge across different populations in Brazil. This study applied IRT for the analysis of HIV/AIDS knowledge among MSM, as it offers important advantages, particularly by the individual analysis of the items and the reliability of the measure, supplementing information provided by the CTT. In addition, IRT psychometrics offers a suitable approach to studying an instrument’s ability to detect change.[17] In particular, IRT offers both the potential for new ways of interpreting individual change and improving scale properties in order to better ascertain the measurement scores.

Initially, both analyses (CTT and IRT) showed that a large proportion (43.2% and 40.7%, respectively) of participants did not reach the average level of HIV/AIDS knowledge, and this is of public health concern. It also indicates that myths about HIV transmission are still present in this population, also reported in other studies.[11]

On the other hand, the IRT analysis provided more useful information with regard to the knowledge score as well as the individual analysis of the items, including the characteristic curves, the item information curves, the identification of items with problems, their distribution on a scale, and how much information each item brought to the measurement of knowledge.

We should note that the most problematic items were those related to AIDS treatment (items 1 and 2). Both require knowledge on two distinct issues, risk or prevention and treatment, and therefore, presented higher degree of difficulty, low discrimination power and also the lowest percentage of correct answers. Most likely, the respondents did not completely distinguish or understood their contents, casting doubt on where the lack of knowledge predominated and also indicating an excessive difficulty for a correct response. On the other hand, item 6, which is also related to the risk of HIV positive mothers infecting their babies and AIDS treatment, did not seem to be totally unknown by the respondents.

The absence of difficult items in this study contributed to the inaccuracy of the measurement of knowledge among those with median level and above, as accurately shown by the total information curve. A recent study that examined the psychometric properties of an HIV/AIDS knowledge scale among adolescents reported that in order to improve the precision of the instrument, new items should be added, particularly those more difficult to answer which provide information about higher trait values. Moreover, in order to improve the overall performance of the knowledge scale, this should include a greater number of items, with higher discriminatory power and different degrees of difficulty, which can provide information along the whole scale.[1]

The analysis of the DIF identified items that tend to benefit one group more than others. Items 3 and 5 were shown to be more favorable to MSM under age 25 and those with higher schooling, showing higher chances of correct answers of these items. The domain of items 3 and 5 with uniform DIF refers to the indirect transmission of HIV (using public restrooms and the sharing of meals). Studies show that the analysis of patterns of items that display DIF is a useful tool to identify and better understand the differences between ethnic and racial groups.[7,16] The analysis of the DIF items may contribute to the development of educational strategies in the approach of the domain of items in which DIF was detected. However, it becomes more complex when trying to understand crossover DIF, as in the case of items 1, 2, 6 and 7. As these items also showed low discrimination power, further analysis is warranted for a better understanding of the DIF effects. We suggest they should be reviewed with regard to content, language and format, since they contain words (e.g., small, less, reduced) that may have caused ambiguity, bringing confusion, indecision and insecurity for respondents.

Although there are substantial problems in the structure of some items, the results of the unidimensional analysis led us to admit the existence of a dominant trait or factor (i.e., HIV knowledge) responsible for the performance of the set of items and that the instrument used was able to measure the levels of latent knowledge. Ideally, one should pursue a higher percentage of total variance accounted for by the first principle component, indicating that the set of items is more associated with the dominant factor, which is the latent trait measured.[h] We emphasize the need to review items 1, 2, 6 and 7 in order to better compose the scale with large loading values in the first factor. In addition, in order to promote unidimensionality, it is recommended to use constructs that are in the same direction, i.e., a set of items strictly negative or positive.[i]

In conclusion, the IRT model was shown to be adequate to measure the level of HIV/AIDS knowledge among MSM and brought important information that can contribute to improving the properties of the items in order to build a knowledge scale suitable for this population. The high proportion (40.7%) of participants with a knowledge score below the average is of concern, considering that the evaluated items are basic and well publicized information about the modes of HIV transmission, particularly among MSM populations. Furthermore, the results revealed some weaknesses of the measuring instrument, and improvement in the quality of the instrument is essential to better ascertain the levels of knowledge of HIV/AIDS.

The results of IRT analyses should be considered before carrying out other evaluations or interventions in this population. It is essential to review the items with lower discrimination, to incorporate new and more difficult items, and to add other relevant topics. Calibration of items and checking differential item functioning should be assessed in preliminary pilot studies. Comparisons between scores of different samples are possible with IRT methodology because item parameters are invariant to the various groups, and both parameters and latent trait are measured in the same metric scale.[b] This way, monitoring the level of knowledge of this MSM population, or others, over time could be measured in different samples, allowing to accurately detect the progress made by the population, as well as assisting in the development of prevention programs and of new interventions.