Some Psychometric Properties of the Brazilian CDC Clear Communication Index.

The Centers for Disease Control and Prevention (CDC) Clear Communication Index (CCI) was cross-culturally adapted to Brazilian Portuguese (BR). It was necessary to analyze the reliability and validity of the BR-CDC-CCI for its use in Brazil. This study aimed to evaluate the psychometric properties of the instrument in its Brazilian version. Four specialists in health education used the BR-CDC-CCI to evaluate a population-level health education material. Primary health care professionals (n = 105) evaluated the same health material using the BRCDC-CCI, and 30 professionals performed the retest 15 to 20 days after the first assessment. Cohen Kappa and area under the receiver operating characteristic (ROC) curve analyses were developed. Inter-rater agreement ranged from moderate to almost perfect, with 90% of the items almost perfect. The percentage of agreement ranged from 8.6% to 98.1%. For the analyzed questions, the area on the ROC curve was 0.9412 (confidence interval [CI] 95%; [0.8259, 1.000]). The BR-CDC-CCI had sufficient validity and reliability for its use in the evaluation of educational/informational materials in health in the Brazilian context. In view of the good results from this psychometric assessment, we anticipated the BR-CDC-CCI could contribute to improvements in Brazilian professionals' skills in developing health communication materials, thereby improving the quality of education and, possibly health outcomes. [HLRP: Health Literacy Research and Practice. 2022;6(2):e84-e87.].

Health communication should be based on effective criteria for clarity (Baur & Prue, 2014). To address the need for standard criteria for health communication and education materials, the Centers for Disease Control and Prevention (CDC) Clear Communication Index (CCI) was designed and tested by staff at the United States CDC in 2014. The CDC-CCI has 20 close-ended questions covering seven domains: main message and call to action, language, information design, state of the science, behavioral recommendations, numbers, and risk. The CDC-CCI presented good validity when tested before and after redesigning three educational materials (Baur & Prue, 2014). It can be used by health and communication professionals to evaluate public health materials intended for audiences of different levels of health literacy (Early et al., 2020; O'Sullivan et al., 2020; Porter et al., 2019). Studies on health literacy in Brazil are incipient and instruments with criteria and objectivity like the CDC-CCI are not yet available to assist health professionals in the development of health education materials. A cross-culturally adapted version to Brazilian Portuguese (BR) (Marinho et al., 2020) was carried out and it is still necessary to analyze its reliability and validity. Hence, this study aimed to evaluate some psychometric properties of the instrument in its Brazilian version.

Methods

This was a methodological study for evaluating some psychometric properties of the BR-CDC-CCI (Guillemin et al., 1993; Hosmer & Lemeshow, 2000; Shoemaker et al., 2014; Wong et al., 2018). We compared the evaluation of one health education material by 105 health professionals with a consensus of experts and performed its reliability assessment. The study was approved by the Research Ethics Committee of the Federal University of Minas Gerais (protocol CAAE 79108017.9.0000.5149).

Data Collection

Three municipalities in Minas Gerais, southeastern Brazil, were chosen by convenience in the neighborhood of the researcher's (A.M.C.L.M.) study center. Two researchers (A.M.C.L.M. and L.E.O.C.) consecutively approached 118 health professionals in their respective workplaces. All health professionals had bachelor's degree in health area and worked in public primary health care units. The number of participants was according to the recommendation to include from 5 to 10 respondents for each question under validation (Shoemaker et al., 2014). The researchers asked the professionals to use the BR-CDC-CCI to assess health education material and to answer a demographic questionnaire. There were 105 professionals who agreed to participate. Data collection was conducted between September 2018 and February 2019.

The Health Education Material

The research team chose the population-level material from the Ministry of Health (Brasil, 2015). This 28-page booklet was publicly available, addressed a topic common to different health areas, and included the seven assessment domains for which the CDC-CCI was designed. This health education material was used in primary health care (PHC) settings in Brazil to inform patients about the rationale use of medicines. This booklet was useful for evaluation of BRCDC-CCI because it was nationally used by all PHC professionals (Brasil, 2015).

Gold Standard

The gold standard for the evaluation of the education material included four specialists with more than 10 years of experience in health education and communication. Two meetings were held, one to explain the evaluation rules and the second to establish the gold standard of the presence or absence of compliance with the criterion for each 20 items of the BR-CDC-CCI (Rutjes et al., 2007). With extensive discussion, each meeting lasted approximately 4 hours until reaching the consensus.

Reliability

Randomly selected, 30% of the professionals reevaluated the health education material 15 to 20 days after the first assessment. This retest allowed for answers to be checked to determine if answers remained stable over time, which ensured that the instrument resisted the time bias (Wong et al., 2018).

Data Analysis

After double entry of data and correction of typing inconsistencies, a descriptive analysis was conducted on demographic characteristics. The percentage of agreement for each question of the instrument was calculated by comparing it with the result of the gold standard, dichotomized between presence and absence of the CDC-CCI instrument criterion. To assess the predictive capacity of the health professionals' responses for each of the items in relation to the gold standard established by the specialists, the area under the receiver operating characteristic (ROC) was used. The curve was constructed using participants' answers about the presence or absence of the criterion and the percentage of positive responses regarding compliance with the BR-CDC-CCI criterion. Theoretically, items with a positive evaluation according to the gold standard were expected to have a higher proportion of positive responses among participants and vice versa. The construction of the ROC curve was based on the balance between sensitivity and specificity and its analysis was synthesized by calculating the area under the curve, with a respective 95% confidence interval (Hosmer & Lemeshow, 2000; Wong et al., 2018). The gold standard assessment indicated that question 20 “does not apply” to the educational material evaluated. Thus, this item was disregarded for the purpose of calculating the area under the ROC curve, since the construction of this calculation requires a dichotomous response. The reliability of the instrument was analyzed through reproducibility using Kappa Cohen intra-examiner concordances (Landis & Koch 1977). The statistical analyses were performed using the STATA version 14.

Results

The average age of the group who selected to assess reproducibility (n = 30) was 35.4 years. The majority (73.3%) were women, and 76.7% had a postgraduate course with a high percentage of nurses and dentists (63.3%). The average age of the 105 professionals was 35.6 years. There was a higher percentage of women (77.1%). Nurses and dentists represented 71.4% of the total.

The inter-rater agreement, which assessed the temporal stability of the BR-CDC-CCI through Kappa, as well as the percentage of positive responses regarding compliance with the CDC-CCI criterion attributed by the health professionals per question, was shown in Figure . Inter-rater agreement ranged from moderate to almost perfect, with 90% of the items almost perfect. The temporal stability for question 20, although moderate, had the lowest value among the participants. The question 17 was considered applicable but with low performance in this study. The percentage of agreement ranged from 8.6% to 98.1%. For the analyzed questions, the area under the ROC curve was 0.9412 (CI 95%; [0.8259,1]) (Figure ).

Temporal stability of the Brazilian Portuguese Centers for Disease Control and Prevention (CDC) Clear Communication Index (CCI) agreement of the evaluators (percentage of positive responses regarding compliance with the CDC-CCI criterion assigned by health professionals).

Validity of the Brazilian Portuguese Centers for Disease Control and Prevention (CDC) Clear Communication Index (CCI). The area under the ROC curve (percentages of positive responses regarding compliance with the CDC-CCI criterion attributed by health professionals by question versus dichotomous values of the gold standard assessment). CI = confidence interval; ROC = receiver operating characteristic.

Discussion

Although problems were detected on two questions, the BR-CDC-CCI showed satisfactory reliability and validity for its use with the chosen health communication material. The Brazilian version of the instrument showed adequate inter-rater kappa values, showing temporal stability (Rodrigues et al., 2019; Shoemaker et al., 2014). The performance of area under the ROC curve showed that the Brazilian version of the CDC-CCI is valid (Hosmer & Lemeshow, 2000; Wong et al., 2018).

The problems with question 20 reinforced the recommendations of the English Index authors that the questions not applicable to certain materials can be excluded during the evaluation (Baur & Prue, 2014). Question 17 was a part of a “numbers” domain, which was difficult even among those who create educational materials. As in other cultural contexts, there were difficulties in the field of “numeracy” (Rothman et al., 2008).

The CDC-CCI was created to be used by professionals involved in creating or evaluating health communication materials (Baur & Prue, 2014), especially those used in public health communication. The participation of PHC professionals in the present study can be considered a right choice for a Brazilian validation study because primary care was where the largest volume of education interactions in health care occurred (Hone et al., 2018). As there were greater links between service and patients, these professionals had a great capacity to evaluate the criteria in the material chosen for the public in general in Brazil.

Additional steps were required to better understand the psychometric properties of the instrument (i.e., the analysis of other health materials with the participation of several audiences as was done by the authors of the English version of the CDC-CCI). In future studies, the inclusion of professionals from other regions could be desirable. On the other hand, we evaluated PHC professionals in smalls towns that were the majority among the municipalities in Brazil (Calvo et al., 2016); we also evaluated health professionals from the Brazilian National Health System, which is the major employer of health professionals in the country. Even though our data were not generalizable to the entire country, this is the first study to assess psychometric properties of the CDC-CCI in Brazil and these results could be considered a useful step for further research in the instrument evaluation. As recommended by the literature, the use of complementary assessment instruments may be required in certain educational/informational materials in health (Early et al., 2020; O'Sullivan et al., 2020; Porter et al., 2019). Finally, these types of instruments did not prove whether the health education material were effective, therefore, testing them with a target population is relevant.

Conclusion

The BR-CDC-CCI had sufficient validity and reliability. In view of the good results from this psychometric assessment, we anticipated that the BR-CDC-CCI could contribute to improvements in Brazilian professionals' skills in developing health communication materials, thereby improving the quality of education and, potentially, health outcomes.