Development and validation of an instrument to assess Brazilian healthcare professional providers' knowledge on sickle cell disease.

OBJECTIVE: To develop and validate DFConhecimento, an instrument to assess Brazilian healthcare professional providers' knowledge on sickle cell disease.
METHOD: Study carried out in four stages: (1) instrument development; (2) content validation by an Expert Committee; (3) cultural adequacy check at pre-test; (4) instrument reliability analysis by healthcare professional providers supported by Intraclass Correlation Coefficient calculation. The data for content validation and reliability analyses were collected through the web tool eSurv and analyzed within the statistical software and environment R.
RESULTS: The instrument, consisting of 13 multiple-choice questions, showed acceptability, with an average Content Validity Index of 0.88. The reliability analysis showed moderate agreement (0.67) indicating that test-retest reproducibility is acceptable.
CONCLUSION: The instrument DFConhecimento showed reliability and internal consistency, proving suitable for measuring Brazilian healthcare professional providers' acquisition of knowledge on sickle cell disease.

Introduction

Sickle cell disease is a chronic and hereditary condition of high incidence in the world and in Brazil, which predominantly affects the black population and presents clinical manifestations in the first years of life, with repercussions on morbidity, mortality and quality of life.1, 2

One of the challenges regarding this condition is to enhance healthcare professional providers’ education so that they may acquire basic knowledge in order to avoid misconceptions and wrong beliefs about it and prepare to lay the foundations for further training aimed at actively helping people to avoid complications and improve their quality of life.2, 3, 4, 5, 6, 7, 8

Addressing this challenge, the Education and Support Center for Hemoglobinopathies (CEHMOB-MG), along with the Center for Newborn Screening and Genetics Diagnosis (NUPAD) and the Hemominas Foundation, have made available the distance education course “Sickle Cell Disease: Care Line in Primary Health Care in the State of Minas Gerais” since 2010. The aim of the course is to enhance healthcare professional providers’ knowledge on the disease and promote critical awareness of their performance as professional providers of care to people with sickle cell disease.2, 3

Distance education is a potentially relevant means of training healthcare professional providers, as it promotes autonomous learning. As this is an emerging educational strategy, it is necessary to assess its impact regarding the knowledge acquired by healthcare professional providers who make use of such a course.3, 9

A review of available literature on instruments to assess the knowledge of professional healthcare providers on sickle cell disease yielded the General Perceptions Of Sickle Cell Disease Patients Scale, which aims to assess healthcare professional providers’ attitudes regarding patient care.10, 11 The aim of the instrument, however, is not to assess knowledge on sickle cell disease, which made evident the need to develop an instrument to serve that purpose.3, 4 This was made by a group of healthcare professional providers at the Center for Newborn Screening and Genetics Diagnosis (NUPAD) and the Hemominas Foundation, together with the School of Nursing, the Laboratory of Experimentation in Translation at the Arts College and the Laboratory of Biostatistics at Universidade Federal de Minas Gerais (UFMG), within the scope of the project Empoder@ – Methodological innovation in educational practices for self-care promotion.

The aim of the study herein reported was to develop and validate the Sickle Cell Disease Knowledge instrument – DFConhecimento – with the objective of assessing Brazilian healthcare professional providers, including, among others, physicians, nurses, psychologists, dietitians, nutritionists, physiotherapists and healthcare managers.

Method

This study was carried out from June 2015 to August 2016, and comprised the following stages: instrument development, content validation, cultural adequacy and reliability analysis.

To establish the domains of sickle cell disease and develop the instrument items, we addressed the main issues associated with the clinical aspects, pathophysiology, warning signs, medication, vaccine schedule, guidance for self-care and follow-up protocols.13, 14 The first version of the instrument was drafted and discussed through online interdisciplinary meetings convening healthcare professional providers (at CEHMOB-MG and the Nursing School), applied linguists, statisticians with expertise in instrument development and specialists in sickle cell disease. Protocols and manuals on assistance to people with sickle cell disease published by the Brazilian Ministry of Health were also consulted.4, 15

For content validation, a sample of eleven professional healthcare providers with experience in the care of patients with sickle cell disease was selected: two hematologists, two pediatricians, two family and community physicians, four nurses and one pedagogue.

A letter explaining the aims, methodology, development process and reasons for the instrument content validation was e-mailed to the sampled professionals, together with a free and informed consent form, inviting them to participate in the study as expert assessors and answer a questionnaire applied through the web tool eSurv. Seven healthcare professional providers agreed to participate as the Expert Committee to assess the first version of the instrument. They were a hematologist, a pediatrician, a family and the community physician, three nurses and a pedagogue.

The experts were asked to assess each item in the first version of the instrument, focusing on the clarity and relevance of its content, by choosing one out of four assessment outcomes: 1 – item requires major modification or substitution; 2 – item requires some modification; 3 – item requires editing to enhance text style; and 4 – item requires no modification. A blank text box was made available in the questionnaire for remarks and observations when experts selected options 1 and 2.

The Content Validity Index (CVI), defined as the sum of the relative frequencies of option 3 (item requires editing to enhance text style) and 4 (item requires no modification), was calculated to verify the level of agreement among assessors regarding the adequacy of the evaluated items. The higher the CVI value, the smaller the number of changes needed for improvement. A CVI greater than or equal to 0.8 was considered indicative of suitability of the item with respect to clarity and relevance of content.

To ensure the cultural adequacy for the target population, the second version of the instrument, validated by the Expert Committee, was used in the pre-test, applied through the web tool eSurv to a group of 3 professional healthcare providers with a college degree in healthcare sciences working in healthcare services. This number is in accordance with the adopted methodological framework, which recommends between 30 and 40 subjects to take the pre-test, without the need for sample calculations related to statistical inferences.19, 20 In the pre-test, besides answering the questions in the instrument, participants were also invited to make suggestions to improve the wording of the items and answer options.

After compiling the pre-test results, a group of specialists working on research focusing on the cultural adequacy of healthcare instruments conducted four meetings. The items in the instrument that showed problems in the pre-test phase were reformulated in order to improve their comprehension by the target population. The pre-test phase yielded a third version of the instrument.

Data collection for reliability analysis of the instrument was carried out with professional healthcare providers working in healthcare services management and projects directly related to sickle cell disease. The inclusion criteria for these professionals were: a college degree in healthcare sciences and the holding of a position in the healthcare area. The interval between the test and retest was 15 days.

The invitation to take part in the study to get data for the reliability analysis was sent to a group of 300 professionals, 50% of whom (150) had taken the distance education course Sickle Cell Disease: Care Line in Primary Health Care. Subjects were randomly selected through a database of 2446 professional healthcare providers. The invitation sent by email to each professional healthcare provider included a free and informed consent form, an introduction to the study and the request to answer an electronic questionnaire on the web tool eSurv to validate the third version of the DFConhecimento instrument. A total of 102 professional healthcare providers agreed to participate, 52% (53) of them having taken the above-mentioned distance education course. This number is in accordance with the adopted methodological framework, which recommends a number of participants between 5 and 10 times the number of the items in the instrument, with no need for sample calculations.21, 22 The objective was to verify if the instrument was reliable to assess the knowledge on sickle cell disease acquired by professional healthcare providers.

In the reliability analysis phase, internal consistency and temporal stability were evaluated. The internal consistency was determined using Cronbach's alpha coefficient (α), considering satisfactory an α greater than or equal to 0.70. In the temporal stability assessment of the instrument, the influence of random selection or random guessing in answering was sought to be minimized. Assuming that a participant can learn, but not unlearn, content within 15 days between the test and retest, correct answers to items in the test, but not in the retest, were considered invalid and the answers to these items were thus excluded from calculations measuring temporal stability. The temporal stability and reproducibility of the instrument were evaluated by means of test-retest, calculating the Intraclass Correlation Coefficient (ICC), considering a satisfactory ICC > 0.5. The inter-item correlations at test and retest times were assessed following standard procedures by the Pearson correlation coefficient, which is largely used to quantify the level of linear correlation between two quantitative variables. In this analysis, the percentage of concordance in responses between the two sessions was also used, which was calculated as the ratio between the number of individuals giving the same response both times and the total number of individuals with valid answers for that item. The level of significance adopted for the statistical tests was 5%.

Participants were asked to answer a questionnaire requesting the following information: sociodemographic data: (1) sex, academic background, city where they work and professional training category; (2) details of their professional performance: type of healthcare service in which they worked and number of years working in healthcare services.

The data were presented in a spreadsheet and analyzed with the statistical software R (R Core Team, 2015).

A flowchart of the development and validation of the instrument sickle cell disease knowledge – DFConhecimento – is shown in Figure 1.

Figure 1

Stages in the development, cultural adequacy check and validation of the DFConhecimento instrument for Sickle Cell Disease.

The project was approved by the Ethics and Research Committee Involving Human Beings of the Universidade Federal de Minas Gerais, according to decision Number. 1.177.817, 2015. The agreement of healthcare professional providers to participate in the study was recorded by using a Free Informed Consent Form available in the initial webpage of the electronic questionnaire in the web tool eSurv.

Results

Fifteen structured and multiple-choice questions comprised the first version of the instrument, covering the following topics: neonatal screening; sickle cell anemia genotype; sickle cell traits; factors that contribute to the process of vessel occlusion and sickle cell formation; warning signs of sickle cell disease; use of antibiotics; health issues of adolescents and pregnant women; factors for prevention of leg ulcers, and; health referral services.

Experts who took part in the interdisciplinary meetings recommended excluding the topic health referral services. This was deemed to depend on regional planning models, which would demand adapting the instrument to each to each particular region of the country.

Experts reached the agreement that the questions in the drafted questionnaire did cover topics related to knowledge regarding sickle cell disease in the context of multidisciplinary healthcare and educational practices, according to the materials consulted and the experience of the experts.

The first version of the instrument was submitted for assessment by an Experts committee with the following academic background: six professionals (85.7%) had course diplomas and one of them had an a graduate degree (14.2%).

Table 1 presents scores of the instrument assessment and the Content Validity Index (CVI). The average CVI value was 0.88.

Table 1

Committee and Content Validity Index of the instrument. Belo Horizonte, MG, Brazil, 2016.

Item topic	Requires major or some modification (1 and 2)	Suitable or in need of minor modification (3 and 4)	Content Validity Index (CVI)
1 – Neonatal Screening Program	0	7	1
2 – Sickle Cell Condition	1	6	0.86
3 – Sickle Cell Anemia Genotype	1	6	0.86
4 – Sickle Cell Trait	2	5	0.71
5 – Sickle cell disease manifestations	0	7	1
6 – Vaso-occlusion	1	6	0.86
7 – Factors that favor red blood cells sickling	0	7	1
8 – Warning Signs	1	6	0.86
9 – Medications used in prevention and/or treatment of sickle cell disease	1	6	0.86
10 – Vaccination	2	5	0.71
11 – Use of antibiotics	2	5	0.71
12 – Adolescent Health	0	7	1
13 – Pregnancy and contraception	1	6	0.86
14 – Care for prevention of leg ulcers	0	7	1
Average CVI	–	–	0.88

The Committee assessment made suggestions regarding three aspects in the questionnaire, namely:

The need to format some of the items by highlighting the text in the commands to make clear what kind of statement assessment was being requested for. This was the case in statements of the kind “All of … EXCEPT FOR”, in which the respondent needs to choose the option that does not apply or is an exception.

Addition and omission of options: With regard to the item focusing on people with the sickle cell trait, the Committee suggested reformulating some of the options and adding an option that stated that people with the sickle cell trait are not considered as carriers of a disease, but rather of a relatively common and clinically benign condition. They also suggested omitting the option “fever” in the item inquiring about vessel occlusion and adding it to the item related to the warning signs of the disease, along with “enlarged spleen and liver”, “coughing or shortness of breath” and “abdominal pain”.

Providing more specific formulation in the wordings for options. For instance, in the item warning signs of the disease, “Worsening of pallor” was suggested to be phrased as “Worsening of skin-mucous pallor”; “They are medicines used to treat sickle cell disease and its complications” was rephrased as “They are medicines used to prevent and/or treat sickle cell disease and its intercurrences”; in the item on leg ulcer, “Watch out for insect bites” was suggested that it be rephrased as “Use of insect repellent and watch out for insect bites”.

Item ten was excluded because the vaccination schedule is regularly updated by the Ministry of Health and, therefore, the instrument would have to be regularly updated in accordance with the Ministry's directives.

The committee's suggestions were analyzed and discussed by an interdisciplinary team of researchers in healthcare, applied linguistics and statistics. Following discussions, a second version with 13 questions was obtained. This version was applied to a sample of the target population comprising 36 healthcare professionals in order to ensure cultural adequacy. It was reported by 47.2% of the sampled professionals that their healthcare units had patients with diagnoses of sickle cell disease.

Regarding the comprehension of the instrument, the sampled professionals provided inconsistent answers. Some difficulty was reported by 58% of the professionals in answering questions eleven (When discussing persons with Sickle Cell Trait, it is correct to state that they…) and thirteen (All of the following are recommended guidelines for preventing leg ulcers in persons with sickle cell disease except for…), which they ascribed to the commands in those questions. This led to the formatting of options by highlighting “EXCEPT”, “CORRECT” and “INCORRECT” in the items, implemented in the third version of the instrument, which became the final one (Table 2).

Table 2

English gloss of the Final version of the DFConhecimento instrument on sickle cell disease.

Sickle cell disease knowledge survey
1. Among the diseases screened by the Neonatal Screening Program, which one has the highest incidence in the Brazilian population?
() Congenital Hypothyroidism
() Phenylketonuria
a() Sickle Cell Disease
() Cystic fibrosis
() Biotinidase Deficiency
() Congenital Adrenal Hyperplasia
() Don’t know
2. When discussing Sickle Cell Disease, it is CORRECT to state that
() It occurs among the black population in Brazil
() It is one of the rarest genetic disorders in Brazil and worldwide.
a() It is characterized by the predominance of hemoglobin S in red blood cells
() Its SS genotype is a mild form of the disease
() Don’t know
3. Which one is the Sickle Cell Anemia Genotype?
() Hb SC
() Hb AS
a() Hb SS
() Hb SD
() Hb AA
() Don’t know
4. When referring to persons with Sickle Cell Trait, it is CORRECT to state that they
() Need monitoring and hematological treatment
() Suffer from anemia
a() Do not have a disease but a relatively common and clinically benign condition
() Are not advised to practice physical activity
() Don’t know
5. All of the following are features of Sickle Cell Disease EXCEPT FOR
() Jaundice
() Frequent Infections
() Hemolytic anemia
a() Iron deficiency anemia
() Biliary Calculus
() Don’t know
6. All of the following manifestations are caused by vaso-occlusion EXCEPT FOR
() Pain cramps
() Priapism
() Leg ulcers
() Retinopathy
() Osteonecrosis
a() Hyperglycemia
() Hand-Foot Syndrome (Dactylitis)
() Stroke
() Don’t know
7. All of the following conditions favor the sickling of erythrocytes EXCEPT FOR
() Dehydration
() Strenuous physical activity
() Exposure to cold
a() Being overweight
() Emotional stress
() Don’t know
8. All of the following signs are considered warning signs in Sickle Cell Disease EXCEPT FOR
() Fever
() Enlarged spleen and liver
() Coughing or shortness of breath
() Abdominal pain
a() Feet and leg Itching
() Worsening in pallor of skin and mucous membranes
() Nausea/vomiting
() Neurological deficit
() Don’t know
9. All of the following drugs are used to prevent and/or treat Sickle Cell Disease and its intercurrences EXCEPT FOR
() Penicillin
() Folic acid
a() Ferrous sulphate
() Hydroxyurea
() Pain killers
() Iron chelator
() Don’t know
10. Persons with Sickle Cell Disease should be prescribed antibiotics and guided as to how to take them in all of the following situations EXCEPT FOR
() Children under 5 years old
a() Pregnant women
() In dental procedures
() In case of presumed infection
() Don’t know
11. In the case of teenagers with Sickle Cell Disease, it is INCORRECT to state that
() They will generally experience delayed growth and puberty
() They are prone to develop self-concept, self-image and self-esteem disorders
a() Priapism in male teenagers is always related to sexual desire
() They may have academic and professional skills deficits
() Don’t know
12. As regards pregnancy and contraception in persons with Sickle Cell Disease, it is CORRECT to state that
() Urinary and respiratory tract infections are uncommon
a() Pregnancy is a potentially serious condition for woman and fetus
() Women with sickle cell disease must be advised against becoming pregnant
() Pre-natal care must be done at a Primary Care Unit
() Women with sickle cell disease must not be on injectable or oral contraceptives
() Don’t know
13. All of the following are recommended guidelines for preventing leg ulcers in persons with Sickle Cell Disease EXCEPT FOR
() Skin nourishing and cleansing
() Use of adequate and comfortable footwear
() Use of insect repellents and insect bite treatment
a() Mandatory use of compression stockings
() Do not know

Correct option as answer to the item.

For the calculation of the total score of the respondent, 1 point was assigned to each correctly answered item and 0 points otherwise. For calculation of the final score of the instrument, it was decided to use the sum of correct answers, considering the following knowledge score ranges: 12–13 (more than 90% accuracy), excellent; 8–11 (between 60% and 89% accuracy), good; and 7 or less (less than 59% accuracy), poor.

In the reliability analysis phase, a sample of 102 healthcare professionals answered the third version of the instrument in two sessions (test and retest). Among respondents, 84% were female; 58% had a course diploma; 55% were nurses; 29% had been in the healthcare area for more than 11 years; 38% worked in Primary Health Care; 80% stated that they had met and tended to patients with sickle cell disease in the area covered by their healthcare unit, and; 92% worked in cities and villages in the State of Minas Gerais.

Table 3 shows the correlation analysis between the responses to items at test and retest (Pearson's correlation coefficient and percentage of concordant responses) between the total scores at both moments and Cronbach's alpha coefficient (α) for the DFConhecimento instrument.

Table 3

Correlation between item responses, scores on subscale, total score on test and retest and Cronbach's alpha coefficient (α) for the DFConhecimento instrument. Brazil, 2016. (n = 102).

Questions	Pearson's correlation coefficient – test and retest	Percentage of concordant responses in the test and retest	Alpha of Cronbach if the item is withdrawn (absence alpha)
1	0.54	83.9	0.8170
2	0.45	88.9	0.8059
3	0.37	83.5	0.8024
4	0.52	85.2	0.8007
5	0.46	63.4	0.8047
6	0.48	76.0	0.7941
7	0.51	67.9	0.8013
8	0.52	71.0	0.8145
9	0.54	70.4	0.8035
10	0.49	67.1	0.8208
11	0.39	73.5	0.8017
12	0.47	67.5	0.8029
13	0.41	79.8	0.8074
Total score	0.67	–	0.8183

The overall value of Cronbach's alpha (α) for the third version of the instrument was 0.818, indicating high internal consistency. The exclusion of question 10 (α = 0.8208) yields an α value of absence greater than the α value of the instrument with item 10 (α = 0.8183), indicating that the internal consistency of the instrument is slightly greater when item 10 is excluded. However, this improvement in the internal consistency of the instrument is small and item 10 was retained. Although the Pearson correlation coefficient was not able to capture the correlation between responses to items at test and retest, the concordant responses percentage at these two moments presented a mean of 75.2%, with a standard deviation of 8.2%, indicating moderate agreement. The reliability analysis of the instrument was supported by the Intraclass Correlation Coefficient calculation, whose value was 0.67 (95% CI = [0.55; 0.76]), also indicating a moderate agreement. Thus, the instrument reproducibility can be considered acceptable.

Discussion

The development of the instrument DFConhecimento was found necessary, given the lack of instruments for its particular purpose in the literature. To that end, an adequate and rigorous methodology was used, recruiting experts in healthcare, applied linguistics and statistics in order to obtain a valid and reliable instrument. All items included in the first version were maintained, except for one that was related to referral services and would demand localizing the instrument to particular regions.2, 3

The content validation allowed for the measurement of the CVI, which presented an average result of 0.88 of acceptance. Studies considered a CVI of less than 0.80 as the cutoff point.18, 20, 23, 24 The Expert Committee's participation made possible the improved content and format of the items of the instrument. The committee agreed to exclude the vaccination schedule item due to regular updates by the Ministry of Health.

Cultural adequacy, verified by the semantic analysis of the instrument during pre-test taken by healthcare professional providers, allowed us to identify and explore problems presented in two questions, favoring instrument readjustment and ensuring the comprehension of the items by the target population.9, 25

Regarding reliability, the results show satisfactory indexes, especially when compared with instrument validation studies, thus guaranteeing reliability of the instrument, making it validated for later studies.22, 26 Cronbach's alpha presented a value of 0.818, indicating a high internal consistency. The reliability analysis indicated a moderate correlation (ICC = 0.67), indicating that the test–retest reproducibility is acceptable. Evaluation instruments are useful and capable of presenting scientifically robust results only when they demonstrate good psychometric properties.16, 28

The final version of the instrument can be considered adequate since it meets satisfactory values in global content validity and reliability analysis and covers topics related to knowledge about sickle cell disease considered basic for healthcare professional providers to avoid misconceptions and wrong beliefs about this condition and essential to lay the foundations for further training aimed at actively helping patients to avoid complications and improve their quality of life.2, 3, 4, 5, 6, 7, 8

Conclusion

The present study contributed to the development and validation of a reliable and validated instrument to assess professional healthcare providers’ knowledge on sickle cell disease.

Conflicts of interest

The authors declare no conflicts of interest.