Validation and reliability of the Care Vulnerability Index: A study by interrater agreement and test-retest method.

AIM: The aim of this study is to determine the validity and reliability of the Care Vulnerability Index (CVI) as a tool to estimate the need and competence of care.
DESIGN: A cross-sectional survey including a longitudinal component.
METHODS: Content validity ratio (CVR) was calculated by interrater agreement of a group of 11 experts in two rounds. The test-retest analysis was measured in an urban population of Colombia with 96 participants through two statistical tests: Pearson's correlation coefficient and the difference in means.
RESULTS: Care Vulnerability Index turned out to be valid with a CVR of 0.879. Reliability by Pearson correlation between test-retest was 0.912 (CI95: 0.872-0.941; p-value <.01) and there was no significant mean difference between test and retest in global score and in clustered groups of variables. Validating CVI will make it possible to prioritize healthcare resources in the population and identify people susceptible to care problems.

INTRODUCTION

It is well known that vulnerable populations are more likely to suffer from health problems, as indicated by the Social Determinants of Health. In this way, structural and political circumstances can affect people's physical and mental health (Willen et al., 2017).

Knowing the vulnerable populations is essential to carry out health promotion strategies in the community, as well as prioritizing interventions, and managing social services and health assets. (World Health Organization (WHO), 2007; Lee, 2005).

BACKGROUND

Vulnerability is an essential characteristic of the human being and can be defined as the possibility of damage and/or loss of health (Angel et al., 2020; Fernandez‐Batalla, González‐Aguña, Monsalvo‐San Macario & Santamaría‐García, 2018; Sellman, 2005). In recent years, this definition has been used in different areas with little precision (Cardona, 2001; Fernandez‐Batalla, 2019), mentioning concepts such as social or environmental vulnerability (Bao, 2015; Bowie, 2018; Flanagan, Hallisey, Adams & Lavery, 2018; United Nations Development Programme [UNDP], 2014).

From the point of view of care, vulnerability is related to the competence and need of people to take care of themselves at a given time (Arribas‐Cachá et al., 2009; Fernández‐Batalla, 2018). Vulnerability accompanies the person throughout life, varying their capacities and needs (Fernandez‐Batalla et al., 2018; [UNDP], 2014). From this perspective, the Care Vulnerability Index (CVI) proposed by Fernández‐Batalla (2018) is based on Orem's self‐care model (Orem, 1993) but without parametric research on its validity and reliability.

The CVI is made up of a 12‐item questionnaire grouped in 5 clusters linked to the Basic Conditioning Factors (because they are descriptive variables of the person and his care) identified by Orem (1993) later developed by Fernández‐Batalla et al. (2018). Table 1 shows the definition of the items and the value of each cluster.

TABLE 1

Definition of the items that make up the scale of Care Vulnerability Index

Cluster (value)	Item	Definition
Cluster 1 (5 points)	Vital stage	Corresponds to the classification of people according to their maturational state from birth to old age.
	Developmental state	The continuous sequential physiological, psychological and moral changes during the life of person
Cluster 2 (1 point)	Perception of gender limitation	Perception of attitudes or behaviours that limit the agency of self‐care based on stereotyped social roles related to the internal experience of gender role
	Level of cultural integration	Level of attachment that a person perceives with their current place of residence in the physical, emotional and functional context. This attachment must be reciprocal by the community with the person
Cluster 3 (3 points)	Family support	System of care that encompasses the knowledge, skills and attitudes that the family gives to the person
	Individual care pattern	System of care that brings together knowledge, skills and attitudes to satisfy the needs and development of capacities of the person
Cluster 4 (4 points)	Mobility limitation	Difficulty in walking from place to place
	Cognition limitation	Difficulty mental processes related to learning, thinking, reasoning and judgment
	Sensory limitation	Difficulty perceiving external and internal sensory stimuli to which the individual responds (sight, hearing, smell, taste and touch)
Cluster 5 (2 points)	Environmental factors	Influence of the elements of the environment to condition in a way the person's ability to exercise self‐care
	Time availability	Amount of time that the person can have or seek for self‐care
	Material resources availability	Amount of material resource that the person can have or seek for self‐care

Each of the items had between 3 and 5 response options categorized from lowest to highest level of vulnerability (the scale of CVI is available in Appendix 1). The final calculation of the level of vulnerability was established based on the methodology described by Arribas‐Cachá et al. (2009) in the Healthcare Severity Index, where the individual scores of the items are combined following a clinical and a mathematical model to obtain the score for each cluster.

In addition, it was evaluated that each item of the scale was part of the computerized medical records of Primary Health Care and that they constituted data that nurses frequently manage in theoretical clinical cases (Fernández‐Batalla & Jíménez‐Rodríguez, 2015).

Research question

The study starts with the questions.

Is CVI a valid scale to measure the vulnerability of people?

Is CVI a reliable scale?

THE STUDY

Design

The study design consists of a panel of experts to evaluate the validity of content in several rounds and a cross‐sectional survey in the participating population carried out in two moments.

Method

This methodological study is part of a larger study carried out through exploratory research of a mixed method (deductive‐inductive) to design and psychometrize the CVI instrument, based on the studies of Fernandez‐Batalla (2018) and Santamaría‐García (2008). The methodology for validation was based on the studies of Zamanzadeh et al. (2015).

The design of the instrument was carried out by determining content domain, item generation and instrument construction (Carmines & Zeller, 1979; Nunnally & Bernstein, 1994; Zamanzadeh et al., 2015). Items were identified by literature review and interviewing a focus group of 11 experts made up of: doctor nurses, specialists in Family and Community Nursing and master's degree nurses.

Calculation of content validity

The content validity method selected was the quantitative one using the content validity ratio (CVR) by interrater agreement. The experts were requested to specify whether an item is necessary for operating a construct or not. Experts were requested to score each item from 1–3 with a three‐degree range: “not necessary,” “useful but not essential” and “essential”‏ (Muhammed et al., 2020; Zamanzadeh et al., 2015).

Study participants

To measure reliability, a stratified cluster sampling was carried out according to five neighbourhoods in Cartagena de Indias (Colombia). People who agreed to participate in the study voluntarily after informing them of the purpose and basic method of participation in the study were included. The inclusion criteria were to be older than 7 years to guarantee that they understood the questions. In the case of people with cognitive or understanding problems, the variables were recorded by the primary caregiver. A sample size of 195 people was calculated (95% confidence interval, precision =0.07). The total initial sample was 211 people. The time between the two questionnaires (test and retest) was two weeks.

Based on the previous works, it was determined that a minimum sample size would include 5–10 times the number of elements (Park, Yoon, Yun & Park, 2017; Steven, 2002). Because 12 items were used, at least 60 participants were needed to assess the difference. In this study, 104 of 211 (49.29%) patients agreed to participate, fulfilling the inclusion criteria. Eight participants were excluded due to incomplete data in more than 10% of the instrument, making up a final sample of 96 participants (45.50% of total sample).

Analysis

Lawshe's criteria were established to assign a minimum value of CVR of 0.59 with an acceptable level of significance (p < .05) (Lawshe, 1975). A CVR between 0.59 and 0.79 was considered “Need for revision” and a CVR above 0.8 was considered appropriate.

Reliability was measured in two ways: through the Cronbach's alpha coefficient of the total sample (n = 211) and using the test–retest method, applying CVI to the same sample at two different times. Two statistical tests were carried out to measure the reliability of CVI: Pearson's correlation coefficient and the difference in means. The Nunnally criteria were chosen for the reliability coefficient values that indicate that they should be higher than 0.8 or 0.9 in clinical context (Barrios & Coscuella, 2013; Nunnally & Bernstein, 1994).

Data were analysed using RStudio (R Core Team, 2019). We checked for data entry errors. The normality of the distribution was evaluated based on skewness and kurtosis and with Lilliefors's normality test. Descriptive statistics were used to assess the participants’ characteristics and item data.

Ethics

Consent was requested from the people to participate in the study. In the case of minors, the consent was given by the parents and verbally by the minor. This research project follows the ethical implications established in Resolution 008,430 of 1993 of the Colombian Ministry of Health, as well as in the ethics regulations of the University of Cartagena in relation to informed consent, treatment and custody of clinical data and confidentiality of the data, and approval by the Office of Epistemology and Bioethics of the Madrid Scientific Society of Care (SOCMAC) with registration number 20190304_VCFF.

RESULTS

Validity

Content validity ratio calculation for the total items was 0.833 in the first round. Two items needed to be checked. After the second round, all items obtained an interpretation of “Appropriate” and the CVR was 0.879. Table 2 shows the CVR value for each item in the different rounds.

TABLE 2

Content validity ratio of Care Vulnerability Index

Item	CVR1 †	Interpretation	CVR2 ‡	Interpretation
Vital stage	0.818	Appropriate	0.818	Appropriate
Developmental state	0.636	Need for Revision	0.818	Appropriate
Perception of gender limitation	0.818	Appropriate	0.818	Appropriate
Level of cultural integration	0.818	Appropriate	0.818	Appropriate
Family support	1	Appropriate	1	Appropriate
Individual care pattern	1	Appropriate	1	Appropriate
Environmental factors	0.818	Appropriate	0.818	Appropriate
Time availability	0.636	Need for Revision	1	Appropriate
Material resources availability	1	Appropriate	1	Appropriate
Mobility limitation	0.818	Appropriate	0.818	Appropriate
Cognition limitation	0.818	Appropriate	0.818	Appropriate
Sensory limitation	0.818	Appropriate	0.818	Appropriate
Total	0.833	‐	0.879	‐

†CVR1: content validity ratio at first round.

‡CVR2: content validity ratio at second round.

Reliability

Cronbach's alpha value for the total sample (n = 211) was 0.460.

To carry out the test–retest study, 96 participants from five neighbourhoods were selected. The mean age was 46.3 years (± 4.4 years). 64.6% were women. The sociodemographic characteristics of the selected neighbourhoods are detailed in Table 3 (Pérez Valbuena & Salazar Mejía, 2007), and the characteristics of the participants according to their neighbourhood, age and sex and are detailed in Table 4.

TABLE 3

Sociodemographic characteristics of the neighbourhoods

Neighbourhood	Low‐income population (%)	Immigrants (%)	Years of education (mean)	Informal workers (%)
Piedra Bolivar	43.7	0	7.8	27.6
Zaragocilla	25.9	6.4	8.9	24.5
Calamares	16.3	4.4	10	25
Escallon Villa	23.3	1.3	9.2	32
Campiña	51.6	3.2	8.1	42.7

TABLE 4

Characteristics of sample

Neighbourhood	Men		Women		Total
	N	Age mean	N	Age mean	Age mean (CI95)†
Piedra Bolivar	3	48.13	6	58.38	54.9 (37.8–72)
Zaragocilla	9	47.72	23	43.00	44.3 (37.1–51.5)
Calamares	7	38.48	12	46.57	43.6 (35.2–52)
Escallon Villa	8	40.32	12	45.78	43.6 (33.2–54)
Campiña	7	55.79	9	49.70	52.3 (41.2–63.4)
Total	34	45.78	62	46.69	46.3 (41.9–50.7)

†CI95: confidence interval of 95%

Global scores were compared in the test and retest (Figure 1). There were no large differences between the measurements.

FIGURE 1

Global Care Vulnerability Index Score comparison (test and retest)

To measure correlation, the scores for each person were compared using Pearson's correlation test. Pearson's correlation coefficient between the two measurements (n = 96) was 0.912 (CI95: 0.872–0.941; p‐value <.01). Less data dispersion was obtained in CVI scores >0.2. Test–retest correlation graph is shown in Figure 2.

FIGURE 2

Correlation of Care Vulnerability Index test–retest

The Bland–Altman diagram (Figure 3) relates the difference between the CVI values obtained in test and retest with the mean of these two values. The limits of greater dispersion are concentrated in the range between 0.06 and 0.21. In all, 64.6% of the participants obtained the same score in the two measurements. The Bland–Altman diagram did not show a systematic pattern in the dispersion of the test and retest measurements. When comparing the difference of the IVC scores in the test and the retest against the mean of these two measurements, it was obtained that 10.4% (n = 10) of the cases were located outside the range of the mean ±2SD ( −0.08 and +0.07).

FIGURE 3

Bland–Altman diagram of Care Vulnerability Index test–retest

The difference in means turned out to be statistically non‐significant (d=−0.006; p‐value =0.6168, CI95: −0.032–0.019), so the hypothesis of the equality of the means of the CVI score in the two applications is accepted (test and retest), because it contains the value zero.

In the analysis of the difference in means for each of the clusters, it was found that there were no significant differences. Table 5 shows the values of the means, the differences and the values of the confidence interval for each of the clusters and the total CVI.

TABLE 5

Means difference test–retest

Cluster	Mean (test)	Mean (retest)	Difference	CI95†	p‐value
1	0.716	0.716	0	−0.227 to 0.227	1
2	0.154	0.154	0	−0.067 to 0.067	1
3	0.391	0.422	−0.031	−0.173 to 0.110	0.664
4	0.479	0.493	−0.014	−0.180 to 0.152	0.869
5	0.462	0.514	−0.052	−0.179 to 0.075	0.418
Total	0.147	0.153	0.006	−0.032 to 0.019	0.6168

†CI95: Confidence Interval of 95%

The normal distribution of the data was verified with the Lilliefors normality test for the two applications: test (d = 0.074; p‐value =.212) and retest (d = 0.079; p‐value =.151).

DISCUSSION

No validity and reliability studies have been found on care vulnerability. In relation to the theoretical model used, we have found the studies by Fernández‐Batalla & Jíménez‐Rodríguez (2015), which evaluated the validity of a computational tool (expert system) based on the same theoretical model used as the CVI, turning out to be valid (in terms of credibility and efficiency) for the simulation of clinical cases in Family and Community Nursing.

The theoretical approach of this research is based on the vulnerability that arises from Basic Conditioning Factors and how these factors have the ability to influence the competence of self‐care. Self‐care involves a continuum throughout the person's life, where they have to activate their competence to satisfy needs, which vary throughout their lives depending on the circumstances of the environment, health, etc.

In previous research, self‐care is only related to the adaptation of behaviour and lifestyle to certain health problems (De María et al., 2019; Buck et al., 2018; Loven‐Wickman et al., 2019).

Riegel et al. (2019) indicate that the limitations in finding studies with the proposed definition of self‐care is due to the fact that multiple terms are used as synonyms for self‐care, (such as self‐management, self‐monitoring, and self‐help) and it is not always clear how the term is defined.

In this respect, the concept of vulnerability that the authors of this study allude to is more related to the concept of pre‐frailty addressed by Rasiah et al. (2020).

In relation to other vulnerability scales, we find the Vulnerability Index, which has been used for studies of social vulnerability in homeless people (Bowie & Lawson, 2018) and validated with a retrospective review of hospital admissions in the homeless population, finding that hospital admissions predicted Vulnerability Index scores, including health conditions and substance use (Cronley et al., 2013).

As indicated by Zamanzadeh et al. (2015) and Pedrosa et al. (2013), some limitations of content validity studies should be considered, since feedback from experts is subjective and may be linked to bias among experts.

In this study, the confidence interval of Pearson's correlation coefficient moves in a narrow range, with a lower limit that exceeds the value of 0.85 as an expression of a good correlation between the two scores (Rodríguez‐Mancebo, López‐Pumar, & Marrero‐Santos, 2013).

From the theoretical model, it is assumed that vulnerability constitutes a characteristic of the person, so it cannot have a “zero” value, however, in our data 4 participants are identified (4.17%) who in the two applications of the test gave a care vulnerability level of zero. The zero value does not represent invulnerability. In this sense, the experts indicate that a zero value corresponds to the minimum level of vulnerability measurable through the Basic Conditioning Factors (Fernández‐Batalla, 2019).

The CVI questionnaire was developed to be an instrument applied by a nurse trained community health, even so, it requires a certain capacity to understand and interpret the questions that comprise it. The application of the questionnaire was assisted by the researcher to ensure a better understanding of the terms of the questions. This procedure, according to Rodríguez et al. (2013), reduces incomplete answers and unanswered questions, and it is considered that it did not affect the reliability result, because the same method was adopted in the two applications.

Most of the study participants (86.46%) obtained vulnerability values <25% of the theoretical maximum value. The population studied corresponds to the general population that lives in a certain area of the city of Cartagena de Indias (Colombia), so that, a priori, they are people with a health and care situation that allows them to live in a home and not be admitted to hospitals or care centres.

The results of our study suggest that there is stability in the total CVI score obtained by the participants. This is indicated by the absence of significant differences in the score obtained in the questions that make up the questionnaire between test and retest. The mean difference between the CVI scores in the two applications of the questionnaire was very small (−0.006), which is indicative that there is no systematic error in these results that can be attributed to the instrument (Rodríguez‐Mancebo et al., 2013).

The data were obtained with an interval of two weeks, considered a sufficiently long time so that the answers to the questions of the questionnaire from the first application were not remembered, and is within the interquartile range recommended by Park (Park et al., 2018) for studies test–retest reliability in patient‐reported outcome measures for older people. In this period, events could occur that positively or negatively affected the vulnerability of participants. These changes may be related to the true dynamics that vulnerability suffers, which can be affected by vital events, the perception of health, as well as factors that act in the environment.

The present study incorporates data from a Colombian population and in the Spanish language, which would allow its application in Spanish‐speaking areas. The validation of this vulnerability determination model to other languages, considering cultural adaptation (Risco et al., 2020), will allow to expand the model in a better way.

Limitations

Unlike other validation studies (Muhammed et al., 2020; Zha et al., 2020), Cronbach's Alpha value was not given special importance for measuring internal consistency, because it constitutes a multifactorial scale, as indicated by Streiner (2003). In this case, the low value obtained would be explained because the vulnerability of care constitutes a multidimensional characteristic (the items that make it up are independent). In other words, the characterization of the person by their Basic Conditioning Factors makes an assessment of essentially different elements (Fernández‐Batalla, 2018).

The scale was designed and used during the study in Spanish. The experts who validated the scale, as well as the participants who completed it, were all Spanish speakers. The translation into English has been carried out for the realization of this article, so it would be necessary to carry out a cross‐cultural adaptation for use in a language other than Spanish.

More studies with different populations are needed to improve the validity of the CVI.

CONCLUSION

This study concludes with some implications. The CVI questionnaire is qualifies as a valid instrument for the purposes of evaluating the vulnerability of people based on how they take care of themselves. The study carried out indicates that CVI has adequate reliability in terms of the stability of the information derived from its application in the general population.

The approach to the vulnerability of the population allows a better approach to the concept of self‐care and it allows to know the capacities and the needs of the person.

It is important to have validated tools to measure the level of vulnerability of people. This will allow us to identify which population groups are susceptible to care problems, both at the individual, family and community level. In this way, the institutions have tools to prioritize healthcare resources.

On the other hand, having population indicators of care vulnerability will allow nurses to work with a theoretical framework that integrates patient care and health decision‐making throughout the life continuum.

CONFLICT OF INTEREST

No conflict of interest has been declared by the authors. All authors have approved the manuscript and declared that this manuscript has not been published before and have not received funding source.

AUTHOR CONTRIBUTION

Marta Fernández, Enrique Monsalvo and José María Santamaría conceived the study, analysed the data and drafted the manuscript. Estela Melguizo, Yolima Manrique and Alexandra González analysed the data and made substantial contributions. Lourdes Jiménez, Sara Herrero and Blanca Gonzalo made a critical review of the manuscript with substantial contributions. As corresponding author, I confirm that all named authors have approved the manuscript and agree with its submission.

Supplementary Material

Click here for additional data file.