Development, validation and reliabilty of a questionnaire to evaluate the changes on the level of physical exercises and in daily life habits due to COVID-19 pandemic social distancing.

BACKGROUND AND AIM: Physical inactivity is expected to happen during the COVID-19 pandemic through home quarantine measures. The aim of this study was to develop, validate and perform the reliability of the questionnaire "Physical exercise (PE) level before and during social isolation (PEF-COVID19)" to evaluate the level of PE during the social distancing due to the COVID-19 pandemic and to try to identify changes in the daily life of the individuals.
METHODS: This transversal study was developed to measure psychometric properties of the questionnaire PEF-COVID19. The survey was divided into 4 sections including subjects' characterization, social isolation update and physical exercise performed, pain, anxiety and stress before and during COVID-19 pandemic. After the survey construction in Portuguese language (Brazil), the survey was transferred to an online digital platform (Google® forms). The Construct, Clarity and Relevance Validation strategy was judged by a panel of experts and the validity index (VI) were calculated. The reliability was evaluated through the test-retest interrater reliability and measured through the intraclass correlation coefficient (ICC) and Kappa coefficient (KC).
RESULTS: Twenty-five experts participated of the survey validity and 34 respondents from the target population participated of the test-retest reliability. The general average measures for VI were all above 0.84 and test-retest ICC and KC were 0.89 and 0.88, respectively.
CONCLUSIONS: This survey was considered valid and reliable to be applied to the general population over 18 years-old to investigate the PE practice and psychological aspects during the social distancing due to the COVID-19 pandemic, a public health problem.

Click here for additional data file.

The literature search was made in order to obtain the articles for the topic of interest synthesis. The Searches werecarried out through the electronic index databases (PubMed, SCOPUS, EMBASE) on 3rd March 2020. The data collection used the following keywords/descriptors/MESH in the English, Spanish and/or Portuguese languages according to the databases instructions. The general search string was: (“covid-19” OR “COVID-19 pandemic”) AND (“physical exercise” OR “physical activity”) AND (“stress and anxiety” OR “mental health” OR “psychological impact”) AND (“pain” OR “pain perception”) AND (“survey and questionnaire” OR “crosssectional study”) were used.

The aim of the literature search was to identify validated surveys in the literature to investigate the effects of the physical exercise practice, pain, stress and anxiety before and during the pandemic.

The studies that fulfilled the eligibility criteria (Table 1) were considered for the analysis and to serve as a base to build a new questionnaire in case of survey inexistence based on the proposed goal.

Table 1.

Questionnaire characteristics, by sections

Domain	Items	Measurements	Questions types
Section 1Characterization	18	Socio-demographic, anthropometric, occupation, level of education, marital status, health condition, drugs use	Open-ended, closed-ended, Yes/No
Section 2PE before SI	13	Auto-perception level of PE, PE practice, frequency and exercise types, levels of pain and regions of pain, stress and anxiety	Closed-ended, Yes/No, Likert scale
Section 3SI update	3	SI update, number of days in social isolation and if not, possible reasons for that.	Open-ended, Closed-ended, Yes/No
Section 4PE during SI	13	Auto-perception level of PE, PE practice, frequency and exercise types, levels of pain, stress and anxiety	Closed-ended, Yes/No, Likert scale

Legend: PE, physical exercise; SI, social isolation

Introduction

Self-administered questionnaires or surveys are tools commonly used in science health area for health research (1,2). Nevertheless, it is necessary to consider (i) the advantage is related to the practicality and number of persons that can be reached with the application and (ii) the disadvantage is the dependence of the subject interpretation, since a health professional will not conduct the questionnaire in case of any doubt.

If a questionnaire is subject dependent, the process to develop this instrument must be careful and consistent, following quality standards and requires verification of its usefulness before implementation (1).

World Health Organization (WHO) classified COVID-19 as a pandemic on March 11th 2020 (World Health Organization, 2020), caused by a respiratory virus called severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), described first time in Wuhan, China, in December 2019 (4). The first COVID-19 case in Brazil was confirmed on February 26th 2020 (5).

As a consequence of the COVID-19 characteristics, fast and well done strategies to reach useful and important information for the health area community is important and required. A survey that can be easily spread in many countries is an alternative to achieve this goal (6,7).

WHO advices the importance of the social distancing to prevent a rapid spread of the disease in the population at the same time; also, to avoid a collapse in the worldwide health systems with the goal to prevent the death of many people (8,9). A consequence of the lockdown is the reduction of the level of physical activity because the individuals reduce their daily activities outside. This condition is not good for heath in general, because it can contribute to sedentary behaviours (4) and in few months this can represent a public health problem. Authors have pointed the importance of physical exercise (PE) practice to prevent overall mortality (10,11), as well as cardiovascular disease-related mortality (Je et al, 2013), or cancer-relate mortality (12–14).

Considering that the physical exercise (PE) practice is committed and a fast source of information is required, the rationale of this study was to develop and validate the questionnaire “Physical exercise level before and during social isolation (PEF-COVID19)” to evaluate the level of physical activity during the social distancing due to the COVID-19 pandemic and to try to identify changes in the daily life of the general population over 18 years-old. Moreover, these findings could permit to identify undesirable behaviours of the population, and to program guidance strategies to minimize the bad consequences of the COVID-19 pandemic.

Methods

This is a transversal methodological study that was carried out in the end of March 2020, and the development of the “Physical exercise level before and during social isolation (PEF-COVID19)” survey, followed three main steps: reliability, validation and feasibility.

This project was approved by the Ethics Committee of the Hospital Universitário Pedro Ernesto (HUPE), Universidade do Estado do Rio de Janeiro (UERJ), under protocol number CAAE 30649620.1.0000.5259.

For this survey development, a seven-step scale design proposed by Artino et al. (1) was followed: 1) literature review; 2) conduct focus groups; 3) literature synthesis and discussion in the focus groups; 4) items were developed; 5) expert validation were conduct; 6) cognitive interviews with few respondents to check if they understood what was proposed; 7) pilot testing.

A literature review on 30th March 2020 was conducted to identify other validated questionnaires in the topic of interest and, to the best of our knowledge, no validated surveys were found at the time of the search (search strategy on Supplementary material). The searches were carried out in important databanks and the search strategy is presented as a supplementary material. This stimulated PhD doctors started to build the questionnaire after to set the objectives. The goals of the questionnaire were to investigate the physical exercise practice, changes in the exercise habits, presence of pain, anxiety and stress before and during the COVID-19 pandemic social distancing. The experts decided to build a self-administered questionnaire considering the worldwide pandemic. The challenge was to write clear and unambiguous questions using a proper vocabulary to the target population (1). The inclusion criteria to define the target sample were people from general population over 18 years old. The exclusion criterion was age below as 18 years old.

Considering the format, some questions were open ended, but the majority close ended with only one possibility of answer, to have answers easier to administer and analyze. Also, Likert scales were used to measure levels of pain, anxiety and stress. Items should be simple, short, and written in language familiar to the target respondents. The survey was divided into 4 sections, presented on Figure 1. Table 1 presents the questionnaire characteristics.

Figure 1.

Physical exercise level before and during social isolation (PEF-COVID19) questionnaire sections

After concluding the survey construct in Portuguese language (Brazil), the survey was transferred to an online digital platform (Google® forms) that was automatically hosted and a pilot testing was done to find language mistakes and errors in the questions format. The URL link data access is password protected through a unique study ID that ensures confidentially of all self-reported data, secured using a “Cloud” database (15). In the sequence, data was automatically scaled and scored by the platform and downloaded to Excel sheets, to analyse the results with a statistical software of choice.

Validity

A variety of methods for analyzing the validation of quantitative data collected exists, but independent of the method used, acceptability of an item or scale and its criterion should be determined in advanced (16). The validity of an instrument is related to the extent to which the interpretations of a test are warranted (17). An instrument is valid when its construction and applicability allow the accurate desired measurement (18).

In general, there are some kinds of validity as content validity, appearance validity, clarity validity and validity of construct. Content validity, one of the types of validation used in this investigation, is based on judgment of a panel of experts in a specific area of interest (19,20). This means that content validation determines if an instrument effectively exploits certain phenomenon to be investigated.

In this study, together with content analysis, the judges with extensive experience on the subject in question that composed the panel of experts scored four sections of the questionnaire with their specific goals to be achieved. The Construct, Clarity and Relevance Validation strategy was also judged by the panel of experts, even though it is considered a subjective and unsophisticated technique, as it provides only judgment on the relevance and adequacy of the items.

The scores from Likert scales and grades from the panel of experts compound the validity index (VI) determination (21).

A flowchart of the items selection and validation is presented on Figure 2.

Figure 2.

Flowchart of the items selection and validation

Panel of Experts

The Judgement and Quantification phase included two evaluations. The first assessment consisted of an expert panel that focused on the relevance and clarity of the questions as well as the significance and completeness of responses. The second evaluation comprised an online discussion with coworkers that focused on the understandability, completeness, plausibility and management of the instrument.

The Panel of experts were composed by doctors in science health who work in practice and science with physical exercise.

The instrument presented to the panel of experts was divided into 4 sections, according to the objectives of the instrument and for the questions, a Likert scale was used for Content, Construct, Clarity and Relevance Validation. The judges should give a score for each question from 1 to 10 (1-4 = not relevant/incomplete/unclear/meaningless; 5-7 = partially relevant/ partially well-constructed/ partially clear/ partially relevant; 8-10 = highly relevant/clear/complete/meaningful) (20,22).

Reliability

The reliability of a questionnaire can be considered the ability to reproduce a result consistently across time and space, or from different observers (19). As measurement error is present in respondents, the reliability can be evaluated using its internal consistency, test-retest reliability (stability test), and inter-rater reliability, respectively (23). The use of these psychometric properties depends on the kind of questions adopted and the application method (with or without raters). The guideline for reporting reliability and agreement studies (GRRAS) was followed in this study (24), as suggested by the Equator Network guideline for the study type.

Considering the specificities of the current questionnaire, the stability test was applied to measure the instrument reliability. The intraclass correlation coefficient (ICC) has become the preferred index to measure the test-retest reliability, as it reflects both correlation and agreement (25). Also, Kappa coefficient quantifies the level of agreement of categorical variables and measures inter- or intra-rater reliability or test-retest reliability in epidemiologic and clinical instruments (26).

A convenience sample from the target population based on the inclusion criteria was invited to answer the survey through social media. Fifteen days after the first time, these respondents were kindly asked to answer the survey again, to avoid the memory interference. An interval of 2 to 14 days can be used, however it depends on the attribute being measured (27). The data from the first and second sets of answers were tabulated in electronic sheets (Excel®, Microsoft, 2010) and the ICC and Kappa was calculated, depending on the nature of the questions if unranked (nominal) or ranked (ordinal).

Feasibility

The first part of the feasibility study was made through the panel of experts (25 doctors specialists in the field of physical exercise). For each section of the survey, the doctors answered through a likert scale, the clarity of the questions and they also could give an opinion about how to improve the questions.

The second part of feasibility study was made based on the guidelines proposed by ARTINO et al. (1) where cognitive interviews with few respondents were applied to check if they understood what was proposed. Three respondents from the target population were invited to fill the survey and to register the time to finish the questionnaire.

The respondents answered 3 questions: 1. “Was the survey clear, regarding its questions?”; 2. “Did you have some doubt during the survey filling?”; 3. “How many minutes did you spend to fill the survey?”.

Data Analysis

A validity index (VI) was calculated after the analysis of a panel of experts, doctors in the health area and human movement. The VI was calculated as the number of experts providing a score of above 8 in the Likert scale (scores were between 1 and 10) divided by the total number of experts. With more than 5 experts, the VI should not be lower than 0.78 (20,22).

The following cutoff points were considered for VI: ≥0.78: valid; <0.6 medium validity and ≤0.40 low validity (20).

For the reliability, respondents from the target population participated of the test-retest measurement. The agreement between the answers was measured through the intraclass correlation coefficient (ICC) and Kappa coefficient (KC). The two-way mixed effects model was used considering people effects are random and measures effects are fixed for the ICC. To calculate the ICC and KC values, the IBM SPSS Statistics for Windows, version 20.0 (IBM® Corp., Armonk, N.Y., USA) was used.

The cutoff points for ICC analysis was based on the guidelines which state that, the ICC below .40, the reliability is poor; between .40 and .59 is fair; between .60 and .74, the reliability is good; and when it is between .75 and 1.00 is excellent (28).

The agreement between nominal answers was calculated through Kappa coefficient analysis.

Researchers (29,30) have proposed the following as standards for strength of agreement for the KC with slightly different descriptors. According to Brennan and Silman (1992), if KC is below .20, the agreement is poor; between .21 and .40 is fair; between .41 and .60, the agreement is moderate; between .61 and .80 is good and between .81 and 1.00 is very good (29). Another proposition from Landis and Koch (30) ≤0=poor, .01–.20=slight, .21–.40=fair, .41–.60=moderate, .61–.80=substantial, and .81–1=almost perfect.

Results

The literature review strategy at the time of the survey construction (Supplementary material) did not find any validated survey on the topics of interest (physical exercise, pain, stress and anxiety before and during COVID-19 pandemic). Few manuscripts were related about the topics of interest on physical activity (31) or physical exercise (32), and psychological impact (32,33). These articles were used to provide ideas to the survey questions.

The results obtained in each of the phases of the validity and reliability process were reported descriptively. Twenty-five experts participated of the survey validity and 34 respondents answered the same survey fifteen days after the first time for the test-retest reliability. The respondents were Brazilians with mean age (±SD) of 46.6(±13.6) years, body mass 76(±17.7), height 1.63(±0.3), 82% highly educated (with masters or doctorate), 70.1% were married and 61.8% works in the health area (physiotherapy, veterinary medicine, nutrition, holistic therapies, nursing). The diagnosis was made on the basis of a sum of points (score) and the means of the scores for the different sections of the questionnaire.

To determine the validity index (VI) the scores provided from Likert scales and grades from the panel of experts, the VI was calculated and the results are presented on Table 2.

Table 2.

Final scores from the validity indexes by the panel of experts in relation to the content, construct, clarity and meaningfulness for the 4 sections of the questionnaire (n=25)

	I	II	III	IV
Content	0.93	0.93	0.84	0.93
Construct	0.92	0.90	0.84	0.92
Clarity	0.90	0.88	0.84	0.92
Relevance	0.95	0.90	0.84	0.93

Legend: I, Subjects characterization; II, Physical exercise performed or not, pain, anxiety and stress before COVID-19; III, Confinement situation update; IV, Physical exercise performed or not, pain, anxiety and stress during COVID-19

The absolute agreement definition for the 4 sections of the instrument are presented on Table 3.

Table 3.

Final scores from the intraclass correlation coefficient (ICC) and Kappa coefficient (KC) values for the 4 sections of the questionnaire (n=34)

	I	II	III	IV
ICC	1.00	0.94	0.70	0.93
KC	0.96	0.91	0.93	0.79

Legend: I, Subjects characterization; II, Physical exercise performed or not, pain, anxiety and stress before COVID-19; III, Confinement situation update; IV,Physical exercise performed or not, pain, anxiety and stress during COVID-19

The general average measures ICC test-retest for the questionnaire was 0.99 and Kappa 0.88. Considering the cutoff points, the 4 sections of the instrument were considered reliable.

The scores from the feasibility study gave by the panel of experts is presented on Table 2 (Clarity indexes). The second part of the feasibility study, with the target population, showed that for the interviewers the questionnaire was clear, they did not have any doubt about the questions and the mean time to fill the survey was about 4 minutes.

Discussion

This study reported the construction and evaluated the validity, reliability and feasibility of a self-administered questionnaire to investigate the level of physical exercise and some undesirable conditions related to the stress, pain and anxiety during the social distancing due to the COVID-19 pandemic, a public health problem. Moreover, it is expected that this survey can permit to identify undesirable behaviours of the population, and to provide information to the definition of program guidance strategies to minimize the bad consequences of the COVID-19 pandemic.

The COVID-19 disease crisis has drastically changed people’s normal routines, since the SARS-CoV-2 virus is still spreading around the world with impact in the global economy and in the population health (34). Many countries decided to lockdown and other social distancing and this all happened over a very short period of time (35), including the different regions from Brazil.

A sedentary life tends to increase with the social distancing and the established virtual contact to the work and community relationships (36). Such changes require a flexible adaptation to new circumstances to avoid sedentary behaviour, and mental illness and this goes in agreement with the purpose of the development of the current survey. Therefore, the need to perform physical exercise is an evidence as the need to guarantee the proper safety of individuals, during this period of crisis caused by the current pandemic (37).

All the steps involving the development of this survey were followed according to Artino et al., (2014). To the best of our knowledge, no validated surveys were found at the time of the instrument construction. Other self-administered questionnaires considering COVID-19 in different topics were also developed, as the desire for parenthood (38), the impacts on anxiety (39,40), the sleep disturbances (41), psychological outcomes (42), telemedicine (43) and so on, considering the worldwide pandemic. The findings of this study showed that clear and unambiguous questions were created using a proper vocabulary to the target population as the validity index (clarity) for each section of the survey, evaluated by a panel of experts.

About the questions format (the majority close ended), other survey also used similar format (44). Likert scales were used in the PEF-COVID19 to measure levels of pain, anxiety and stress; a study about knowledge, attitude, anxiety and perceived mental healthcare need in Indian population during COVID-19 pandemic also used the same format (40). This study used the Google® forms digital platform as it is a free and reliable platform (15), as other studies (40,43,45).

Evaluating validity (20,22) and reliability (19,23) is important to know the accuracy of an instrument and if similar results are reproduced under the same methodological conditions. One of the ways of evaluating the validity is through a panel of experts and the reliability of self-administered questionnaires is through test – retest. High levels of reliability and validity of a health assessment instrument, denote that it is an instrument that can be used at different social, economic and cultural levels in a given population (46). This fact will allow an excellent identification of the health problems in that population and, consequently, the more adequate elaboration and decision making to the reality.

Related to the validity, this instrument was considered valid by the panel of experts. They evaluated content, construct, clarity and relevance. Content relevance, representativeness, and technical quality are required to achieve content validity and this is mainly assessed through the panel of experts (18). The content validity is considered as the “theoretical analysis” (47). Boateng et al, 2018 suggest the same validation strategy used in this study and Goni et al., (2020) and Zamanzadeh et al., (2015) used them to develop their questionnaire.

Reliability shows how the measurement tool is reproducible and determines its internal consistency (49). The test-retest reliability with a diagnosis made on the basis of a sum of points (score) for each section of the questionnaire thought ICC and Kappa coefficient, depending on the nature of the question showed how reliable the instrument is. Technically the PEF-COVID19 is considered an instrument ready to be used to the general population and to answer the proposed goals.

The strength of this study is, to the best of our knowledge, this is the first survey to evaluate the level of physical exercise and psychological impact as stress and anxiety and pain of the Brazilian population during the COVID-19 outbreak. Moreover, it is expected that with the comparison of some parameters before and during the outbreak, the results can aid in the definition of policies to help the Brazilian population.

As a limitation, the answers that are given on the second occasion of the test-retest reliability, sometimes can be influenced by the ratings of the first measurement, threaten the assumption of independence.

Conclusion

This survey construction and validation followed a systematic, seven-step design process with the goal to develop a high-quality questionnaire. During a pandemic occurrence, a specific-related questionnaire requires agility and commitment between the involved researchers because the instrument can be applied only during this event.

This instrument was considered valid and reliable to be applied to the general population over 18 years-old to investigate the physical exercise practice and psychological aspects during the social distancing due to the COVID-19 pandemic.