The validity and reliability of the second korean working conditions survey.

BACKGROUND: The aim of this study was to evaluate the quality of the Second Korean Working Conditions Survey (KWCS), focusing on its validity and reliability.
METHODS: The external validity was evaluated by the assessment of sampling procedures and the response rate, in order to investigate the representativeness of the sample. The content validity was evaluated by the assessment of the development of the questionnaire, and the consistency of questions for the selected construct. The test-retest method was used to evaluate the reliability by means of a phone call survey of 30% of the respondents, who were randomly selected. The respondents' satisfaction regarding the survey procedures and interview time were analyzed to evaluate the quality of survey data.
RESULTS: The external validity was assured by an acceptable sampling procedure, rigid multi-stage stratified cluster random sampling. The content validity was also guaranteed by a reasonable procedure for the development of the questionnaire with a pretest. The internal consistency of the questions for work autonomy was maintained, with 0.738 of Cronbach's alpha. The response rate of 36% was lower than that of the European Working Conditions Survey (EWCS), with a contact rate of 66%, compared to 76% for the EWCS. The matching rates of the five retested questions were more than 98% reliable.
CONCLUSION: The quality of the second KWCS was assured by the high external and content validity and reliability. The rigid sampling procedure and development of the questionnaire contributed to quality assurance. The high level of reliability may be guaranteed by the sophisticated field survey procedures and the development of a technical manual for interviewers. The technical strategies for a high response rate should be developed for future surveys.

Introduction

The Korean Working Conditions Survey (KWCS) was developed to survey the working conditions of the nation's working population, and to assess their exposure to working factors, including the state of this exposure according to job type, type of occupation, and employment status. As a reference, the survey used the European Working Conditions Survey (EWCS), which has been carried out every 4–5 years since 1991.

The Occupational Safety and Health Research Institute conducted the first census-based Working Conditions Survey in 2006 to contribute towards creating occupational safety and health policies for improved working conditions for employees [1]. The institute then carried out a second survey in 2010 to gather sufficient data on changes in working conditions and to contribute to policy setting. Although social interest in working conditions has increased as a result of various social issues, there is still a lack of data that would be useful in setting policies to improve working conditions. To date, the focus of working conditions has been on mechanical, physical, and chemical hazard factors. However, interest has increased recently in work-related risk factors, such as musculoskeletal burdens [2] and socio-psychological [3] factors. In addition, interest has increased in working conditions according to employment status, such as those of vulnerable social groups [4], as well as differences in working conditions according to social class or target group, such as those between male and female workers [5]. Information on how employees are exposed to physical and chemical risk factors can be acquired through working environment measurements, or from the quinquennial National Occupational Exposure Census [6]. However, there is no information system that contains data on exposure to working factors, such as musculoskeletal pressures and socio-psychological factors, or working conditions of specific classes or target groups. The KWCS data can help with setting policy objectives and targets, by (1) assessing the exposure rate to health risk factors and identifying occupations with high risk factors; (2) helping to set safety and health policies that reflect changing working conditions by assessing changes in working conditions through periodic surveys, in addition to the cross-sectional investigation of working conditions; and (3) including survey items that may help detect changes in the labor market. By identifying the links between the labor market and health risk factors, it can help in drafting policies that effectively reduce occupational health risk factors.

This paper was written to investigate the validity and reliability of the second KWCS, in order to improve the quality of future KWCSs, and to guide the secondary analysis of KWCS data. The authors tried to assess the validity and reliability, with provision of information on the sampling frame, research procedures, survey items, and response rate. Among the various validity items, external and content validity and reliability were assessed by matching the rate of the result of the test and retest, interview time variation, and the satisfaction of respondents towards the interview process.

Materials and methods

The quality of survey data can be assessed by its validity and reliability. Validity is the degree to which it measures what it is supposed to measure. Reliability is the extent to which a measurement gives results that are consistent. Validity is subdivided into four different types, namely external validity, internal validity, content validity, and conclusion validity [7]. However, because the working conditions survey was oriented not towards causal questions but towards the monitoring of the state of working conditions, and the survey design was cross-sectional, the authors assessed only the external and content validities, and not the internal and conclusion validities. The reliability may be analyzed by two different approaches: one is the test-retest method, and the other is comparison between the reference and observation.

This study assessed external validity by focusing on the sampling procedure and response rate of the target population and allocation. The sampling design can be assessed by analyzing the sampling procedure, in order to guarantee the representativeness of the population. The response rate is a useful indicator for the quality of sample data. The authors compared the response rates of the KWCS to those of the EWCS.

The content validity was also assessed by focusing on the development of the questionnaire. The translation and back translation of the questionnaire of the fourth EWCS was used in the first KWCS. Another procedure for developing the questionnaire for the second KWCS was analyzed, namely the procedure for adding new questions and the changing of old questions. The changing of questions in the fifth EWCS in 2010 by the European Foundation for the Improvement of Living and Working Conditions (Eurofound) was considered. The KWCS was developed to monitor the state of working conditions, namely working hours, workload and autonomy, and exposure to various hazards, etc., which were used in the EWCS. The questionnaire development for KWCS was focused on monitoring the working conditions of Korean workers and comparing these conditions with those of European workers. The authors assessed how these two different focuses reflected on the development of the questionnaire.

The test-retest method was used to assess the reliability of the KWCS, including the internal consistency of some questions for selected constructs, such as work autonomy. The sample for the retest was selected randomly from the surveyed sample. Selected questions for which answers could not be changed just after the survey were retested by phone call survey, for example, address, gender, birth year, employment status, and a question about the attitude of the respondent towards the risk of work. The home call survey for the retest was processed by new interviewers who had not participated in the main survey. The attitudes of respondents towards the KWCS, for example satisfaction with the survey, were included in the retest survey, because this kind of attitude may be one of the indicators of the reliability of survey data.

Results

External validity

Target population

The target population of the second KWCS can be defined as “The entire working population, aged 15 years and over, in all households residing in Korea, as of the time of the survey.” However, as a result of practicality issues, the target population was selected as households from the apartments and general enumeration districts of the 2005 census. These excluded islands, dormitories, special welfare institutions, tourist hotels, and foreigner enumeration districts. Then, those who fit the definition of “working” from the sampled households were chosen as the final survey targets.

From the sample enumeration districts extracted, “ordinary households” set as survey targets were defined as “households consisting of family members, households consisting of both family members and five persons or less without blood relationship, households with five members or less without blood relationship, and one-person households,” and “institutional households” were excluded, which was defined as “households with six members and over without blood relationship, and households living together in such facilities as an orphanage, a dormitory, and a child-care institution”, according to the National Statistical Office definition [8].

The final survey subjects were people aged 15 years and over who were working and living permanently within the sample household. Here, “workers” were defined as “those who have worked for at least an hour, in return for some form of compensation, in the week preceding the survey.” This definition is in line with the EWCS and the Economically Active Population Survey (EAPS) from the definition of survey term by Statistics Korea. Possible employment states include “self-employed without salaried employees,” “self-employed with salaried employees,” “employee,” “unpaid family-member workers,” and “temporarily unemployed.” To reflect the working structure of an aging society [9], in contrast to 2006, the upper limit of age 65 years was removed. To select the final survey targets, after visiting the households a “household member list” was composed to check for those who were eligible. When two or more people were eligible, we used the “most recent birthday method” [10] to select the person with a birthday nearest to the research date as the final survey target. A professional surveyor visited homes for a face-to-face interview with the selected household member.

Sampling design

The sample frame was the 2005 census. The preliminary sampling unit was the population and housing enumeration district, and the secondary sampling unit was the household and household members. The sampling of the sample enumeration district, the preliminary sampling unit, used probability proportion [11] to size systematic sampling in proportion to the number of households within the enumeration district. In the sampled enumeration district, the principle was to systematically investigate 10 sampled households [12]. We selected the 10 target households using systematic sampling from the household lists and district maps of the enumeration district. Professional surveyors conducted interviews with a total of 10,000 workers in 10,000 households (1 per household) in the 16 major Korean cities/provinces, selected through stratified cluster sampling [13]. In all, 127 professional surveyors carried out the surveys from June 20, 2010 to October 10, 2010 (15 weeks, excluding Chuseok holidays).

In total, there were 265,350 population enumeration districts, and 15,887,128 households. In the sample design, the primary sampling unit was the enumeration district of the census (Table 1). The secondary sampling unit was the household and household members. The enumeration districts of the census, the primary sampling unit, are formed by grouping adjacent households geographically, and comprise 60–70 households. In the sample design, stratification was based on the characteristic information of the enumeration district, the primary sampling unit. Seven metropolitan cities and nine provinces were primarily stratified to produce the statistics for each city and province. The nine provinces were then further stratified into neighborhoods (dong), townships (myeon), and towns (eup). Depending on the characteristics of an enumeration district, it was secondarily stratified into apartment enumeration districts and general enumeration districts, or into agriculture/forestry/fishery (AFF) household enumeration districts, and non-AFF household enumeration districts. Within the 28 primary strata, grouped by administrative division, a detailed stratification was undertaken by dividing the strata in Seoul, metropolitan cities, and provinces into apartment enumeration districts and general enumeration districts. In addition, the strata in counties, towns, and townships were divided into two groups, at the boundary of 50% of AFF household enumeration districts. This stratification process yielded 56 strata for the sample design.

Table 1

Distribution of sample and census enumeration districts by administrative and enumeration district characteristic (sample/census)

Administrative division	Apartment/AFF household stratum	General/non-AFF household stratum	Total	Administrative division	Apartment/AFF household stratum	General/non-AFF household stratum	Total
National	466/118,240	534/1,447,110	1,000/265,350	Chungbuk-dong	17/2,605	15/2,400	32/5,005
Seoul-Northwest	7/2,043	21/7,599	28/9,642	Chungbuk-eup/myeon	5/1,658	5/1,768	10/3,426
Seoul-Northeast	20/6,642	32/11,024	52/17,666	Chungnam-dong	13/2,058	12/1,778	25/3,836
Seoul-Southwest	18/5,668	32/11,073	50/16,741	Chungnam-eup/myeon	11/3,482	12/3,789	23/7,271
Seoul-Southeast	14/4,618	17/5,892	31/10,510	Chunbook-dong	21/3,822	16/3,033	37/6,855
Busan	30/8,105	41/11.960	71/20,065	Chunbook-eup/myeon	7/2,641	3/1,104	10/3,745
Daegu	24/5,586	31/7,797	55/13,383	Chunnam-dong	16/2,420	12/2,037	28/4,457
Incheon	25/6,218	30/7,970	55/14,188	Chunnam-eup/myeon	14/4,478	7/2,408	21/6,886
Gwangju	23/4,217	17/3,293	40/7,510	Gyeongbuk-dong	18/3,467	22/4,248	40/7,715
Daejon	19/33,822	21/4,196	40/8,018	Gyeongbuk-eup/myeon	11/4,422	9/3,764	20/8,186
Ulsan	17/2,593	18/2,967	35/5,560	Gyeongnam-dong	21/4,561	26/5,532	47/10,094
Gyeonggi-dong	73/22,368	69/22,577	142/44,945	Gyeongnam-eup/myeon	8/3,315	10/4,445	18/7,760
Gyeonggi-eup/myeon	4/2,218	16/7,792	20/10,010	Jeju-dong	6/523	16/1,618	22/2,141
Gangwon-dong	16/2,641	16/2,656	32/5,297	Jeju-eup/myeon	4/572	2/290	6/862
Gangwon-eup/myeon	4/1,476	6/2,100	10/3,576

AFF, agriculture/forestry/fishery.

Allocation

One of the important objectives of the sample design of the second KWCS was to produce relatively precise statistics on the working conditions per statistic publication unit. To achieve this, a sample above a certain size was allocated per city and provincial unit. In this study, the anticipated maximum limit of the error margin of the population proportion estimate at the 95% confidence level was approximately 1.5%.

The distribution of the sample enumeration district per administrative division (the primary stratum) was decided by comparing the proportional allocation method, the square proportional allocation method. The sample distribution method ultimately used for the sample design was the proportional allocation of households in each city and province followed by the priority allocation of 20 enumeration districts to each city and province. This was identical to the sample distribution method used in the first KWCS sample design.

The sample distribution of detailed strata of each regional stratum followed the household number proportion allocation principle. Other than in towns and townships (eup-myeon) with a high AFF ratio (Strata 1), the sampling rate was lower than that of other strata. The reason for reducing the sampling rates of strata with relatively high AFF household ratios (Strata 1 at the eup-myeon level) was to increase the efficiency of estimations because these strata have high homogeneity in the classification by job type or industry sector. Table 1 shows the sample size status of each stratum within each region.

The sampling of the enumeration districts (the primary sampling unit) was done using the probability proportional to size systematic sampling method, which extracts samples in proportion to the number of households within each enumeration district. When applying systematic sampling, the most effective method is to arrange the sampling units by classification index first before sampling. Here, the enumeration districts of the 56 strata were first aligned according to administrative district number and then sampled using the probability proportional to size systematic sampling method so that each stratum could be a geographical representative. The selection method used to identify target households was to select 10 households using systematic sampling from the district maps and household lists of the sample enumeration district. We then needed to visit each household to identify eligibility.

In cases in which an entire sample enumeration district was unavailable for survey because of an incident between sampling and surveying (e.g., rebuilding, redevelopment) or because of changes in characteristics (e.g., residential districts turned into shopping districts), the enumeration district was replaced by a district that had the same classification indicators at the time of sampling. The number of substituted sample enumeration districts in the second KWCS was 53 out of 1,000. By region, Seoul had 13, Gyeonggi-do had six, Busan had seven, Gangwon had seven, and the rest had less than five. The main reasons for the substitutions were either that there were no households to survey because of redevelopment or a similar event (17 districts) or the apartment management offices refused the survey (13 districts). The maximum number of visits for each household was four visits, each with different days and hours (twice on weekdays, 4 times on weekend days). Each surveyor recorded the contact conditions for each household in the Household Visit Record Sheet. If, even on the fourth visit, the interview could not be carried out, the reason was recorded. Then, based on the household list, a substitute household was selected, following the preset contact order. In the case in which contact was made with any one of the household members, the affiliation of the surveyor and the purpose of the visit were clarified. The surveyor then checked how many “eligible members” were in the household (working household members aged 15 years and over), and if there was at least one member, the household member list was completed, according to the survey form, and the eligible member was identified. If a household contained two or more eligible members, the final eligible member was chosen based on the “most recent birthday method.” If the eligible member was not at home, the surveyor established the nearest possible interview date and time to revisit and conduct the survey.

Response rate

Records on various rates, such as response rates, are core measurement indicators that allow the quality of the survey to be assessed. In the KWCS, a seven-code recording method, developed by the Standard Definitions (2011) of the American Association for Public Opinion Research (AAPOR) [14] was used during the field survey. Using this coding system, the response rates, cooperation rates, refusal rates, and contact rates for the KWCS were calculated (Table 2). In the second KWCS, surveyors visited a total 49,248 households from the 1,000 sample enumeration districts. Of these, 10,019 were successfully interviewed and included in the final analysis (I). There were 6,244 households with which contact was made and that had eligible members but that refused or were unavailable for interviews (R). There were 1,214 households that were visited at least four times, but could not be interviewed because of the absence of the eligible member (NC). Despite contact with the eligible household member, an interview could not take place because of situational circumstances (O) in 661 households. There were 5,416 households that had no eligible members. In the case of 660 households, it was not possible to confirm if there were any eligible members (UO). In 12,428 households, residence or eligibility could not be confirmed because the members (UH) were absent. The remaining 12,606 households refused to participate in the survey.

Table 2

Response rate, cooperation rate, refusal rate, and contact rate of the second KWCS

Rates*	First KWCS	Second KWCS	Fourth EWCS	Fifth EWCS
Response rate (RR3)	0.35	0.36	0.47	0.44
Cooperation rate (COOP3)	0.59	0.62	0.66	0.60
Refusal rate (REF2)	0.24	0.22	0.24	0.30
Contact rate (CON2)	0.59	0.60	0.77	0.76

*Based on the standard definitions of the American Association for Public Opinion Research.

EWCS, European Working Conditions Survey; KWCS, Korean Working Conditions Survey.

Although the contact rate was relatively lower in the second KWCS (CON2 0.60) than in the fifth EWCS (0.76), the cooperation rates (COOP3: KWCS 0.62, EWCS 0.60) were higher, and refusal rates (REF2: KWCS 0.22, EWCS 0.30) were lower. Despite this, the overall response rate of the KWCS (RR3 0.36) was lower than that of the fifth EWCS (RR3 0.44), not reaching the de facto EU standard [15,16].

Content validity

The original survey of the fifth EWCS, currently held in Europe, was acquired and translated for the survey design of the second KWCS. The “Injury/Disease” questionnaire was drafted by reorganizing the questions of the 2009 Labor Force Survey of England [17]. The survey questionnaire of the EWCS was developed in close cooperation with a questionnaire development expert group. This group included members of Eurofound's Governing Board, representatives of the European Social Partners, other EU bodies [the European Union (EU) Commission, Eurostat, European Agency for Safety and Health at Work], international organizations (OECD: Organization for Economic Cooperation and Development, ILO: International Labor Organization), national statistical institutes, and leading European experts in the field. The survey has been translated into 27 languages and 15 language variants [18]. To solve the problem with translation, the English questionnaire items of the EWCS were translated by two professional translators, cross-checked, and back-translated for the final KWCS questionnaire items. The main topics covered by the questionnaire were: job context; working time; work intensity; physical factors; cognitive factors; psychosocial factors; health and well-being; skills, training, and career prospects; work organization; social relationships; job fulfillment; work-life balance; financial security; violence, harassment, and discrimination; consumption of alcohol/tobacco; experience of work-related injury and diseases; and respondent characteristics. New questions from the fifth EWCS were also introduced to enable more in-depth analysis of psychosocial risks, worker participation, precarious employment and job security, place of work, work-life balance, leadership styles, and health.

Content validity may be assessed by the questions on concept of working conditions. Most of the questions were translated from those of the fifth EWCS. Notwithstanding the sophisticated development of the fifth questionnaire of the EWCS, every question should be validated by a professional before the survey. Some of them were already applied to the first KWCS, and the result of the first KWCS was evaluated by comparison with those of the fourth EWCS [1]. In the back-translation, emphasis was placed on conceptual and functional (rather than linguistic) equivalence. Some discrepancies were discussed until a satisfactory version was reached. Other questions translated from the questions of the Labor Force Survey of the UK had been reviewed by professionals before the second KWCS with the above procedures. An expert review on the additional questions focused on the wording and relevancy of questions.

The draft questionnaire was finalized into a version that could be pretested. The pretest was held and comprised cognitive interviews and ‘real life’ interviews. The respondent and interviewer feedback from the pretest helped to establish the final questionnaire. The aim of the pretest was, firstly, to assess whether the questionnaire was relevant to and easily understood by the respondents in terms of the concepts, and the way they were phrased in the questions; and secondly, to assess the technical functioning of the questionnaire. There were two stages of testing. In the first stage, ‘real life’ interviews were carried out using the draft version of the questionnaire. This part focused on the interviewer–respondent interaction, and technical functioning of the questionnaire (interviewer instructions, order of questions, filters, response categories, and interview duration). In the second stage, cognitive interviews were conducted aimed at gaining an understanding of the answering processes and finding out whether the new survey questions were understood as intended. Based on the results of the pretesting, the final version of the survey questionnaire was compiled. The modifications were made, for example, by revising the structure of the questionnaire, adding more instructions, and rephrasing questions. The sample of the pretest was composed of 40 respondents selected from different areas and types of household.

Reliability

Quality management of the retrieved survey sheets from each region was conducted in three stages: (1) the first preliminary inspection by the regional field overseer; (2) the second preliminary inspection by professional editors at headquarters; and (3) the 30% random verification by phone. In each stage, problematic survey sheets were either discarded or supplementary surveys were conducted. The content of the 30% random phone verification confirmed whether the survey answers and the actual answers of the address, gender of respondent, birth year, employment status, and health risks at the current job coincided. Every retested item showed more than 98% of the matching rate, which means high reliability of the survey data (Table 3). There were little variations in the matching rate of each retested item as the coefficient of variation among areas was less than 1.3%. The question about the attitude towards the risk of work had a high matching rate, in spite of its variable characteristic. In addition, the interview time and satisfaction about the interview process were asked in the retest (Table 4). The average interview time was 32.1 minutes, which is less than that of the fourth and fifth EWCS (35 minutes, 42 minutes) [15,16]. The satisfaction of respondents about the interview may have influence on the quality of the survey data. The percentage of respondents satisfied with the interview was a total of 94.3%, with a 3.2% coefficient of variation among the survey areas.

Table 3

Matching rate of answers of the test-retest

Characteristics	Retest item
	Address	Gender	Birth year	Employment status	Q66*
Matching rate (%)	98.6	99.6	98.9	98.5	99.0
Coefficient of variation of matching rate among sampling areas (%)	0.6	0.4	0.8	1.3	0.5

* Do you think your health or safety is at risk because of your work?.

Table 4

Average interview time and the percent of respondents satisfied with the interview

Characteristics	Average interview time (min)	Respondents satisfied with the interview (%)
	32.1	94.3
Coefficient of variation among the sampling areas	8.7%	3.2%

Problematic surveys discovered in the follow-up validation results were sent back to the regional field overseers for the reconfirmation process. If errors such as “false entry” were found again, even in the reconfirmation process, the survey sheet was scrapped, and another household was visited. Of the 10,132 sheets retrieved, 113 sheets were discarded after the first and second inspections. Thus, 10,019 sheets were included in the final study, exceeding the target of 1,000 enumeration districts and 10,000 sample households planned in the sample design.

The internal consistency test was applied to some questions, for work autonomy. Work autonomy was one of the concepts for socio-psychological working conditions. Questions for work autonomy were measured by a five-point scale of agreement, as follows. The Cronbach's alpha value of work autonomy questions in the first KWCS was 0.767 and that of the second KWCS was 0.738, therefore this was reliable as a construct.

You can take your break when you wish

You are free to decide when to take holidays or days off

At work, you have the opportunity to do what you do best

Your job gives you the feeling of work well done

You are able to apply your own ideas in your work

Discussion

The quality of the KWCS may be assessed by its validity and reliability. This paper assessed the external and content validity among the four validity categories. The internal validity and conclusion validity were not evaluated because the KWCS was oriented towards monitoring the state of working conditions by cross-sectional survey design. The external validity means the validity of generalized inferences in scientific studies, usually based on experiments, as experimental validity [19]. In other words, it is the extent to which the results of a study can be generalized to other situations and to other people [20].

The external validity of the KWCS means the degree to which the results of the sample can be generalized to the Korean worker population. The representativeness of the sample may be evaluated as the external validity by analyzing the sampling procedures with the sample framework and response rate. The external validity can be assured by sound sampling procedures. The sample of the second KWCS was selected by rigid sampling procedures from the sample frame to select respondents of sampled households. The response rate may be also one of the indicators for the representativeness of the sample. The response rate (RR3) of the second KWCS was 0.36, which is relatively lower than that (0.44) of EWCS. However, the response rate of the second KWCS was increased from that of the first KWCS, when compared with the decrease of the response rate of the fifth EWCS from that of the fourth EWCS. The refusal rate of the KWCS (0.22) was lower than that of EWCS (0.30). In the case of KWCS, the external validity was influenced by the sampling procedures and field survey. In the next survey of KWCS, technical strategies for the improvement of the response rate should be developed in order to assure the external validity. Recently, the conditions for household interview surveys in Korea have been getting more difficult because the number of one-person households has increased, and the protection of privacy is more recognized by Korean people. In the future, these difficulties for household interview surveys should be considered in the development of a technical manual for interviewers.

The sampling frame of the KWCS was a popular method for a household survey, especially for a labor force survey. The response rate is one of the indicators for the quality of the survey result. The critical value of the acceptable response rate cannot be ascertained because the acceptable response rate was assessed by so many factors, such as sampling frame, questions, survey method, etc. [21]. Babbie [22] reported that a response rate of 50% is adequate for analysis and reporting. He also mentioned that an acceptable response rate was only a rough guide; it has no statistical basis and the demonstrated lack of response bias is far more important than a high response rate. In the future, the nonresponse bias of the KWCS should be identified by analysis of refusal cases.

Content validity is one of the construct validities referred to in the general case of translating any construct into an operationalization [7]. In particular, content validity may be checked by the operationalization against the relevant content domain for the construct. This paper assessed the procedures of questionnaire development, including translation and back-translation of those of the EWCS and the Labor Force Survey in the UK, and the pretest of the questionnaire. However, as another aspect of content validity, domain validity could be assessed by expert review in the Korean situation because there is a difference between those in the Korean and UK surveys. The domains of working conditions of Korean society may be different from those of the UK. If some of the domain is unique to Korea, the KWCS should include those kinds of domain. However, in comparing the actual situation of working conditions of Korean workers with those of UK workers, the comparability of the survey content should be maintained.

The three components of content validity during the translation were assessed by expert review. Functional equivalence as semantic equivalence involves the choice of terms and sentence structures so that the meaning of the source language statement is preserved in the translation. Conceptual equivalence refers to the degree to which a concept, independent of the words used to operationalize it, exists in the same form in the source and target cultures. But normative equivalence could not be assessed by experts because the results of the KWCS will be compared with those of the EWCS. Some of the differences between the results of the KWCS and the EWCS may be interpreted as societal rules and norms.

The result of the test-retest has shown the high reliability of the KWCS. Even though the scope of retest questions was limited to five questions, including one attitude question, a high matching rate was produced by phone call survey, with a 30% sample of total respondents. The coefficients of variation of the matching rate by sampling area were very low, at under 1.4%, which means the matching rate was shown to be homogenous by area. This paper did not cover the validity and reliability of the statistical output. The matching rate derived from the sample also assessed its generalization of total respondents with an error term. The authors do not include any kind of statistical evaluation of validity and reliability. In future surveys, the quality of the statistical output will be evaluated. Other approaches may be applied to evaluate the level of reliability, such as the comparison of some indicators with other reference values. The KWCS has used a similar sample frame to that of the EAPS in Korea. For example, the composition of the Korean labor force estimated from the KWCS can be compared with that of the EAPS. However, this paper did not cover the comparison between the KWCS and the EAPS. The KWCS was used as the source data for assessing factors related to the quality of working conditions. The approval of the National Statistical Office was received to continue the study and to analyze the survey data and trends in the generated information.

The quality of the KWCS may be assessed by the validity and reliability. The authors tried to evaluate the external and content validity by focusing on the procedures for sampling and questionnaire development. The reliability was evaluated by the test-retest method, with a phone call survey of 30% of the total respondents. The sound procedures for sampling of the second KWCS could ensure the external validity is high. Including the pretest, the similar process for questionnaire development, with expert review and back translation, also ensures the high content validity of the second KWCS. But the response rate was low because of the conditions for the household interview survey, in spite of the sophisticated technical guidance provided for the interviewers. The other indicators, such as interview time and respondent's satisfaction about the interview survey, have shown the assurance of the high quality of the KWCS. This paper has some limitations in analyzing the validity and reliability. The first limitation may be found in the methodological approaches for the analysis of validity and reliability. The authors did not include the other method for validity and reliability. For example, the comparison between the survey output and reference data is not included. Another limitation may be associated with the interpretation of the results of the KWCS. For example, weighting and estimates of the sampling error are not discussed in this paper. In particular, statistical analysis for the validity and reliability test are not covered in this paper. The third limitation may be found in evaluating the reliability of internal consistency with only one concept.

The quality of the second KWCS was assured by the high external and content validity and reliability. The rigid sampling procedure and the development of the questionnaire with a pretest have contributed to the quality assurance. The high level of reliability may be guaranteed by the sophisticated field survey procedures and the development of a technical manual for the interviewers. Even though the response rate of the KWCS was similar to that of the EWCS, the technical strategies for high response rate should be developed for future surveys.

Conflict of Interest

No potential conflict of interest relevant to this article was reported. The findings and conclusions in this report are those of the authors and do not necessary represent the views of the Occupational Safety and Health Research Institute.

Acknowledgments

This study was supported by the intramural research fund of the Occupational Safety and Health Research Institute (OSHRI).