Impaired work functioning due to common mental disorders in nurses and allied health professionals: the Nurses Work Functioning Questionnaire.

PURPOSE: Common mental disorders (CMD) negatively affect work functioning. In the health service sector not only the prevalence of CMDs is high, but work functioning problems are associated with a risk of serious consequences for patients and healthcare providers. If work functioning problems due to CMDs are detected early, timely help can be provided. Therefore, the aim of this study is to develop a detection questionnaire for impaired work functioning due to CMDs in nurses and allied health professionals working in hospitals.
METHODS: First, an item pool was developed by a systematic literature study and five focus group interviews with employees and experts. To evaluate the content validity, additional interviews were held. Second, a cross-sectional assessment of the item pool in 314 nurses and allied health professionals was used for item selection and for identification and corroboration of subscales by explorative and confirmatory factor analysis.
RESULTS: The study results in the Nurses Work Functioning Questionnaire (NWFQ), a 50-item self-report questionnaire consisting of seven subscales: cognitive aspects of task execution, impaired decision making, causing incidents at work, avoidance behavior, conflicts and irritations with colleagues, impaired contact with patients and their family, and lack of energy and motivation. The questionnaire has a proven high content validity. All subscales have good or acceptable internal consistency.
CONCLUSION: The Nurses Work Functioning Questionnaire gives insight into precise and concrete aspects of impaired work functioning of nurses and allied health professionals. The scores can be used as a starting point for purposeful interventions.

Introduction

Mental health complaints such as stress, mild depression, and anxiety disorders, often referred to as common mental disorders (CMDs), can lead to impairments in work performance (Aronsson et al. 2000; Hilton et al. 2008; Lerner et al. 2004; Lerner and Henke 2008; McKnight and Kashdan 2009). These impairments result not only in lower productivity; but in certain occupations, they can have serious consequences as well, e.g., in the work of nurses and allied health professionals. In these professions, consequences of impaired work functioning can affect the health of the caregiver as well their patients. Examples of these deleterious effects include medication errors, needle stick injuries, near errors, and decreased patient satisfaction (Gartner et al. 2010). These consequences are even more noteworthy given the high incidence of CMDs in this occupational group. The relative risk of depression is highest for nurses, RR = 3.5, 95% CI (1.3, 9.6), as compared with other human service workers and other healthcare workers (Wieclaw et al. 2006). A forecasted increase in the shortage of nursing personnel may result in an even higher risk for the development of CMDs, due to increased job demands and patient load or lack of supervisor support (Gershon et al. 2007; Stansfeld and Candy 2006; Sundin et al. 2007; Virtanen et al. 2008).

Both the high prevalence of CMDs and the high risk of serious adverse events in these occupations call for action. If we know the exact aspects of work functioning that are impaired, we can purposefully intervene in a proactive manner. In the short run, knowledge of impairments could result in increased awareness on the part of the employee, the supervisors, and the managers, which might be a starting point for discussion and personal support. Also, help-seeking behavior might be stimulated by the insight into impaired work functioning. Finally, detection of problems in work functioning due to CMDs can guide in developing purposeful interventions to improve work functioning and contribute to solutions for underlying mental health problems. For this purpose, sound measuring instruments can be helpful.

Examples of measuring instruments such as questionnaires for assessing impairments in work functioning do exist: the Work Limitation Questionnaire (WLQ)(Lerner et al. 2001), the Stanford Presenteeism Scale (SPS)(Koopman et al. 2002; Turpin et al. 2004), and the Endicott Work Productivity Scale (EWPS) (Endicott and Nee 1997). However, the detection ability of these scales has not been studied (Nieuwenhuijsen et al. 2010). We assume that mild CMDs can also result in impaired work functioning, even though the worker might not always be aware of the presence of mental health problems and their consequences. Many of the existing work functioning scales, e.g., the WLQ and the SPS, explicitly refer to health problems in their items. However, these questionnaires are less suitable for detecting new cases of workers with impaired work functioning due to mental disorders.

Furthermore, existing instruments were developed for the work context in general, rather than for a specific occupational group (Sanderson et al. 2007). An advantage of focusing on specific occupations is that items in a measuring instrument can refer more directly to the actual work practice and to concrete experiences of the employees. This approach enables the detection of specific aspects of work functioning that are impaired and thus enables subsequent concrete interventions. Therefore, we aim to develop a questionnaire for the early detection of impaired work functioning due to CMDs in nurses and allied health professionals. Our research questions are as follows:

Which self-report questionnaire items can be formulated to detect CMD-associated impairments in the work functioning of nurses and allied health professionals and how is the content validity of these questionnaire items evaluated by the target population?

Which subscales can be distinguished from a pool of formulated items and which items can be selected for definite use in the questionnaire?

Since a universally accepted definition of work functioning is missing, different concepts and perspectives were discussed by the authors to guide the development process of the detection instrument. Based on these various conceptions, for the purposes of our study, we consider work functioning as a comprehensive concept, encompassing a wide range of aspects measurable by self-reports. We include aspects of the work process and work outcome (Sonnentag and Frese 2002), as well as aspects of task execution and of organizational functioning, such as behavior within the team and toward the environment of the work organization (Motowidlo and Van Scotter 1994; Viswevaran and Ones 2000). Additionally, the extra effort to complete work tasks is included where appropriate (Dewa and Lin 2000). Furthermore, in the present study, rather than expressing impairments of work functioning solely in terms of quantity, qualitative aspects of work functioning will be addressed as well (Haslam et al. 2005; Suzuki et al. 2004; Yassi and Hancock 2005).

Following this description, we assume work functioning to be a multidimensional construct; therefore, no prior limit was set on the number of subscales and items the instrument should contain. Yet, we strive to develop a self-report questionnaire based on the classical test theory assumptions.

In the following, the methods and results of the two research questions will be described separately as part 1 and part 2.

Methods

Methods part 1: development of the item pool

Design

In order to develop a sound questionnaire with high content validity, a protocol based on recommendations by Haynes (Haynes et al. 1995) and by Terwee (Terwee et al. 2007) was followed. The development of the item pool comprised of three phases: the preparation phase, the item generation phase and the revision phase, is described in detail below. Figure 1 presents an overview of the study design with the methods and results for each step.

Fig. 1

Overview of the study design and the results of each step

Preparation phase

Procedure of the preparation phase: In the first phase, we conducted two systematic literature searches in four databases: PubMed, PsycINFO, Embase, and Cinahl. We aimed to inventory all literature about effects of CMDs on work functioning in general (first search) and nurses and allied health professionals in particular (second search) (Gartner et al. 2010). Subsequently, five focus group interviews were held. Following a multiple category design (Krueger and Casey 2000), three focus groups were held with nurses and allied health professional and two with experts on work functioning in the health sector.

The focus group interviews with a duration of 2 hours were conducted by two researchers (FG & KN) who alternately moderated or observed. The group interviews were structured by three cases, which were presented to the participants. The cases, written in the second person, described, respectively, an employee with fatigue and stress, depression and anxiety, and alcohol abuse. Participants were asked to reflect on aspects of the work that might be affected by the mental health complaints described. By working with these cases, participants of the employee focus groups were not forced to disclose whether mentioned examples were derived from own experiences or from the behavior of colleagues. In the beginning of each focus group, the discussion was explorative in nature. Later on, aspects of impaired work functioning derived from our literature review were validated and supplemented with illustrative examples. The moderator ensured that for each aspect of impaired work functioning mentioned, the different occupations and specialties present gave concrete examples. The moderator explicitly asked for differences in experiences between the various occupational groups present. Also, the moderator asked to clarify any ambiguities in the examples of participants.

Each focus group discussion was audio taped. The Medical Ethics Committee of the Academic Medical Center Amsterdam decided that approval of the research protocol by the committee was not required.

Subjects of the preparation phase: Focus group members were recruited from one academic medical center using a purposive sampling procedure, with variation in wards and occupations as a major criterion. Nurses and allied health professionals for the three employee focus groups were invited via head nurses. For the selection of participants in the focus groups, we asked for a mix between healthy participants and participants with current or past mental health complaints. We assumed that every employee can deliver input on the research question either based on own experiences with mental health complaints in the presence or past or otherwise based on observations in co-workers with mental health complaints. Employees from the same ward were assigned to different focus groups. Information was collected about the participants’ history of mental health complaints. Of the 19 participants, 16 had experienced a difficult period in life with effects on their mental health in the past and three currently experienced problems. Nine participants had (mild) mental health complaints in the past and one currently had.

Participants for the expert focus groups, such as senior nurses and occupational physicians, were personally invited. Informed consent was obtained from each participant, and all participants were compensated with a 25 Euro voucher for their 2-h participation.

Analysis of the preparation phase: Audiotapes of the focus groups were transcribed verbatim. The analysis of the focus group interviews followed a purpose-driven approach, aiming to distinguish as many different signals of impaired work functioning as possible and to organize all signals into themes (Krueger and Casey 2000). First, each interview was open coded. In this inductive step, all examples of impairments in the work functioning were indexed. During the coding procedure, we aimed to be as inclusive as possible. Therefore, in case of inconsistencies between codes, no exclusion or broadening of codes was performed but inconsistent codes were preserved. Second, codes were refined and reduced within a process of re-reading and constant comparison (Pope et al. 2000). Third, the obtained codes were categorized into themes covering related aspects of work functioning. One researcher (FG) performed the coding of the data; subsequently, a second researcher (KN) checked the coded data of each interview. For the analysis of the literature review, see Gärtner et al. (2010).

Item generation phase

Procedure of the item generation phase: In the second phase, items were formulated based on the results of the literature search and focus groups. For each theme that resulted from the preparation phase, sufficient items for possible subscales were formulated (minimum of seven). Each item had to refer to a clear, concrete single action or behavior. To connect with the actual behavior and perception of nurses and allied health professionals, item formulation had to reflect expressions from focus group participants as much as possible. Where possible, items had to be applicable for the different tasks and jargons of the various occupations and specialties as well. A four-week timeframe was chosen for all items. Response formats were chosen according to the content of the associated themes with a minimum of five and maximum of seven categories (Streiner and Norman 2008). Response scales utilized were the following: Likert-type scales (from 0 = totally disagree to 6 = totally agree), an adjectival scale asking for difficulty (from 0 = no difficulty to 6 = great difficulty), relative frequency categories (from 0 = almost never to 6 = almost always), or absolute frequency categories from (0 = not once to 6 = on average more than 1x per day).

Analysis of the item generation phase: The results of the focus groups together with the information derived from the literature reviews were synthesized into themes and all signals of impaired work functioning were translated into items. These were discussed several times by all of the authors, which resulted in the first pool of items. In this phase, we adhered to the principle of being as inclusive as possible (Terwee et al. 2007).

Revision phase

Procedure of the revision phase: As part of the revision phase, the first pool of items was submitted for an expert check. Six experts (head nurses and occupational health professionals) were asked to identify items that were unclear or irrelevant. They were asked to rate the relevance of each theme and the completeness of the questionnaire as a whole on a 5-point Likert scale ranging from 1 = not at all relevant/complete to 5 = highly relevant/complete. On item level, the relevance was rated on a 2-point scale (yes, no). In addition, participants were invited to suggest supplementary themes and items.

Subsequently, verbal probe interviews were conducted with six nurses and allied health professionals who reviewed the individual items in a 1-hour interview (Willis 2005). Participants were asked to identify any item that was unclearly formulated, difficult to respond to, or not applicable to all nursing wards and allied health professions. Additionally, the preference for response formats was discussed.

Subjects of the revision phase: For the expert checks, six key persons (head nurses and occupational health professionals) were invited. For the verbal probe interviews, six nurses and allied health professionals were invited personally. The sampling in this phase was again purposive and we aimed to have as many different professions represented, e.g., also (head) nurses form anesthetic and surgical nursing wards and allied health professionals. The experts, nurses, and allied health professionals invited were partly already participated in the focus group interviews and partly were newly recruited.

Analysis of the revision phase: Possible changes in the item pool resulting from the expert checks and verbal probe interviews were proposed by one researcher (FG) and discussed by the research team until consensus was reached. Items and response categories that were reworded where when possible checked in subsequent interviews. Expert comments on missing signals of impaired work functioning led to the formulation of additional items. In order to draw conclusions on the content validity, the quantitative results about the relevance and clarity of themes and items were summarized by frequencies of the given answers.

Methods part 2: item reduction and subscale generation

Procedure part 2

The second part of our study has a cross-sectional design. Respondents were contacted by e-mail and asked to fill out an electronic version of the item pool, which took approximately 45 min for completion on a computer. It was possible to log out half way through the survey and to continue after logging in again later on. However, the questionnaire had to be fully completed within 3 days. It was not possible to skip questions. Two reminders to complete the questionnaire were sent by e-mail. For each completed questionnaire, we donated 2.50 Euro to a charity that the respondents could select from among three options.

Subjects part 2

A random sample of 1,200 nurses and allied health professionals in one Dutch academic medical center was taken, as we expected a response rate of 25% and strived to recruit 300 respondents. This sample was stratified by age, gender, and occupation.

Information was collected about the participant’s gender, age, and the history of their mental health complaints. Mental health status was measured using two questionnaires. First, the General Health Questionnaire (GHQ-12) was used, a 12-item self-report questionnaire developed to detect common mental disorders in the general population (Goldberg et al. 1988). Following earlier studies in the working populations, a cut-off point of ≥4 was applied to identify individuals reporting sufficient psychological distress to be classified as probable cases of minor psychiatric disorder (Bultmann et al. 2002). Second, the 16-item distress subscale of the Four-Dimensional Symptoms Questionnaire (4DSQ) was used (Terluin 1998; Terluin et al. 2006). For case identification, a cut-off point of ≥11 was applied (van Rhenen et al. 2008).

Analysis part 2

A first reduction in items was based on the variation in answers. In the case of minimal variation (≥95% of answers given in one response category), exclusion of the item was discussed in the research team (Streiner and Norman 2008). Further reduction in items and determination of the underlying factors were based on explorative factor analysis with an orthogonal rotation approach, using principal component analysis (PCA) and Varimax Rotation (Stevens 2002; Tabachnick and Fidell 2001). To determine the optimum number of factors, we considered Catell’s screetest (1966). Kaiser’s criterion (retain factors with Eigenvalue >1) (Kaiser 1960), and parallel analysis, following the criterion that the PCA Eigenvalue of our dataset had to exceed the mean Eigenvalue of 100 random datasets with the same number of items and sample size (Horn 1965). In cases where these methods led to different numbers of components, we preferred the most interpretable component structure, with the least number of components.

Subsequently, we performed a sequence of PCA Varimax rotations and the analysis of internal consistency (using Cronbach’s alpha coefficients), to give meaning to the selected factors, to distribute items to the factors, and to further reduce the number of items (Ruiz et al. 2008; Stevens 2002). Items were assigned to a factor if their factor loading was 0.40 or greater (Stevens 2002). In case of cross-loadings, they were assigned to the factor with highest factor loading. The selection of items forming the definite subscale was based on the following considerations:

The content of the items: selected items should clearly represent the subconstruct with as many different facets as possible.

Factor loading: items with higher factor loadings were preferred.

Cronbach’s alpha: items with highest contribution to the scale’s overall alpha were proposed for selection.

The analyses were repeated after each deletion of items until the unidimensional structure of each subscale was stable without further improvement in the alpha coefficient. A Cronbach’s alpha of at least 0.70 was regarded sufficient and above 0.80 as good (Nunnally 1978; Streiner and Norman 2008).

Since the item pool was too large (231 items) to analyze in one PCA, we analyzed four clusters of themes that are related to each other from a theoretical point of view. This division is in line with existing models of job performance (Viswevaran and Ones 2000). Our first cluster, “cognitive aspects of work functioning”, corresponds with the idea of task performance. The second cluster, “causing incidents”, corresponds with counterproductive behavior, although we do not regard causing incidents as voluntary, which is part of the definition of counterproductive behavior. Our third cluster, “interpersonal behavior”, and fourth cluster, “energy and motivation”, are in accordance with organizational performance and the extra effort needed to perform the work, respectively. See Table 2 for the allocation of themes to the clusters.

Table 2

Participant characteristics (N = 314)

Demographic characteristics
Gender [N (%)]
Female	257 (81.2)
Male	57 (18.2)
Age in years [mean (SD)]	44.5 (12.0)
Marital status [N (%)]
Married/living together with a partner	227 (72.3)
Being in a relationship	21 (6.7)
Single	54 (17.2)
Divorced	11 (3.5)
Widow/widower	1 (0.3)
Ethnical background [N (%)]
Dutch	261 (83.1)
Immigrant first generation	35 (11.1)
Immigrant second generation	18 (5.7)
Occupation [N (%)]
Nurse	220 (70.1)
Surgical nurse	23 (7.3)
Anesthetic nurse	13 (4.1)
Allied health professional	58 (18.5)
Working experience in years [mean (SD)]	20.8 (12.2)
Kind of contract [N (%)]
Permanent position	301 (95.9)
Fixed-term contract	9 (2.9)
Temporary employment	4 (1.3)
Work hours per week [mean (SD)]	30 (6.3)
Mental health complaints	83 (26)

Finally, to test whether the selected subscale structure remained stable, a confirmatory factor analysis with all remaining items from all clusters was carried out, using the Oblique Multiple Group Method (Stuive et al. 2008; Stuive et al. 2009). Based on the highest item test correlations for each item on each subscale, it can be determined for which subscale the individual items have the best fit. Possible incorrect assignments of items to subtests were corrected in this step.

All statistical analyses were performed using SPSS version 16.0, except for the Parallel Analysis, which was conducted using Monte Carlo PCA for Parallel Analysis (Watkins 2006).

Results

Results part 1: development of the item pool

The literature reviews together with the five focus groups initially yielded 13 themes of impaired work functioning with underlying items. The themes resulting from the systematic literature review and the focus groups overlapped to a large extent. However, the focus group data provided more detailed themes on task execution and comprehensive examples of behavior for all themes.

The focus groups further revealed that reflection on and insight in ones own behavior and insufficiencies at work is difficult for employees with mental health complaints. As a result, we formulated the proposed items as concrete examples of behavior, whenever possible. Also, where appropriate, we chose for response formats with frequency categories, to help respondents rely on memories of past behavior rather than on an interpretation of it.

The 13 themes with underlying items were evaluated in the revision phase by the expert checks. For the relevance element of content validity, themes were evaluated by means of a 5-point scale question. For nine themes, 100% of the scores were given in categories either 4 or 5. For the other four themes, at least a two-thirds majority of answers were in categories 4 or 5 (see Table 1). Thus, all themes were assessed from relevant to highly relevant. The second element of content validity, comprehensive representation of the targeted construct, was evaluated by means of a 5-point scale question. Two of the six given answers were in category 4, and the other four answers were in category 5. Therefore, the representativeness of the item pool was assessed as highly comprehensive. One new theme derived from the expert checks was “execute work less independently”, under which relevant items from other themes or newly formulated items were organized.

Table 1

The generated themes grouped by four clusters to be used in the principal component analysis (PCA)

Cluster (number of items in item pool)	Themes (number of items in item pool)	Behavior described by the items of the theme	Relevancy [frequencies of given answers (five response categories: 1 = ‘not at all relevant’ to 5 = ‘extremely relevant’) (experts: n = 6)]
Task execution (91)	Attention and concentration problems (14)	Concentration, focusing on tasks, being able to multitask, working precise and accurate	4 (n = 2) 5 (n = 4)
	Losing the overview (17)	Being able to prioritize in tasks, thinking ahead, having the overview of all work tasks, being able to anticipate to unforeseen situations	4 (n = 3) 5 (n = 3)
	Impaired memory (8)	Forgetting important things at work, control behavior due to doubts about which tasks are done yet	3 (n = 1) 4 (n = 1) 5 (n = 4)
	Impaired decision making (12)	Not being able to make decisions themselves, hand decision-making tasks over to coworkers, lack of trust in the quality of own decisions at work	4 (n = 2) 5 (n = 4)
	Working less efficient (11)	Not being able to complete work in time, needing more time to finish tasks, being less productive at work	3 (n = 1) 4 (n = 3) 5 (n = 2)
	Limiting one’s tasks (23)	Avoiding busy shifts or difficult patients, giving up special tasks like supervision tasks, the absence of extra role behavior like helping co-workers	2 (n = 1) 3 (n = 1) 4 (n = 3) 5 (n = 1)
	Execute work less independently (6)	Being in need for help from co-workers to ask questions, to finish work, or to avert incidents	Added after revision phase
Causing incidents (46)	Causing incidents (46)	Causing incidents in general as well as different sorts of incidents, i.e., medication administration, documentation, communication	4 (n = 2) 5 (n = 4)
Inter-personal behavior (65)	Contact with patients and their relatives (26)	Speaking in an inappropriate tone to patients or relatives, being impatient, having lack of empathy, avoiding difficult or emotional situations with patients, not being able to prevent conflicts with patients or relatives	2 (n = 1) 4 (n = 4) 5 (n = 1)
	Aggressive behavior (11)	Rough treatment of patients and co-workers, blaming patients for unsuccessful care	4 (n = 3) 5 (n = 3)
	Impaired contact with colleagues and supervisors (19)	Avoidance of contact with co-workers, becoming irritated and angry about organisational issues, conflicts with co-workers	4 (n = 1) 5 (n = 5)
	Avoid work and colleagues while on the job (9)	Avoidance of talks, contact and collaboration with co-workers and supervisors, withdrawal from common rooms to be alone	4 (n = 5) 5 (n = 1)
Experience of work and emotions at work (29)	Experience work to be more demanding (8)	Having trouble managing the work load, more energy needed to execute work, feeling the need for extra days off	4 (n = 2) 5 (n = 3)
	Emotions (21)	Having feelings of losing control at work, being anxious, being short tempered, becoming emotional, being unsure about the own skills, being unmotivated	4 (n = 2) 5 (n = 4)

On the item level, the revision phase led to the addition of eight new items and the deletion of 20 original items, mainly due to overlap or ambiguity. Further comments in this phase led to re-wordings of items. One example of rephrasing was the change of the term “errors” into “incidents”, as this term more explicitly indicates the involuntary nature of these unintended actions.

After the revision phase, the item pool consisted of 14 themes with a total of 231 items. These themes were grouped into four clusters. See Table 1 for the themes and a description of the items. Figure 1 presents an overview of the results for each step of this study.

Results part 2: item reduction and subscale generation

The socio-demographic characteristics and the mental health complaints of the sample with 314 subjects are presented in Table 2. The sample is representative of the occupational groups, working in the academic medical center where our sample was recruited.

Item reduction by explorative factor analysis

As expected, all 231 items had a highly skewed distribution of answers. First, 19 items were deleted because of too little variance in answers. The data of all four clusters were suitable for the PCA. However, the PCA for the second cluster (causing incidents) had to be performed without the data of the allied health professionals, as too many “not applicable to my job” answers were given in this group, leading to too many missing values. The Kaiser–Meyer–Olkin values for the four clusters were 0.73, 0.72, 0.80, and 0.90, respectively; all exceeding the recommended value of 0.60 (Kaiser 1970, 1974). Bartlett’s test of sphericity was significant in all cases (with P < 0.0001) (Bartlet 1954). Table 3 presents an overview of PCA results and a description of the content of the items included per selected factor. In the supplemented files, we present the rotated component matrix with the factor loadings for each cluster.

Table 3

Results of the principal component analysis for all four clusters

* Number of respondents who answered all items

** Percentage of variance explained by the first factor in each subscale

*** This subscale is a selection of items from the subscale ‘causing incidents’ which are applicable to allied health professionals

The PCA of the first cluster was performed with 82 items, of which 19 remained. Based on the scree-plot and the interpretability of the factors, a three-factor solution was chosen. It accounted for 32% of the explained variance. The following subscales were identified: “cognitive aspects of task execution”, “withdrawing from responsibilities”, and “impaired decision making”.

The PCA of the second cluster was performed with 41 items, of which 15 remained. An interpretable one-factor solution was chosen based on the scree-plot, explaining 23% of the total variance. The identified subscale was “causing incidents at work”.

For the third cluster, out of 61 items, 19 remained. The scree-plot of the PCA pointed to four factors, which were highly interpretable. It accounted for 36% of the overall variance. Subscale one is “avoiding contact with colleagues” and two is “conflicts and irritations with colleagues”. Subscale three and four are “impaired contact with patients and their family”; because of their overlap in underlying content, they were combined.

In the PCA of the fourth cluster, with 28 items of which six remained, we chose the one-factor solution, based on the scree-plot and the good interpretability. It explains 35% of the variance. This subscale is called “lack of energy and motivation”.

For each cluster, a final PCA was performed with the selected items. For all clusters, the selected number of factors was corroborated.

Confirmatory factor analysis

A confirmatory factor analysis was performed on all 55 items of the eight subscales that remained after the PCA. Overall, the distribution of items into the subscales was confirmed. Some items have high scores on a subscale with which their own subscale is highly correlated. We regard these correlations as acceptable, as long as the score on its own subscale is higher or close. The results of the Oblique Multiple Group Method led to combining of two subscales, “withdrawing from responsibilities” and “avoiding contact with colleagues”, into a new subscale named “avoidance behavior”. Also, a total of four items were replaced and five were removed. In the supplemented files, we present the rotated component matrix with the factor loadings for each cluster. At the end of this study, a questionnaire with seven subscales and a total of 50 items was derived (Table 4). The internal consistency is good in four subscales (0.81–0.94) and acceptable in three subscales (0.70–0.78).

Table 4

Psychometric properties of the definite seven subscales

Subscale	# of items	N*	Cronbach’s α	Theoretical range of sum score	Range of sum score in sample (median)
Cognitive aspects of task execution and general incidents	11	308	0.94	0–100	0–82 (5)
Impaired decision making	3	310	0.88	0–100	0–100 (0)
Causing incidents at work**	8	176	0.78	0–100	0–40 (4)
Avoidance behavior	8	294	0.70	0–100	0–81 (0)
Conflicts and irritations with colleagues	7	311	0.77	0–100	0–61 (4)
Impaired contact with patients and their family	8	223	0.81	0–100	0–42 (4)
Lack of energy and motivation	5	307	0.81	0–100	0–73 (7)

* Number of respondents who answered all items, this N is used for Cronbach’s α and the range of the sum score in the sample

** Data of nurses only is analyzed

The first subscale was “cognitive aspects of task execution and general incidents”, covering eleven items on working efficiently, alertly, accurately, independently, keeping track of the tasks, and causing incidents in general. The second subscale is “impaired decision making”. This subscale encompasses three items regarding the ability to make important and quick decisions in stressful situations. The third subscale was “causing incidents at work”, consisting of the eight items covering different types of incidents: medication administration, documentation, and interpretation. This scale was not suitable for the allied health professionals, as too many of them answered “not applicable to my job” on more specific incidents items. The fourth subscale was “avoidance behavior”, which encompassed eight items about avoiding particular tasks and responsibilities as well as avoiding contact and cooperation with co-workers. The fifth subscale was “conflicts and irritations with colleagues”, its seven items described feelings of anger and irritation regarding co-workers and conflicts and tensions in the team. The sixth subscale was “impaired contact with patients and their family”, that included eight items about lack of time, patience, and empathy for patients and their family. The seventh subscale was “lack of energy and motivation”. Its five items refer to lack of motivation and experiencing work as more demanding. Table 4 presents the psychometric properties of the seven subscales. We present the definite questionnaire in the "Appendix".

Discussion

Aim of this study was to develop a job-specific detection questionnaire for impaired work functioning due to CMDs in nurses and allied health professionals. In the first part of this study, various signals of impaired work functioning due to CMDs were identified, using literature research and focus group interviews and later translated into items. These signals covered 14 themes of work functioning impairments and described concrete behavior or actions of the work of nurses and allied health professionals. In the second part, seven clear and interpretable factors were distinguished by factor analysis, grouping 50 items of the original 231 items in the item pool. Four of the seven subscales have good alpha’s (above 0.80), three have acceptable alpha’s (above 0.70). Based on the evaluations from the expert check and verbal probe interviews, we conclude that the content validity of our instrument is high. The newly developed questionnaire is called the Nurses Work Functioning Questionnaire (NWFQ).

The development of the questionnaire followed a clear step-by-step procedure, planned in advance. In the development process, we used literature as well as qualitative data presenting knowledge and experiences of employees and experts as input sources. Furthermore, in the quality assessment of possible items and the choice of definite items and subscales, both expert opinions and statistical analyses were used. In conclusion, the procedure employed exemplifies the requirements for the development of a scientific questionnaire that is relevant for practice (Haynes et al. 1995; Terwee et al. 2007).

The focus group interviews were applied as one step in this development study. Using a purposive sampling strategy, the focus group data include experiences from diverse nursing specialisms and experts’ professions. Therefore, we assume that the focus group results are applicable to the whole spectrum of the work of nurses and allied health professionals. This comprehensive approach is an important aspect of quality in qualitative research methods.

Unlike existing work functioning scales, the NWFQ aims to be job-specific. It comprises aspects of work functioning that are not, or are to a lesser extent, included in generic work functioning instruments. One specific aspect is “causing incidents”. In healthcare service, incidents can have serious consequences for the health of patients as well as for the health of the workers. Therefore, detecting a high risk of incidents is indispensable when assessing (impaired) work functioning in nurses and allied health professionals. A second aspect, which exemplifies the value of job-specific scales, regards interpersonal behavior. Most generic questionnaires do not include this aspect of work functioning as a separate subscale, such as the Endicott Work Productivity Scale (EWPS) (Endicott and Nee 1997) or use only a few items without the distinction between co-workers and other persons, e.g., the Work Limitations Questionnaire (WLQ) (Lerner et al. 2001). The quality of communication with patients and their family forms a crucial element of the NWFQ, as this work aspect is essential in the health service sector. Not only does the job-specific approach lead to more concrete examples of behavior in the items itself, it also leads to a better coverage of the most relevant aspects of the work. Therefore, the job-specific approach used here is of additional value to similar measurement instruments that approach work functioning more generally.

Based on insights from the focus groups that reflection on ones own behavior is sometimes insufficient when suffering from mental health complaints, we aimed to formulate items that present behavior as concrete as possible. However, as the items also had to be broad enough to be applicable to the different nursing wards, some items give room for broader interpretation. For example, the item on assessing which (nursing) care a patient needs (item 30) can relate, e.g., to giving the right decubitus prophylaxis, delivering the right medication, or choosing correct patients’ transport implementation of the questionnaire should await the results of further research on its construct validity and reproducibility. Also, to draw conclusions about the detection ability of the NWFQ, results on the discriminative validity are necessary.

The multidimensionality of the instrument and the nature of the items allow for more accurate assessment of the nature of impairments in work functioning. High scores provide a starting point for purposeful interventions. Depending on the specific aspects and severity of impairments, interventions can be tailored. Interventions can be of small scale, such as paying more attention to the specific (impaired) work aspects or by a temporarily adjustment of tasks. Interventions can also be of larger scope, guided by professional counselors such as psychologists or occupational health physicians. Future research should focus on (1) the implementation of various interventions using the NWFQ and (2) the effectiveness of those interventions.

Conclusion

The Nurses Work Functioning Questionnaire (NWFQ), a 50-item multidimensional measure of impaired work functioning in nurses and allied health professionals due to CMDs, was developed. Its seven subscales, with high-content validity and good internal consistency, cover the full range of impaired work functioning of nurses and allied health professionals with CMDs. The individual subscale scores give insight into the precise aspects of impaired work functioning, allowing for tailoring of interventions for individual needs. Therefore, the NWFQ demonstrates both breadth and depth of measurement, while allowing for self-administration within a reasonable amount of time.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOC 209 kb)

Textbox: Cases used for the focus group discussion

Case1: Try to imagine yourself in the following situation: Due to conflicts at home you have not been feeling well the past weeks. You have much less energy than usual and after a long day at work you feel too exhausted to do your everyday activities and to relax. This morning you arrive at work feeling stressed already, today will be a very busy day again. Just the idea of all the work you have to do makes you tired. What difficulties do you expect to face during this workday?

Case 2: Try to imagine yourself in the following situation: Since a few months you have not been feeling very well. In the last few weeks you have been feeling especially bad. You feel depressed, there is nothing you want to do or what excites you. The only thing you feel like doing is to stay in your bed all day long. At work you sometimes feel anxious without any reason; you can’t tell where the anxiety comes from, the feelings just comes over you. In the past weeks you have had more and more difficulties to accomplish your tasks at work. Can you describe how your working day goes in these circumstances?

Case 3: Try to imagine yourself in the following situation: You have a nice team you work with, with many different people and you get along with each other very well. Since a while you have noticed that one of your colleagues behaves differently. Regularly, you have the feeling she smells of alcohol. What has changed in the behavior of your colleague?

Table 5

Instructions for sum score calculation

Subscales		Items	Calculation of standardized sum score	# of items
				Total	Minimum
1	Cognitive aspects of task execution and general incidents	1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 16	(sum of item scores * 100)/(# of items × 6)	11	9
2	Impaired decision making	48(R), 49(R), 50(R)	(sum of item scores × 100)/(# of items × 4)	3	3
3	Causing incidents at work*	14, 26, 27, 28, 29, 30, 31, 32	(sum of item scores × 100)/(# of items × 6)	8	6
4	Avoidance behavior	36, 37, 38, 39, 40, 41, 42, 43	(sum of item scores × 100)/(# of items × 4)	8	6
5	Conflicts and irritations with colleagues	33, 34, 35, 44, 45, 46, 47	(sum of item scores × 100)/(# of items × 4)	7	6
6	Impaired contact with patients and their family	10, 11, 12, 13, 22, 23, 24, 25	(sum of item scores × 100)/(# of items × 6)	8	6
7	Lack of energy and motivation	17, 18, 19, 20, 21	(sum of item scores × 100)/(# of items × 6)	5	4

Technical details

– Items followed by (R) need to be recoded before sum score is calculated

– Item score counting starts with 0 on the outer left category, add 1 point for each category further to the right (e.g., disagree = 0; disagree a little = 1; not agree/not disagree = 2; agree a little = 3; agree = 4)

– Calculation of standardized sum scores follows the principle: (sum of item scores × 100)/(# of items × maximum score per item)

– For sum scores calculation, subjects need to have filled out at least ¾ of all items of a subscale

– The range of the standardized sum score is 0–100 for each subscale

* The subscale “Causing incidents at work” is not suitable for allied health professionals