Instrument to assess the nursing care product: development and content validation.

OBJECTIVES: develop and validate the content of a tool about nursing care production.
METHOD: the data were collected between 2011 and 2013, based on focus groups, the application of semistructured questionnaires (prototype test) and the Delphi technique. The focus groups were used to produce the instrument items and held at three hospitals in the interior of the State of São Paulo, involving 20 nurses. A panel of 10 experts evaluated the instrument.
RESULTS: after two phases of the Delphi technique, the tool consisted of eight items. The content validity index of the scale corresponded to ≥0.9 and the content validity of the items ranged between 0.8 and 1.0, indicating the maintenance of the structure and content. The assertion on the applicability in daily nursing practice showed a content validity index of the scale equal to 0.8.
CONCLUSION: this study permitted the development and content validation of scale on nursing care production, equipping the nurses in their management practice.

Introduction

The nursing activities can be understood as a care production system 1. The conception of this product involves interaction among human beings, dynamic, non-linear and emerging processes, besides the ability to self-organize and adapt to the needs of the patient/family member, the team and the institution, characterized as a complex adaptive system 2.

Nurses are responsible for managing care through expressive and instrumental direct and indirect care actions. This practice implies staff dimensioning, leadership and training of the team, resource projection and provision, care coordination and execution, as well as the planning and assessment of the interventions made 3. The service ability and complexity of the services provided in hospital organizations influence the activities performed. The incorporation of quality improvement and cost control processes modifies the organization of nursing work and can limit the time spent on direct care delivery to patients(4-5). Thus, the managers have faced severe problems around the world, related to insufficient human capital, an excessive work burden, lack of qualification, absenteeism and professional evasion, which influence the quality of care and may result in care errors and/or omission 6.

To guarantee the delivery of this service and grant further visibility to the profession, an accelerated expansion of studies is ongoing that emphasizes the standardization and classification of patient-centered nursing activities, including diagnoses, interventions and desired outcomes 7. In parallel, the safety movement in health care highlights the relation between the burden of the nursing professionals and the care results obtained 8. Based on these dimensions, another research line considers the performance of this service as the ability to obtain the resources and use them sustainably to produce care and discusses aspects to measure and assess its contribution in organizations 9.

Although the increasing use of management tools in nursing services drives the researchers to develop specific, valid and reliable tools 10, however, no discussions on tools to assess the care product were found, neither in the Brazilian nor in the international literature. In that sense, this research departs from the following problem: can a tool be developed that contains activities and/or situations of nursing practice to assess the product at the end of the work shift? By exploring this knowledge, the efficiency and efficacy of the nursing activities can be measured, contributing to decision making and the improvement of processes. Thus, in this study, the aim was to develop and validate the content of a tool about nursing care production.

Method

The study was developed in three phases: production of items and development of the tool, test of the prototype and content validation.

Phase 1: production of items and development of the tool

The focus group technique was used to produce the items of the tool to assess the production of nursing care 11. This phase was undertaken at three large hospitals in the interior of the State of São Paulo, with a view to joining subjects from distinct realities. The data were collected between October 2011 and July 2012. Four focus groups were constituted involving 20 nurses and the meetings took between one hour and a half and two hours at most. Further details on this phase have been described in another study.

The discussions were conducted by one of the researchers (moderator) and guided by the following questions: What activities do you assess at the end of the workday to consider your shift was excellent? In what situations is your shift good? When do you consider your shift regular? and What was your shift like to be considered bad?

The discourse was recorded and filmed, with the participants' consent, and then transcribed and investigated using thematic content analysis 12. The theoretical framework was based on the concept of complex adaptive system 2and care production1.

To construct the prototype tool, reference frameworks on nursing care management(3,13) and

quality process in health(6,8) were used, among others. The grading was based on the measures used in the classification of nursing outcomes 14and a patient classification tool 10.

Phase 2: prototype test

Intentional sampling was used to select nine nurses to assess the prototype, representing six medical and surgical wards, one pediatric ward and two adult and pediatric intensive care services at one of the hospitals, in the different work shifts.

After they agreed to participate, the subjects received individual orientations about the composition of the prototype and its use. The participants were expected to apply it at the end of at least two shifts. The nurses received a questionnaire, which initially contained the respondent's profile and seven proposals, considering the relevance, clarity, simplicity and applicability of the prototype. To assess each proposal, a five-point Likert scale was used, ranging from one (I completely disagree) to five (I completely agree), besides space for the subjects to add other comments.

The data were collected in March 2013. Seventeen assessments were obtained, as one of the participants went on leave.

Phase 3: content validation

This type of validity indicates whether the instrument items are appropriate to measure the specific construct and whether they adequately address their domain, being necessarily judgment-based 15. Therefore, the Delphi technique was applied, which aims to transform the experts' opinions on a theme into a group consensus, through the validation of structured questionnaires, conducted in phases or cycles. Based on the answers to each questionnaire, the subsequent phase is reformulated and the process continues until a pre-established agreement is obtained among the participants 16.

To compose the expert panel (judges), 15 nurses were eligible 17, who had at least ten years of professional experience, acted as teachers, preferably as leaders of research groups in the management area, certified by the Brazilian Scientific and Technological Development Council (CNPq) and nurse managers from Brazilian hospitals. The experts were invited electronically, addressing information on the research objectives, method, ethical aspects and access link. The questionnaires were constructed online, using the software Google Drive(online panel). The data were collected between May and December 2013, totaling 10 judges.

The questionnaire contained a pretext with a short explanation letter about the study objectives, followed by the consent form. Access was only permitted after consent had been obtained. Then, demographic and professional data were collected to determine the respondents' profile.

The first part consisted of items corresponding to the nursing care production, assessed individually through eight assertions, involving: pertinence of contents, understanding among the professionals, extent and complexity of the item, clarity of the statements, grading, applicability in daily nursing practice and contribution to management decision making. The second part permitted assessing the structure of the tool (nine assertions), considering the abovementioned aspects and whether they represented the most expressive dimensions of this production. What the content of each item is concerned, the experts were asked to indicate the maintenance, addition, modification of the grades, elimination, merger or others, justifying their decision or any other considerations. The questionnaire was organized in a Likert-type format, similar to what was done in the prototype analysis.

Score five was attributed to the answers "I completely agree" and one to "I completely disagree". In case of assertions formulated as negatives (assertions 4 and 5 in the care production items; and 5 and 6 in the assessment of the tool structure), the score was inverted.

The consensus criterion in this study was previously determined, considering the content validity index of the items (IVC-I) equal to 0.8 or higher and, for the structure (scale), the content validity index of the scale (IVC-E) equal to 0.9 or higher(13,15) was considered. For each assertion, IVC-I and IVC-E were calculated by adding up the answers "4" - I agreeand "5" -I completely agree, using the total number of judges as the denominator 17. For a new phase (cycle of questions), as a form of feedback to the judges, a report was included in the questionnaire that described the results and the items that were revised, eliminated or validated, highlighting what should be reassessed.

The statistical analysis of the data corresponding to the analysis of the prototype (Phase 2) and content validation (Phase 3) was undertaken using Microsoft Office Excel 2007. The Likert scale was considered as the ordinal measuring level and median and quartiles were calculated (Q1 and Q3). The subjective data were grouped and analyzed according to the research objective.

Approval for the research was obtained from the Institutional Review Board - Process 0050/12.

In the content analysis of the discourse, obtained through the groups, four thematic categories emerged: planning, intervention and assessment of care; dimensioning and qualification of the nursing team; resources needed to deliver care and multiprofessional interaction.

Based on the categorization of the themes, subcategories were delimited, constituting items to compose the prototype: 1. nursing care planning, 2. patient/family care, 3. care needs, 4. nurse supervision and care transition, 5. interaction and multidisciplinary activity, 6. resources needed and support services, 7. problems and emergency situations, 8. staff dimensioning according to workload and 9. qualification and professional development.

Based on the guiding questions, item grades were generated (from one till four), showing the increasing intensity concerning the best nursing care product. Thus, at the end of the work shift, the nurse could assess all items at one of the four grades, considering the option that best approached the reality experienced. The individual item scores are added up and lead to a classification according to the following intervals: 9-12 points (bad), 13-21 (regular), 22-30 (good) and 31-36 (excellent).

To elaborate the score intervals, it was considered that the classification of the nursing care product remains in a given category if the minimum score for one grade is obtained on all items and up to about 40% for the higher grade 10.

Nine nurses participated in the prototype test ( Table 1), predominantly women (n=7), with a mean age of 32 years (Standard deviation sd=5.0) - ranging between 27 and 40 years, and mean professional experience of seven years (sd=3.0) - ranging between 2 and 14 years. Concerning the education, six subjects had concluded specialization courses in the activity area or related areas. The lowest percentage of agreement (50% - Md 3.0) was found for the assertion related to the applicability of the tool in daily nursing practice.

Table 1

Nurses' opinion about the prototype tool. Campinas, SP, Brazil, 2014

Assertion	%	Md (Q1-Q3)*
Covers most expressive items	89	4.0 (4.0-5.0)
Addresses pertinent factors in each item	100	4.0 (4.0-4.0)
Presents clear statements	89	4.0 (4.0-5.0)
Permits common language among the professionals	88	4.0 (4.0-5.0)
Is complex	33	2.0 (2.0-4.0)
Is very long	40	2.0 (2.0-4.0)
Can be introduced in daily nursing practice	50	3.0 (2.0-4.0)

*Md: median; Q1-Q3: quartiles

The panel consisted of 10 female experts, with a mean age of 57.7 years (sd=8.3), 33.7 years of professional experience (sd=7.2). Eight were Ph.D.'s in nursing, one held an M.Sc; and one was a hospital administration specialist. Seven nurses taught, one was a faculty member and nurse coordinator, one was a manager and one a nursing service management consultations, including different institutions in seven Brazilian cities in the South and Southeast.

The Delphi technique was applied in two phases to reach the pre-established consensus. The answers to the first questionnaire showed an agreement index in the different items (IVC-I) of nursing care production ranging between 0.6 (Md 4.0) and 1.0 (Md 4.0-5.0). Assertion 4 (extent) showed an IVC-I ranging between 0.0 (Md 2.0) and 0.4 (Md 2.0) and assertion 5 (complexity of the tool) ranged between 0.0 (Md 2.0) and 0.3 (Md 2.0), representing the lowest agreement levels ( Table 2).

Table 2

Experts' opinion on nursing care production items in the Delphi 1 phase. Campinas, SP, Brazil, 2014

*Score ranges from 1 to 5: the higher the score, the higher the agreement

†IVC-I: content validity index of items

‡Md: median

§Q1-Q3: quartiles

Although the judges mostly agreed to maintain the indicators, they suggested several modifications in the structure and content of some items. Thus, concepts were revised, terms were removed that indicated subjectivity and statements that were unclear were explained better.

The item "nurse supervision and care transition" was (re)named "monitoring and care transfer", evidencing the educational aspect of the supervision and the systematization of information when the shift is transferred. It was argued that the support services can be considered as resources. Therefore, the title of this item was altered to "resources needed to deliver care", correlating the time the team consumes to deliver care. The indicator "problems and/or emergency situations" was excluded because it is considered redundant in view of the changes made in the items "resources needed to deliver care" and "nursing staff dimensioning".

The title of the tool - classification of the nursing work process - was also changed, aiming to further clarify the research problem. The order in which the items were presented was also modified. The tool was now identified as Assessment of the Nursing Care Process ( Avaliação do Processo de Cuidar de Enfermagem- APROCE) and included eight items or indicators with revised scores.

As regards the second questionnaire (Delphi 2), the answers showed a variation in IVC-I between 0.8 (Md 4.0) and 1.0 (Md 4.0-5.0) for the different items. Agreement with assertion 4 (extent) ranged from 0.0 (Md 2.0) to 0.4 (Md 2.0), and with assertion 5 (complexity) between 0.1 (Md 2.0) and 0.3 (Md 2.0) ( Table 3).

Table 3

Experts' opinion on nursing care production items in the Delphi 2 phase. Campinas, SP, Brazil, 2014

*Score ranges from 1 to 5: the higher the score, the higher the agreement

†IVC-I: content validity index of items

‡Md: median

§Q1-Q3: quartiles

In view of the experts' contributions, the following changes were made: inversion of grade "3" by "4" in the item "nursing care planning"; exclusion of the term that indicated subjectivity in grade "4" - almost all- for the item "interaction and multidisciplinary action" and an increasing time interval was adopted in response to the requests/calls in "patient and/or family care".

As regards the structure of the instrument (scale), the IVC-E was higher than 0.9 in the first and second phase of the Delphi technique. Scores below the established level were highlighted for assertion 4, about the clarity of the statements (0.7 - Md 4.0) (phase 1) and for assertion 8, which refers to the applicability in daily nursing practice (0.8 - Md 4.0), in phases 1 and 2 ( Table 4).

Table 4

Experts' opinion on the structure of the tool in the phases Delphi 1 and 2. Campinas, SP, Brazil, 2014

Assertion*	IVC-E†	Delphi 1 Md‡	Q1-Q3§	Delphi 2	IVC-E†	Md‡	Q1-Q3§
AF1 - addresses more expressive areas	1.0	4.0	4.0-5.0	1.0	4.0	4.0-4.7
AF2 - item content	1.0	4.0	4.0-5.0	1.0	4.0	4.0-5.0
AF3 - shared language among professionals	0.9	4.0	4.0-4.0	1.0	4.0	4.0-4.7
AF4 - clear statements	0.7	4.0	2.5-4.0	1.0	4.0	4.0-4.0
AF5 - very long	0.4	2.0	2.0-4.0	0.1	2.0	2.0-20
AF6 - complex	0.4	2.0	2.0-4.0	0.2	2.0	2.0-2.0
AF7 - increasing intensity in grades	1.0	4.5	4.0-5.0	0.9	4.0	4.0-4.7
AF8 - can be introduced in daily practice	0.8	4.0	4.0-4.7	0.8	4.0	4.0-4.0
AF9 - permits management decision	0.9	4.0	4.0-4.7	1.0	5.0	4.0-5.0

*Score ranges from 1 to 5: the higher the score, the higher the agreement

†IVC-I: content validity index of items

‡Md: median

§Q1-Q3: quartiles

The title of the tool was again changed to Assessment of Care Product in Nursing (APROCENF), in view of the experts' validation, so as to further approach the theoretical framework adopted in the research. And, in this second phase, they indicated the maintenance of the tool structure and content and suggested applying the scale at regular instead of daily intervals. Thus, in its final form, the tool consisted of eight items (described earlier), with scores ranging between 8 and 32 points.

Discussion

In this study, the development and content validation of a tool to assess the product of care in nursing is described.

To build a measure, items need to be produced that are relevant to the research question and target public. Besides the literature review, interviews are the most used source to obtain qualitative data. In particular terms, the focus group allows the participants

to explore what they have in common instead of

presenting individual viewpoints 18. Therefore, the use of these groups at different institutions allowed for the identification of several aspects that influence the care production and are common in the different realities the nurses experience.

These aspects were analyzed from the complex system perspective, centered on the best nursing practices(2,19). The items produced were ordered in view of the application of this science to the traditional management concepts: planning, organization, direction, coordination and control 20. The understanding of the factors inherent in the care system expands the managers' view on the processes, which permits forecasting their behavior and determining measures to mitigate and/or solve problems(20-21).

A study demonstrates that the nursing work environment and actions affect the patients' perception of care quality. The nurses' clinical competency, the relations among professionals, the appropriate composition of the team, a more autonomous practice with management support, besides patient-centered care, can contribute to a more positive experience of this care consumer 22.

The grades of each item (from one to four), indicating an increasing level of intensity in terms of the best care product, represented the main challenge in the development of this tool. The discourse obtained through the guiding questions supported these grades, but the measures to classify nursing outcomes 14, initially used in this construction, did not delimit the scale answers objective and clearly. Therefore, the most important changes in its content were made in the first phase of the Delphi technique.

To male the tool more objective, some values were mentioned in each items to better understand and distinguish its grades. No studies were found that evidenced the consumption of the nursing team's time to provide for the necessary resources and, hence, estimates values were used, related to the distribution of nursing time and to associated activities 23.

Similarly, the nursing team's response time to the requests/calls, corresponding to the item "patient/ family care", was based on international literature on the assessment on nursing care not delivered/delayed 24.

As a possibility to approach theory and care practice, the preliminary assessment of the prototype tool was chosen, involving clinical nurses who are directly active in patient care. The pretest evidenced the users' understanding of the items produced and appointed no substantial changes in the format or content. Nevertheless, the feasibility of daily applications represents a reflection on the researchers' initial proposal, as it supports the experts' considerations (IVC-E corresponding to 0.8 in Delphi 1 and 2) and represents an important aspect to be tested.

The daily completion of the tool would help the clinical nurse to assess the product delivered at the end of the work shift. Nevertheless, some experts considered the scale too long to be included in the work routine and suggested its periodical application, at regular intervals. Thus, the analysis of each item would be based on direct observation (supervision), on the survey of data through care records and interviews with patients/relatives and professionals.

To quantify the content validity of this tool, the IVC was chosen as the index of inter-rater agreement or estimated consensus because of its easy understanding and because of the distinction among the information on the items (IVC-I) and the scale (IVC-E). Thus, each item could be highlighted and refined and the scale could be made more accurate(15,17).

The results demonstrated the validity of the tool's content, but should be further analyzed to assess the other psychometric properties. Further research is also needed to investigate the different possible applications so as not to consume much nursing time to complete it and produce results for inter and extra-institutional comparison.

Conclusion

This study permitted a better understanding of nursing care production from the perspective of nurses working at hospital institutions and permitted developing and validating the content of a tool to measure this

product. Its application permits both the identification of the main dimensions involved in care and the measuring of their influence, contributing to management decision making and effective care management.