Analysis of Two Diverse Nursing Records Applications: Mixed Methods Approach.

Introduction: Poor adoption of electronic health records among healthcare workers can diminish their impact. Healthcare informatics solutions development should diligently acknowledge end-user needs. This study compares a user experience and perceived quality of the nursing process integration in two different applications for electronic documentation of the nursing care plan. Both applications were designed and tested in Slovenia.
Methods: In the first phase, final year undergraduate nursing students were recruited (n=73) and randomly assigned into two groups. Each group used one of the applications for a duration of five hours. A survey among students was conducted. In the second phase, additional students were recruited (n=40) and invited to participate in qualitative analysis of the unfavourably rated application.
Results: The modern, visually improved application was favourably rated by students in terms of all aspects of application usability. However, students reported a significant number of inadequacies regarding the nursing process methodology integration. On the contrary, the students using the poorly rated and visually outdated application reported no such concerns. Qualitative analysis of student reflections identified additional positive features of software design that were not detected in survey results analysis.
Conclusion: This study showed that a user-centred approach can be used to compare diverse electronic solutions. Detected discrepancies in findings using qualitative and quantitative analysis show the importance of integrating diverse research approaches for adequate evaluation of software solutions. Furthermore, this study design promotes empowerment of healthcare workers to participate in the development and critical evaluations of software solutions.

Introduction

The first healthcare computer-based information systems were used mainly for physician order entries, test results reporting, financial management and pharmaceutical and radiological activities support (1). Lately, healthcare organisations have been adopting information systems capable of storing and managing all clinical practice data. The integration of modern information and communication technologies (ICT) in patient care and the use of electronic health records (EHR) for health data management have shown some positive impacts on healthcare quality and patient outcomes (2, 3, 4). Nevertheless, ICT design should be adequately adapted for the specific clinical environment. In a study of 3900 hospitals in the United States, ICT implementation showed no positive effects on mortality, cost savings, adverse drug events or readmission rates. The reasons for this were attributed to a lack of organisational changes, poor adoption and poor utilisation among healthcare professionals (5). Efficient strategies for the implementation of user-friendly ICT solutions are crucial and should consider healthcare workers’ needs, to reduce their resistance to adoption and ensure optimal staff support for the best clinical results (6).

The nursing process is a fundamental method for planning and providing integrated nursing care, enabling nurses to identify patients’ needs and plan nursing care for those needs. The nursing process is cyclical and consists of four to six phases according to different sources. Our study used the five-phase nursing process in the theoretical background consisting of assessing, diagnosis, planning, implementing and evaluating (7). This definition of the nursing process establishes the nursing diagnosis as an individual phase.

The organisation NANDA International, Inc. (NANDA-I) defines the nursing diagnosis as a clinical judgment about individual, family, or community responses to actual or potential health problems or life processes, forming the basis of nursing care planning (8). The entirety of the nursing process is reflected in nursing documentation as documented nursing health history, nursing diagnosis, nursing care plan, nursing-related outcomes and planned interventions (6, 7).

Electronic nursing record (ENR) systems are designed for electronic documentation of nursing-specific data and should enable documentation of nursing care plans according to the nursing process. The nursing process contributes significantly to nurses’ participation in patients’ treatment, but in Slovenia, it is often limited to education and theory and is not adequately represented in contemporary clinical nursing documentation (9). It should be fully implemented in contemporary patient care and ENR systems (10), enabling an efficient implementation of the nursing process in clinical practice to improve the quality of healthcare, improve nursing documentation and support holistic interdisciplinary patient care (11).

New technology acceptance can be enhanced if a user-centred approach is used in product development and evaluation (12). A user-centred approach has been used before, to evaluate the perceived usability of diverse health-related electronic systems’ functionalities and user satisfaction (13, 14). It could also be employed to assess and improve the adequacy of nursing process methodology integration and usability of diverse ENR systems. An extensive review of software usability evaluation studies reported that the most commonly used methods are usability testing, heuristic evaluation and questionnaire methods, while also highlighting the importance and lack of studies using both qualitative and quantitative methods for evaluation of usability (15).

In this study, qualitative and quantitative methods were used to conduct a user-centred analysis of two Slovenian electronic documentation applications that differed in basic functionalities, graphical interface and nursing process incorporation. Our primary objectives were to analyse the opinions and attitudes of end-users regarding the usability of two systems and the ability of those systems to support the nursing process. Our secondary objectives were to generate a list of key system features to support usability and to demonstrate selected methods of evaluating systems.

Methods

Quantitative and qualitative methods were used to conduct a user-centred analysis of two diverse applications for documentation of nursing practice. To achieve an in-depth analysis of results an explanatory sequential mixed method was used.

Research participants

All students in the last, sixth, semester of an undergraduate nursing programme at the University of Ljubljana, Faculty of Health Sciences were recruited (n=73) and randomly assigned into two groups by a coin toss. Among participating students, 61 of them submitted a completed survey after interacting with the evaluated application. Of 61 students, 53 were women and 8 were men, most of them using ICT every day (n=58; 95%). On computer-related skills, 56% of students rated their skills as “competent”, 26% as “advanced” and 18% as “beginner”. Most students had never used software designed for documentation of healthcare-related data (n=50; 82%). Among the students that completed the survey, 26 students used application B and 35 students used application A. No significant differences in demographic data were detected between the groups. A qualitative analysis of the unfavourably rated application followed. For qualitative analysis, an additional 40 students, who did not participate in previous study procedures, were invited. Among them, 26 students agreed to participate in the study, with 25 of the submitted student reflections being suitable for analysis.

Data collection

Quantitative data for the conducting of our study were gathered using a survey. Each group used one of the applications during the study for a total of five hours. First, the lecturer (one of the researchers) guided the students through the creation of a uniform nursing care plan to provide basic knowledge concerning the functionalities of the application used. Students then designed a nursing care plan by themselves, for one patient they had encountered during clinical placement in community settings, and prepared a semi-structured written case report.

After completion, an on-site anonymised online survey among the students was conducted, regarding the user experience and the perceived suitability for quality nursing care plan documentation. The questionnaire contained 47 statements regarding seven different aspects of the application use. Participants rated statements regarding the application’s speed, visual design, overall impression and learnability on a nine-point semantic differential table. The sets of statements and their design were adopted from a questionnaire previously already used in a national research project (16) for evaluating ICT user experience. The statements regarding suitability for patient data documentation, suitability for nursing care plan formulation and satisfaction with the formulated nursing care plan were specific for the used applications and were designed for this study. The students were asked to express their agreement with statements using a five-point Likert scale (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree). The questionnaire also included demographic questions and two open-ended questions, in which students could express up to three best and three worst attributes of the application. Beforehand the questionnaire was evaluated by an interdisciplinary expert group and pilot tested with 11 students. Statistical analysis of the pilot study was not carried out. Following the students’ interaction with the questionnaire, a short focus group was undertaken. Students reported fatigue when evaluating a large number of statements on a single page of the questionnaire. The questionnaire was restructured so that a lower number of statements were presented per page of the questionnaire. For an in-depth understanding of the students’ attitudes, further qualitative analysis of the unfavourably rated application was undertaken. Students were provided with written key information regarding the application and a demonstrational video featuring an experienced user formulating a nursing care plan. The students were then asked to submit their opinions related to the application observed in the video in the form of a reflection. Participants were instructed to write a holistic evaluation and were encouraged to be honest and critical regarding the observed application.

Data processing

The software IBM SPSS Statistics 23 was used for data analysis. Data were not distributed normally according to the Shapiro-Wilk test; therefore in the results section we report the median (M) and the first and third quartile values (Q1–Q3) for representation of data, and we used the Mann-Whitney U test for comparison of the included applications. Before aggregating the data, the internal consistency of the individual cluster of content-related statements was estimated using Cronbach’s alpha coefficient. Statistical significance was acknowledged for p values lower than 0.05. Responses to open-ended questions were analysed with a deductive qualitative approach using appropriate predefined code categories (17), which were defined according to the application usability aspects included in previous items of the questionnaire. For qualitative data analysis of students’ written reflections, thematic analysis of a text was used (18). All suitable student reflections were analysed. The saturation of data was reached with 19 units.

Evaluated applications

Two applications were compared in this study: e-Documentation for nursing care and Clinical Information System, referred to as applications A and B respectively. In this study, the versions of the applications used were designed exclusively for educational purposes and contained no actual clinical data, but were identical to their clinical practice counterparts in design and functionalities.

Application A is an ENR system developed as a joint effort of academic and clinical experts. Under development since 2002, it presents an information model for organisation and management of patient data exclusively for nursing care (9). It aims to provide a professionally suitable ICT model to promote nursing quality (19, 20). The students formulated the nursing care plan by documenting the patient’s needs according to an adapted Henderson theoretical model (21), and used the problem aetiology and signs/symptoms principle for nursing diagnosis formulation (7). All subsequent elements of the nursing care plan were manually typed into predefined dialogue boxes.

Application B was designed as a complete EHR system for healthcare and administrative data management in a hospital setting. Deployed for clinical use in 2011, it aims to minimise documentation time, make data management more efficient and prevent adverse drug events (22). The students formulated the nursing care plan by selecting desired statements from a list of NANDA International, Inc. (NANDA I) standardised nursing diagnoses organised according to Gordon’s functional health patterns (23). They then selected appropriate aetiological factors, nursing-related outcomes and interventions from lists, tailored according to the selected nursing diagnosis. Nursing interventions, nursing-related outcomes and aetiological factors are preformulated but not linked with standardised nursing language.

Consent to conduct research

All participating students were informed beforehand about the nature of the study, participated voluntarily and were able to withdraw from the study at any point. This research project involved human participants, therefore approval from an Ethical Committee for Research was obtained beforehand.

Results

Of 61 students, 53 were women and 8 were men, most of them using ICT every day (n=58; 95%). On computer-related skills, 56% of students rated their skills as “competent”, 26% as “advanced” and 18% as “beginner”. Most students had never used software designed for documentation of healthcare-related data (n=50; 82%). Among the students that completed the survey, 26 students used application B and 35 students used application A. No significant differences in demographic data were detected between the groups.

The user experience and perceived quality of the nursing process methodology integration were addressed in 47 statements rated by students. The aggregated results for the observed aspects of each application’s use are presented in Table 1.

Table 1

Comparison of survey results for the observed aspects of application use.

Application usability aspect	Cronbach’s alpha	Application	M (Q1–Q3)	Mann-Whitney U
Overall impressiona	0.942	AB	4.0 (3.0–5.0)7.0 (5.0–8.0)	p<0.001
Learnabilitya	0.943	AB	5.0 (2.5–6.0)7.0 (5.0–8.0)	p=0.002
Operabilitya	0.849	AB	5.0 (3.0–5.5)6.8 (5.5–8.0)	p<0.001
Visual designa	0.906	AB	5.0 (3.8–6.5)7.0 (6.0–9.0)	p<0.001
Usability for patient data documentationb	0.938	AB	3.0 (2.8–4.0)4.0 (4.0–4.5)	p=0.002
Usability for nursing care plan formulationb	0.950	AB	3.0 (2.6–4.0)4.0 (4.0–5.0)	p=0.001
Satisfaction with formulated nursing care planb	0.948	AB	3.0 (3.0–4.0)4.0 (3.5–4.0)	p=0.015

Statements regarding the overall impression, learnability, operability and visual design were rated on a nine-point differential table.

Statements regarding usability for patient data documentation, usability for nursing care plan formulation and satisfaction with the formulated nursing care plan were rated on a five-point Likert scale. A higher score indicates greater favourability towards that aspect of application use. M – Median, Q1–Q3 – interquartile range.

Although application B was rated favourably in all aspects of application usability, students’ answers to open-ended questions showed different results. Participants were asked to identify the three best and three worst attributes of the application used. Students reported 258 attributes: 136 concerned application B and 122 application A. Figure 1 represents the number of reported positive and negative attributes for both applications, thematically categorised according to the questionnaire-based aspects of application usability.

Figure 1

Quantitative representation of reported negative and positive attributes for the observed applications.

The most frequently reported positive attributes of application B were the transparent overview of data and documentation speed, while the most frequent negative attributes were inappropriateness for older staff. Regarding the nursing care plan formulation, the students reported positive attributes that included promotion of holistic nursing care plan formulation, integration of preformed listed terms and simplicity of the formulation process. Reported negative attributes regarding the nursing care plan formulation included difficulty in navigating listed terms and scarce selection of listed terms. Additionally, participants reported some professional concerns regarding application B, which included inadequate individualization of a nursing care plan, inappropriate listings of preformed terms and inadequate patient data security.

The most frequently reported positive attributes of application A were nursing needs assessment transparency, while the most frequent negative attributes were documentation speed, outdated visual design and inability to correct documentation errors. Regarding the nursing care plan formulation, the students reported positive attributes that included integration of the nursing care process, promotion of holistic nursing care plan formulation and support of individualized formulation of nursing diagnoses. Only one negative attribute regarding the nursing care plan formulation process in application A was detected. A student reported difficulties formulating adequate nursing diagnosis using the problem, aetiology and signs/symptoms principle.

Unfavourably rated application A was included in additional in-depth analysis. In a qualitative analysis of students’ reflections, we used open coding thematic analysis of text to formulate codes, which were then grouped into eight code categories and three themes, as shown in Table 2.

Table 2

Detected themes codes categories and codes in qualitative analysis of submitted student opinions.

Theme	Category	Codes
Professional suitability	Nursing theory compatibility	Follows the nursing process, Holistic, Universal, Supports nursing theory, Secure, Relatable to existing paper-based documentation
	Flexibility	Flexible for enabling free text, Flexible for allowing limited data documentation, Diverse nursing diagnosis creation, Individually fitted nursing care plan, Efficient compared to paper-based documentation,
	Structured data representation	Clear data organization, Colour coded problem documentation, Easy to use, Inhibits incompleteness of documented data, Supports patient progress tracking, Practical graphical representations, Clear patient data overview
Functionalities and design	Non-user-friendly functionalities	Does not automatically save data, Does not flag logical inconsistencies, Does not provide pre-structured elements of a nursing care plan, Unintuitive to use, Poor standardized languages integration, Poor overview of medical data
	Poor visual design	Monotonous colour scheme, Outdated visual design, Sharp edges, Small buttons, Small font
	Time-consuming	Does not provide pre-structured elements of a nursing care plan, Unintuitive to use, Overuse of manual typing
Improvements recommendations	Functionalities enhancements	Include images documentation, Improve interdisciplinary use, Formulate data to enable research, Include risk screening tools, Include notifications and warnings, Include standardized nursing languages
	Ensure user training	Provide nursing process training, Provide user training

The students were dissatisfied with the visual design and some functionalities of the application. Regarding visual design, the colour scheme seems to play an important role, as one of the students wrote:

“The programme would benefit from some aesthetic improvements since the outlook should feature livelier colours as opposed to the current grey design. I would recommend for example light blue, white or light green. Something unobtrusive but friendly to the user’s eyes.”

Besides poor visual design, the reported limitations of the application’s functionalities were focused on time consumption and the amount of manual typing required for documenting patient data. Both those aspects can be summed up by a student that wrote:

“In the majority of cases documentation of data requires manual input of nursing care plan elements, which takes the healthcare worker an enormous amount of time.”

Additionally, the qualitative analysis provided some information regarding the positive aspects of application A. In students’ opinions, the nursing process is suitably integrated into the inner workings of the applications, as two students wrote:

“The application does a good job of guiding the user through the stages of nursing documentation. All the parts of the documentation (anamnesis, needs evaluation, goals and so on) are clearly designed and the user is able to easily switch between different parts of the application.” “In my opinion, the application is easy to use since it is clearly structured, understandable and guides you through the nursing care process.”

Additionally, participants reported some professional concerns. Concerning reported inadequacies of the application, the students suggested a stricter use of standardized nursing languages. In their opinion, the use of preformed statements and standardized languages such as NANDA-I is beneficial for efficient documentation and promotion of quality care. As examples of this, students wrote that the use of preformed statements could decrease the time needed for documentation and improve the quality of documented data.

“What I believe is even more important is the implementation of NANDA diagnosis into the application, which would additionally ease users’ workload and save time.”

“We should consider abolishing the option of nursing diagnosis creation according to the problem, aetiology and symptoms/signs principle. Manual creation of nursing diagnosis is not suitable for in-depth statistical analysis and objective patient condition evaluation, since there is no universal way of interpreting data.”

As seen above the students recognized standardized nursing languages to be important in ensuring suitable formulation of patient data for research purposes.

Discussion

The study participants comprised a relatively homogeneous student group that frequently uses ICT but rarely for healthcare-related data documentation. Although the students were in the final semester of a nursing science degree, a large percentage (82%) had never used any application for patient data documentation. This indicates the lack of implementation of nursing informatics in the Slovenian nursing curriculum, which should be addressed in response to the increasing importance of informatics competencies in a modern clinical environment (24). In their reflections students also reported the need for staff training as a recommendation for improved ENR implementation.

Application B provided a better user experience and was perceived more suitable for nursing care plan documentation, since according to our survey it was rated favourably in all seven analysed aspects of application usage. According to the questionnaire results, application B performed better in operability, learnability, attractiveness and understandability, factors that in literature characterise greater software usability (25). The application is visually well designed and enables data input with minimal manual typing, making the application fast and easy to use. Its visual design features a colourful user interface with large letters and tabs. Some patient data can be documented using interactive images, and vital signs can, without additional effort, be presented in charts or tables (22). On the other hand, the user interface of application A, last updated in 2005, is poor in visual design and requires an excessive amount of manual typing. Answers to the open-ended questions also pointed out the superior visual design of application B, which could be associated with an improved user experience and would explain the preference for it in the survey results.

Open-ended questions results show that students using application B were predominantly dissatisfied with the level of individualisation of the nursing care plan and reported inadequate collections of listed aetiological and diagnostic terms. This points to an incomplete implementation of nursing standardized languages, which are normally associated with an increase in nursing care plan quality (26, 27). Based on the participants’ answers to open-ended questions, we could assume a more satisfactory integration of the nursing process in application A. Students using this application reported predominantly positive attributes related to nursing care plan formulation. Previous research (19) on the use of electronic documentation application A showed that its design prompts the user towards consistent documentation of individualised aetiological and diagnostic terms related to nursing diagnosis statements. Although application B was more recently developed and represents a modern EHR system, we can speculate that its design focuses on simplicity of use, visual design and incorporation of user needs while not ensuring patient individualisation and adequate integration of the nursing process.

Qualitative analysis of written reflections further confirms that visual design and time needed for documentation significantly influenced the perceived professional suitability of the evaluated applications. Students recognized numerous positive aspects of application A, predominantly regarding the organization and presentation of nursing-specific data and nursing care plans. This leads us to believe that the overwhelmingly inferior survey score of application A could be a consequence of poor visual design and excessive need for manual typing. Specifically, students disliked the sharp edges, small font and monotonous colour scheme that made the overall appearance of application A look outdated. Time spent documenting was another concern regarding the application use, with students urging the use of preformed statements and integration of standardized nursing language for faster and easier formulation of nursing care plans.

A user-centred approach can be used to evaluate the user experience and quality of nursing process integration in diverse ENR systems. Furthermore, our study highlights the importance of using qualitative and quantitative approaches in software evaluation.

Firstly, diverse opinions regarding the aspects of evaluated software solutions are difficult to adequately include in a structured questionnaire. While the main aspects of the applications’ use were included in the structural parts of the questionnaire, some were not predicted. For example, in open-ended questions, the students using application B reported concerns about inadequate patient data safety. Students did not consistently give reasons for their concerns, but one of the students reported that enabled access to other hospitalised patients indicated a lack of patient data security. Such functionality is not enabled in application A, for which no such concerns were detected. This aspect of application use was not included in any item of the structured part of the questionnaire and could therefore not be subjected to quantitative analysis.

Secondly, the survey results could prove to be misleading. Answering open-ended questions, students using application B reported the inadequate scope for individualising nursing care plans, which was not evident in the rated statements of the questionnaire. Although the questionnaire included items regarding the applications’ usability for the creation of an accurate and individually appropriate nursing diagnosis, the students did not rate these aspects of the application significantly lower. The qualitative analysis of student reflections also confirmed that students were satisfied with multiple aspects of application A that include data organisation, holistic care promotion, nursing process integration and so on, however, this was not evident in the survey results. Again, the students did not rate the applications’ usability for nursing care plan formulation and quality of formulated nursing care plan significantly higher in comparison to items describing visual design in rated statements of the questionnaire.

This could indicate that students might have reported a relatively lower level of favourability for all aspects of application use, in response to the visually poorly designed and slowly working application, while in the case of application B assigning relatively higher levels of favourability in response to the novel aesthetically appealing design. This is supported by the undesirably high internal consistency coefficient of the entire questionnaire (Cronbach’s alpha=0.97). These concerns are a significant result of our study since they indicate that future user-centred evaluations of software solutions should be carried out and interpreted cautiously. This study also highlights the importance of empowering end-users for competent participation in the development and critical evaluations of future digitalization of the nursing process. Users may be unable to objectively evaluate ENR systems’ professional suitability if they are influenced by a poor or excellent software design and therefore provide misleading results. Open-ended questions and in-depth qualitative analysis should be included in future studies since they can provide relevant information regarding the professional suitability of evaluated software solutions that otherwise cannot be adequately addressed in a structured quantitative questionnaire.

The study limitations include the low number of participants and its educational setting. Participating students lacked clinical expertise and did not use the evaluated applications before the study took place. This made them unable to provide the best possible feedback while evaluating the software in a strictly controlled environment. Each student had access to a computer, was familiar with the patient beforehand and had no time restriction while documenting patient data. Our study could be improved if the students were placed in a simulated environment and interacted with the evaluated applications for a longer time. This could better mimic the intended use of the application in actual clinical practice by an experienced user. Future research should take place in a clinical environment among a larger number of experienced users. An important application usability aspect is usage in the actual clinical environment scenarios, where users may be faced with a lack of appropriate hardware, environmental disturbing factors and time constraints (28). Furthermore, participants in this study were limited to a single university. Nevertheless, this study offers important insights for future nursing documentation applications development.

Conclusion

We used a user-centred approach to compare two diverse applications for electronic documentation of the nursing process. The results of this study provide new information on the user-centred evaluation of nursing-specific applications for clinical practice documentation. Additional findings using qualitative analysis further show that future user-centred evaluations of software solutions must consider the possible inability of users to objectively rate professional suitability under the influence of a satisfactory or dissatisfactory experience using the evaluated software. Such studies could lead to the empowerment of healthcare professionals to participate in the development and critical evaluations of future digital solutions.