Analysis of the quality of hospital information systems in Isfahan teaching hospitals based on the DeLone and McLean model.

BACKGROUND: Quality is one of the most important criteria for the success of an information system, which refers to its desirable features of the processing system itself. The aim of this study was the analysis of system quality of hospital information systems (HIS) in teaching hospitals of Isfahan based on the DeLone and McLean model.
MATERIALS AND METHODS: This research was an applied and analytical-descriptive study. It was performed in teaching hospitals of Isfahan in 2010. The research population consisted of the HIS's users, system designers and hospital information technology (IT) authorities who were selected by random sampling method from users' group (n = 228), and system designers and IT authorities (n = 52) using census method. The data collection tool was two researcher-designed questionnaires. Questionnaires' reliability was estimated by using Cronbach's alpha was calculated. It was 97.1% for the system designers and IT authorities' questionnaire and 92.3% for system users' questionnaire.
RESULTS: Findings showed that the mean of system quality score in a variety of HIS and among different hospitals was significantly different and not the same (P value ≥ 0.05). In general, Kosar (new version) system and Rahavard Rayaneh system have dedicated the highest and the lowest mean scores to themselves. The system quality criterion overall mean was 59.6% for different HIS and 57.5% among different hospitals respectively.
CONCLUSION: According to the results of the research, it can be stated that based on the applied model, the investigated systems were relatively desirable in terms of quality. Thus, in order to achieve a good optimal condition, it is necessary to pay particular attention to the improving factors of system quality, type of activity, type of specialty and hospital ownership type.

INTRODUCTION

Due to the extensive developments in medical technology and increasing the patients’ expectations, it has been emerged a growing need for the use of information systems in hospitals. Hospital information system (HIS) as an information powerful tool can help the handling process of hospitals and dramatically increase the correct decision-making in positive performance of the hospitals.[1] This system provides the possibility of data collection, processing, analysis, reporting the health information and leads to the appropriate indicators for monitoring and evaluation of health system performance.[23] The system will not only improve health care decision-making in action but also plays the greatest role in the development of effective organizational performance through the provision of information and patient records to service providers.[4] The objectives of the HIS are included:

Standardization of work processes in the hospital

Improving the patient care

Savings in hospital costs

Information preparation for performance evaluation

Monitoring of health and medical care[5]

Generating relevant and high-quality information to support decisions.[6]

An information system is effective when it is able to respond to users’ information needs. Otherwise, it would step into the vanity and in order to be prevented from entering the early stage of information systems futility, it is required to assess the effectiveness of the system periodically to realize the possible failures in order to improve the system.[7] Systematic process of data collection about the activities, characteristics and outcomes of the activities are to be used by the specific individuals and the main goal is the evaluation of uncertainty reduction and improving the effectiveness.[8] Evaluation phase is important in the development of any system. Most systems are developed well and completely, but the systems have never been evaluated. For fulfilling the goals of the system implementation to achieve the maximum benefits of the system, all systems need to be evaluated based on previously developed criteria.[9] Quality is the most important criterion for the success of an information system[10] which refers to the desirable features of an information system[11] as: Ease of access, flexibility, system integration, system response time, system reliability, the use of new technologies, detecting user expectations, ease of learning and use, and so on.[1012] Mahdavi in his thesis entitled “Proposed system for evaluation of quality evaluation systems,” has pointed out that the quality of an information system can be assessed as the actual participation of a specific information system in achieving organizational goals.[13] Khalesi as stated in the research entitled “Attitudes of users about the application of Admission Discharge System-9 software (ADS-9),” that the most important method of improving the quality and quantity of ADS-9 software was attention to the information needs of its users and particularly at the level of hospitals and medical centers.[14] For the evaluation of health information systems, different models can be used and one of them is the successful model of information systems of DeLone and McLean.[15] DeLone and McLean proposed their model after a comprehensive survey on the effective criteria for evaluating the information systems. This model had six criteria and one comprehensive evaluation model of information systems. The model focused on six major issues involved the whole system.[16] Quality was one of the axes and referred to the desirable features of an information system.[11] The scales related to quality were included as follows:

Ease of access

Flexibility of the system (Flexibility of an information system is the ability to be adapted to changes[1718])

System integration

Response time (response time was defined in this study as the period that an information system responds to a specific request. The system response (reaction) should be fast and at the least possible time to the user request (demand).[19] In this study, the researcher assessed the response time of the system with several questions, including: (1) Is the search passing for a long time in some parts of the system due to the increase for information of the database? (2) Is there an easy and quick reporting system?

System reliability

Benefits derived from the information system (the purpose of the derived benefits of the system in this study was the advantages of the system in comparing to the manual system)

Usefulness of decision support system features

System usefulness functions

Utilization of resources

The use of new technologies

Error recorded

Recognizing the user expectations (at the design time of a system, several questions are discussed including the users anticipation of the system,[20] comfortable learning, effectiveness of maintenance services, to be comfortable with the program, system user friendly and so on. The mentioned cases were among those that the users expected from the system and were investigated in this study)

Easy to learn and use.[1011]

The purpose of the evaluation was to assess the system quality, determining strengths and weaknesses of the information systems in this field and guidance for improving the use of these systems.[21] In recent years, many studies have been conducted regarding the quality assessment of the importance of HIS. The results of the studies in this field are indicative of inappropriate system quality and the lack of software flexibility to use.[22] According to the viewpoint of World Health Organization, it is essential to perform researches in order to investigate the cause or causes of this problem and the empirical researches for reducing the problems in this field too.[23] According to this principle that assessment plays an important role in software development[21] and also due to the weaknesses and numerous problems of HIS, they need to be assessed based on the comprehensive health community models. The aim of the present study was to perform an analysis of the quality of HIS in teaching hospitals of Isfahan based on the Delone and MacLean model.

MATERIALS AND METHODS

This research was an applied and analytical-descriptive study, which was conducted based on the Delone and MacLean model quality criteria. HIS in teaching hospitals of Isfahan were analyzed in 11 locations in Isfahan medical teaching hospitals (Shahid Beheshti, Shahid Chamran, Noor and Ali Asghar, Imam Musa Kazem (AS), Isabne Maryam, AlZahra(SA), Ayatollah Kashani, Feiz, Saied-alShohada(AS), Farabi and Amin) during June 2010 until March 2011. The study population was included HIS designers, hospitals’ information technology (IT) authorities and the users of HIS in Isfahan teaching hospitals (it is noteworthy that Najaf Abad Modarres public hospital was faced with the lack of HIS and Hazrat Zahra (SA) public hospital was at the system startup stage and pilot implementation). These two hospitals were excluded from the study research community. Sampling was performed on a census form in the system designers and IT authorities of the hospitals due to the limited statistical study community and community-based sampling for the users of HIS group. Initially, the total number of users in each hospital was determined and based on the minimum and maximum number of users in the studied hospitals, the number of samples obtained at least 20 users for each hospital according to the following equation:

In total, at least 220 people were obtained for all the hospitals (users group was sampled randomly). The collection data tool in the present study was a researcher-made questionnaire based on the quality criteria and coincident with the Delone and MacLean model. Two separate questionnaires were designed (system designers and hospital IT authorities’ questionnaire and HISs’ users questionnaire). The method of data collection was visiting the researcher from the studied hospitals. IT authorities and users’ questionnaires were distributed among them. The questionnaire for the system designers delivered directly if they were available and otherwise, the questionnaire and the required comments were sent by e-mail. Questionnaire content validity was confirmed according to the teachers’ point of views, computer scientists, hospitals HIS authorities, information management practitioners and the field of health informatics at headquarters. In order to estimate the reliability of the questionnaire, Cronbach's alpha coefficient was used in SPSS software. The calculated Cronbach's alpha coefficient was 97.1% for the system designers’ questionnaire and hospital IT authorities and was 92.3 for the system users’ questionnaire. After collecting and ultimate control of the gathered data, they were entered in the SPSS software version 18. The followings were used for analyzing the collected data: Measurement tools, converting the qualitative responses into quantitative values, weighting method (valuation) for each of the options, five-point Likert scale to measure the respondents’ comments and one-way ANOVA test for comparing the mean score values.

RESULTS

These demographic characteristics were investigated: Gender, age, education and field of study for the users of HIS. Most of them (42.5%) were in the age group of 20-30 years. 70.6% of the users were female and 49.6% of them had a bachelor's degree. Most of hospitals’ IT designers and authorities (53.8%) were in the age group of 20-30 years. 63.5% of them were males and 69.2% of them had BA degree. Table 1 shows the findings of the study about the mean score of the quality components in a variety of HISs in the research community. It should be noted that in the Table 1, number 1 was allocated for the system users, and number 2 for the group of hospitals’ designers and IT authorities. With regard to this fact that the investigated components were based on the account type of user research community, Therefore, some of the components were common in both of questionnaires and some were dedicated.

Table 1

Mean scores of system quality components in a variety of hospital information systems in the research community

The findings in Table 1 about the mean quality scores in a variety of HISs in the research community were indicative that in relation to the quality criteria, Kosar system (new version) gained the highest mean score (57.8%) and Rahavard Rayaneh system had the lowest score (45.2%) based on the users’ questionnaires. According to the system designers and IT authorities questionnaire and in this criterion, the highest mean score (92.1%) was for the Kosar system (new version) and the lowest score (52.9%) was gained by Rahavard Rayaneh system. ANOVA test showed that there was a significant difference in the quality scores in the HISs and were not the same (P value ≥ 0.05). Table 2 shows the findings of the study about the comparative final mean scores of quality in a variety of HISs. In total, Kosar system (new version) has gained the highest score and Rahavard Rayaneh system had the lowest score.

Table 2

Comparison of the final scores of system quality in a variety of hospital information systems

ANOVA test showed that the final mean score of the system quality in a variety of HISs was significantly different and was not identical (P value ≥ 0.05). The findings of the study about comparing the mean score of system quality among different hospitals showed that Farabi hospital had allocated the highest mean score (59.1%) and Feiz Hospital had the lowest mean score (45.2%) based on the users’ questionnaires. Al-Zahra and Farabi Hospitals with the mean scores of 92.1% and 52.9% had gained the highest and the lowest mean score in this criterion, respectively based on the system designers and IT authorities’ questionnaire.

As it is indicated in Table 3 and in relation to system quality criterion, Al-Zahra Hospital with the mean of 64.8% had the highest mean score and Feiz Hospital with the mean of 47.4% had the lowest mean score. ANOVA test also indicated that there was significant difference in the final mean score of system quality among different hospitals and it was not identical (P value ≥ 0.05).

Table 3

Comparison of the mean final score of system quality among different hospitals

DISCUSSION

According to the research results into practice, in association with system quality components in a variety of HISs, the “ease of learning and use” component was the most desirable and the “response time” was the lowest due to the users’ questionnaires. It seemed that the developers of information systems noted to the component of “ease of learning and use” in the design of these systems. That way, the users can use the system with ease but the component of “response time” was not enough. However, it is one of the most important and most essential factors of a system and especially in the medical environments. Ahmadi et al. in a study titled “Usability Evaluation of HISs based on Isometric 9241 Standard” stated that the developer companies of HISs in Iran should pay more attention to the synchronization of software response time with its speed.[21] The study results were consistent with the present study. According to the system users’ questionnaires, the system quality criterion was quite desirable in a variety of HISs. Due to the system designers and IT authorities’ questionnaire, the reliability (validity) component had the greatest desirability and “beneficial features of decision support system” allocated to the lowest. The investigated systems had weaknesses in terms of the decision support system component and it should be paid attention to this component at design. The findings of Wangpipatwong study entitled “Factors affecting in adoption Web sites based on information quality and system quality” in 2005 showed that the reliability (validity) and effectiveness components had the greatest influence on adoption of websites among the system quality criterion components. It was consistent with the results of the present study.[24] According to the system designers and IT authorities’ questionnaire, the system quality criterion was desirable in a variety of HISs. The investigated systems were quite desirable in terms of the system quality but had some defects. The overall system quality mean score obtained in the studied systems was 59.6%, which it had a significant difference with the highest mean (100 score). In general, the level was relatively desirable, but it was far from ideal situation. What causes that the investigated systems were quite desirable in terms of system quality criterion summarized and included in order of preference as follows:

Slow response time

Lack of system design based on user expectations

Lack of adequate budget and sufficient funding for the implementation of the system

Lack of full and comprehensive training of related personnel to the system

Systematic errors occurring while working with the system

Low system reliability.

Ahmadi et al. in a study entitled “HIS system compliance with medical record department users’ needs and expectations” showed that the existing HISs covered only less than half of the users’ needs and expectations. This was consistent with the results of the present study.[25] The aim of the study of Ribiere et al. (1999) entitled “The quality of HISs: An assessment of user satisfaction” was to identify the requirements for HISs in order to improve the quality of health care level. Finally, the researcher in this study has achieved this result that the best way to maintain the user satisfaction of HIS was the system design to be based on the opinions and needs of the users, not the views and the needs of system designers.[26] MohsenZadeh in his study fully described the Boehm's quality model and pointed out that in this model, the quality system was defined as a set of specifications and criteria. In highest-level of the model, three fundamental requirements were defined as the end user, reliability (validity) and the effectiveness of the system. It was noted to the significance of reliability (validity) component in assessing the quality of an information system.[27] Khalesi has stated in his research that regarding to record the cost information of patients; the software was not desirable and just was responsive to the needs of users as much as 38.5%.[14] This finding was consistent with the results of the present study. The results were not consistent with the study of Hanmer (2004) entitled “Evaluation of computerized HISs of public hospitals in South Africa based on the Delone and MacLean model.” Hanmer achieved to the conclusion that the HISs of public hospitals in South Africa have been able to fulfill the expectations of the system quality requirements.[28] According to the studies and researches mentioned above, the installed software in medical centers has failed to meet expectations in terms of system quality requirements. In relation to the system quality components at different hospitals, similar results were obtained from reviewing the system quality factors in a variety of HISs. Regarding the system quality criterion in relation to the various hospitals, all hospitals were considered quite desirable. The overall system quality mean score among different hospitals was located in a desirable level (57.5%) and there was a significant difference between the maximum mean score (100 points). The obtained results from the evaluation of the system quality criterion of the hospitals showed that the followings certainly played direct roles on the HISs: Type of activity (medical, education-medical), specialty (general, technical) and property type (university, private sector). In the present study, the mean score of system quality criterion among different hospitals was not identical. This was probably due to the influence of the type of activity, type of specialty and type of hospital ownership. Therefore, in designing the HIS, it should pay specific attention to the above-mentioned factors and particularly to the specialty of the hospitals. Designing and evaluation of HISs should be based on well-known standard models in order to benefit from the systems, qualified information and users’ satisfaction. In total, the results of the present study showed that the six investigated HISs were significantly different in terms of system quality and they were not identical. Kosar system (new version) had the highest mean score on the investigated criterion. In comparison between different hospitals, Al-Zahra hospital was allocated the highest mean score on the investigated criterion to itself. This was probably due to the type of activity, type of specialty, and type of hospital ownership. In most of the obtained findings in the present study, Rahavard Rayaneh system had the least mean values. Being old and working under Dos operating system could be stated as the reasons for the weaknesses of the system. Despite the fact that Kosar system (new version) gained the highest mean scores on the information quality criterion, but it did have some gaps that needed to be resolved. In this study, the evaluations of results were different and non-consistent in both users and system designers. Regarding this difference, it could be noted to the followings:

Different expectations of users and system designers

Different positions of users and system designers

Different knowledge level and understanding between users and system designers

Sense of belonging and ownership of the system by the system designers.

Suggestions

Providing user authentication by allocating username and dedicated password for each user

Generating a periodic change of the password management system

Systems must contain firewalls, auditing capabilities, safeguards and setbacks in order to meet security rules and patients’ expectations about the privacy and confidentiality of records

System to be based on advanced operating system and Web

Evaluation of daily system performance in medical centers

Creating the on-line guide for the new and experienced users, on-line help should be comprehensive, accessible and available to all of the users. The individuals should have tested the ability to access all of the system functions through the online manual

The system should be flexible to meet the demands of the new users

Using NET framework technologies in system design

Designing the subsystems conforming to the requirements and demands of the users.