A meta-analysis on the effectiveness of computer-based education in nursing.

OBJECTIVES: The purpose of this study was to conduct a comparative analysis of education effectiveness between computer-based education and traditional education methods in nursing.
METHODS: Medical and nursing literature databases were searched to identify studies regarding the effectiveness of computer-based education in nursing. Overall effect sizes for three outcome variables (knowledge, attitude, and practice level) were calculated. The effects of study characteristics on the outcome variables were analyzed.
RESULTS: Twenty-seven studies published from 1990 to February 2009, which that met the inclusion criteria, were included in the analysis. The meta-analysis showed that computer-based education generally had positive effects on knowledge, attitude, and practice, with overall effect sizes of 0.43, 0.35, and 0.34, respectively. This study also showed that the type of learner and the total education period had different effects on knowledge.
CONCLUSIONS: This metaanalysis found that computer-based education in nursing had positive effects on knowledge, attitude, and practice.

I. Introduction

As the availability of computers and the accessibility of the internet have improved, computer-based educations (CBE) have gained new ground as legitimate instructional methods replacing traditional educational methods such as lectures and textbooks [1,2]. CBE was developed to present teaching content using information and communication technologies. It can range from simply presenting lecture slides to integrating content into virtual learning experiences using computers and other electronic devices. One of the merits of CBE is that it can be delivered at any time and from any location. CBE can also reduce variability in educational outcomes by providing standardized education with no variability in educational providers. On the contrary, one of the disadvantages of traditional teaching methods is that a student must attend a lecture at a particular time and place. Another disadvantage is that it does not allow self-paced learning.

Nursing educators have utilized non-traditional teaching methods, such as audio-visual devices or computer-based education (CBE) for more than two decades. Owing to the development of personal computers in the 1980s, many nursing educators adopted CBE or the e-learning methods, and their use in nursing is growing exponentially [1,2].

Along with increasing demand for CBE in nursing education, many studies have been conducted to compare the effectiveness of CBE compared to traditional teaching methods. Those studies can be categorized by the type of learner, who may be a nursing student, nurse, patient, or other nursing clients. Various course contents were delivered to nursing students using CBE and its effectiveness was explored in many studies, such as in teaching clinical calculation [3], relieving student anxiety in psychiatric practicum [4], teaching intramuscular injection [5], and teaching maternity nursing practicum [6]. These studies concluded that CBE is an effective teaching method for outcomes such as knowledge, skill, learning attitude, and self-efficacy. Several studies have been conducted to investigate the effectiveness of CBE in teaching clinical nurses. The study of Harrington and Walker [7] used it for fire safety education, Kim and Jang [8] for ventilator training, and Choi [9] for infection control skills. These studies have also found that CBE is an effective teaching method to increase the level of knowledge and to produce a better job performance.

However, despite these research findings, it is difficult to conclude that the effectiveness of CBE differs significantly from or is even better than that of traditional teaching methods, because most studies were conducted on a relatively small scale. To address this concern, a meta-analysis -- the statistical analysis of a collection of results from individual studies for the purpose of integrating findings -- was applied. Meta-analysis has been widely used for evidence-based practice, which combines the results of multiple experiments or quasi-experiments to obtain a composite estimate of the size of the effect.

Several meta-analyses on the effect of CBE in health sciences concluded that CBE has at least similar [10] or even better outcomes [11] when compared to traditional teaching methods. However, Cohen's study is the only meta-analysis which explored the effect of CBE in nursing [11]. Cohen's study concluded that CBE made a moderate contribution to the effectiveness of teaching, yet the results showed that effect sizes were varied by the characteristics of the studies and called for more studies to be conducted on different educational outcomes [11,12]. Also, considering the fact that the study was published more than 15 years ago, it is worthwhile to conduct another meta-analysis, as information technology advances very rapidly.

The purpose of this study was to investigate the effectiveness of CBE compared to that of traditional teaching methods in nursing. This study focuses on three questions. First, how effective is CBE for three different outcomes of education such as knowledge, learning attitude, and practice performance? Second, is CBE especially effective for certain outcomes of education? Third, under which conditions does CBE appear to be most effective? It is intended that by addressing these questions from a meta-analytic framework, more precise conclusions concerning the effect of CBE can be reached.

CBE in this study is limited to the type of education in which a student learns by executing special learning programs on a standalone computer or on the internet. Traditional education (TE) is defined as education delivered using face-to-face lectures, textbooks, booklets, brochures, or verbal interactions.

II. Methods

This study used meta-analysis, which required locating studies, coding study features, and quantifying outcomes. The methods were similar to those used in previous meta-analyses of instructional technologies in higher education by Kulik et al. [13].

1. Locating Studies

The first step in the meta-analysis was to collect a number of studies that compared the effects of CBE vs. TE. For this study, CBE was limited to computer-assisted education and computer-managed instruction using a standalone computer or an internet service. TE was defined as education delivered using face-to-face lectures, textbooks, booklets, brochures, or verbal interactions.

The initial search was performed among articles published during 1990-2009 in three international electronic literature databases such as OVID, PubMed, and Proquest, and three Korean electronic literature databases such as RISS by Korea Education and Research Information Service, Research Information Center for Health (RiCH), and Korean Studies Information Service System (KISS), using the keywords 'computer', 'nursing education', 'computer education in nursing', 'knowledge', 'learning attitude', and 'practice performance'. Because we were interested in the effectiveness of computer-based education in nursing, we restricted our data search further with the following inclusion criteria. First, the article had to describe a study that was conducted with nursing students, nurses, patients, or other nursing clients using a quasi-experimental or experimental design. Second, the study had to compare outcomes of education such as knowledge, learning attitude, and practice performance in two groups of education, CBE and TE. Third, a nurse had to be involved in the research as a primary author or a co-author. Fourth, descriptive and inferential statistics had to be available.

After examining citation information and the abstracts of searched articles, a total of 49 articles were identified and we then retrieved the full texts of these articles for further screening. A total of 11 articles were excluded primarily because they did not have a quasi- or true experimental design. Seven articles were eliminated because they either used different learning outcomes measures, or did not include nurses as a primary author or a co-author. Four articles were then removed due to unavailability of descriptive and inferential statistics for meta-analysis. The remaining 27 articles were retained for final analysis. The 27 articles reported separate comparisons of CBE vs. TE. The 27 studies are listed in Table 1.

Table 1

A list of articles used in the meta-analysis

Out of 26 articles on knowledge, 19 articles were analyzed after articles #7, #9, #14, #17, #18, #19, and #25 were removed based on the homogeneity test. Out of 6 articles on attitude, 4 articles were analyzed after articles #17 and #25 were removed based on the homogeneity test. Out of 11 articles on achievement, 8 articles were analyzed after articles #9, #17, and #25 were removed based on the homogeneity test.

2. Description of Studies

The 27 studies located for this analysis had different learners and measured various outcomes. The next step in the meta-analysis was to describe the relevant study characteristics and outcomes of the studies identified.

1) Study characteristics

To characterize the studies more precisely, five variables were identified. They were coded following the coding schemes of several published studies [11,14,15]. The coded general characteristics of articles included publication features such as publication place and year; types of learners such as nursing students, patients, and learners other than nursing students or patients; and study features, such as sample size and education period. The coding categories of these variables and the number of comparisons in each category are listed in Table 2.

Table 2

Characteristics of 27 studies analyzed in this study

2) Study outcomes

The next step in the meta-analysis was to express the outcomes of each comparison in quantitative terms. Outcomes used in the studies were knowledge, attitude, and practice performance.

3. Data Analysis

Cohen's [16] measure of effect size d was calculated as the basic index of effect for three outcome variables. For studies which reported more than one outcome variable, we calculated the effect size for each outcome variable. When a study included both a conventionally taught control group and a no-treatment control group, results from the comparison with the conventionally taught group were coded for analysis. This procedure controlled for the possibly confounding effects of differential time-on-task. For studies that reported the means and standard deviations for both experimental and control groups, we calculated d from the means and the pooled standard deviation. For the studies without any information on means or standard deviations, we calculated d from statistics such as t and F, using procedures described by Song [17], and Lipsey and Wilson [18]. To make our study more similar to traditional reviews, we also examined the direction of the differences in outcomes of computer-based and traditional teaching: + for differences that favored computer-based instruction; and - for differences that favored traditional instruction.

Statistical analyses were performed with Comprehensive Meta-Analysis, ver. 2.0 [19]. All statistical tests were carried out using a two-tailed test with a significance level of .05. Heterogeneity among studies was explored before combining the effect sizes of all studies by applying the Q test as defined by Cochran [20]. A random or fixed effects model was used based on the heterogeneity test. Before combining all articles, each effect size was weighed to avoid the undue influence of studies with small sample sizes [21]. The 95% confidence interval of standardized effect size, Cohen's d, was provided. The binomial effect size display (BESD) (r), representing the difference in outcome measure between experimental and control groups, was computed. To identify the presence of potential publication bias, we computed the Fail-safe N suggested by Rosenthal and Rosnow [22]. Statistical analyses were carried out separately for each outcome variable. One study feature - the type of CBE - was dropped from the statistical analysis because of the lack of studies with computer-managed instruction (CMI).

III. Results

This section reports the results of two different analyses. The first analysis examined the overall size and significance of the effects of computer-based education on knowledge, attitude, and practice performance. The second analysis was conducted to determine whether the reported effects of computer-based education on knowledge, attitude, and practice performance were different for different types of learners and the total education period.

1. General Characteristics of Studies

The general characteristics of 27 studies analyzed in this study are presented in Table 2. Fourteen studies (51.9%) were published in Korea, and 13 studies (48.2%) were published in countries other than Korea. Four studies (14.8%) were published in the 1990s, and 23 (85.2%) were published after 2000. Ten studies had a sample size greater than 80. Eleven studies had nursing students as learners, six studies had patients as learners, and 10 studies had other types of learners. Twenty-six studies used computer-assisted instruction (CAI) and only one study used CMI. Seventeen studies had a total education period of less than 3 weeks and 10 studies had more than 3 weeks.

The 27 studies located for this meta-analysis looked at three different outcomes: knowledge in 26 (96.3%) studies, attitude in 6 (22.2%) studies, and practice performance in 11 (40.7%) studies. The effect size on knowledge, attitude, and practice performance for the 27 studies varied from study to study. Thus, before we combined the effect size of CBE on knowledge, attitude, and practice, the homogeneity of these outcome measures was tested. Based on the homogeneity test of the outcome measures, seven studies on knowledge, two studies on attitude, and three studies on practice performance were removed.

2. Overall Effects

1) Effect size for major outcome measures

In the first set of analyses, simple descriptive statistics were used to compare the results of computer-based education to those of traditional education. Results were compiled on three outcomes: (1) knowledge; (2) learning attitude; and (3) practice performance. Table 3 presents the weighted mean effect sizes, 95% confidence interval, BESD (r), homogeneity statistics Q (p), total effect size, and fail-safe number.

Table 3

Effect size for major outcome measures

ad+(SE): mean effect size weighted by the inverse of their random-effects variance (standard error), b95% CI: 95% confidence interval, cBESD (r): binomial effect size display, dQ (p): homogeneity statistics, eZ (p): total effect size, fNfs: fail-safe number.

The weighted mean effect size on knowledge for the 19 studies was 0.42. This indicated a modest effectiveness of CBE on knowledge. Also the BESD (r) representing the difference in knowledge between experiment and control groups was .21. This indicates that knowledge in the CBE group had improved by 21% more than in the traditional instructor-led group.

The weighted mean effect size on attitude for the four studies was 0.35. This indicated a small effectiveness of CBE on attitude. The BESD (r) was .17, indicating that attitude in the CBE group had improved by 17% more compared to the traditional instructor-led instruction group.

The weighted mean effect size on practice performance for the eight studies was 0.34, representing a small effectiveness of CBE on practice performance. The BESD (r) was .17, representing that practice performance in the CBE group has improved by 17% more compared to the traditional education group.

2) Effect size of major outcome measures by study characteristics

Although the effect of computer-based education was moderate in the typical study, the size of the effect varied from study to study. The effect size of computer-based education ranged from a high of 0.47 with knowledge [14] to a low of 0.23 with attitude [10].

It seemed possible that this variation in study outcomes might be systematic, and we therefore carried out further analyses to determine whether different types of learners and total education period were producing different results. The effect sizes for knowledge, attitude, and practice performance by study characteristics are presented in Table 4.

Table 4

Effect size on major outcome measures by study characteristics

Effect size from only one article was removed.

ad+: mean effect size weighted by the inverse of their random-effects variance, b95% CI: 95% confidence interval, cQ (p): homogeneity statistics.

The weighted mean effect size of CBE on knowledge for nine studies with nursing students was 0.30, four studies with patients 0.77, and six studies with other subjects 0.50. The effect size of CBE on attitude for one study with nursing students was 1.01, and the weighted mean effect size of CBE on attitude for three studies with other subjects was 0.29. The weighted mean effect size on practice for two studies with nursing students was 0.18, two studies with patients was 0.25, and four studies with other subjects was 0.42.

For knowledge outcome measures, studies with patients showed the largest effect size, followed by studies with other subjects and studies with nursing students. For attitude outcome measures, it is hard to compare because there was only one study on attitude with nursing students. However, a study with nursing students showed a much larger effect size compared to studies with other subjects. For practice outcome measures, studies with other subjects showed the largest effect size, followed by studies with patients and nursing students. However, it is hard to compare because there were only two studies on practice outcome measures with nursing students and patients.

The weighted mean effect size of CBE on knowledge for 11 studies with a total intervention period of below 3 weeks was 0.50, and for eight studies with a total intervention period of over 3 weeks was 0.32. The weighted mean effect size of CBE on attitude for three studies with a total intervention period of below 3 weeks was 0.49, and the effect size of CBE on attitude for one study with a total intervention period of over 3 weeks was 0.14. The weighted mean effect size of CBE on practice performance for seven studies with a total intervention period of below 3 weeks was 0.27, and the effect size of CBE on practice performance for one study with a total intervention period of over 3 weeks was 0.59. Education below 3 weeks had a bigger effect on knowledge than education over 3 weeks. It was hard to compare the effect size of CBE on attitude and practice performance between education periods of over 3 weeks and below 3 weeks, because there was only one study on attitude and practice with an education period of over 3 weeks.

IV. Discussion

As computer and telecommunication technology advances, the use of computer-based education is growing at a rapid rate. The general benefits of computer-based education are that it is usually self-paced and highly interactive. Web-based education, especially, is regarded as one of the greatest innovations in education compared to traditional instructorled education, due to the fact that access is available anytime and anywhere around the globe [23]. Due to these benefits, computer-based education is regarded an assistive and alternative educational method to traditional education. Nurse educators are facing many challenges in preparing themselves to cope with these changes in educational methods.

This meta-analysis examined the results of 27 studies that compared computer-based education with traditional education in nursing. We compared the effect sizes of knowledge, attitudes, and practice performance from computer-based education and traditional instructor-led education. We also compared the effect sizes of knowledge, attitudes, and practice performance by types of learners and total education period.

This meta-analysis showed that for the most part, computer-based education has made a small but significant positive contribution to knowledge, attitudes, and practice performance, with respective effect sizes of 0.42, 0.35, and 0.34. These were similar to the findings of Cohen's study [11] in which the overall achievement effect size was 0.45. The effect size of knowledge was a little higher than those of attitudes and practice performance in this meta-analysis. This could be due to the fact that changes in knowledge precede changes in attitudes, and changes in attitudes precede changes in practice performance [15,24-26]. Characteristics of CBE or the content provided could contribute to the higher effect size of knowledge than those of attitudes and practice performance.

The effect sizes of knowledge, attitudes, and practice performance in this study were smaller than those of studies comparing an experimental group with computer-based education to a control group with no education [20,26]. These results are expected, since in this study, CBE was compared with traditional instructor-led education rather than with no education. According to BESD (r) statistics, the computer-based education group improved knowledge by 21% compared to the traditional education group. The computer-based education group improved attitude and practice performance by 17% each, compared to the traditional education group.

The second set of analyses reported on the relationship between study characteristics and study outcomes. First, we divided the studies in this meta-analysis into three groups based on the type of learner (patient, nursing student, and other subjects), and compared the effect size of outcome variables by the type of learner. However, we were able to compare only the effect size of knowledge by the type of learner, due to the limited number of studies on attitude and practice performance. The patient group showed the largest effect size of knowledge, followed by the other subjects group and nursing student group. This could be due to the fact that patients have a higher motivation, demand, and readiness for learning compared to nursing students and other subjects [27]. Further study is needed to compare the effectiveness of CBE on outcome variables by the different types of learners.

Secondly, the studies in this meta-analysis were divided into two groups based on total education period (a group with an instructional duration of less than 3 weeks and another group with an instructional duration of greater than 3 weeks), and we compared the effect sizes of outcome variables by total instructional period. Again, we were not able to compare the effect sizes of attitude and practice performance by total instructional duration due to the limited number of studies. The average effect size of knowledge in the group with an instructional duration of less than 3 weeks (d = 0.50) appeared to be larger than the average effect size in the group with an instructional duration of longer than 3 weeks (d = 0.32). Kulik et al [14]. compared the average effect size of knowledge with three different instructional durations. In the Kulik et al.'s study, the average effect sizes by duration of instruction were 0.47 for the one semester or less category, 0.49 for the one semester - one year category, and 0.25 for the more than one year category. Since most of the studies included in our meta-analysis tend to have shorter instructional durations compared to Kulik et al. [14], with only 3 studies with an instruction duration between 8 to 16 weeks, it is very difficult to compare the findings of this study with that of Kulik et al.'s study. Thus, further study is needed to investigate the effect of instruction duration as study results with various instructional durations accumulate.

Of the 27 studies used in this meta-analysis, 26 reported results from CAI programs. Only one study reported results from CMI [28]. Thus, it is impossible to compare the effects of CAI with those of CMI. In this meta-analysis, only knowledge, attitude, and practice performance were included as outcome variables. We would like to recommend further meta-analysis with various physiological outcome variables such as plasma glucose levels, as Kim [29] did in her study [29], and various psychological variables such as depression and self esteem of Hill et al. [30] did in their study [30].

This meta-analysis study showed that CBE by nurses has greater positive effects than traditional instructor-led education, even though the effect size was small. This study showed that the effect of CBE on knowledge was a little higher compared to the effect of CBE on attitudes and practice performance. This study also showed that the type of learner and the total education period had different effects on the level of knowledge. This meta-analysis has proven that CBE, which can be delivered at any time from any location, is indeed very effective and a legitimate instructional method to replace traditional educational methods such as lectures and textbooks, as the availability of computers and the accessibility of the internet have improved.