Online public health preparedness training programs: an evaluation of user experience with the technological environment.

OBJECTIVES: Several public health education programs and government agencies across the country have started offering virtual or online training programs in emergency preparedness for people who are likely to be involved in managing or responding to different types of emergency situations such as natural disasters, epidemics, bioterrorism, etc. While such online training programs are more convenient and cost-effective than traditional classroom-based programs, their success depends to a great extent on the underlying technological environment. Specifically, in an online technological environment, different types of user experiences come in to play-users' utilitarian or pragmatic experience, their fun or hedonic experience, their social experience, and most importantly, their usability experience-and these different user experiences critically shape the program outcomes, including course completion rates. This study adopts a multi-disciplinary approach and draws on theories in human computer interaction, distance learning theories, usability research, and online consumer behavior to evaluate users' experience with the technological environment of an online emergency preparedness training program and discusses its implications for the design of effective online training programs. .
METHODS: Data was collected using a questionnaire from 377 subjects who had registered for and participated in online public health preparedness training courses offered by a large public university in the Northeast.
RESULTS: Analysis of the data indicates that as predicted, participants had higher levels of pragmatic and usability experiences compared to their hedonic and sociability experiences. Results also indicate that people who experienced higher levels of pragmatic, hedonic, sociability and usability experiences were more likely to complete the course(s) they registered for compared to those who reported lower levels. DISCUSSION: The study findings hold important implications for the design of effective online emergency preparedness training targeted at diverse audiences including the general public, health care and public health professionals, and emergency responders. Strategies for improving participants' pragmatic, hedonic, sociability and usability experiences are outlined.
CONCLUSION: There are ample opportunities to improve the pragmatic, hedonic, sociability and usability experiences of the target audience. This is critical to improve the participants' learning and retention as well as the completion rates for the courses offered. Online emergency preparedness programs are likely to play a crucial role in preparing emergency responders at all levels in the future and their success has critical implications for public health informatics.

Introduction

Preparing the public health workforce to mitigate, respond to, and recover from natural and man made disasters is not a minor undertaking. Both governmental and non governmental organizations have called on universities and other educational institutions to develop programs to efficiently and effectively train our public health workforce [1,2,3,4]. Many educational institutions across the country have responded to this by developing and offering virtual or online programs that incorporate ‘canned courses’—i.e. courses that do not require an instructor and instead allow students to download the materials and self-learn at their time of convenience [5].

The effectiveness of such online courses depends on delivering rich learning experiences for the students. However, unlike traditional classroom-based education, the online environment is not under the control of an instructor. Students’ learning experience in such online situations could be affected by not only the structure and content of the course but also the student interactions facilitated by the technology-based infrastructure and the usability of such infrastructure. Thus, to measure the effectiveness of online courses, we need to go beyond the evaluation tools that are currently used to evaluate offline or classroom-based courses and use tools that provide a more holistic view of users’ online learning experience.

Specifically, to understand and evaluate the learning experience in an online program, we need to draw on our understanding of people’s behavior in online environments. Prior studies in consumer psychology and human computer interaction offer an appropriate foundation for this. Research in consumer psychology indicate that experience has two primary dimensions—a utilitarian (or cognitive) dimension and a hedonic (or affective) dimension [6,7,8,9,10]. However, in an online environment, factors that are either related to the technology itself or to the interactions of the people with the technology could also shape such experience. Prior studies in the area of human-computer interaction and computer-mediated communication [11, 12, 13] indicate the relevance of two other dimensions—sociability experience and usability experience. In this study, the online offerings of an emergency preparedness program offered by a public university in the Northeast was evaluated on the above four dimensions of online user experience.

In addition, in this study we also examine whether online user experience had any impact on course completion. Prior research in this area has shown that online distance education courses often have higher non-completion rates than traditional in-class courses [14, 15]. The reasons cited for this include student isolation and technological barriers which in turn de-motivates students and lead to course drop out [16, 17, 18]. The current study will provide insights into how the technological environments can be developed so that users (i.e. students) would not only learn but also have a positive experience that in turn enhances the probability of course completion.

Further, we empirically show that higher levels of student self-motivation do not translate into course completion, which in turn emphasizes the need to focus on student’s experience during the course to enhance program success.

The remainder of this paper is organized as follows: The next section reviews the background literature and theories for this study: 1) workforce development for public health emergency preparedness, 2) online learning environment and online consumer experience, 3) cognitive affective learning, 4) social Learning theory, 5) usability in distance-learning environments, and 6) motivation and course completion. Following that we formally define our study research questions. This is followed by the methodology section which includes details on data collection and data analysis. Next, we discuss the study results and their implications. The report ends with a brief conclusion and key recommendations for improving the users’ online experience and thereby enhancing program effectiveness. .

Background

Workforce development for public health emergency preparedness

In 1997, the U.S. Department of Health and Human Services issued a report titled ‘The Public Health Workforce: An Agenda for the 21st Century’, which highlighted the gap in training and preparation for public health professionals for emergency preparedness [2]. It is estimated that there are around 500,000 people in the public health workforce at the federal, state and local levels. In addition, there are around 3 million people working in the healthcare system (private and non profit) who play a key role in public health emergencies [1]. In case of an emergency situation, be it an epidemic, terrorist attack or a natural disaster, these are the people who will be deployed to the front lines and the report raised concerns regarding their training and readiness. According to their assessment this “compelling and urgent programmatic forces are making enhanced training and education opportunities for public health professionals a necessity.” [1].

As a result, in September 2000, the Center for Disease Control (CDC) and the Association of Schools of Public Health together brought out a plan to develop a national network of public health preparedness centers. As part of the plan, they funded several University programs to start Centers of Public Health Preparedness (CPHP) in around 10 regions across the country. CPHPs in all these regions have been offering relevant courses to train the public health workforce for emergency preparedness [4].

The Institute of Medicine later released a report in 2003 titled “Who Will Keep the Public Healthy?” that not only reiterated the need for education and training for the public health workforce, but also stated that online distance learning was the best solution to train this large number of diverse public health workers in a cost-effective manner. This has led several state universities and local governmental agencies to start their own online education programs [19, 20] for training the public health workforce in emergency preparedness. Despite the growing numbers of such programs, there have been very few initiatives focused on evaluating the online learning environments of these programs, especially for the CPHP offerings, other than the evaluations done by CDC itself.

Online (or distance) education is definitely a cost effective and efficient way of training such large numbers of public health workforce. However, in order to evaluate such programs, one needs to adopt an interdisciplinary perspective as diverse aspects (technology, social, etc.) assume importance. This study offers a theory based framework drawn from multiple disciplines to evaluate the online environments of such distance education programs.

Online learning environment and online experience

Online classrooms and learning environments are inevitable to meet the demands of training requirements for public health emergency preparedness. It provides the economies of scale and convenience that will not be available in traditional classroom settings. There are innumerable benefits for students from online distance learning, flexibility and convenience perhaps being the most important [16]. However, this is not without many disadvantages and problems [21,22,23]. Student isolation [17] and student frustrations [18] have been found to be two of the major disadvantages with online distance education. A recent study on an online medical self-paced course noted that the major challenges were technological problems and the fact that opportunities for social interaction was much lower [16]. Another study found that there were eight main factors that impede online distance education: administrative issues, lack of social interaction, academic skills, technical skills, learner motivation, time and support for studies, cost and access to the Internet, and technical problems [24].

Sustained frustrations and isolation can impede learning, especially the cognitive and affective dimensions of the learning experience [25]. Studies in this area show that these would also decrease the storage and processing capacity of working memory [26, 27]. In addition, frustration and anxiety are major factors that lead to de-motivation among students [25]. Motivation is critical for this kind of online learning environments [28]. It becomes even more critical when training public health emergency preparedness workers as many students are much older, have full time jobs and other work and family commitments, as compared to young college students [20].

Cognitive Affective learning

Similar to the research in consumer psychology, where pragmatic and hedonic component of experience received much attention, in the education and learning literature, the cognitive (pragmatic) and affective (or hedonic) dimensions of learning has been the focus of many researchers. The cognitive dimension was considered most critical for learning in many of the earlier studies.

While the cognitive dimension is critical, researchers also began discovering that there is an affective dimension that impacts learning, memory, retention and inference making. More recently this component received even more focus in the context of online learning which led researchers in the MIT Media Lab to work on affective agents where a robotic computer aims to improve user’s motivation to learn. The robotic computer is capable of expressing affect by rewarding or showing pleasure when the learner does something right, and when the learner gets distracted, it would try to entertain the learner and so on. There has also been significant work done in developing affective interface agents that are capable of working as teaching assistants in monitoring and managing online distance learning [29, 30, 31]. The objective of this line of research is to detect the affective or emotional state of the learner and provide appropriate affective or hedonic support to keep the learner engaged in the content and also motivate them to complete the tasks before them.

Research in consumer online behavior shows that when users are engrossed in the online activity, they do not keep track of time and get into a state of “flow” [32, 33, 34]. This stream of research suggests that when people are provided with activities that they get engrossed in and start deriving fun from, they reach a state of flow [34]. In the online learning environment, if students are provided with activities that they could get immersed in and achieve a state of flow, it would not only improve learning but also enhance the course completion rates and the student retention rate. Hence, while the cognitive dimension of learning leads the student to evaluate the pragmatic value or the usefulness of the course content, it is the affective component that enables the user to have a better hedonic experience.

Social Learning theory

Another relevant stream of research adopts the social learning perspective in which the conjecture is that knowledge is socially constructed and occurs when individuals engage in discourse about a subject matter [35, 36, 37]. Knowledge is embedded in individuals, and by providing effective communication channels and opportunities to interact with one another—either socially or in a classroom setting—it would lead to more knowledge transfer and creation, and in turn offer a richer learning environment [36, 37].

This perspective has been widely accepted in the context of online distance education and it is often emphasized that student interactions are central and critical for a successful learning experience and consequently the success of online courses [35, 38]. These interactions could be with other students or with the instructor. In the context of online public health emergency preparedness courses, especially CPHP courses, almost all the courses are ‘canned courses’ without an instructor or fellow students. This could potentially affect the sociability experience and thereby impact learning and course completion rates.

Usability in distance-learning environments

As mentioned previously, technological barriers and usability issues are the two most often cited reasons for student frustrations and poor completion rates. Several studies have considered the usability issues of different online courses [39, 40, 41] and have broadly concluded that usability is a critical factor in determining the success of any online course.

Usability is the extent to which a user can successfully accomplish the tasks with effectiveness and efficiency [42]. In the distance education context, usability would be the effective and efficient accomplishment of learning related tasks or goals in the online environment (with or without using specified tools for that system). In the context of emergency preparedness training courses, it is a critical evaluation component as users’ interaction with the system is more than users’ interaction with the instructor.

Usability issues are more widely accepted by course providers as a potential problem and many understand the need to rectify them. However, usability issues are much more difficult to evaluate as users often attribute usability issues to their own lack of skills or a problem at their end (for example, their problematic home computer or Internet connection). In addition, many specific usability questions such as “is navigation through the website easy or difficult?” can be answered in two different ways – navigation through the website is easy or difficult for ‘everybody else who is skilled in computing technology’; or navigation through the website is easy or difficult for ‘me’ specifically. Analysis of the results also becomes difficult as users may hold different technology standards, different levels of skills, and access to different levels of technological assistance.

To overcome these measurements issues, in the current study, we used a simple pre-validated scale to evaluate whether the overall technological environment was easy/difficult; confusing/not confusing; consistent/inconsistent; stressful/not stressful; simple/complicated and tiring/not tiring. This usability tool has been found to be effective in understanding whether the overall usability experience was satisfactory to the user [43, 44].

Motivation and course completion

Lack of motivation has been cited as one of the major impediments to online learning [15, 24, 45, 46]. Motivation to enroll for courses could come from both internal forces and external forces [47].

Intrinsic (or internal) motivation has been indicated as one of the key factors that drive people to register for courses as it reflects a person’s need to enhance their skill set, their market value, self-esteem, etc. Extrinsic (or external) motivation relates to one’s profession including mandatory job requirements, CPE credits, suggestion from one’s boss and colleagues, etc. Extrinsic motivation would also include motivation from educational institutions or the course providers (e.g. instructors, universities, and program administrators). However, the CPHP is not organized to provide this kind of motivation. Hence, the main sources of extrinsic motivation seemed to be from their own professional life.

While both intrinsic and extrinsic motivation could certainly lead students to register for courses, there is no evidence yet that this would lead to course completion. In this study we empirically examine whether there is any significant difference in the intrinsic and extrinsic motivation levels of students who completed the courses and that of students who did not complete the courses.

Research Questions:

The above literature review suggests that poor online course experience (that in turn may arise from a lack of instructors, lack of social interactions, technological problems in the online courses, etc) could de-motivate students and lead them to drop the courses that they had registered for. The discussion also suggests that motivation to enroll for a course, while an important factor, may not be enough to ensure that the student completes the course.

Thus, in our empirical study, we address two research questions that reflect the above two issues. First, are there any significant differences in students’ online course experiences (pragmatic, hedonic, sociability and usability) based on their course completion status? Second, are there any significant differences in students’ intrinsic and extrinsic motivation levels based on their course completion status?

Based on the theories and concepts outlined previously, we define student’s online course experience—i.e. the overall experience a student derives from his or her interaction in the online course environment—along four dimensions: pragmatic, hedonic, sociability, and usability.

Pragmatic experience is the pragmatic or utilitarian value the student experiences in the online learning environment. This dimension is related to goal-oriented behavior [33] of the student and reflects whether the student found the experience in the online learning environment useful, valuable, and/or worthwhile [43, 44]. The hedonic dimension is the intrinsic value the customer derives from the interactions in the online learning environment. It reflects the enjoyment and excitement students derive during the learning process as well as during their interactions in the online learning environment. The sociability dimension is the social experience students derive from the interactions in the online learning environment. It captures students’ perceptions regarding the overall openness, friendliness and politeness of the community in the learning environment [11,48]. Even though there weren’t much human-human interactions in this study context, there were human-computer interactions and such interactions can also lead to sociability experience [48]. The usability dimension is defined as the students’ experience in navigating and using the online materials. As such, this dimension captures the ease of use and clarity of the technological features of the online learning environment. Higher levels of usability experience reflect the ability of the student to navigate and participate in the online learning environment smoothly and effortlessly and without any obstructions or annoyances that might distract them from their goals or interests [11].

Next, we describe our empirical study..

Method

Data collection and Data analysis

Data was collected using a Web-based questionnaire from students who had registered for the courses offered by a CPHP based in a large public university in upstate NY. Emails were sent to approximately 2700 students who had enrolled in one or more of the courses during the past one year. Each email briefly described the study and invited the student to respond to a survey—the link to the survey was included in the email (the survey was available from the CPHP’s Web site). There were 415 responses to the email invite. 38 responses had to be excluded from the analysis due to high amount of missing data. Thus, there were a total of 377 usable responses.

Data was collected on different aspects of the online program, including, student motivations, student profile, and their overall experience with the CPHP Web site and with the courses (specifically, the 4 dimensions of user experience—pragmatic, hedonic, sociability and usability). The questionnaire was built using existing scales for measuring each of the variables. Student’s online course experience was measured using an existing validated scale designed to measure online experience [43, 44]. A tool to measure student motivation to enroll was developed by the CPHP staff for an earlier study and was adapted and used in this survey. Course completion data was collected using a simple yes/no question as to whether they completed all the courses they had registered for.

A factor analysis of the data related to student motivation yielded two distinct factors— intrinsic motivation and extrinsic motivation. See Table 1 for items and their factor loadings. The intrinsic motivation factor included 4 items and extrinsic motivation factor also had 4 items.

Table 1:

Factor scores for ‘motivation’

Intrinsic motivation
I registered to gain more knowledge	.922
I registered myself for personal development	.922
I registered myself for professional development	.887
I registered to do something useful/constructive	.810
Extrinsic motivation
It was required for other educational programs	.839
It was recommended by someone outside my workplace	.703
It provided continuing education credit	.620
It was required/highly recommended for my job	.588

Similarly factor analysis for each of the online experience dimensions were done separately. Items and factor loadings are provided in Table 2. As can be seen in Table 2, pragmatic experience was measured using a 7 item scale (reliability α = 0.96), hedonic experience by using a 9 item scale (α = 0.95), sociability experience by using a 5 item scale (α = 0.87), and usability experience by using 6 item scale (α = 0.91).

Table 2:

Factor scores for Online Experience

Pragmatic	Scores
Valuable/Not Valuable	.938
Practical/Impractical	.918
Relevant/Irrelevant	.915
Informative/Not Informative	.905
Worthwhile/Worthless	.904
Productive/Not Productive	.903
Useful/Not Useful	.893
Hedonic
Stimulating/Boring	.914
Exciting/Not exciting	.892
Captivating/Not captivating	.872
Fun/Not fun	.856
Satisfying/Unsatisfying	.846
Enjoyable/Not enjoyable	.831
Entertaining/Not entertaining	.809
Deeply engrossing/Not deeply engrossing	.803
Pleasant/Unpleasant	.802
Sociability
Inviting/Uninviting	.851
Friendly/Unfriendly	.840
Polite/Impolite	.808
Personal/Impersonal	.799
Social/Unsocial	.748
Usability
Simple/Complicated	.866
Easy/Difficult	.858
Confusing/Not Confusing	.827
Not Tiring/Tiring	.827
Consistent/Inconsistent	.826
Stressful/Not Stressful	.807

An independent sample t test was used to compare the means of the four experience dimensions (pragmatic, hedonic, sociability and usability) between students who completed all the courses they registered for and students who did not complete one or more of the courses they registered for. The data was analyzed using SPSS, all the experience dimensions were entered as test variables and the item ‘Did you complete all the courses you registered for’ was entered as the grouping variable. Similarly, an independent sample t test was used to compare the means of intrinsic and extrinsic motivation between students who completed all the courses they registered for and students who did not complete all the courses they registered for.

Results and Discussion

Majority of students had registered for just one course. Specifically, 157 people (41.6%) registered for 1 course; 73 people (19.4%) registered for 2 courses, and 24 people (6.4%) had enrolled for a course that was not listed in the survey. The study sample also included students from 31 countries although the large majority was from the United States. The number of female students was much higher (61%). This represents the actual student population ratio at this CPHP. Racial distribution was as follows: 73.7% White non Hispanic, 7.6% black non-Hispanic, 5.4% Hispanic or Latino and 5% South East Asian. This distribution also mirrors the student population distribution at this CPHP.

The mean and standard deviation for all the 4 dimensions of online experience and the two factors of motivation are provided in Table 3.

Table 3

Means and Standard Deviation of Study Variables

Variables	Mean	S.D
1. Pragmatic experience	6.1	1.12
2. Hedonic experience	5.0	1.26
3. Sociability experience	4.9	1.22
4. Usability experience	5.6	1.15
5. Intrinsic motivation	5.4	1.96
6. Extrinsic motivation	3.8	1.52

Online experience & course completion

Analysis of the data indicates that, overall, participants had higher levels of pragmatic and usability experience compared to hedonic and sociability experience (see mean values in Table 3).

The results from the independent sample t-test showed that there was significant difference in the scores for all the 4 dimensions of experience between students who completed the course and students who did not. Results are shown in Table 4.

Table 4

T test results for online experience & course completion

Experience	Means	N	Std. deviation	DoF	T value
Pragmatic	Yes  6.2No  5.5	27567	.961.48	79.9	3.89***
Hedonic	Yes  5.1No  4.4	27467	1.151.53	84.9	3.67***
Sociability	Yes  5.1No  4.3	27567	1.071.57	81.6	3.68***
Usability	Yes  5.8No  4.9	27567	1.041.32	87.3	4.92***

DoF – Degrees of freedom

Yes – Completed all the courses they enrolled

No – Did not complete all the courses they enrolled

p<.001;

p<.01;

p<.05

The mean scores for pragmatic experience for students who completed the courses (M =6.2, SD =.96) was significantly higher than for those who did not complete the courses (M=5.5, SD=1.48); t(79.9)=3.89, p<.001. The mean scores for hedonic experience for students who completed the courses (M =5.1, SD =1.15) was significantly higher than for those who did not complete the courses (M=4.4, SD=1.53); t(84.9)=3.67, p<.001. Similarly, the mean scores for sociability experience for students who completed the courses (M =5.1, SD = 1.07) was significantly higher than for those who did not complete the courses (M=4.3, SD=1.57); t(81.6)=3.68, p<.001. Finally, the mean scores for usability experience for students who completed the courses (M =5.8, SD =1.04) was significantly higher than for those who did not complete the courses (M=4.9, SD=1.32); t(87.3)=4.92, p<.001.

Overall, the results support the broader study thesis that students who experience higher levels of pragmatic, hedonic, sociability and usability experiences are more likely to complete the course(s) they registered for compared to those who report lower levels. In other words, these results indicate that people who dropped out had less positive online experience on all the four dimensions—pragmatic, hedonic, sociability and usability. The four-dimensional online experience questionnaire is useful in such situations where one can capture the underlying experience and derive insights on what aspect of the user experience really leads to non completion.

Our analysis also shows that students rated ‘hedonic’ experience and ‘sociability’ experience lower than ‘pragmatic’ and usability experience. For sociability experience, a sizeable number of the students gave a rating of 4 (neutral) on a scale of 1 to 7, which indicates that they did not perceive sociability to be either negative or positive. It could also indicate the lack of sociability experience in this CPHP online program.

There was no statistically significant difference in students’ extrinsic motivation levels between those who completed all the courses they registered for and those who didn’t (See Table 5). There was mild statistically significant difference in students’ intrinsic levels between those who completed all the courses they registered for and those who did not (M=5.2, SD =2.01); t(106.7) = −3.2, p<.05. However, the results from the independent sample t test indicate a negative effect. In other words, students who completed all the courses they registered for had lower intrinsic motivation levels compared to those who did not complete all the courses they registered for. This indicates that lower levels of intrinsic motivation do not imply that they would drop out from the course. On the same lines, higher levels of intrinsic motivation do not imply that they would complete the course. In short, the results from this study indicate that student motivation (both intrinsic and extrinsic) is not a good predictor of course completion.

Table 5

T test results for Motivation and course completion

Motivation	Means	N	Std. deviation	DoF	T value
Intrinsic	Yes  5.2No  6.0	23859	2.011.62	106.7	−3.2*
Extrinsic	Yes  3.8No   3.9	21351	1.451.78	66.7	−.259

DoF – Degrees of freedom

Yes – Completed all the courses they enrolled

No – Did not complete all the courses they enrolled

p<.001;

p<.01;

p<.05

This finding combined with the earlier finding further indicates the importance of students’ online experience (all the four dimensions) for maintaining student interest and ensuring that they complete the courses. In other words, while motivation may play a key role in bringing the student to the program (i.e. enrolling for the course), it is their perceived experience during the online course that critically determines whether or not they would complete the course.

Study Implications

The results from this study have several implications for CPHPs, and more generally, for similar online training programs. First, this study indicates the need to focus on the four key dimensions of user’s online experience (i.e. their underlying feelings and perceptions) rather than on ad-hoc issues. Prior studies and evaluations have mainly considered specific problems perceived by the course provider rather than the actual user experience. The evaluation tool described here brings out users’ sentiments about different aspects of the program and gives a much more fundamental and holistic understanding of the program’s potential weaknesses and areas for improvement.

The study also highlights the importance of hedonic and sociability experience for students in such online training environments. Many online courses focus mainly on the pragmatic value of a particular course for the students and neglect the potential hedonic experience. While pragmatic experience is important and should be the primary focus, boredom and lack of fun can make students weary and de-motivated. Including elements that enhance fun and entertainment as part of the learning experience would be invaluable. Even in face-to-face classroom situations, instructors constantly try to include fun activities such as field trips, role-playing, including videos and movie clips etc that are relevant for the content of the course. The type of fun activities would be different in an online course (a few suggestions are provided in Table 6), but necessary especially for training programs that use ‘canned courses’.

Table 6

Key strategies for improving course completion rates and overall program success

Key Strategies	How to	Benefits for students	Benefits for CPHP
1) Improving pragmatic experience	Expectations management: - make clear what the content of the course is - make it clear upfront who would benefit from the course and who should be taking it. - collect feedback from students at the end of each course on how useful and valuable the course was.	- will not unnecessarily register for a course that they do not need.	- will be able to target courses better at the right individuals - will be able to improve the content of the course
2) Improving hedonic experience	Improve fun and entertainment: - Add more interactive elements in the courses - Include video clips made with ‘Second Life’. E.g. Play2train http://play2train.us/wordpress/ - Include pictures and graphics (pictures speak a thousand words)	- will capture the attention of students - will improve learning and retention of the material. - will see courses as more fun than as a chore. - will keep student engrossed (time flies when you are deeply engrossed). - will keep them from getting distracted.	- will improve the success of the overall program. - will improve student ratings - will be able to attract more students (such online programs don’t have any boundaries, so the potential is immense). - will be able to retain students and get them to come back for more courses.
3) Improving sociability experience	Improve possibilities for social interaction. - Provide an online community/forum for students to interact - Allow students as well as CPHP staff to interact in the community - Offer some courses in ‘blended format’ – i.e. part online and part in-class.	- will improve networking potential - will improve their social experience - will improve learning and retention (collective learning seems to improve information processing) - will feel part of the CPHP community - will not feel that they are on their own - Blended format offers the convenience of online courses but will provide some f2f time that will enhance sociability experience. However, this will be limited to local students.	- will improve CPHP’s relationship with students (strong ties). - will be able to attract more students through ‘word-of-mouth’ marketing (which is a potential outcome of such online communities). - will be able to understand student needs by keeping abreast of the ongoing discussions in the community (instant feedback loop). - online communities have been found to improve motivation as well (Huett et al, 2007) - Blended format will allow CPHPs to improve the variety of courses offered. - It will allow CPHPs to get to know their students better. - Will improve student retention in the local region.
4) Improving usability experience	1) Improve usability experience by using some of the standard usability practices (Nielson, 2000). - Update the websites regularly (at least every 2 years or so if not more frequently) using the latest technologies - Use simple designs (Nielson, 2000) - Remove unnecessary content and avoid clutter. 3) Make CPHP courses accessible through Mobile phonesMobile phones are now very well equipped with fast connections and readable interfaces.	- improved usability would make it easier for students to access the course materials and reduce the learning curve related to the course technologies -Convenience would be the biggest benefit for students. - Beneficial for public health workers who are always travelling. - Very beneficial for people who use public transportation and have lot of time while travelling as well as during wait times.	- improved usability can improve student retention and continued enrollment. - Benefits for CPHP include improving versatility of courses. - Offering courses using more than one platform will improve the reach and enrollment levels - Will improve completion rates and continued enrollment.

As discussed previously, social learning theory suggests the importance of, sociability experience in learning; the current study findings indicate that sociability experience is equally important to ensure higher course completion rates. Good sociability experience prevents students from “feeling lonely”, and more importantly, enables them to engage in “active learning”. Indeed, student interactions have been found to be critical for the success of many online distance education courses [49, 50, 51]. Such interactions allow students to feel that they are part of a community of learners and share experiential knowledge that enhance the overall quality of learning.

Finally, this study found that majority of the students who enrolled in these programs were self-driven or self-motivated. Intrinsic factors such as professional and personal development seem to drive these public health professionals to enroll in such training courses. At the same time, such motivation did not translate into ensuring course completion. This implies that rather than depend on student motivation, course providers would need to provide such self-motivated individuals with a positive and engaging online learning experience to ensure high levels of course completion.

Conclusions and Key Recommendations

Key recommendations that follow from the study findings are given below (also summarized in Table 6).

: In this study, the majority of the students found the courses to be useful and valuable (the mean score for pragmatic experience was higher than those for the other three experience dimensions). However, this is still relative to the very low hedonic and sociability experiences and indicates the potential for improvement. An important means to enhance pragmatic experience is effective student expectations management. Students should be able understand upfront what they will be getting out of each course. This can be done by bringing more clarity to course descriptions and also detailing as to what specific goals students will be able to accomplish by taking each course. It will also help to indicate who would benefit by taking a particular course.

: Hedonic or fun and entertainment from these courses were rated quite low. It is true that fun and entertainment is not one of the primary objectives of these courses. However, as mentioned previously, when people get engrossed in the learning material, their learning and retention of the material are typically much higher [52]. In addition, they would try to finish the courses, instead of procrastinating and/or getting distracted with other things. An effective way to improve hedonic experience is to create more interactive and fun user interface. For example, one could incorporate video clips made in ‘Second Life’ that will give the user a personal view of a disaster and how things could be as he/she approaches a disaster area in addition to being a fun experience. Play2train http://play2train.us/wordpress/ developed by Idaho Bioterrorism Awareness and Preparedness Program using ‘Second Life’ is a good example of this.

: Sociability experience was another weak factor in CPHP courses... As noted previously, positive sociability experience would enable students to feel that they are part of the overall community of students who are enrolled in the program. One solution would be to provide students with access to an online community/forum within the CPHP that will enable interactions with fellow students as well as with the CPHP staff. This would not only improve students’ sociability experience, but also enhance their learning and networking potential, and in turn, improve student retention. Developing such forums is a very cost-effective solution with proven benefits given the low cost of associated information technologies.

: It is important to ensure that the design of the online environment provides seamless and enjoyable navigation experience for the user. Best practices in usability include offering simple and clutter less user interface, intuitive navigational features, and avoiding technological jargons in user guidance. . In addition, usability can be significantly improved by offering online programs on mobile platforms and thereby catering to today’s public health worker who is likely to be very mobile. If courses can be accessed through smart phones (this would require redesigning the interface to fit the mobile device) it would improve the convenience factor significantly.

In conclusion, there are ample opportunities to improve the pragmatic, hedonic, sociability and usability experiences of the target audience. This is critical to improve the participants’ learning and retention as well as the completion rates for the courses offered. Online emergency preparedness programs are likely to play a crucial role in preparing emergency responders at all levels in the future and their success has critical implications for public health informatics.

However, we need more studies in the future to understand the factors that affect students’ overall experience in the online learning environment of CPHP courses. Future research could focus on understanding how the experience (pragmatic, hedonic, sociability and usability) would impact student performance in the courses, student learning and retention of the subject matter, and more importantly, their real life job performance. In addition, conducting qualitative studies with a cohort group of students could help us better understand the factors that shape the overall experience specific to this set of population as well as whether such online training is an effective long term solution for training our public health workers.