Online survey software as a data collection tool for medical education: A case study on lesson plan assessment.

Background: There are no general strategies or tools to evaluate daily lesson plans; however, assessments conducted using traditional methods usually include course plans. This study aimed to evaluate the strengths and weaknesses of online survey software in collecting data on education in medical fields and the application of such softwares to evaluate students' views and modification of lesson plans.
Methods: After investigating the available online survey software, esurveypro was selected for assessing daily lesson plans. After using the software for one semester, a questionnaire was prepared to assess the advantages and disadvantages of this method and students' views in a cross-sectional study.
Results: The majority of the students (51.7%) rated the evaluation of classes per session (lesson plans) using the online survey as useful or very useful. About 51% (n=36) of the students considered this method effective in improving the management of each session, 67.1% (n=47) considered it effective in improving the management of sessions for the next semester, and 51.4% (n=36) said it had a high impact on improving the educational content of subsequent sessions. Finally, 61.4% (n=43) students expressed high and very high levels of satisfaction with using an online survey at each session.
Conclusion: The use of online surveys may be appropriate to improve lesson plans and educational planning at different levels. This method can be used for other evaluations and for assessing people's opinions at different levels of an educational system.

Introduction

Evidence indicates that an educational system is capable of achieving its objectives when it has optimal educational quality. Accordingly, finding appropriate ways to improve the quality of education is clearly necessary. Evaluating educational programs at various levels ensures quality and creates academic standards. Quality is mainly defined by customers’ opinions. Therefore, one way to evaluate the quality of educational programs is to investigate the viewpoints of the learners at different levels of educational programs because learners are the first and the most important customers of educational systems (1-3). Teachers use daily lesson plans to plan and organize a group of activities in relation to educational objectives and content as well as the students’ activities. A lesson plan divides the lesson content into appropriate, defined levels and steps to be taken in a given period based on the objectives and desired educational outcomes. This tool plays a key role in shaping education (4).

A variety of tools and techniques can be applied for evaluation. Traditional methods like paper questionnaires have some disadvantages such as high costs. Moreover, much time is needed to design, reproduce and distribute the questionnaires as well as data collection and analysis (5).

Nowadays, established innovations in information technology and the internet are useful tools for collecting and distributing information in all industrial and service organizations (6). Medical professions are increasingly dependent on electronic information (7), and selecting the appropriate information depends on accessing the desired information in the fastest and most appropriate way (8). It seems that one main mission of educational systems is to use these innovations to improve the quality of content and educational programs (9). One such method is to use online survey software to evaluate and modify programs (10,11). The online survey method is a useful tool for collecting data quickly. This survey can be performed in two ways: a) e-mail-based methods and b) web-based methods, each of which has its own advantages and disadvantages (12). Web-based methods allow the direct guidance of respondents to a uniform resource locator (URL). Other benefits include saving in time and costs, ease of use, the ability to prevent errors when entering and editing data, and rapid transmission of survey results (13,14).

To be used in education, online survey softwares should ensure students’ privacy, provide convenient URLs so that students have easier access to the system, have multiple-choice questions, and provide students the opportunity to openly send their comments (15). According to various studies, there are many reasons to use such softwares, some of which are as follows: The ability to review large samples; limited available financial and human resources; the capability of the internet to access groups and individuals who are difficult or impossible to access through other ways; the ability to survey people who have physical problems or particular diseases; the ability to survey people who have concerns about meeting in person; the ability to use templates to evaluate the satisfaction of customers, students, professors, and so on; the ability to export data from the software to other analytics softwares such as SPSS; the ability to prevent errors when entering and editing data, as well as the rapid transfer of survey results (14-16). Despite these advantages, online surveys also have several disadvantages: They may be presumed to be spam or affected by computer problems, and they are hindered by the lack of experience and expertise of the participants in the use of online softwares (16).

However, various studies have shown that students prefer online surveys. One study of 1,000 students revealed that they preferred an electronic evaluation option, because they felt it was more confidential, easier, and had less time pressure (17). According to other studies, using online survey tools for evaluation eliminates many of the barriers of traditional methods such as the problem of data entry, analysis, and data management (15).

Online tools have also been used in medical fields to investigate students’ viewpoints. Some studies have shown that a simultaneous and online assessment during a semester eliminates a lot of trouble with evaluations for medical students at the end of a semester (18). Online survey tools have been used by the Residents Review Committee to assess the programs of internal medicine residents in southern California in 2010. In this survey, questions were designed to assess clinical rounds, residents' attitudes about working hours, workload, and how to improve patient care and hospital services. In this study, about 1,000 sheets of paper and 50 hours of runtime were saved in an annual holding. SurveyMonkey software was used in this program. The results revealed that the quality of responses to feedbacks from residents improved through faster responses to their comments. Residents stated that web survey was easier to use (5).

Evaluations are usually done at the end of a semester, and thus professors do not have an opportunity to get feedback from students. Moreover, delayed assessments may not be accurate because of the passage of time. Evaluations made during a semester while a course is being taught removes many of the problems of end-of-course evaluations such as a reluctance to complete questionnaires and forgetting the details of the course. Forgetting a subject has a particular effect in cases of having multiple professors for one course and in cases of professors who teach in the early sessions of a semester. These professors are deprived of timely feedback from their students. This is especially evident in medical sciences where oftentimes several professors are responsible for teaching one course (18).

According to the results of the initial investigation, lesson plan assessments are often performed at the end of a course and with traditional methods that expend a lot of money, time, and manpower. The objectives of this study were to examine the strengths and weaknesses of online survey software in collecting data for a lesson plan assessment and to examine students' views on it.

Methods

Description of the Selected Online Survey Software

Approximately 300 online survey software products are listed in www. websm.org, including SurveyMonkey, Zoomerang, Free Online Survey, and esurveypro (14). Each of these four tools has its own strengths and weaknesses. After reviewing the available literature and considering the limitations of available tools in terms of payments and the use of full functionality of some of these tools, the free version of esurveypro software was selected for use to assess the daily lesson plans of various courses in the fields of health information technology and medical informatics in Mashhad and Iran Universities of Medical Sciences. The free version of this software does not need any specific permission to use. This study was approved by the Ethics Committee of Mashhad University of Medical Sciences (Code: IR.MUMS. REC.1394.752).

Some of the most important features of this software include unlimited and varied surveys, the capability of designing and selecting different types of questions (e.g., multiple-choice, open-ended, etc.) and their unlimited numbers, the capability of editing the structure and content of the questions, the capability of creating and managing e-mail contact lists (for follow-up e-mails), the capability of simultaneously collecting and analyzing responses, and the capability of reviewing the details of answers and creating summary reports (19). This study was conducted in two phases as described below.

Phase 1: Developing the Survey of Students’ Attitudes towards Daily Lesson Plans

After determining the objective of the survey and formulating questions for evaluating daily lesson plans, an exclusive page for the assessment process was created on the esurveypro to use the online survey software and to create questionnaires. This software offers the capability of choosing the title and providing explanations below the title for each specific survey in this section. Using the “questionnaire designers”, one can also add a logo and choose the desired color and structure, apply security restrictions, including passwords for access to any of the questionnaires, as well as add unlimited numbers of various types of questions. Other features available in the “design section” are the capabilities of adding an obligation to respond to an intended question, specifying the maximum and minimum responses for each question, and editing, deleting, moving, or copying any of the questions. Moreover, the capabilities to add restrictions to respond to questionnaires, editing, viewing incoming responses, sending a questionnaire to collect information, deleting a questionnaire, and receiving responses at any time are available in the “administration section”. There is also the capability of sending the questionnaires to specific individuals or creating lists of email addresses and managing them in the “invitation administration” section.

In this study, the questionnaire was designed by the software. Validity was assessed by two medical education specialists, two health information management specialists, and one medical informatics specialist who provided their feedbacks on the format, clarity, and meaning of the questions and response options. Their suggestions were applied. Reliability was assessed by the test-retest method prior to the main study. Ten students were included in a pilot study conducted within two weeks. They were also invited to comment on the clarity and comprehensibility of the questionnaire. Reliability was 80%. Appendix 1 shows that the different surveys were developed for different classes, and Appendix 2 shows some of the questions developed for evaluation of each session of the classes.

The students gave their consent verbally to participate in the study. Next, a list of student email addresses was created in the “email list” section of the software. The questionnaire was sent to students who have willingness to participate in the study. At the end of each session, the URL address of the mentioned questionnaire was sent to the students by the software. Responses collected by the website were given to the related professor in the form of detailed and summarized reports so the professors could modify their next daily lesson plans for the same period and the subsequent periods.

The questionnaire to assess students’ attitudes towards daily lesson plans included the following topics: Observing the continuity of the taught contents at each session; providing clear and concrete examples of the contents of each session; permission for students to participate in the discussions; mastering the subject of the session and the ability to answer questions; taking time to answer students' questions; using teaching aids; asking questions and providing opportunities for students to think; respect for students; appropriate communication and interaction with students; creating enthusiasm and interest in research and further study on the teaching subject in that session; observance of the timely presence of professors and class time; paying attention to regular attendance and the participation of the students; and the appropriateness of the session content with the course plan offered at the beginning of the semester.

Phase 2: Students’ Attitudes towards Advantages and Disadvantages of using an Online Survey Software

After the software had been used for one semester, a questionnaire was designed to assess the advantages and disadvantages of this method. The students’ attitudes were evaluated in a cross-sectional study, using the online survey software. The questionnaire consisted of two parts: a) close-ended questions in which students should provide their opinions, using a five-point Likert scale, b) two open-ended questions in which students were asked about the strengths and weaknesses of using the software to evaluate lesson plans. Validity and reliability (78%) were assessed similar to the previously-discussed questionnaire. Appendix 3 displays how this questionnaire was developed in the software.

The research population consisted of undergraduate students in four classes of health information technology at Mashhad University of Medical Sciences (70 students), three classes of master’s students in health information technology, and one class of master’s students in medical informatics at Iran University of Medical Sciences (25 students), of which 70 responded to the questionnaire (response rate =73%).

Descriptive statistics using available features in the software itself was reported for data. Students’ responses to open-ended questions were analyzed, using a conventional content analysis. Main themes were identified based on the students’ comments.

Results

In general, results revealed that 57.1% (n=40) of the students believed that evaluating a class per session (daily lesson plans) was useful (high and very high), and 34% believed this evaluation to be partly useful.

As demonstrated in Table 1, only 11.5% (n=8) believed that the software application was not useful for evaluating daily lesson plans, and only 9% did not believe this method to be easy. Respectively, 51.3% (n=36) and 51.4% (n=36) of the students believed that the application of this method was highly and very highly effective in improving the management and educational contents of the subsequent sessions. Of the students, 67% and 65.7% (n=46), respectively, believed that this method was useful in improving management methods and the content of that lesson in the subsequent semesters. Eventually, 61.4% (n=43) of the students expressed high and very high levels of satisfaction with using online surveys in each session.

Table 1

Students’ Viewpoints on the Use of Online Survey Softwares in Evaluating Daily Lesson Plans

Items	very HighN (%)	HighN (%)	Partly HighN (%)	LowN (%)	Very LowN (%)
Software’s usefulness in the evaluation of daily lesson plans	17(24.2)	24(34.2)	21(30)	8(11.4)	0(0)
Use of online assessment being better than traditional paper methods	28(40)	26(37.1)	12(17.1)	3(4.2)	1(1.4)
Use of an online survey at each session being simple and feasible	18(25.7)	24(34.2)	22(31.4)	4(5.7)	2(2.8)
Improving management method of subsequent sessions using an online survey at each session	12(17.1)	24(34.2)	28(40)	4(5.7)	2(2.8)
Improving educational content of subsequent sessions using an online survey at each session	14(20)	22(31.4)	25(35.7)	2(2.8)	1(1.4)
Improving management method of sessions in subsequent semesters using an online survey at each session	19(27.1)	28(40)	16(22.8)	1(1.4)	1(1.4)
Improving educational content of sessions in subsequent semesters using an online survey at each session	19(27.1)	27(38.6)	17(24.2)	5(7)	2(2.8)
Overall satisfaction of the students with using an online survey at each session	14(20)	29(41.4)	21(30)	5(7)	1(1.4)

Table 2 demonstrates that more than 65% (n=46) of the students believed this method was suitable for class evaluation at the end of a semester, 61.3% (n=43) believed this method was suitable for class evaluation by students during a semester and were willing to participate in mid-term evaluations and 75.6% (n=53) believed this software was extremely easy to use.

Table 2

Students’ Viewpoints on Using Online Survey Softwares in Other Educational Evaluations

Items	Very HighN (%)	HighN (%)	Partly HighN (%)	LowN (%)	Very LowN (%)
Online evaluation method’s suitability for evaluating professors at the end of a semester	21(30)	25(35.7)	19(27.1)	5(7)	0(0)
Online evaluation method’s suitability for evaluating student learning during the semester	12(17.1)	31(44.2)	17(24.2)	6(8.5)	2(2.8)
Willingness to participate in evaluation programs during semester using online survey software	17(24.2)	26(37.1)	23(32.8)	3(4.2)	1(1.4)
Use of online survey software for educational evaluation being simple and practicable	36(51.4)	17(24.2)	17(24.2)	4(5.7)	1(1.4)

Based on the students’ answers to open-ended questions, the main themes were extracted. These themes and some students’ comments are shown in Table 3. The main strengths and benefits were as follows: Improved performance of the teachers; quick comments and feedback; considering students’ opinions; improving the learning experience of the students; quick evaluation, and improving student participation. In addition, the main weaknesses were accessibility; need for more questions in evaluation; limited time; and concerns about privacy.

Table 3

Main Themes Extracted on Strengths and Weaknesses of Online Surveys

	Themes	Students’ Opinions
Strengths	Improved performance of teachers	“This method will improve the quality of educational sessions.”“This method will make professors change their teaching method in ways that will be beneficial to students.”“It will improve the educational content.”“Providing feedback from questionnaires completed by students at each session to professor and solutions devised by professor and collaboration with students can be helpful.”
	Quick comments and feedback	“If there is a weakness in education, the relevant professor can be quickly informed through this software.”“Questions can be answered quickly and easily and the ability to express opinions about details exists.”“Its positive point is the fast transfer of comments and feedback of the students from the classroom who may not be able to provide face-to-face comments.”
	Considering students’ opinions	“Using this method will be very useful if professors pay attention to the opinions of the students.”“Questions can be answered quickly and easily and the ability to express opinions about details exists.”“Professor can evaluate his/her performance by considering the views of the students in each session and understand his/her strengths and weaknesses in different subjects.”“Its positive point is the fast transfer of comments and feedback of the students from classroom who may not be able to provide face-to-face comments.”
	Improved students’ learning experience	“Passion and focus of the students to learn will increase when they are allowed to participate in the planning and presentation of educational content.”“Use of new teaching aids and familiarity with these methods is necessary for the new generation.”
	Improved students’ participation	“Passion and focus of the students to learn will increase when students are allowed to participate” .“Providing feedback from questionnaires completed by students at each session to the professor and solutions devised by the professor and collaboration with students can be helpful.”
	Quick evaluation	“We will have more presence of mind and our attention at meetings will increase because this evaluation is done at the end of each session.”
Weaknesses	Accessibility	“Sometimes there is no Internet access for students.”“Email service provider server problems and lack of access to email for off-campus students.”
	Need for more questions in evaluation	“Given that questions about each session are repetitious, answers will be similar to each other.”“It is better that questions about a professor’s behavior in each session and the amount of learning and student's attention also be asked.”“Using a common questionnaire is not very useful for evaluation of all sessions; thus a little change can be made in the questionnaire according to each session.”“In this form, it is better to consider some questions in the framework of issues in that session so that the level of a student’s understanding of the course content is measured.”
	Limited time	“Filling out the questionnaire and commenting about each session leads to a waste of time.”“There are too many questions, and answering them needs patience.”“In my opinion it is better to carry out the survey every few sessions instead of every session.”
	Privacy	“Students may think that they will be identified from their IP or email address and may not provide their full opinion in all cases.”

Discussion

In general, the findings of this research revealed that the majority of the students believed that using online survey tools and methods are useful and suitable for improving lesson plans. Given that the main mission of educational institutions is to increase the quality and use of new technologies in this regard (9), the advantages and difficulties in implementing this technology should be considered from students’ perspective.

Results revealed that the majority of the students believed that using this method to evaluate lesson plans was easier and better than traditional methods, and most of them were more satisfied with the online evaluation method. These results correspond with the results of previous studies on the subject (5,20). In a study by Anderson, about 90% of the students agreed or fully agreed to make online evaluation a priority over traditional methods (21). Saving paper and time are other benefits of this method form the students’ perspective (5).

One of the main students’ concerns was the confidentiality of their responses and the possibility of being identified by professors. According to Duncan, ensuring students of the confidentiality of their responses is one of the important principles of applying online survey software (15). Fleming et al. also considered this matter to be one of the disadvantages of and a challenge to this type of software (16).

Despite the fact that internet access in the world is leading to the increased use of online surveys (22), students’ comments in response to open-ended questions showed that a major issue is a lack of adequate access to the internet. This shows that lack of access to the internet and the lack of an appropriate infrastructure in developing countries still create problems in the use of web applications. Timely access to information is one of the success factors of information systems (23).

Some technical problems such as difficulties in accessing university e-mail accounts outside of the college and e-mails being delivered to junk mailboxes were mentioned by students as weaknesses of this method. In previous studies, students have also mentioned issues such as lack of access to the internet and technical problems as barriers to participate in online surveys (21).

The majority of the students believed that using online surveys to evaluate professors at the end of a semester and during a semester is useful, and most of them intended to participate in evaluations during a semester using this method. Students believed that using this method to evaluate professors during the semester is suitable and believed that with this, professors can receive timely feedback about conditions of classes and students. In a 2010 study, performing evaluations at the end of and during a physiology course were assessed. A web-based system was used to evaluate students’ viewpoints. In this study, medical students had access to a web-based system from the first session for the evaluation of physiology lessons. Sixteen professors were evaluated. Students were able to change their opinions about a professor and his/her method of teaching during the semester. Evaluations were performed two weeks after the end of the course with the same system and the same questions, and 26% of the students participated in evaluations during the semester and 65% participated in evaluations after the semester. During the semester, 38 students submitted 305 evaluations, and 82% sent comments along with answers to the close-ended questions. Only 62% of the students who participated at the end of the semester provided comments. Each professor received an average of 5.6 comments in the simultaneous evaluation method, but only 3.1 in the other method (p<0.05). Comments of the simultaneous evaluation group were longer and clearer in terms of text (p<0.05) and had more qualitative information. The average score given by those who participated in simultaneous evaluation were significantly higher at the end of the semester than the scores given by students in the other group. Those students who did not participate in either of these evaluations gave lower scores than the other two groups. Those who sent more comments had a higher average score. Despite all of this, the evaluation method did not influence the scores given to the professors by the students (18).

One limitation of this present study was the reluctance of some students to participate in the study; only 73% of them participated in final evaluations. However, this was also considered as one of the challenges of using online surveys. Differences in response rates to web-based survey and other survey modes indicate that, on average, web-based surveys are used approximately 11% less than other evaluation modes (14). Additionally, in this research we used only one online survey tool, which might have affected the findings. Application of other tools is highly recommended in future studies.

Conclusion

According to the findings, using online surveys can be suitable for modifying and improving lesson plans. Softwares can be used to execute other evaluations and review the comments of people in different levels of an educational system. Furthermore, the use of these tools can influence decision-making patterns and change it to student-oriented models. This method and the software features can be used to evaluate other parts of educational systems such as evaluation of the learners in the form of diagnostic (prior to beginning the course), formative (during the course), and final (end of course) evaluations, evaluation of the viewpoints of the learners regarding different teaching methods, evaluation of the viewpoints of professors and colleagues about different methods and training courses, and so on.