Pilot of a questionnaire study regarding perception of undergraduate medical students towards online classes: Process and perspectives.

BACKGROUND: The COVID-19 pandemic situation has forced a shift in medical education from physical classroom to virtual online teaching. However, students and teachers perceive this differently. It is important to study these perceptions to improve the teaching-learning process, and thus to validate the role of online learning in the country.
OBJECTIVE: To document the process of a pilot for a questionnaire-based study regarding perception of undergraduate medical students towards on-going online classes.
METHODS: Medical Education Department of a teaching hospital developed a new questionnaire with the primary objective of studying the satisfaction and usefulness of the online classes (e-Learning). Content validity was done. For the pilot work, 10% of the total student strength was targeted. Random purposive sampling from each phase of the undergraduate course was done to choose the participants. The questionnaire was administered via Google Classroom. It was an external, undeclared type of pilot. All the responses were documented and analysed for both changes in the questions as well as for statistical sample size derivation for the main study.
RESULTS: Responses from 30 students were analysed for the pilot study. Based on the proportion of the level of satisfaction (23.3%) and usefulness (23.3%) of the on-going online classes observed, and with 20% relative precision and 95% confidence, the minimum sample size for the main study was calculated. The responses revealed the need for minor changes in the questionnaire tool for overall feasibility and achievement of the objectives of the main study.
CONCLUSIONS: A pilot study is a necessary component for a research project, especially when it involves the use of a new tool. This paper shows the relevance of the same. The authors intend to convey the importance of documenting the processes of conduct of a pilot study, the issues involved therein, and the steps taken to resolve the same. Copyright:

Introduction

The UNESCO estimates that around 32 crore college students are affected in India as all educational activities have been halted due to COVID. This has worked as a catalyst for the educational institutions to adapt to the changing times, and try platforms and technologies not earlier used. What has thus emerged is the liberal use of online delivery of classes. The teachers assigned work to students via the internet, delivered lectures video conferencing using different Apps like Zoom, Google meet, Facebook, YouTube and Skype, etc.[1] The situation is no different in the field of medical education. The use of emergent technology for education, such as artificial intelligence for adaptive learning and virtual reality, is very likely to be essential components of the transformative change and the future of medical education. A major challenge for medical teachers in the present times is to replicate the experience of clinical exposures. At present, videos, podcasts, simple virtual reality and computer simulations are beginning to be used to assist teachers facilitate student learning and training in these areas. Simple online platforms, such as websites and blogs, provide basic information and also offer opportunities to host videos for demonstrating essential procedural clinical skills and communication.[2] With the increased use of technology in medical education, there is a need for teachers/learners to develop and implement innovative solutions to keep pace with the present demand.

Online learning has been reported to help students attain better understanding of the subject and skills with easy accessibility and flexibility. However, lack of interaction with the teacher and in-depth group discussion for clarification of concepts are disadvantages. Students have hitherto preferred e-learning as a supplemental tool rather than replacement one, whereas teachers have perceived e-learning to save time in editing and updating of content without worrying about variation in content delivery.[3] This has been offset by the concern and mistrust of technology especially by senior faculty. In India, prior to COVID pandemic, most of the e-learning activity in the medical colleges was limited to data search for thesis and research work. Presently e-learning in medical education in India is growing at a good pace.[3] There is opinion that conventional teaching should be supplemented with computer and web-based e-learning.[4] Studies suggest that e-learning should be further utilized in a blended learning environment to support face-to-face teaching and allow for flexibility in delivery of medical education.[5]

The challenge before medical educators is to use these new technologies effectively to transform learning into a more collaborative, personalized and empowering experience.[67] Communication and care are the two categories identified for the use of mobile technology. Communication could be between physicians and patients, physicians and healthcare workers or amongst physicians themselves. Care of the patient can be provided through newer mobile devices using software applications. They can provide diagnosis and treatment through patient specific analysis.[8] This will be useful for healthcare providers practicing in rural, remote and primary care settings in the future to improve a range of professional competencies viz. attitudes, knowledge, skills, behaviours among healthcare professionals.[8]

The National Medical Commission has put forth a module on the online learning and assessment with due guidelines for teaching procedural skills. It recommends the usage of online instructional videos taught either as small groups or through break rooms with apt discussions. Training kits, mannequins and online screen based virtual reality simulators have also been advised depending on availability.[9]

We, as faculty members of the Medical Education Department (MEU) of a teaching hospital, felt the need to study the success of the new schedule, specifically with respect to the students' perception of the classes. The primary objectives were to analyse the satisfaction and usefulness, as perceived by the students. The easiest modality of survey being a questionnaire, this was developed, the details of which have been published.[10] This was followed by the content validation and pilot study.

Pilot study has been defined in many ways. As per the Collins English Dictionary, it is “a small-scale experiment or set of observations undertaken to decide how and whether to launch a full-scale project”. More to the context, the Association for Qualitative Research defines it as “a small study conducted in advance of a planned project, specifically to test aspects of the research design (such as stimulus material) and to allow necessary adjustment before final commitment to the design”.[11] These statements are indicative of the importance of the “pilot” in research studies or scientific projects. Many online resource materials and literature provide reasons and guidelines for conducting pilot studies.

Pilot studies, although frequently done in research, are rarely documented separately in published literature. This aspect has been detailed by Malmqvist et al.[12] In the present article, our endeavour has been to describe the methodology and utility of the pilot work of a new questionnaire tool administered in a single institution, with the aim of augmenting the literature on this aspect of research.

Why a Pilot?

The institution from where the present study is being reported is a Teaching Hospital under a Deemed University, with an intake of 100 undergraduate students every year. There are four phases in the medical curriculum. So, any student questionnaire survey would, by default, have around 400 as the sample size. However, two issues presented minor hurdles. At the institutional research proposal meeting, it was pointed out that as we had developed a new questionnaire, there had to be some sort of validation. It was also suggested that the sample size calculation on the basis of a pilot would add statistical power to the study.

This motivated the authors to do a literature search. Quite a few advantages of, and indications for, doing a pilot study have been documented.[1314151617] Table 1 summarises the list of advantages Table 1.

Table 1

Summary of advantages and indications of pilot study

Helps define and/or refine the research question

Tests the adequacy of research instrument/tool

Allows decision making about sampling and recruitment strategies

Enables testing of the proposed design, and modality of the study

Checks the study feasibility, and detect possible hurdles

Provides preliminary data

Tests safety of treatment outcomes in clinical trials

Enables training of research assistants

With regards to a new questionnaire tool, there are some specific indications for conducting pilot, as put forth by McLeod.[18] These include the need to check clarity of terminologies used in the questionnaire, check the appropriateness of questions with respect to the study target group, exclusion of leading questions and to ensure that the questionnaire can be completed in a reasonable time frame.

So, the decision was taken to do a pilot work with our new questionnaire, with three specific objectives:

To calculate the sample size for the main study

To check whether the new questionnaire tool would achieve the intended purpose

To check feasibility of the conduct of the main study

Methodology

The proposal of the pilot work along with the main work was presented in the IRB (Institutional Review Board). The Scientific Committee approved the work and gave directions to begin the pilot work. This study was exempted from Ethics Committee review as it belonged to one of the exceptions viz. “Comparison of instructional techniques, classroom methods and curricula”, in accordance with Section 4 (sub-section 4.8) of the ICMR (Indian Council of Medical Research) Guidelines for Biomedical Research.

The first step was validation of the questionnaire tool. It was a 20 item tool developed specifically for studying perception of the undergraduate students towards the on-going online classes. Face validity of the tool was done by experts in the field, drawn from the medical education department and curriculum committee members of the institution. They concurred that the questions were having a logical link with the study objectives. For content validity, the questionnaire, along with the study plan and objectives, was sent to seven members who were not directly involved in the tool development. They were asked to rate each item (question) on a scale of one to five under three criteria: Alignment with objectives, clarity of framing and language. Thus, each item would merit a maximum score of 15, and each criterion, 100. The submitted rating showed that no item score was less than 13 (86.67%). In the tool, 16 items rated >90% (average >13.5). Content validity [Table 2] was thus established. Criterion validation was not deemed applicable for the present topic, being a subjective phenomenon, and construct validity was not possible as there was no established instrument to be correlated with our new tool.

Table 2

Scale rating scores of the items/questions for content validity

Content type	Evaluators							Mean
	1	2	3	4	5	6	7
Alignment with Objectives	99	91	82	94	97	97	95	93.57
Clarity of framing	97	90	81	97	93	92	92	91.71
Language	95	98	85	100	91	100	92	94.43
Question 1	15	15	11	12	13	14	13	13.28
Question 2	15	14	10	15	15	15	15	14.14
Question 3	14	13	15	11	14	14	12	13.28
Question 4	14	14	15	15	14	14	15	14.42
Question 5	15	14	12	15	15	14	15	14.28
Question 6	15	12	11	15	15	13	15	13.71
Question 7	15	15	14	15	15	15	15	14.86
Question 8	14	12	11	15	15	15	14	13.71
Question 9	15	14	12	15	14	15	15	14.28
Question 10	14	12	13	15	15	15	15	14.14
Question 11	15	14	15	15	12	14	15	14.28
Question 12	13	14	11	15	14	13	12	13.14
Question 13	13	15	10	15	13	15	15	13.71
Question 14	15	15	15	15	13	15	12	14.28
Question 15	15	14	13	15	15	15	15	14.57
Question 16	15	15	12	15	13	15	12	13.85
Question 17	14	14	11	15	13	15	12	13.43
Question 18	15	14	11	15	15	15	15	14.28
Question 19	15	14	13	14	14	14	12	13.71
Question 20	15	15	13	14	14	14	15	14.28

The second step was to conduct the pilot work. The target group was chosen from the study population itself, as the questions were very specific to students' experience. A decision had to be made regarding the number of students to be involved in the pilot. Literature search provided multiple options, some feasible, some not so. Tsang et al., in an article on questionnaire validation, suggest that it is “advisable to test the tool on a small sample before conducting a pilot test”.[19] The number cited in this article is in the range of 30–50. The lower limit of this range was acceptable for the present work, but not the suggestion of doing a pre-pilot sampling, as the same benefits could be derived from this planned pilot itself. Two other publications cited feasible numbers. Connelly LM cites experts' recommendation “that a pilot study sample can be 10% of the sample projected for the larger parent study”.[15] Perneger, et al. suggests that “Thirty participants is a reasonable starting point for pre-tests of questionnaires”.[20]

With this background information, it was decided to include 10% of the total population size, viz. 40 students. Discussions with the biostatistician indicated that 30 would suffice. This meant that any attrition in the pilot group could be covered. The selection of students was done via equal representation from all the phases; approximately 10 from each phase. As a form of random sampling, we decided to take multiples of the roll number eight (8, 16, 24 etc.).

As a prior intimation and to ensure connectivity, the class representatives of each phase were informed about the study plan. A Whatsapp™ group of the selected participants and authors was created, so that all were in the loop. The students were informed about the course of research work, with instruction to respond within a week's time. After setting the tone for the study, the administration of the questionnaire was done. An introduction about the study was placed at the beginning, which included consent too.

Mode of administration was via the Google Classroom™. This portal was deemed suitable and convenient for the given number of participants. All the investigators were given access in this platform, so as to reply to any clarifications sought by the respondents and to keep tab of the responses. These were documented, followed by submitting the same to the statistician. There was also a post-pilot meeting of investigators to discuss any changes to be made in the tool and/or administration arising from the findings of the pilot.

Results

A total of 31 students responded in the given time frame, of which one refused to give consent. This gave us 30 pilot respondents; nine were from first year, nine from second year, five from third year and seven from final year (Phases I to IV). There were 20 females and 10 males. All the respondents agreed that it took less than 15 min to complete the questionnaire.

Among the questions, two were chosen to specifically represent the primary objectives of the study, related to perception. These were used for calculating the sample size. Based on the proportion of level of satisfaction (23.3%) and usefulness (23.3%) of the on-going online classes observed in these responses, and with 20% relative precision and 95% confidence, the minimum sample size was computed as 316. This was feasible, as there would be 370 respondents in the main group.

The following issues were noticed with regard to technicalities and logistics:

A question was not answered by more than 20 respondents. On re-trialling amongst the investigators, it was noticed that this question had not been marked mandatory. So, it was intentionally or otherwise, skipped.

One among the two questions that were selected by the statistician to calculate the sample size was misplaced with the baseline set of questions in the tool.

Two questions with free text response had minor lack of clarity in responses.

There was an unpredictable period of waiting for individual responses.

Technical aspects of the Google Classroom™ were new and unfamiliar to few of the authors.

Accordingly, relevant corrections were made to the questionnaire, including re-framing of one of the free text response questions, re-ordering of one question and marking all the questions as being mandatory. Administration of the tool for the main study was decided to be done utilizing Google Forms™.

All the responses were scrutinised by the investigators. This showed that the objectives of the main study would be met with this questionnaire. Two questions were specifically framed keeping in mind the constructs of satisfaction and usefulness. Both received a majority of decisive responses; 23, and 25, out of 30, respectively. One question was to give an overall score of satisfaction, on a scale of 1 to 10. Majority of the responses were in the middle scoring range, with only four on either extreme [Figure 1].

Figure 1

Responses to three questions related to the main objectives of the study

Discussion

Baker has referred to a pilot study as “pre-testing or 'trying out' of a particular research instrument”.[21] Pilot works serve important purpose in academic studies, especially in the medical field. However, very few pilot studies are reported as standalone publications. Few articles have commented on this and suggested reasons. These include publication bias, results focusing only on statistical outcome and the design itself being unclear.[2223] A comprehensive tutorial on pilot studies has stressed the importance of making every attempt to publish these, with valid reasons.[24] This article is an attempt to publish the data of our study to correct the existing anomaly in literature.

Pilot studies have been classified based on organization, into “internal” and “external”; and based on respondent participation, into “undeclared” and “participatory”.[14] The present pilot was external and undeclared type, i.e., it was administered to a small group of participants who were then not included in the main survey, and as if it was the actual full-scale survey. Internal pilot mandates thorough planning at the outset, and the awareness that no change can be made after the pilot.[25] Our intention was to check the responses and to calculate the sample size, thereby not necessitating a participatory type of pilot work.

Analysis of responses of the pilot showed that the primary objectives of the main study could be met. There was adequate distribution of respondents from all the phases, indicating that expectance of a similar participation in the main study. The overall conduct of the pilot also showed the feasibility of the main study that was being planned along similar lines. These are the primary aims of a pilot study as stated by Hassan et al.[17] We could also obtain a feasible sample size, thus helping to attain the statistical power needed for reporting the findings of the main study. In addition, with the input of the statistician, three questions were earmarked for the statistical analysis of the main study, and a grading system (low, moderate, high) was decided for the responses to these questions. One article has cautioned about the possible pitfall in estimating sample size for the main study from a pilot, due to the wide confidence interval.[23]

The various possible interpretations of results of a pilot study have been detailed by In J., as follows14: 1) termination, i.e., cannot proceed with main study, 2) can proceed with the main study after modifying its design, 3) not necessary to modify the study design, but requires thorough monitoring throughout or 4) can proceed without any modification. In the present work, we could proceed (to the main study) with some modifications in the study design, namely minor changes in the questionnaire and administering modality. An article on pilot study in qualitative inquiry (Kim Y.) reported four ways, in which, implementation of a pilot proved useful.[25] One of these was the modification of the interview questions, which was applicable in the current study.

Ultimately, the fact that the pilot and the subsequent main study is conducted for a single institution makes it a limitation to generalize the results obtained and extrapolate to larger groups of students and/or institutions. A similar concern was cited by Fraser et al.[16]

The entire process has been summarised in a flow chart [Figure 2].

Figure 2

Flow chart of the entire pilot process

Conclusions

The pilot work for a study on students' perceptions towards online classes, using a new questionnaire, was planned and conducted in a stepwise fashion as recommended in literature.[13]

The results from our pilot study clarified the fact that the proposed main study, with intended objectives, will be feasible, after making minor changes in the questionnaire tool.

Calculation of the sample size could be achieved after analysing the responses from the pilot study.

As faculty members of the Medical Education Department, we felt that it is worthwhile to publish the details of this pilot study. The investigators wish for other colleagues and/or institutions to take up similar research work. We also hope that this encourages more studies looking into the effectiveness of online learning and online healthcare delivery in rural and primary care areas of our country.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.