"What do Ayurveda Postgraduate Entrance Examinations actually assess?" - Results of a five-year period question-paper analysis based on Bloom's taxonomy.

BACKGROUND: The standards of Ayurveda education in India are being questioned in the recent years and many suggestions related to educational reforms are being put forth by educators and health policy experts. However, the Post Graduate Entrance Examinations (PGEEs) that are carried out to select the candidates to pursue postgraduate programs have received little attention in this context.
OBJECTIVES: The objective of this study was to classify the Multiple Choice Questions (MCQs) from Ayurveda PGEEs conducted in different universities of India during the five year period (ranging from 2010 to 2014) into six levels of Bloom's Taxonomy in cognitive domain.
METHODS: This is a retrospective observational study. The sampling method followed was purposive sampling. Totally, 3299 MCQs obtained out of 25 question papers from seven universities spread across four zones of India (North, South, West and East) were included in the study and were classified based on the Bloom's taxonomy.
RESULTS: About 93.3% of MCQs assessed only the 'recall' component whereas 6.2% of the MCQs assessed 'comprehension'. Percentage of MCQs that assessed 'application' level was a mere 0.3% whereas the percentage of MCQs that assessed the 'analysis' component was found to be only 0.2%. There was not even a single question to assess the 'synthesis' and 'evaluation' components.
CONCLUSIONS: We conclude that an appropriate proportion of MCQs assessing 'higher order thinking' are required to be included in Ayurveda PGEEs. While it is possible to frame MCQs to assess all six levels of Bloom's taxonomy in cognitive domain, the teachers are required to be trained well in the skills of MCQ writing. We propose that our study may be taken as a lead to introduce the required reforms in PGEEs. Clinical Trial Registration No.: Not applicable.

Introduction

A postgraduate medical entrance examination (PGEE) is the qualifying examination that is offered to the medical graduates willing to get enrolled in postgraduate programs. Any PGEE that aims at selecting candidates with highest academic and professional excellence must incorporate different measures to evaluate the knowledge (cognitive domain), skills (psychomotor domain) and attitudes (affective domain) of an aspirant. Various methods of assessment have been employed from time to time by institutions and universities across the globe to fulfill these expectations [1]. In India, however, thinking abilities of the candidates in the cognitive domain (knowledge) alone are mostly evaluated in all kinds of PGEEs. Multiple Choice Questions (MCQs) are the usual kinds of items that are presented in these examinations to accomplish this objective. It is generally presumed that MCQs, if framed carefully, can assess one's thinking skills in all the six levels of knowledge domain, namely, 1. Recall, 2. Understanding, 3. Application, 4. Analysis, 5. Creation (Synthesis) and 6. Evaluation [2]. These levels actually represent the degrees of difficulty: i.e., the lower levels being simpler and higher levels being more complex and difficult. This categorization of levels from lower order to higher order in the knowledge domain is one of the vital components of Bloom's taxonomy. Bloom's taxonomy of educational objectives happens to be one of the most often used models while designing the different training, learning and examination methods. It was created in 1956 under the leadership of educational psychologist Dr. Benjamin Bloom in order to promote higher forms of thinking in education, such as analyzing and evaluating concepts, processes, procedures, and principles, rather than just remembering facts [2], [3], [4]. Though there are many arguments highlighting the limitations of MCQs, they are the most commonly used testing items for undergraduate and postgraduate medical entrance examinations. It is generally accepted that MCQ testing is an efficient, objective and reliable way of assessing the abilities in the cognitive domain.

However, in the current scenario of Ayurveda education, the picture appears to be quite different. During the postgraduate education (after completing which, the students become eligible for teaching assignments), the evaluation and assessment skills do not form a part of training in general [5]. Further, there exists no formal mechanism to train Ayurveda teachers in framing MCQs. A few of the previous studies have shown that graduate level of Ayurveda education is more ‘memory-oriented’ than being competence based [5], [6], [7], [8]. Therefore, the kind of postgraduate scholars who get enrolled in different institutions through competitive process often may not meet the expected levels of academic and professional excellence. Keeping this possibility in view, the present study was conceptualized to evaluate the quality of various PGEE question papers.

Objective of the study

We conceived the present study with an objective of evaluating the levels of thinking skills that are actually assessed through MCQs in different PGEEs of Ayurveda, conducted in different universities across India.

Study design

This is a retrospective observational study aimed at evaluating the Ayurveda PGEE question papers of last five years (2010–2014).

Sampling and data collection

The kind of sampling method followed in this study was purposive sampling. We divided India into four conventional zones: North, West, East and South. These were the strata from where we selected different universities. We consulted the online repositories of PGEE question papers such as www.ayurvedpg.com and www.liveayurved.com, and also the specific websites of different universities spread across India, to access the authentic papers. We ensured the authenticity of the question papers by choosing to include the original scanned copies only. We did not include the various ‘PG Entrance Guides’ available in the market deliberately because the authenticity of the question papers given in these books is often unreliable.

The total number of questions presented in PGEEs in different universities was not constant; it varied from 80 (in Jodhpur Ayurveda University – JAU) to 250 (in Maharashtra University of Health Sciences-MUHS) per year. However, except for MUHS, the average number of questions being presented per year was between 100 and 200 in most of the universities. The guiding principle followed in this study was either to have full sets of question papers for five years from each zone (North, South, West and East), or, to have at least 700 questions for five-year period from each zone. (The number 700 was arrived at after taking the average of 100 and 200 questions, i.e., 150 questions per year for 5 years, and then allowing a deletion rate of 10 questions per year for possible ambiguity). Whenever we could not access the full sets of question papers from a single university for 5 years, we clubbed those question papers with other universities in the same zone so that we could achieve our target number. Finally, seven different universities spread across North, West, East and South zones of India were shortlisted: Banaras Hindu University (BHU), Dr. Sarvapalli Radhakrishnan Rajasthan Ayurveda University (DSRRAU), Gujarat Ayurveda University (GAU), Maharashtra University of Health Sciences (MUHS), Rajiv Gandhi University of Health Sciences (RGUHS), University of Jammu (UJ), and Uttarakhand Ayurveda University (UAU).

Assigning the level in the cognitive domain of Bloom's taxonomy

Though the prescribed ‘action verbs’ were taken into consideration during this exercise to assign the ‘Bloom's taxonomy level’, they did not serve the purpose completely since we were assessing only MCQs. Therefore, a group of two investigators, previously trained in the concept of Bloom's taxonomy, read out aloud each of these MCQs and came to a consensus unanimously regarding the level of thinking skills the particular MCQ assessed. Whenever there was a lack of consensus, a higher level was agreed upon among the two levels proposed. Ambiguous and incorrect MCQs were eliminated during the process. Finally, a total of 3299 MCQs were included and were assessed.

Data characteristics

Table 1 shows the details of zonal distribution, number of questions, number of question papers, and the specific universities that were included in the study. It also shows the number of questions included in each year as per different universities in each zone.

Table 1

Showing the year-wise distribution of the questions in each zone along with the universities covered.

Zone	Year					Universities	Number of questions from each university	Number of question papers	Number of questions from each zone
	2010	2011	2012	2013	2014
East	174	180	179	180	179	BHU	892	5	892
North	141	149	150	150	324	UJ	740	5	914
						UAU	174	1
South	280	100	98	0	249	RGUHS	297	3	766
						MUHS	430	2
West	160	160	191	180	75	GAU	340	4	727
						DSRRAU	426	5
Total	755	589	618	510	827		3299	25	3299

Abbreviations: BHU- Banaras Hindu University, DSRRAU- Dr. Sarvapalli Radhakrishnan Rajasthan Ayurveda University, GAU- Gujarat Ayurveda University, MUHS- Maharashtra University of Health Sciences, RGUHS- Rajiv Gandhi University of Health Sciences, UJ- University of Jammu, and UAU- Uttarakhand Ayurveda University.

Results

MCQs according to Bloom's taxonomy

The Table 2 shows the assigned Bloom's taxonomy levels of the studied sample of questions. Table 3 shows the number and percentage of questions in each level. We observed that out of 3299 MCQs, 3079 (93.3%) assessed only the ‘Recall’ component (level-1) and 204 (6.2%) MCQs assessed ‘Comprehension’ component (level-2). The percentage of MCQs that assessed ‘Application’ (level-3) was only 0.3% (11), whereas the percentage of MCQs that assessed the ‘Analysis’ component (level-4) was a mere 0.2% (5). It may be noted that there was not even a single question to assess the ‘Synthesis’ and ‘Evaluation’ (level-5 and level-6) components for the entire five-year period in our sample.

Table 2

Showing the zone-wise distribution of the studied sample of questions as per Bloom's taxonomy.

Zone	Bloom's taxonomy	Year					Total
		2010	2011	2012	2013	2014
East	Recall	163	177	175	165	161	841
	Understanding	11	3	4	14	13	45
	Application	0	0	0	1	0	1
	Analysis	0	0	0	0	5	5
North	Recall	130	127	135	143	303	838
	Understanding	11	22	15	7	21	76
South	Recall	272	96	92	–	237	697
	Understanding	8	4	6	–	11	29
	Application	0	0	0	–	1	1
West	Recall	130	156	175	172	70	703
	Understanding	24	4	15	7	4	54
	Application	6	0	1	1	1	9

Table 3

Showing the overall number and percentage of questions in each level of Bloom's taxonomy.

Bloom's level	Frequency	Percent
Recall	3079	93.3
Understanding	204	6.2
Application	11	0.3
Analysis	5	0.2
Total	3299	100.0

Other relevant observations

We observed that 78.9% of the questions were from Ayurveda subjects and 21.1% were from the current biomedical sciences. Among Ayurveda subjects, a significant number of questions were from Chikitsa (14.5%), Siddhanta (11.3%), Dravyaguna (8%), Shaarira (7.5%) and Vikriti Vijnyana (7.1%). The subjects that contributed most to the ‘biomedical sciences’ category were Medicine (5.6%), Physiology (3.8%), Anatomy (2.7%) and Surgery (2.5%). However, it was interesting to note that only 2.5% of questions were related to the integrative knowledge of both current biomedical sciences and classical Ayurveda.

It was further interesting to note that, out of all questions evaluated, there were 28.8% of questions that required the candidate to know a specific reference/chapter of a specific classical Ayurveda textbook, and 13.4% of questions required the candidate to know the numerical values such as the correct number of Shastra Karma (surgical procedures), the correct number of chapters in a given Ayurveda textbook, etc.

Discussion

Our study confirms the notion that Ayurveda education, in general, is more memory-oriented than being competence based. This is because, the ‘Recall-based’ questions dominate the present Ayurveda PGEEs, irrespective of the region and university. Many of the categories of MCQs, such as those focusing on the numeric values, may not have much clinical applicability. Even remembering the specific references and specific chapter numbers of a textbook, may not help in gaining practically useful knowledge because the searchable digital versions of Samhitas are already available. Therefore, the results of our study suggest that a ‘higher order of cognition’ is not being assessed in these examinations and hence, the system seems to have been designed to specially select the students with only a ‘good ability to memorise’ over those who have critical thinking skills to pursue postgraduate programs in Ayurveda.

The standards of Ayurveda education are being challenged over the past few years repeatedly by the experts, researchers and policy makers [5], [6], [7], [8], [9], [10], [11], [12]. The seriousness of this challenge is reflected by the fact that the governing body, Central Council of Indian Medicine (CCIM), has repeatedly attempted to introduce numerous changes to the curricula of graduate and postgraduate programs during last ten years or so. The situation is of real concern because, the education system, when not healthy, is likely to produce mediocre physicians and teachers. Mediocrity not only dilutes the present day standards, but also gets perpetuated among the coming generations of students. In this context, selecting academically and professionally excellent students for pursuing postgraduate programs becomes essential to keep up the standards of education and practice.

It must be noted that, till recently, there existed no mechanism to train Ayurveda teachers in examination and evaluating skills, including the preparation of MCQs. The recent initiative, ‘Training of Teachers’, taken up by the CCIM is an admirable step in this regard. We propose that such training programs must include the following topics and that all the teachers must be trained in these domains on a compulsory basis: a. Fundamental principles of assessment in medical education, b. Good practices of question paper setting, c. Good practices of preparing multiple choice questions, d. Good practices of conducting viva voce examination, and e. Good practices of conducting practical examination. It has been shown in a study that effective faculty development programs are useful in training the teachers in framing good MCQs [13].

There is in fact, a scarcity of standard peer-reviewed scholarly curriculum-oriented textbooks in Ayurveda. There are ambiguous contents and inter-author differences in the currently available textbooks [8]. Because of the compulsion to provide ‘evidence’ in the form of ‘references’ to substantiate the proposed answer in the ‘answer key’, the question paper setters often prefer playing safe by asking recall-based questions to stay away from being dragged into any such controversies. Further, there is also a dearth of literature related to effective teaching, learning, and evaluation methods relevant for Ayurveda. Therefore, there is a need for generating literature related to “methods in Ayurveda education” while also sensitizing the teachers regarding these issues [8]. This will possibly pave way for betterment of the quality of Ayurveda education.

Proposed style for framing MCQs to evaluate all levels of thinking abilities

In the following section we provide examples for MCQs that assess various levels of thinking skills in the cognitive domain of Bloom's taxonomy, taking the ‘applied Kriya Sharir (Ayurveda Physiology)’ as the area of study. We have also taken into consideration the need for integrative approach while framing these items [14]. We have also explained the mental processes involved in solving these questions and the reasons for assigning the specific levels of Bloom's taxonomy.

A. Recall (level-1)

The two types of Pliha Vriddhi (splenic enlargement) described by Chakrapani are:

Mridu Vriddhi (soft enlargement) and Kathina Vriddhi (hard enlargement)

Sashoola Vriddhi (painful enlargement) and Ashoola Vriddhi (painless enlargement)

Chyuta Vriddhi (splenic displacement) and Achyuta Vriddhi (splenic enlargement in situ)

Ghana Vriddhi (solidified enlargement) and Aghana Vriddhi (liquefied enlargement)

Explanation

The correct option in this case is C. The common words that are used to evaluate the recall component are: define, label, state, list, match, write, recall, name, underline, repeat, choose, record, mark, identify and recognize. To answer this question correctly, a student needs to recall the classification of Pliha Vriddhi (splenic enlargement) as described by Chakrapani in Charaka Samhita, Chikitsa Sthana, 13th Chapter. Thus, this question requires the knowledge of classification and nomenclature, which is of first level in Bloom's taxonomy.

B. Understanding (level-2)

Based on which of the following information you can draw the relationship between ‘water homeostasis’ and ‘Vrikka’ according to Ayurveda physiology?

Vrikka is related to Meda, and Sveda is also related to Meda. The functions of Sveda and Mutra are related to Kleda. Kleda refers to water content.

Vrikka and Basti –both are located in Koshtha. Vrikka is said to drain water into Basti. Basti is related with Ambuvaha Srotamsi. Ambu represents water content.

Vrikka is a synonym of Kloma. Kloma is related to Udakavaha Srotas. Udaka means water.

All the above options are wrong and there is no relationship between Vrikka and water homeostasis.

Explanation

The correct option in this case is A. To answer this question correctly, the student needs to identify the following relationships: 1. Vrikka is the Mula of Medavaha Srotas, 2. Sveda is the Mala of Meda, 3. Mutra and Sveda-both are related with the excretion and retention of Kleda. Further, the student needs to recall that the sweat and urine represent the two important routes of water excretion. The student further needs to identify that the following distracting phrases, i.e., ‘Vrikka is said to drain the water into Basti’ and ‘Vrikka is a synonym of Kloma’ are incorrect. The common words that are used to evaluate the comprehending ability are: describe, rephrase, restate, discuss, explain, express, locate, tell, indicate, condense, outline and report [15]. This item actually requires the student to mentally outline and restate the concept of water homeostasis in Ayurveda.

C. Application (level-3)

Praseka (excessive salivation) may manifest as a feature of Kapha Vriddhi, and Dhattura (Datura metel) has been described to possess ‘Kaphaghna’ property. Now, choose the most appropriate statement among the following:

Dhattura cannot block excessive salivation because salivation is not a muscarinic action

Dhattura may be effective in conditions such as Parkinson's disease and motion sickness because its' chemical constituents are muscarinic antagonists

Dhattura can block salivation because salivation is mediated through nicotinic receptors

Dhattura can cause contraction of gut smooth muscles and can stimulate the secretion of HCl because they are muscarinic actions

Explanation

The common words that are used to evaluate the application ability are: apply, use, demonstrate, illustrate, chart, solve, operate, implement, practice, employ, dramatize and show [15]. To answer this question, a student needs to know that the plant described (D. metel) contains alkaloids such as Atropine and Hyoscine. He/she must also know that they are muscarinic receptor antagonists. The student must have an understanding that salivation is a muscarinic action and occurs in Parkinson's disease and in motion sickness. This is how the student's ability to remember, understand and apply the knowledge in a new situation is tested. All other options are factually incorrect, and hence, the correct option in this case is B. This item expects the student to actually ‘use’ and ‘apply’ the understanding related to the physiology of autonomic nervous system and the pharmacology of D. metel.

D. Analysis (level-4)

Ayurveda employs ‘Kedari-Kulya’ and ‘Khale-Kapota’ theories to describe the processes of the tissue nourishment. Two different analogies/instances have been used to illustrate these processes. Considering the ‘nature of energy expenditure’ involved, these processes can be categorized into:

Sympathetic and Parasympathetic effects respectively

Passive and Active transportation respectively

Inspiration and Expiration respectively

Systole and Diastole respectively

Explanation

The correct option in this case is B. The common words that are used to evaluate the analyzing ability are: distinguish, discriminate, analyze, compare, contrast, diagram, differentiate, relate, classify, examine, categorize [15]. To arrive at the correct answer, the student must have an understanding of the analogies that have been used to describe these theories. Kedari-Kulya theory makes use of a situation where the water from a reservoir flows passively through small canals into different pieces of land where grains are grown. Khale-Kapota theory uses an instance where pigeons from different places approach a heap of collected grains and return to their nests after picking up the grains of their choice. The student must be able to break down this information into two distinct aspects of energy expenditure: the flow of water occurs ‘passively’ in the first instance, whereas the pigeons are required to ‘spend energy’ in the second analogy. Student must also know that the technical terms used to describe the two methods of transport in biology are Passive and Active respectively and must be able to relate these terms with the above analogies. The student must also be able to eliminate other confusing distractors, which are again related to energy expenditure but in an improper order. This item therefore, requires a student to break down the elements and to find the relationships among two different concepts to arrive at a final answer. Thus this question assesses understanding, comparison, categorization, and analytical skills in a student which is of fourth level in Bloom's taxonomy [15].

E. Synthesis (Level-5)

Read the following paragraph carefully and select the most appropriate hypothesis that can be generated on the basis of the information provided:

Pakvashaya (large intestine) is the major site of Vayu and Vayu has been ascribed with many functions of brain. The large intestine has a unique composition of microorganisms. Microbial flora in the gut has been functionally linked with the development of brain. Antibiotics often destroy gut microbial flora. Recent research suggests that diet can affect certain autoimmune diseases by inducing changes in gut microbe communities. Takra, (buttermilk) a known probiotic substance, is indicated in the management of gastrointestinal disorders in Ayurveda. Necrotizing entero-colitis in infants is being implicated in subsequent neurodevelopmental defects such as cerebral palsy and microcephaly. The dysbiosis in the gut microbiota likely contributes to this condition. Rajayapana Basti has been reported to be effective in improving motor disabilities among children suffering from cerebral palsy. Asava and Arishta preparations, being fermented products, contain microbes.

Now, read the following hypotheses

Considering the fact that the diet can influence the gut microbe communities, Takra may be effective in non-infective diarrhea associated with lactose intolerance

Because microbial flora is linked with the development of brain, Asava and Arishta may be effective in treating ischemic infarction in brain

Because Rajayapana Basti has been reported to be effective in cerebral palsy, its action may be through correcting the dysbiosis in the gut

Because overuse of antibiotics impairs the composition of gut microbiota, it may lead to Dysautonomia in future

Options

All may be plausible but 1 is most appropriate

All may be plausible but 2 and 3 are most appropriate

All may be plausible but 3 is most appropriate

All may be plausible but 1 and 3 are most appropriate

Explanation

Many experts believe that framing MCQs to assess ‘synthesis’ skills is actually difficult. However, carefully framed questions can assess this ability to certain extent. To answer this question, the student must evaluate the intra- and inter-relationships within the given points. For example, he/she must be able to recognize that Takra is contraindicated in lactose intolerance and ischemic infarction is not a neurodevelopmental disorder. Further, antibiotic overuse leading to Dysautonomia is not a plausible assumption considering the given information, because Dysautonomia has not been shown to have a neurodevelopmental link. Therefore, the correct option in this case is C since Basti is directly administered into the large bowel. This involves combination, comparison, and analysis of diverse elements described in the paragraph and synthesis of new hypothesis. Thus, this question assesses the knowledge, understanding, application and analysis with skills of generation, or creation (synthesis) which is of level-five in Bloom's taxonomy. The common words that are used to evaluate the synthesizing ability are: create, design, plan, organize, generate, develop, propose, assemble, compose, make, construct, formulate, prepare, invent, adapt and elaborate [15].

F. Evaluation (level-6)

A study was carried out among healthy male adult volunteers. Jatharagni in these volunteers was assessed thrice in a year using a validated questionnaire in spring (Vasanta), rainy (Varsha) and winter (Hemanta) seasons. These volunteers were also tested for their serum T3 and T4 levels during these seasons to see if there is a relationship between the Agni, season and serum thyroid hormone levels. The major observations of this study have been given below:

Observations

The mean serum T3 and T4 levels showed a strong tendency towards an increase in spring season in comparison to the other two seasons. The mean Jatharagni scores showed a strong tendency towards a decrease in spring season in comparison to the other two seasons.

Now, read the following interpretations proposed by three students based on the above observations, evaluate the appropriateness of these interpretations, and choose the correct option from the given list that follows.

Considering T3 and T4 levels to be corresponding to Dhatvagni, it can be stated that the relationship between Dhatvagni and Jatharagni may be inversely proportional and there could be a ‘negative feedback’ type of mechanism operating between Dhatvagni and Jatharagni

Considering T3 and T4 levels to be corresponding to Dhatvagni, it can be stated that the relationship between Dhatvagni and Jatharagni may be directly proportional and there could be a ‘positive feedback’ type of mechanism operating between Dhatvagni and Jatharagni.

The mean Jatharagni scores do not increase along with serum T3 and T4 levels. Therefore, the results of this study appear to be wrong. Agni must intensify as serum T3 and T4 levels increase. Either the questionnaire to assess Agni was inappropriate, or the methods employed to record serum T3 and T4 levels were wrong. Sample size too may have been very small.

Options

3 is the most appropriate interpretation

Both 1 and 3 are possible interpretations, but 1 is most appropriate

2 is the most appropriate interpretation, while 1 and 3 are wrong

Both 2 and 3 are correct interpretations, while 1 is wrong

Explanation

This question requires student's knowledge about basic concepts of metabolism according to Ayurveda and contemporary science. He/she should also have skills to find out and generate possible links between two sciences. The student must also analyze the research results given and the way in which the research problem was investigated. The option C is wrong because the 2nd interpretation contains factual error. The option A is inappropriate because there is no substantial information to justify the 3rd interpretation in the given paragraph. Finally, after making judgments of the given possible interpretations the student must be able to evaluate the alternatives and must arrive at the correct option, which is B. The commonly used words to assess the evaluating ability are: appraise, critique, judge, weigh, evaluate, select, rate, defend, prioritize, value, score, measure and assess [15]. In this question, the solving process involves the ability to evaluate and judge, hence is of level-6 in Bloom's Taxonomy.

Conclusions

The study supports the notion that the current formal Ayurveda education is more memory-oriented than being competence based. Disproportionately excess of recall-based MCQs dominate the PGEEs at present. We therefore propose that, there is a need for including an appropriate proportion of the MCQs to assess ‘higher order thinking’ in the PGEEs. Further, Ayurveda teachers are required to be trained well in the skills related to writing good MCQs since MCQs are able to assess more or less all the six levels of Bloom's taxonomy in cognitive domain. We further suggest that our study may be taken as a lead to introduce the required reforms in PGEEs.

Ethical committee approval

Was not required.