All India AYUSH post graduate entrance exam 2019 - AYURVEDA MCQ item analysis.

BACKGROUND: AIAPGET 2019, an all India ranking entrance Test for MD/MS courses of Ayurveda, Unani, Siddha and Homeopathy stream was conducted by joint collaboration of National Testing Agency (NTA) and All India Institute of Ayurveda (AIIA). In this article, we present the item analysis of AIAPGET 2019 Ayurveda stream MCQs.
OBJECTIVES: The aim of this article was to analyse the MCQs of AIAPGET 2019 of Ayurveda stream.
MATERIALS AND METHODS: This exam was computer based conducted all over 25 centers across India. The question paper had 100 MCQs with 1 correct answer and 3 distractors for each item (Problem statement).
RESULTS: AIAPGET 2019 question paper of Ayurveda stream had a Difficulty index of 37.32 ± 16.11 Discriminatory Index of 0.46 ± 0.27 and Distractor Index of 89 ± 17.8.
CONCLUSION: Our analysis showed that though ideal, the question paper trended towards difficulty side.

Introduction

All India AYUSH Post Graduate Entrance Test (AIAPGET) 2019 is an All India ranking entrance examination for to all the AYUSH MD/MS Courses in Ayurveda, Unani, Siddha & Homeopathy systems of medicine. The syllabus for the exam comprises of subject/knowledge areas as per the Graduate Level Education Regulations of respective discipline issued by Central Council of Indian Medicine (CCIM)/Central Council Homeopathy (CCH). The exam is being conducted with joint collaboration of All India Institute of Ayurveda, New Delhi and National Testing Agency. AIAPGET 2019 was LAN Based computer test, with 100 multiple choice questions (MCQs) and duration of 90 min.

Multiple choice questions are the most common method of formative and summative assessment in Undergraduate, Post graduate Medical Programs. MCQs do not only assess the knowledge recollection, but also comprehension, application, analysis, synthesis and evaluation [1]. Constructing MCQs is a challenge and time consuming, but when compared with other methods of examination, MCQs are more objective and minimal human error in evaluating. Earlier papers published have analyzed the Ayurveda PG Entrance question papers based on Bloom’s taxonomy approach [2,3]. However, item analysis which is based on the response of candidate post-exam has not yet been published for any of the AIAPGET exams till date. Post-exam, item analysis assesses the reliability and validity of MCQs by assessing student performance with regards to every MCQ and applying statistical inferences to determine how better was the MCQ and its distractors. It is an assessment tool beneficial both for the candidate appearing exam and examiner [12]. The most common parameters used are Difficulty Index for assessing difficulty level of MCQs, Discrimination Index (DI) to differentiate between students of higher and lower abilities and Distractor Efficacy (DE) for the efficacy of distractors. These parameters are assessed to check the validity and reliability of MCQs so that they can be kept, edited or discarded during the development of MCQs. In case of National level entrance exams though repetition of MCQs are not preferred, item analysis will help in better framing of MCQs. The aim of this article was to analyse the MCQs of AIAPGET 2019 of Ayurveda stream.

Materials and methods

A total of 14,229 students had appeared for the exam for Ayurveda stream of AIAPGET 2019. The question paper had 100 MCQs. Each MCQ (Item) had 1 answer and 3 Distractors. Each correct response had 4 marks and wrong response had a negative marking of 1 mark. Marks obtained by the students were arranged in descending order in Microsoft office excel sheet 2016. The total marks of the candidates when arranged in descending order, the upper one-third students (4743) were considered as high achievers (H) and lower one-third (4743) as low achievers (L) [4]. The response of each of the candidates (n = 14,229) was analysed for Difficulty index, Discrimination index and Distractor efficacy in this article (See Table 1).

Table 1

Showing interpretation of Discrimination index, Distractor Efficacy and Non-functional Distractors.

	Cut-off points	Interpretation
Difficulty index	<30	Too Difficult
	30–40
	40–50
	50–60
	60–70
	>70	Too Easy
Discrimination Index	<0.15	Poor
	0.15–0.35	Good
	>0.35	Excellent
Non-functional distractors	4 NFD	0% Efficacy
	3 NFD	25% Efficacy
	2 NFD	50% Efficacy
	1 NFD	25% Efficacy
	0 NFD	100% Efficacy

Difficulty index

Difficulty Index is the percentage of students in high and low achievers group who answered the item correctly. It ranges between 0% and 100%. It was calculated using the formula.Difficulty Index [

H = number of students answering the item correctly in the high achieving group,

L = number of students answering the item correctly in the low achieving group, and

N = total number of students in the two groups (including non-responders).

Discrimination index or d value

DI is the ability of an item to differentiate between students of higher and lower abilities and ranges between 0 and 1. It was calculated using the formula [5].DI = 2 × (H–L)/N where,

H = number of students answering the item correctly in the high achieving group,

L = number of students answering the item correctly in the low achieving group, and

N = total number of students in the two groups (including non-responders).

Distractor analysis

Distractor efficiency (DE) was determined on the basis of the number of Non-functional distractors (NFD) in an item. NFD is the distractor which is selected by less than 5% of students in an item [5]. DE is ranged from 0 to 100%.

Results

After analysis, it was found that 34 questions had difficulty index <30% and nearly equal distribution in 30%–40% and 40–50%. Only 2 questions had Difficulty index >70% (See Table 2, Table 3, Table 4).

Table 2

Showing results of item analysis of AIAPGET 2019 ayurveda MCQs.

	Cut-off points	No. of Questions (n = 100)	Interpretation
Difficulty index	<30	34	Too Difficult
	30–40	20
	40–50	24
	50–60	13
	60–70	7
	>70	2	Too Easy
Discrimination Index	<0.15	17	Poor
	0.15–0.35	15	Good
	>0.35	68	Excellent
Non-functional distractors	Items with 4 NFD	0	0% Efficacy
	Items with 3 NFD	2	25% Efficacy
	Items with 2 NFD	7	50% Efficacy
	Items with 1 NFD	24	25% Efficacy
	Items with 0 NFD	67	100% Efficacy
	Total Distractors - 300Non-Functional Distractors - 44 (14.6%)Functional Distractors 256 (85.4%)

Table 3

Showing mean ± sd Difficulty index, discrimination index and distractor efficiency.

Parameter	mean ± sd	Range
Difficulty Index	37.32 ± 16.11	2.37–85.25
Discriminatory Index	0.46 ± 0.27	−0.09–0.93
Distractor Efficacya	89 ± 17.8	0–100%

Distractor Efficiency = Distractor efficiency ranged from 0 to 100% and was determined on the basis of the number of NFDs in each item. Three NFD: DE = 25%; 2 NF-D: DE = 50%; 1 NF-D:DE = 75%; No NFD: DE = 100%.

Table 4

Showing the distribution of Score of Ayurveda Stream of AIAPGET 2019.

% of Marks	No. of students (%)
>80%	0.28
70%–80%	2.83
60%–70%	7.17
50%–60%	10.63
40%–50%	13.31
30%–40%	16.09
<30%	49.68

Only 10.28% of the students could score more than 60% and 79.09% of the students scored less than 50% score.

68 questions had DI more than 0.35 which is considered as excellent and 15 questions had DI 0.15 to 0.35 which is considered as good. Only 17 questions had DI < 0.15 which is considered as poor±.

By analyzing the distractors, it becomes easier to identify their errors. In AIAPGET 2019 for Ayurveda, 357 were functional distractors among 400 distractors. The details of Distractor efficacy is given in the table (Table 2).

Discussion

MCQs have their own strengths and weaknesses. MCQs assess the cognitive domain of learning though psychomotor and affective domains cannot be assessed [6]. The advantage of MCQs is that it can be used when there is large number of students in a quick and user friendly. Item analysis can provide valuable information on the validity and reliability of the exam/test. It can also give a valuable information for the experts while framing the MCQ for the next exam.

Item difficulty is relevant for determining whether students have learned the concept being tested. Higher the Difficulty index, lower the difficulty of the question. Though AIAPGET 2019 exam had negative marking, Difficulty index and Discrimination index do not change with negative markings [7]. The mean Difficulty index of the AIAPGET exam 2019 for Ayurveda stream was 37.32 ± 16.11 (Mean ± Standard Deviation). Our analysis showed that the question paper was trending towards the difficult items. Difficult items lead to decreased score while easy items lead to increased score.

Choosing an appropriate level of Difficulty index is one of the crucial factors. It depends on the type of exam. If only a creamy layer candidates are to be selected like in National level competitive exams, items are made to be difficult and if MCQs are to be answered by most of candidates, the items tend to be easy.

Discriminatory index is another parameter of item analysis which helps in discriminating between high scorers and low scorers. MCQs with higher Discriminatory index are considered as excellent. i.e. the answering the questions by the candidates by guessing is minimal. The difficulty index and discrimination index are inversely proportionally related [8]. Discriminatory index ranges from 0 to 1 but sometimes it may also be negative called as Negative DI [9]. AIAPGET 2019 Ayurveda Question paper had a mean discriminatory index of 0.46 ± 0.27. 17 questions had discriminatory index <0.15. 68 questions among 100 MCQ had excellent discriminatory index.

A good MCQ should also have good distractors i.e. should be closely placed near correct answers. Distractors are the one which are usually chosen by low performer while high performing candidate should ignore them [10]. Analysis of distractors can also be a valuable indicator of item difficulty. Non-functional distractors are those which are selected by less than 5% of the candidates. AIAPGET 2019 Ayurveda Question paper had only 44 ((14.6%) NFD among 300 distractors. Items with 0 NFD were 67 items while with 1 NFD were 24 items with overall distractor efficacy of 89 ± 17.8 (Mean ± Standard Deviation).

An ideal MCQ should have 31% to 60% Difficulty index, high discrimination (>0.25) and maximum distractor efficacy with three functional distractors [11,12]. AIAPGET 2019 Ayurveda stream question paper fulfilled all these parameters.

If distribution of Score of Ayurveda Stream of AIAPGET 2019 is observed, we can find that only 0.28% of the students could secure more than 80%. Also, there was increasing percentage of candidates as percentage of the marks goes down. Though ideal, our analysis showed that question paper of AIAPGET 2019 – Ayurveda was trending towards difficult question paper. This data shows extensive scope for training of teachers/paper setters for framing quality based MCQs keeping in view of Students’ intellect appearing for the test and purpose of AIAPGET.

Conclusion

Item analysis gives an information on reliability and validity of an item/test by using Difficulty Index, Discriminatory index and Distractor efficacy tools. Item analysis gives an idea for the exam/paper setters to design the MCQ for further competitive exams based on the purpose of the exam and intellect of the students. This paper gives an idea for the conducting body of the exam or the exam/paper setters for AIAPGET to keep in view the student’s intellect and purpose of the exam while setting paper for AIAPGET in future. It would also help the candidates appearing for AIAPGET in future and prepare accordingly.

Source(s) of funding

None declared.

Conflict of interest

None.