Neurology Exit Examination System in India: A Survey of Examiners' Perceptions and Recommendations.

Background: The traditional Neurology exit examination in India has remained unchanged over the last few decades. In developed countries, objective evaluation methods have replaced the traditional ones. A need for such methods has not been explored in India. Objective: We aimed to study the perceptions and key recommendations of Neurology examiners on the existing examination pattern. Material and
Methods: We conducted an online survey of examiners perceptions and recommendations using a set of 10 multiple-choice questions and an open-ended question.
Results: 46 examiners provided completed responses suitable for analysis. Nearly equal proportions (30%) of the examiners had 10 years, 10-25 years and >25 years' experience. 92% were not satisfied with current system, 95% did not find adequate time for correction of theory scripts, 90% felt that theory questions were random, and 95% had legibility issues. 84% felt that the practical exams do not test true learning, 98% felt the examination stress impairs the performance and 85% felt that there are no objective criteria to pass the candidate. 83% felt the current system-needed changes. The key suggestions provided by the examiners to improve the system included objective assessments like MCQ, OSCE, OSLER and DOPS, inclusion of larger number of short answer type questions and periodic internal assessments of the candidates. Conclusions: A vast majority of examiners favoured changes to the current examination system and provided key recommendations. A larger study is needed to extrapolate these findings to the rest of India. Copyright:

INTRODUCTION

The purpose of an exit examination at the completion of training is to ensure a minimum level of competency to grant a license to practice Neurology. Until a few decades ago, medical examinations in the developed countries were in the form of essays and oral assessments.[1] Evaluations derived from such assessments were subjective, arbitrary and not reproducible. Subsequently, standardized tests such as multiple-choice questions (MCQs) or Objective Structured Clinical Examinations (OSCE) were developed.[23] Worldwide, the Neurology residency exit examination system is highly varied,[456] and in some countries, there is no formal exit examination.

In India, the neurology exit examinations for granting DM (Doctor of Medicine) degree generally consist of written tests (usually 10 questions each carrying 10 marks) covering basic sciences, clinical neurology, investigations, therapeutics and recent advances, and practical clinical examinations consisting of a “long case” and 2-3 “short cases”. This is followed by neuroradiology spotters, electrophysiology spotters, neuropathology slides and viva voce. There are often variations in this scheme from one university to another. Though this examination pattern appears elaborate and comprehensive, the theory examination and evaluation, viva voce and case presentations are largely unstructured and without a clear blueprint for objective evaluation. During the conduct of the DM Neurology examinations, several opinions and suggestions have been put forward and discussed informally by the examiners regarding the need to revise the current system. In a recent editorial (2020), Pauranik noted the wide variability of quality in DM training and assessment in India, emphasized the need for systematic research in this area and urged the Indian Academy of Neurology to take the lead in this direction.[7] As the number of DM Neurology seats in the country continues to increase over time, it is important to have a system of exit examination that ensures uniform minimum national standards of the trainees. The objectives of our study are to systematically study the perception of examiners from various parts of India on the existing examination pattern for DM Neurology and to know their key recommendations.

METHODS

This study is an online survey of DM Neurology examiners in India, conducted by the senior neurology faculty of a tertiary care medical college hospital in Bengaluru, Karnataka state in South India. This study did not probe the DNB (Diplomate of National Board) Neurology examination system, which is also a Neurology certification exam conducted by the National Board of Examinations. A Google survey form was created with a set of 10 single-response-multiple-choice questions [Table 1] covering theory and practical examination pattern (five questions each). The questionnaire was designed according to the guidelines of the Association for Medical Education in Europe[8] and were internally validated. For each question, there were 4 Likert-type response options, and the responses were set in such a way that the score value was proportionate to the degree of satisfaction with the existing system. In addition to these 10 questions, an open-ended question was posed seeking any suggestions and recommendations from the examiners to improve the examination system. The questionnaire was electronically mailed to 60 examiners who have been in the examiner data base of the college. The university to which the college is affiliated, requires the examiners to be preferably chosen from geographically proximate locations, and consequently, the data base is heavily weighted towards Karnataka and other southern states of the country. Incomplete responses and those without the respondent details were excluded from analysis. In addition to the answers to the questions, respondent data was collected including years of experience in DM examinations, and city/town of practice. The data were subjected to descriptive analysis. The final manuscript was approved by all the respondents with relevant inputs. Descriptive statistics including median, inter quartile and percentages were used to summarize the data. Fisher's exact test was used to find associations. Software version 4.0.2 was used for data analyses (R Core Team, 2020, Vienna, Austria; www.R-project.org/) A value of P ≤ 0.05 was considered statistically significant.

Table 1

Online survey questionnaire on DM neurology examination pattern: Responses

Survey questionnaire	Responses (n=46)
	1 (n %) (>75)	2 (n %) (50-75)	3 (n %) (25-50)	4 (n %) (<25)
On a scale of 0-100, how satisfied are you with the current model of DM Neurology final examination?(>75 for extremely satisfied, <25 for extremely unsatisfied)	4 (8.7)	21 (45.7)	14 (30.4)	7 (15.2)
Do you get enough time to read theory answer papers thoroughly and allot marks appropriately? >75 for very adequate time and<25 for very inadequate time	3 (6.5)	18 (39.1)	13 (28.3)	12 (26.1)
On a scale of 0-100, do you think the theory questions are set thoughtfully or randomly to assess students’ learning? >75 for very thoughtful and <25 for very random	5 (10.9)	13 (28.3)	18 (39.1)	10 (21.7)
Is legibility of students’ handwriting an impediment to assess the theory answers appropriately? >75 for not at all, <25 for most of the time	2 (4.3)	19 (41.3)	15 (32.6)	10 (21.7)
By how much percentage do the practical exams reflect the true learning and competence of the students?	8 (17.4)	20 (43.5)	13 (28.3)	5 (10.9)
To what extent do stress and anxiety of the practical exam ambiance effect the students’ performance adversely? >75 for “not at all” and <25 for “ very much”	2 (4.3)	15 (32.6)	17 (37.0)	12 (26.1)
Do the examiners set clearly defined criteria on when to pass or fail a candidate in a practical exam?>75 for ‘very clear criteria’ and <25 for ‘absolutely no defined criteria’	7 (15.2)	17 (37.0)	10 (21.7)	12 (26.1)
Are there chances that competent candidates may be failed or not competent candidates passed due to subjective decision making? >75 for never, <25 for very often’	6 (13.0)	19 (41.3)	14 (30.4)	7 (15.2)
Is there a need to revamp the existing examination pattern (theory and practicals) >75 for ‘no need’ and <25 for ‘very much needed’	8 (17.4)	8 (17.4)	8 (17.4)	22 (47.8)
Do you think an objective exam (both theory and practicals) on the lines of UK/US boards may be a better method? >75 for ‘ definite no ‘and <25 for ‘definite yes’	10 (21.7)	9 (19.6)	12 (26.1)	15 (32.6)

RESULTS

Of 60 respondents, 46 (77%) provided complete data suitable for final analysis. The years of experience as an examiner ranged from 3 to 40 years with a median of 20 years and interquartile range of 12-34 years. Approximately equal proportion had experience of less than 10 years (32.5%), 10–25 years (32.5%) and more than 25 years (35%) as examiners [Figure 1]. There were 40 men and 6 women among the respondents.

Figure 1

Distribution of the neurology examiners by the number of years of experience

Nearly half the respondents (24/47) were from the state of Karnataka. Of these, 20 were from the city of Bengaluru. 17 were from various other states of south India and 5 were from the northern states [Figure 2]. This distribution reflects the database of examiners of the college, which is dominated by examiners from proximate geographical locations. In all, 11 states were represented in this survey. The responses to each of the 10 questions are discussed below.

Figure 2

State wise distribution of the neurology examiners in India

Only 8% of the respondents were extremely satisfied with the current examination system. Nearly half (46.9%) of the respondents are unsatisfied or extremely unsatisfied. Just 5% felt they get adequate time to read the theory answer papers thoroughly and allot the marks appropriately. 55% strongly felt that they do not get adequate time to perform this role. Only 10% felt that theory questions are set thoughtfully to assess the students’ learning. The remaining majority felt that the questions are set in a random fashion. 95% had some difficulty to extreme difficulty with regards to the legibility of the answer scripts. Only a minority (16.3%) felt that the practical exams reflect the true learning and competence of the students. About 40% felt that they do not reflect true learning and competence to a large or extremely large extent.

98% felt that the stress and anxiety of practical examination ambience adversely affected the students’ performance. Only about 14% felt that the examiners set clearly defined criteria to pass or fail a candidate in the practical examination. Nearly 25% felt there are absolutely no well-defined criteria to reach this decision. Nearly 88% felt that often to very often, there are chances of competent students failing or non-competent students passing the exams. About 49% felt that revamping the current system of theory and practical exams is very much needed. 34% felt that it is needed to some extent to modify the existing system. Only 16% felt that the current system should continue unchanged. 60% strongly felt that an objective system of examination is needed. 20% felt that objective system of examination is not an appropriate method. There was no statistically significant association between years of experience and the responses to any of the questions. Key recommendations provided by the examiners to improve the examination system are shown in Table 2.

Table 2

Key recommendations of Neurology examiners

Key recommendations	Total=40 n (%)
Assessment has to be throughout the year or in semester intervals	7 (17.5%)
OSCE stations to be included in the practical examinations	7 (17.5%)
Theory and Practical exams should include a clear scoring (objective) system for the examiners	6 (15%)
Theory examination papers should have more number of questions (say, 25) with short answers to cover wider areas of Neurology	5 (12.5%)
The internal assessment has to carry a set percentage (example: 20-25%) weightage in the final score	5 (12.5%)
Directly observed practical skills like nerve conduction studies, EMG, EEG to be included	3 (7.5%)
In theory exams, a proportion of questions have to be objective type and the rest, to be descriptive.	3 (7.5%)
Need to have clearly laid out syllabus with definition of what is ‘must know’, ‘important to know’ and ‘good to know	2 (5%)
A feedback from the residents on each exam to be collected, analysed and issues if any to be addressed	1 (2.5%)
A training/orientation program for examiners especially the newly appointed examiners to be considered	1 (2.5%)

DISCUSSION

For an assessment of a medical student, no single method is ideal and a mixture of methods is available based on the context and purpose of assessment. The consideration of utility criteria of the assessment tool like objectivity, validity, reliability, educational impact, acceptability, and feasibility can be helpful in making an appropriate choice.[9] ” In real life, there is always some swapping between validity and reliability. For high stakes, competitive examinations, more reliable instruments are used, whereas, for formative assessments, tools with higher educational impact should be chosen. There is a need to design tests that ensure that a license to practice is given only to competent and qualified physicians who have achieved a minimum standard. An apt statement by Gleeson needs our attention: “the examiner has a duty not to release a candidate, who is not properly prepared, on an unsuspecting public.”[10]

From this survey, it is apparent that a majority of the examiners are not satisfied to some or a large extent with the current pattern of theory and practical examinations and are in favour of a more objective, valid and reliable system of assessment. A number of suggestions have been put forward for consideration, though no system is perfect and each one has its merits and demerits, For theory examinations, multiple-choice questions provide a more objective assessment than long and short answer type questions and they also eliminate the answer script legibility issues. Including several short answer descriptive questions, instead of few long answer questions, provides an opportunity to cover wider areas of the field of neurology.[11] However, a long essay type of question assesses the ability of a trainee to construct a coherent account of a given topic in a limited time and this cannot be assessed by short answer or multiple-choice questions. Such writing skills are important for academicians and researchers in the field of Neurology. Thus a combination of long answers, short answers and MCQs could strike a balance between the breadth and depth of knowledge of the examinees.[11] Careful structuring and a pre-set marking scheme will be required to avoid the inter evaluator variability in the marking and evaluation of short and long answer questions.

For practical examinations, one of the many suggestions put forward is implementation of OSCE stations. It is noteworthy that some institutes in India (e.g., the All India Institute of Medical Sciences-AIIMS, New Delhi) already have this system in place.[12] and it would be important for other centres and universities to assess the feasibility of replicating this system.

Each of the OSCE stations will have one patient or a scan or EEG result or set of blood results with a pre-set questions. The student will read the question and perform the task in a fixed time. The performance could be recorded electronically. The student will be assessed by at least one examiner. Hence, the results will be the sum of multiple performances assessed by multiple examiners. This method will make it more objective and reduce the chance of individual bias from affecting the results of assessment. The system also provides legal protection for the examiners, in case the trainee legally challenges the results.

The National Board of Examinations has been conducting case scenario-based, objective questions to assess the clinical skills. Some suggestions given by the survey participants are concurrent with this model. It will be interesting to see if this model can be incorporated in the DM examination system without diluting the existing system of bedside clinical assessment. Some colleges have been using OSLER (objective structured long examination record) for assessing the long case.[10] This could be adopted by all universities to bring in some uniformity and objectivity in the assessment of bedside cases. A major lacuna in the current system is that a trainee's skills in procedures like nerve conduction studies, electromyography and electroencephalography are not assessed and only the interpretation of the recorded graphs is assessed. This could be overcome by using DOPS (directly observed procedural skills) method[13] Here the examiner observes the candidate performing a task like EMG. The task is divided into different components like consent, explanation of the procedure to the patient, actual conduct of the test, interpretation of the results, and explanation of the results to the patient. Examiner will score each of these components. A minimum of DOPS done during training and signed by a qualified faculty member in a logbook or electronic portfolio could also be used. This type of work-based DOPS provides an opportunity for the trainee to get feedback and develop the skills further. In addition, a comprehensive review and recommendation by Ranjan et al. on National Exit exam (NEXT) discusses these issues in detail and is applicable to Neurology DM examination as well.[1112]

During an oral examination numerous sources of error occur, to which examiners are subject in the framework.[1415] For example, with the primacy effect, first impressions dominate over later impressions and, with the recency effect, later impressions are more lasting. In the halo effect, the perception and evaluation of one property outshines the perception and evaluation of other properties. Antipathy, sympathy and the composition of the examiners also have an influence on the evaluation of the test performance. Jayawickramarajah et al.[16] demonstrated that two-thirds of the questions in an unstructured oral examination exclusively examined factual knowledge. An additional problem of an unstructured oral examination is the high probability of an occurrence of Construct Irrelevant Variance (CIV) due to the fact that too few examiners are used. CIV could occur, for example, when the testing of the competence in “clinical decision making” is influenced by the appearance, fear, language skills or clothing of the examinee. Construct Underrepresentation (CU) is a further hurdle that must be considered in the context of an unstructured oral examination, since, for example, two to three clinical scenarios that are tested cannot cover the entire range of the subject area to be tested.[17] Concerns about the validity of this traditional form of examination have led to it being replaced by written examinations or structured oral examinations in some countries.

There is a need to explore these options to bring in more objectivity, validity, and reliability to oral examinations.

The strength of this study is that about 65% of the respondents had experience of more than 10 years and 35% had greater than 25 years of experience. Thus, their perceptions and suggestions would be of immense value in guiding policy makers in their efforts to bring about a change in the examination system. The respondents are also from different states of the country, giving a reasonably broad coverage on the examination pattern in these regions. The main limitation of this survey is the restricted geographical representation of the respondents in a vast country like India. Thus, these results cannot be extrapolated to the country without conducting larger surveys of examiners from other parts of the country. Another limitation is the small number of questions provided. We have also not included questions on other important domains like communication skills, humanistic qualities and counseling skills. However, this study is still relevant and valuable as a starting point for the universities and examiners to review the current system of examinations and as a stimulus for a gradual shift to a more objective, valid and reliable system.

To conclude, this online survey of DM Neurology examiners showed that a vast majority of the examiners are not satisfied with the current system of assessment. Their key suggestions to improve the system include objective assessments like MCQ, OSCE, OSLER and DOPS, inclusion of larger number of short answer type questions in addition to the long answer type questions, periodic internal assessments and training or orientation of examiners. A larger detailed survey from other parts of the country followed by consensus guidelines developed by a working group of the Indian Academy of Neurology are essential to bring about the much-needed improvements. Implementation of such guidelines hopefully ensures uniform minimum national standards for the neurology exit examination in India.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.