Competency-based tool for evaluation of community-based training in undergraduate medical education in India - a Delphi approach.

INTRODUCTION: A community-based training (CBT) program, where teaching and training are carried out in the community outside of the teaching hospital, is a vital part of undergraduate medical education. Worldwide, there is a shift to competency-based training, and CBT is no exception. We attempted to develop a tool that uses a competency-based approach for assessment of CBT.
METHODS: Based on a review on competencies, we prepared a preliminary list of major domains with items under each domain. We used the Delphi technique to arrive at a consensus on this assessment tool. The Delphi panel consisted of eight purposively selected experts from the field of community medicine. The panel rated each item for its relevance, sensitivity, specificity, and understandability on a scale of 0-4. Median ratings were calculated at the end of each round and shared with the panel. Consensus was predefined as when 70% of the experts gave a rating of 3 or above for an item under relevance, sensitivity, and specificity. If an item failed to achieve consensus after being rated in 2 consecutive rounds, it was excluded. Anonymity of responses was maintained.
RESULTS: The panel arrived at a consensus at the end of 3 rounds. The final version of the self-assessment tool consisted of 7 domains and 74 items. The domains (number of items) were Public health - epidemiology and research methodology (13), Public health - biostatistics (6), Public health administration at primary health center level (17), Family medicine (24), Cultural competencies (3), Community development and advocacy (2), and Generic competence (9). Each item was given a maximum score of 5 and minimum score of 1.
CONCLUSION: This is the first study worldwide to develop a tool for competency-based evaluation of CBT in undergraduate medical education. The competencies identified in the 74-item questionnaire may provide the base for development of authentic curricula for CBT.

Introduction

Community-based training (CBT) program is a vital part of undergraduate medical education (UGME) where teaching and training are carried out in the community outside the teaching hospital.1–3 In India, CBT is managed by the Department of Community Medicine or Preventive and Social Medicine. CBT is offered from the first year of Bachelor of Medicine and Bachelor of Surgery (MBBS) with the objective of orienting the students to community-based health care services. Through CBT, students are trained in all 4 core disciplines of community medicine: family medicine, epidemiology, health promotion, and health management.4

In a developing country like India where the predominant section of population is in the villages and suburban areas, teaching in tertiary care hospitals alone does not equip the students with skills essential to work in the community. They have to be trained to work at all levels of health care delivery system. The Reorientation of Medical Education scheme, though not as successful as it was conceived to be, is one of the notable attempts to deliver CBT effectively.5,6

In the UGME scenario worldwide, there is a shift toward competency-based training, and the same is also recommended by an expert group commissioned by the World Health Organization.7,8 Medical Council of India (MCI), the apex body which regulates medical education in India, has in its Vision 2015 document recommended a shift toward competency-based approach.9 The change in the approaches to teaching also necessitates a change in the assessment methods used.10

The National Health Mission in India emphasizes the need for competent health care providers in rural areas. Unlike secondary or tertiary care systems, medical officers working under primary care system do not have the opportunity/privilege to work under experienced health care team. They have to be equipped with skills in clinical judgment and administration of the health center. Apart from this, they have to train their team of paramedical workers and frontline workers in the community. In this case, it is the responsibility of the medical education system to ensure that the candidates have acquired these essential competencies before they graduate and venture into the community on their own.

Currently, there are no competency-based assessment tools available for CBT. We attempted to develop a tool that uses competency-based approach for assessment of CBT. In this paper, we describe the development of a 74-item competency-based questionnaire using Delphi technique. The psychometric properties of the 74-item questionnaire and the development of an abridged 58-item self-assessment questionnaire using exploratory factor analysis are described elsewhere.11

Methods

Study setting

In India, medical graduation or MBBS is covered over 4 and a half years (9 semesters) followed by a year of internship. The subject community medicine is taught since first year till 7th semester through theory sessions and CBT. There are 3 clinical postings under CBT where the undergraduate students are posted in rural/urban health training centers each for 4 weeks. In addition to clinical postings, students also undergo Family Health and Advisory Programme during which they follow up a family (in the community they serve) allotted to them through weekly home visits. The students appear for a final theory and clinical/practical examination in community medicine at the end of the 7th semester. Then, the students are posted in the community as part of Compulsory Rotatory Residential Internship (CRRI) for a period of 2 months where they are expected to practice all the competencies gained.

Questionnaire development – Delphi technique

Based on a review on competencies to be acquired under a CBT program, we developed a conceptual framework using 6 core competencies after adapting it to Indian context and prepared a preliminary list of major domains with items under each domain.12 While deciding on the items, we considered the exposure of the student to the competency during his/her training and the practical requirement of the competency in his/her day-to-day practice of community medicine in future. We had designed it as a self-rated questionnaire where the student will be required to rate his/her competencies. The rating scale for each item was a Likert scale ranging from “much above average” (score=5) to “much below average” (score=1).

We used Delphi technique to develop the questionnaire and arrive at a consensus.13–16 We purposively selected a panel of experts (n=8) in community medicine. The prerequisites were that the expert should have a minimum of 3 years’ experience in health service provision and/or teaching and training and/or research, post his/her postgraduation (MD) in community medicine. The list of members who constituted the expert panel is presented in Table 1. The principal investigator (HDS) facilitated the process but was not part of the panel. Email was the medium of communication with the experts. All experts were aware of the list of experts constituting the panel, but anonymity of the responses was maintained.

Table 1

Panel of experts for consensus building (Delphi technique) to develop competency-based self-assessment questionnaire in community-based training program

SN	Affiliation	Age/sex	Place	Years of post-MD teaching experience
1	Professor, Department of Community Medicine, Post Graduate Institute of Medical Education and Research (PGIMER)	59/male	Chandigarh, India	33
2	Professor, Department of Community Medicine, Sri Manakula Vinayagar Medical College (SMVMC)	41/male	Puducherry, India	13
3	Public Health Specialist, John Hopkins School of Public Health	36/male	Baltimore, MD, USA	10
4	Professor, Department of Preventive and Social Medicine, Jawaharlal Institute of Postgraduate Medical Education and Research (JIPMER)	56/male	Puducherry, India	27
5	Assistant Professor, Indian Institute of Public Heath	37/male	New Delhi, India	10
6	Assistant Professor, Department of Preventive and Social Medicine, Jawaharlal Institute of Postgraduate Medical Education and Research (JIPMER)	35/male	Puducherry, India	6
7	Assistant Professor, Department of Preventive and Social Medicine, Jawaharlal Institute of Postgraduate Medical Education and Research (JIPMER)	34/female	Puducherry, India	8
8	Associate Professor, Department of Community Medicine, North Eastern Indira Gandhi Regional Institute of Health and Medical Sciences (NEIGRIHMS)	40/male	Shillong, India	10

Abbreviation: SN, serial number.

Each Delphi round spanned over 2 weeks. During the 1st week, the Delphi panel experts gave their comments. During the 2nd week, the facilitator compiled the comments. In the light of the comments, the questionnaire was revised and recirculated among the experts.

In round 1, we shared the preliminary draft of the questionnaire prepared by us and a rating sheet. Experts rated each item (close-ended response) in part I of the rating sheet and gave suggestions (open-ended) in part II. The facilitator requested the experts to rate each item (rating scale was between 0 [poor] and 4 [good]) under the heads of relevance, sensitivity, specificity, and understandability (Box 1). We sought suggestions to improve understandability, language of items, and wording. Comments on adequacy and any additional items for inclusion were also welcomed. We requested the experts to also suggest alternate options for the response scale used in the questionnaire. At the end of each round, the median ratings received by each of the items were shared with the experts. The comments were also anonymously shared.

The iterative process continued up to the consensus point. Consensus was predefined as when 70% of the experts gave a rating of 3 or above for each item under relevance, sensitivity, and specificity (content validity). An item that did not achieve consensus was allowed to be rated again by the experts in the light of the compiled ratings and open-ended comments at the end of the previous round. If an item failed to achieve consensus after being rated in 2 consecutive rounds, it was excluded. Any item that achieved consensus was not allowed for rating again in the consecutive rounds. Where they had given poor rating for items under the head “understandability”, we requested the experts for suggestions to improve the same.

Relevance: Item considered relevant if the given item is found appropriate and important in assessing the student’s competency under the given domain.

Sensitivity: Item considered sensitive if it shall be rated high by those who possess the competency under the given item and vice versa.

Specificity: Item considered having good specificity if it exclusively measures the student’s competency under the given domain.

Understandability: Item considered having good understandability if the wording is simple and unambiguous to the student evaluating himself/herself.

Pretesting

The draft of the questionnaire at the end of Delphi process was shared with 3 students (2 female, 1 male). He spent 45 minutes with each discussing the relevance; adequacy of concepts, language, and responses; understandability; and any other difficulties he/she faced in interpreting the questionnaire.

Data entry and analysis

The scores assigned to each item under the 4 criteria were entered into Microsoft Excel. Median ratings were calculated at the end of each round. The extent of consensus arrived at the end of each round was calculated as the percentage of experts assigning a score of 3 or more under each characteristic of an item.

Ethics

The Institute Research Committee of Indira Gandhi Medical College and Research Institute (IGMCRI), Puducherry, approved the study. Written informed consent via email was obtained from each of the Delphi panel experts before including them in the panel.

Results

Questionnaire development

All eight experts participated in each round of Delphi. The preliminary draft of the questionnaire had 6 domains namely “Public health”, “Family medicine”, “Cultural competence”, “Community development and advocacy”, “Research and evidence-based practice”, and “Generic competence” and 81 items.

The details of the modification of the tool in each of the Delphi rounds are presented in Table 2. At the end of round 1, 59 out of the 81 items had attained consensus. Nineteen new items were suggested. So, a total of 41 items were presented for rating by the expert group in the second round along with the compiled ratings and open-ended comments (blinded) from the previous round. There was rearrangement of items under some domains and rephrasing of certain items as suggested by the experts. Two domains namely “Public health” and “Research and evidence-based practice” were regrouped to form 3 domains namely “Public health – epidemiology and research methodology”, “Public health – biostatistics”, and “Public health administration at PHC level”.

Table 2

Details of the modification of the tool in each of the Delphi rounds

Characteristics	Baseline	Round 1	Round 2	Round 3 (final)
Number of items that were presented for rating	81	81	22	0
New items added	NA	19	0	0
Number of items that were presented for re-rating	NA	NA	19	9
Total number of items for which consensus was achieved	NA	59	59+14 (4 out of 22 and 10 out of 19) =73	73+1
Number of items deleted	NA	0	18	8
Number of domains at the end of the round	6	7	7	7
Number of items at the end of the round	81	100	82	74
Any other	NA	Domains rearranged*	Changed to modified Miller’s response scale	–

Note:

Two domains, namely “Public health” and “Research and evidence-based practice” were regrouped to form 3 domains namely “Public health – epidemiology and research methodology”, “Public health – biostatistics”, and “Public health administration at primary health center level”.

Abbreviation: NA, not applicable.

At the end of round 2, 4 out of the 22 items rated for the second time achieved consensus leading to deletion of the remaining 18 items. Out of the 19 newly added items, 10 had achieved consensus. The remaining 9 were presented for re-rating in round 3. No new items were added in round 2. The response scale was changed to a modified form of Miller’s response scale as suggested by the Delphi panel. The final response scale was an adaptation of the Miller’s triangle to assess competency: “don’t know”, “know”, “know how”, “show how”, and “do”.17 Miller’s triangle, which may be applied as a part of Objective Structured Clinical Examination (OSCE) for rater assessment, was adapted and used in this self-assessment questionnaire. Each item in the questionnaire under all domains except “Generic competence” had this response scale. The domain “Generic competence” had the following response scale: “strongly agree”, “agree”, “neither agree nor disagree”, “disagree”, and “strongly disagree”.

At the end of round 3, only 1 out of the 9 items had achieved consensus. The remaining 8 items were excluded. Like in round 2, no new items were added during round 3. Hence, the Delphi process came to a conclusion at the end of 3 rounds.

The final questionnaire prepared after completion of Delphi process is presented in Figure S1. All excluded items that did not achieve consensus are listed in Table S1. The questionnaire had 7 domains and 74 items. The domains (number of items) were “Public health – epidemiology and research methodology” (13), “Public health – biostatistics” (6), “Public health administration at PHC level” (17), “Family medicine” (24), “Cultural competencies” (3), “Community development and advocacy”(2), and “Generic competence” (9). Each item was given a maximum score of 5 and minimum score of 1. Higher score indicated better skill score for each item in the questionnaire. Hence, maximum and minimum possible scores for a student were 370 and 74, respectively.

There were no significant changes made to the questionnaire after pretesting with the students.

Discussion

This is the first study from India and worldwide to develop a tool for competency-based evaluation of CBT in UGME. Drawing from an existing conceptual framework, we designed the preliminary draft of the questionnaire. Though various inventories of competency classifications are available,18–26 we have based our tool on a conceptual framework specific to CBT.12 This draft evolved through 3 rounds of Delphi into the final version composed of 7 domains and 74 items.

Appropriateness of Delphi technique

With increasing use of Delphi to address different research questions, there have been variants of the classical Delphi, thus necessitating the term “Delphi techniques” or the “Delphi approach”.27,28 Fink et al have prescribed a clear “decision trail” as one of the key goodness criteria to judge the credibility of the evidence generated by Delphi.29

Delphi technique is considered as a structured way of assessing and synthesizing/combining human judgment. Rowe et al state that Delphi can be used when the researcher is convinced that the technique will generate more accurate assessments and judgments compared to that provided by individuals.30 Delphi technique has also been used in UGME scenario worldwide.31–33 We resorted to Delphi technique given that our objective was to develop a comprehensive tool for competency-based evaluation of CBT.

Delphi technique limits the inhibition that the participants may face in other informal group situations by promising anonymity. Thus, the technique encourages the expert to offer his/her frank and candid opinion(s) which is termed as “process gain”.30 We are quite sure that we would not have been able to achieve this using any other technique given the issues of seniority, interfering or inhibiting personality traits that are quite evident in other face-to-face meetings of experts.

We understand that Delphi does not offer a fool-proof solution to these issues. But it does circumvent these to a great extent. Hence, we had chosen Delphi as one of the steps in the development of the tool. Delphi was preceded by the use of a conceptual framework to develop a preliminary draft derived from review of literature in the field. After Delphi, the tool was informally discussed with students to obtain their feedback on the tool. This was followed by psychometric analysis to assess the validity and reliability of the tool, the results of which are reported elsewhere.

Recruitment of Delphi panel

We recruited a heterogeneous group of experts from across the country to contribute toward the development of the same. We aimed to draw from all their knowledge and experience while also achieving consensus. Given that there is no prescribed minimum size of the panel, we recruited 8 experts.13

Data collection procedures

We had adhered to all the 4 essential prerequisites of a Delphi technique namely anonymity of participants, iteration, controlled feedback, and statistical aggregation of the results of the rounds. Our panelists could modify their judgment based on feedback without being influenced by others in the group.13,30 Although anonymity is reported to cause a lack of accountability of one’s views, we believe that it is not a major drawback in our study as the outcome of this study has direct application and relevance to the practice of experts themselves who were chosen based on their experience and expertise.

We encouraged qualitative feedback also, so as to not restrict the experts to rating the existing items in the tool. This qualitative feedback enabled us to reclassify certain items under the domains, rephrase certain items and domain names, add relevant items, and delete redundant items or items beyond the scope of the tool.

Means of implementation

The competencies identified in the 74-item questionnaire may provide the base for development of authentic curricula for CBT. In India, community medicine is taught from 1st semester to 7th semester: the period is divided as preclinical (1st–2nd semester), para-clinical (3rd–5th semester), and clinical (6th–7th semester). Competencies pertaining to CBT may accordingly be divided over the preclinical, para-clinical and clinical training periods, with clinical competencies under CBT covered in the 6th and 7th semester. Practice of these competencies, under supervision, is expected during the CRRI period. The tool may find its best application at the end of the CRRI period, though it may also be administered at the end of the 7th semester.

Of the 74 competencies in 74-item questionnaire, 41 competencies (55%) were pertaining to “Family medicine” or “Public health administration at PHC level”. This draws our attention to the primary domains of focus for faculty of community medicine–family medicine and community health administration.34 It is their primary role to impart knowledge and skills in these domains to the students, and faculties need to be sensitized and reoriented in this regard.35–37

Competency-based CBT is likely to face challenges in terms of curricula design, faculty training, student assessment, and systematic institutional change, all of which require sustained, long-term commitment.38

Limitations

There were some limitations. This is a self-rated questionnaire which captures a student’s perception about his/her competencies. Currently, self-assessment or rating does not figure in UGME scenario in India. We propose to develop an instructor-/teacher-rated version of this tool which can be routinely employed in tandem with the existing assessment methods.

Conclusion

This tool can be seen as the sign of entering into the realm of competency-based assessment in community-based UGME in India. It is a valuable addition to the existing assessment methods in India and can guide experts in a need-based design of curriculum and teaching/training methodology.

The 74-item competency-based self-assessment questionnaire for assessing community-based training of undergraduate medical students

Abbreviations: AFB, acid fast bacillus; IFA, iron folic acid; L, laboratory; NCD, non-communicable disease; P, presumptive; PHC, primary health center; RNTCP, Revised National Tuberculosis Control Program; NVBDCP, National Vector Borne Diseases Control Program; RTI, reproductive tract infection; S, syndromic; SN, serial number; STI, sexually transmitted infection; WHO, World Health Organization.

Table S1

Item(s) that did not attain consensus after 2 rounds of Delphi, and hence, were excluded from the questionnaire (n=26)

SN	Item
1	Frame null and alternate hypothesis for your research question
2	Look for bias in a given research article
3	Interpret percentiles in WHO growth charts
4	Making a social map
5	Evaluating beat schedule of health workers
6	Knowledge of registers at PHC
7	Fund raising
8	Accessing and using information for health
9	Appraise available scientific evidence
10	Write a research proposal
11	Skills in scientific writing
12	Presentation of research results
13	Team work with staff
14	Life table
15	Use of multimedia/PowerPoint
16	Utilization and management of RKS fund
17	Disaster relief
18	Patient empowerment
19	Conduct training of staff on different aspects of national programs
20	Conduct rapid survey assessment method like for immunization coverage
21	Integrated medicine
22	Self-care
23	Counseling skills especially for adolescent health problems
24	Primary health care (general OPD)
25	Ability to carry out a cross-sectional study (descriptive/analytical)
26	Ability to prepare microplan for IPPI or any special activity

Abbreviations: IPPI, intermittent pulse polio immunization; OPD, outpatient department; PHC, primary health center; RKS, Rogi Kalyan Samiti; SN, serial number; WHO, World Health Organization.