| Literature DB >> 25828516 |
Hamza Mohammad Abdulghani1, Farah Ahmad1, Mohammad Irshad1, Mahmoud Salah Khalil1, Ghadeer Khalid Al-Shaikh2, Sadiqa Syed3, Abdulmajeed Abdurrahman Aldrees1, Norah Alrowais4, Shafiul Haque5.
Abstract
The aim of this study was to assess the utility of long term faculty development programs (FDPs) in order to improve the quality of multiple choice questions (MCQs) items' writing. This was a quasi-experimental study, conducted with newly joined faculty members. The MCQ items were analyzed for difficulty index, discriminating index, reliability, Bloom's cognitive levels, item writing flaws (IWFs) and MCQs' nonfunctioning distractors (NFDs) based test courses of respiratory, cardiovascular and renal blocks. Significant improvement was found in the difficulty index values of pre- to post-training (p = 0.003). MCQs with moderate difficulty and higher discrimination were found to be more in the post-training tests in all three courses. Easy questions were decreased from 36.7 to 22.5%. Significant improvement was also reported in the discriminating indices from 92.1 to 95.4% after training (p = 0.132). More number of higher cognitive level of Bloom's taxonomy was reported in the post-training test items (p<0.0001). Also, NFDs and IWFs were reported less in the post-training items (p<0.02). The MCQs written by the faculties without participating in FDPs are usually of low quality. This study suggests that newly joined faculties need active participation in FDPs as these programs are supportive in improving the quality of MCQs' items writing.Entities:
Year: 2015 PMID: 25828516 PMCID: PMC4381327 DOI: 10.1038/srep09556
Source DB: PubMed Journal: Sci Rep ISSN: 2045-2322 Impact factor: 4.379
Specification of examination (Total number of MCQs = 80; Marks = 30)
| Courses | Students passed, n(%) | Students failed, n(%) | Mean score | Reliability coefficient |
|---|---|---|---|---|
| Respiratory | 24/64(37.5) | 40/64(62.5) | 16.89 | 0.93 |
| Cardiovascular | 27/63(42.9) | 36/63(57.1) | 17.02 | 0.93 |
| Renal | 41/61(67.2) | 20/61(32.8) | 20.96 | 0.94 |
| Respiratory | 46/80(57.5) | 34/80(42.5) | 18.7 | 0.92 |
| Cardiovascular | 40/73(54.8) | 33/73(45.2) | 20.94 | 0.93 |
| Renal | 39/69(56.5) | 30/69(43.6) | 21.31 | 0.92 |
Different factors associated with the item analysis
| Pre | Post | Pre | Post | Pre | Post | Pre | Post | |||
|---|---|---|---|---|---|---|---|---|---|---|
| Factors | Categories | n(%) | n(%) | n(%) | n(%) | n(%) | n(%) | n(%) | n(%) | χ2(p) |
| Difficulty Index (P) | Difficult (<20%) | 1(1.3) | 0(0) | 0(0) | 2(2.5) | 1(1.3) | 1(1.3) | 2(0.8) | 3(1.3) | 11.61(0.003) |
| Moderated (20–70%) | 55(68.8) | 61(76.3) | 59(73.8) | 68(85.0) | 36(45.0) | 54(67.5) | 150(62.5) | 183(76.3) | ||
| Easy (>70%) | 24(30.0) | 19(23.8) | 21(26.3) | 10(12.5) | 43(53.8) | 25(31.3) | 88(36.7) | 54(22.5) | ||
| Total | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 240(100) | 240(100) | ||
| Discrimination Index (DI) | Non-DI(≤0.15) | 6(7.5) | 3(3.80) | 7(8.8) | 4(5.0) | 6(7.5) | 4(5.0) | 19(7.9) | 11(4.6) | 2.27 (0.13) |
| DI(>0.15) | 74(92.5) | 77(96.3) | 73(91.3) | 76(95.0) | 74(92.5) | 76(95.0) | 221(92.1) | 229(95.4) | ||
| Total | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 240(100) | 240(100) | ||
| Non-function distractors (NFD) | NFD | 4(5.0) | 2(2.5) | 8(10.0) | 7(8.8) | 16(20.0) | 4(5.0) | 28(11.7) | 13(5.4) | 6.00 (0.02) |
| FD | 76(95.0) | 78(97.5) | 72(90.0) | 73(91.3) | 64(80.0) | 76(95.0) | 212(88.3) | 227(94.6) | ||
| Total | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 240(100) | 240(100) | ||
| Bloom's taxonomy levels | K1 | 45(56.3) | 40(50.0) | 54(67.5) | 40(50.0) | 57(71.3) | 37(46.3) | 156(65.0) | 117(48.8) | 12.91 (<0.0001) |
| K2 | 35(43.8) | 40(50.0) | 26(32.5) | 40(50.0) | 23(28.8) | 43(53.8) | 84(35.0) | 123(51.3) | ||
| Total | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 240(100) | 240(100) | ||
| Items writing flaws (IWFs) | IWF | 22(27.5) | 1(1.3) | 15(18.8) | 0(0.0) | 10(12.5) | 4(5.0) | 47(19.6) | 5(2.1) | 38.04 (0.<0001) |
| Without-IWF | 58(72.5) | 79(98.8) | 65(81.3) | 80(100) | 70(87.5) | 76(95.0) | 193(80.4) | 235(97.9) | ||
| Total | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 80(100) | 240(100) | 240(100) | ||
*Pre-training, **Post-training, ***Functional distractor, K1 = Non-scenario bases, K2 = Scenario bases.
Analysis of variance between pre- and post-training test items
| Subjects | Factors | F | ANOVA (p) |
|---|---|---|---|
| Respiratory | Difficulty Index (P) | 4.964 | 0.027 |
| Discrimination Index (DI) | 1.053 | 0.306 | |
| Non-function distractors (NFD) | 0.687 | 0.408 | |
| Bloom's taxonomy levels | 0.622 | 0.431 | |
| Items writing flaws (IWF) | 25.711 | 0.000 | |
| Cardiovascular | Difficulty Index (P) | 6.253 | 0.013 |
| Discrimination Index (DI) | 0.872 | 0.352 | |
| Non-function distractors (NFD) | 0.073 | 0.788 | |
| Bloom's taxonomy levels | 5.154 | 0.025 | |
| Items writing flaws (IWF) | 18.231 | 0.000 | |
| Renal | Difficulty Index (P) | 7.852 | 0.006 |
| Discrimination Index (DI) | 0.422 | 0.517 | |
| Non-function distractors (NFD) | 8.566 | 0.004 | |
| Bloom's taxonomy levels | 10.889 | 0.001 | |
| Items writing flaws (IWF) | 3.833 | 0.050 |
Figure 1Flow-chart of MCQs items' writing training workshop program structure according to the Kirkpatrick's levels of evaluation (adopted from Abdughani et al., 2014).