| Literature DB >> 21197366 |
Eric J Maclaughlin1, David S Fike, Carlos A Alvarez, Charles F Seifert, Amie T Blaszczyk.
Abstract
PURPOSE: Formal presentations are a common requirement for students in health professional programs, and evaluations are often viewed as subjective. To date, literature describing the reliability or validity of seminar grading rubrics is lacking. The objectives of this study were to characterize inter-rater agreement and internal consistency of a grading rubric used in a grand rounds seminar course.Entities:
Keywords: evaluation; grand rounds; public speaking; seminar
Year: 2010 PMID: 21197366 PMCID: PMC3004600 DOI: 10.2147/JMDH.S12346
Source DB: PubMed Journal: J Multidiscip Healthc ISSN: 1178-2390
Baseline student demographics
| Age (yrs)a | 29.9 ± 5.8 | 31.2 ± 6.3 | 28.1 ± 4 | 28.3 ± 7 | 0.006 |
| Gender | |||||
| Male (%) | 44.6 | 49.1 | 35.6 | 53.8 | 0.124 |
| Race | |||||
| Caucasian (%) | 60 | 72.7 | 42.5 | 76.9 | <0.001 |
| Asian (%) | 16 | 3.6 | 31.5 | 2.6 | |
| Hispanic (%) | 17 | 20 | 13.7 | 17.9 | |
| Other (%) | 7.2 | 3.6 | 12.3 | 2.6 | |
| Pre-enrollment | 88.5 ± 4.1 | 87.4 ± 3.8 | 89.2 ± 4.3 | 88.6 ± 4 | 0.05 |
| GPA (%) | |||||
Notes:
Expressed as mean ± SD;
Difference exists between Amarillo and Dallas;
Dallas differs from Amarillo and Lubbock;
Difference exists between Amarillo and Dallas.
Figure 1Histogram depicting the differences in score between faculty graders.
Grade difference of faculty pairs stratified by differences in academic rank
| 0 | 83 | 4.36 | 3.03 | 3.6996 | 5.0233 |
| 1 | 78 | 4.69 | 4.07 | 3.7681 | 5.6047 |
| 2 | 59 | 4.95 | 4.07 | 3.8864 | 6.0051 |
| 3 | 32 | 3.89 | 2.91 | 2.8427 | 4.9385 |
Notes:
Academic rank of faculty pair: 0 represents same rank, 1 represents a difference of one level of rank (eg, assistant professor and associate professor), 2 represents a difference of two levels of rank (eg, assistant professor and professor), 3 represents a difference of three levels of rank (instructor and professor);
No difference in scores were noted across faculty ranks by one-way analysis of variance (P = 0.553).
Correlation between student self-evaluation and faculty presentation scoresa (n = 252)
| Content scores | 0.513 | <0.001 |
| Communication scores | 0.455 | <0.001 |
| Overall presentation scores | 0.539 | <0.001 |
Note:
Faculty presentation score was the student’s final presentation grade (ie, average of faculty grader 1 and 2 scores).
Figure 2Differences between student self-evaluation and faculty presentation scores by quintile.a
Notes: aFaculty presentation score was the student’s final presentation grade (ie, average of faculty grader 1 and 2 scores). Statistical significance assessed by Dunnett’s T3 post hoc test. A negative value indicates that the student’s score was less than the mean faculty grader score and a positive value indicates that it was greater; †Differs from quintiles 3 (P = 0.001), 4 (P < 0.001), and 5 (P < 0.001); ‡Differs from quintiles 3 (P = 0.023), 4 (P < 0.001), and 5 (P < 0.001).
| Topic Selection | Relevant to current pharmacy practice Interesting to broad audience Timely/cutting edge (eg, new data or controversy or applicable to current practice) Scope/focus appropriate (not too broad or narrow) | Relevant to current pharmacy practice Interesting to broad audience Timely/cutting edge (eg, new data or controversy or applicable to current practice) Scope/focus appropriate (not too broad or narrow) | Relevant to current pharmacy practice Interesting to broad audience Timely/cutting edge (eg, new data or controversy or applicable to current practice) Scope/focus appropriate (not too broad or narrow). | Relevant to current pharmacy practice Interesting to broad audience Timely/cutting edge (eg, new data or controversy or applicable to current practice) Scope/focus appropriate | 5% | 0 | |
| Objectives | All objectives clearly described and use measurable terms AND No overlap of objectives AND All objectives addressed AND Appropriate number of objectives (∼4) | Most objectives clearly described and use measureable terms Little overlap in objectives Most objectives addressed Number of objectives reasonable | Objectives unclear and ill defined Objectives overlap considerably in action verbs Most objectives not addressed Inappropriate # for presentation length | No objectives identified OR Objectives do not relate to presentation | 5% | 0 | |
| Introduction | Intro captured audience attention AND Thesis/purpose exceptionally clear AND Intro concise and well organized AND Provided clear overview of talk | Captured some of audience attention Thesis/purpose somewhat clear At times wordy or too brief; mostly organized Generally clear overview of talk | Did not capture audience attention Thesis/purpose not clear Too wordy or brief too and vague Preview of talk confusing and disorganized | No introduction presented in talk OR Intro not relevant to presentation | 10% | 0 | |
| Organization | Concise and complete intro and conclusion AND Clear and logical progression throughout AND All facts linked to topic and objectives AND All major points highlighted | Somewhat brief introduction and conclusion Mostly clear and logical progression Most facts linked to topic and objectives Most major points highlighted | Minimal intro and conclusion Progression throughout difficult to follow Little link between facts and topic/objectives Major points sparsely highlighted | No introduction or conclusion used No logical progression of ideas Facts not linked to topic and objectives Major points not highlighted | 20% | 0 | |
| Primary Literature Citation and Analysis | Comprehensive incorporation of primary literature with most relevant/timely references elaborated upon AND Analysis of literature and/or trial design insightful and accurate | Most key primary literature cited and incorporated Most literature current/timely Analysis of literature and/or trial design limited to provided author’s conclusion(s) | Little primary literature used in talk Some key articles missing Much literature out-of-date Little analysis of literature and/or trial design; recited data | Relied on secondary or tertiary literature (key primary literature missed) OR No current literature cited OR No analysis of literature and/or trial | 20% | 0 | |
| Statistical Interpretation of Data | Tests named, explained, justified, and critiqued with alternative tests identified if appropriate AND Number needed to treat (NNT) or harm (NNH) calculated for all appropriate data | Tests named, explained, and justified if appropriate NNT or NNH calculated for some data if appropriate | Tests named but not explained or justified if appropriate No NNT or NNH calculated if appropriate | No statistical tests named if appropriate | 10% | 0 | |
| Analysis and Application in Practice | Addressed both contemporary and future practice AND Gave well thought-out, detailed recommendation on how to apply including additional data needed Broad perspective given | Addressed both contemporary and future practice Perspective limited | Addressed only one specific setting or perspective Superficially addressed setting and/or perspective | Did not address a specific setting or perspective | 15% | 0 | |
| Response to Questions | All questions were answered correctly AND Was able to justify answers AND Paraphrased understanding of all questions | Majority of questions were answered correctly Most answers were justified Paraphrased understanding of most questions | Majority of questions were only partially answered or not answered correctly Majority of answers poorly justified Paraphrased few questions | Questions were not answered or justified Paraphrased understanding of no questions | 15% | 0 | |
| 0% | |||||||
| Professionalism | Exceptionally dressed (business attire) AND Formal tone and attitude displayed AND May serve as a positive role model for future presenters | Mostly appropriately dressed Acceptable tone and attitude displayed Possibly could serve as a positive role model for future presenters | Somewhat appropriately dressed Tone and attitude too informal Questionable ability to serve as a positive role model for future presenters | Inappropriately dressed Would not serve as a positive role model for future presenters | 10% | 0 | |
| Transitions | All transitions between major areas in talk exceptionally clear and appropriate AND Brief summaries of key points provided for all major topic areas | Most transitions between major areas in talk clear and appropriate Brief summaries of key points provided for most major topic areas | Few transitions between major areas in talk clear and appropriate Few brief summaries of key points provided for major topic areas | Transitions between major areas in talk unclear inappropriate No brief summaries of key points provided for major topic areas | 15% | 0 | |
| Slides and Graphics | Amount of material on slide facilitated understanding of presentation AND Slides contained quality pictures, diagrams, tables, and/or animations AND Slide background and font were professional and enhanced readability AND Slides free from typos and grammatical errors, abbreviations defined | Some slides contained too much or too little information Slides mostly text, some inclusion of a few basic tables, diagrams, or clip art as pictures Slide background and font was acceptable and readable Mostly free from typos and grammatical errors, most abbreviations defined | Most slides contained too much or too little information Slides consisted almost entirely of text; tables, diagrams, or pictures rarely used Background and font unprofessional and/or distracting and/or compromised readability Many typos and grammatical errors, few abbreviations defined | All slides contained either too much or too little information OR All slides were text; no tables, diagrams, or pictures used OR Slide background and font was unreadable and completely distracting OR Many errors and unreadable | 20% | 0 | |
| Presentation Style | Maintains eye contact with audience AND Rarely returns to notes AND Exceptional and consistent facial expressions, gestures, and posture. No distracting movements or gestures | Eye contact maintained most of the time Returns to notes occasionally Acceptable facial expressions, gestures, and posture. Minimal distracting movements or gestures | Eye contact made rarely Most of presentation read Inconsistent and incongruent facial expressions, gestures, and posture Some distracting movements or gestures | Does not make eye contact Reads entire presentation Consistently poor and incongruent facial expressions, repetitive, distracting gestures, and poor posture | 20% | 0 | |
| Elocution | Always articulate with no pronunciation or grammatical errors AND Always uses correct medical/scientific nomenclature AND All word fillers (eg, “um”) appropriate and not distracting AND All attendees can hear presentation Rate of speech ideal | Mostly articulate with few (2–3) pronunciation or grammatical errors Rarely uses incorrect medical/scientific nomenclature Word fillers mostly appropriate and rarely distracting Most attendees can hear presentation Rate of speech slightly too fast or slow | Mostly inaudible and inarticulate with several (3–5) pronunciation or grammatical errors Frequently uses incorrect medical/scientific nomenclature Word fillers frequent and distracting Many attendees can not hear presentation Rate of speech significantly too fast or slow | Inaudible and nonarticulate with numerous errors (>5) OR Constantly uses incorrect medical/scientific nomenclature OR Constant use of word fillers Rate of speech so fast or slow that presentation is not comprehendible | 15% | 0 | |
| Accuracy and Completeness of References | Bibliography complete, in proper format, and no errors AND All graphs, charts, and tables appropriately referenced | Bibliography mostly complete, in proper format, with few (<2) errors Most graphs, charts, and tables appropriately referenced | Bibliography mostly incomplete, not in proper format, with several (>2) errors Most graphs, charts, and tables not appropriately referenced | No bibliography provided OR No graphs, charts, and tables were appropriately referenced | 5% | 0 | |
| Time Management (Goal 40 min, | Spends an appropriate amount of time on the major sections of the presentation AND Presentation within 2.5 minutes of target | Spends an appropriate amount of time on a majority of the major sections of the presentation Presentation within 2.5–5 minutes of target | Spends an inappropriate amount of time on the majority of sections of the presentation (too much or too little) Presentation within 5–10 minutes of target | Inappropriate time spent on all of major sections. | 15% | 0 | |
| _____________________________________________________________________________________________________________________________ | |||||||
| _____________________________________________________________________________________________________________________________ | |||||||