Emma Claire Phillips 1,2,3 , Samantha Eve Smith 3 , Benjamin Clarke 2 , Ailsa Lauren Hamilton 2 , Joanne Kerins 2 , Johanna Hofer 2 , Victoria Ruth Tallentire 1,2,3 Show Affiliations »
Abstract
Background: The Medical Students' Non-Technical Skills (Medi-StuNTS) behavioural marker system (BMS) is the first BMS to be developed specifically for medical students to facilitate training in non-technical skills (NTS) within immersive simulated acute care scenarios. In order to begin implementing the tool in practice, validity evidence must be sought. We aimed to assess the validity of the Medi-StuNTS system with reference to Messick's contemporary validity framework. Methods: Two raters marked video-recorded performances of acute care simulation scenarios using the Medi-StuNTS system. Three groups were marked: third-year and fourth-year medical students (novices), final-year medical students (intermediates) and core medical trainees (experts). The scores were used to make assessments of relationships to the variable of clinical experience through expert-novice comparisons, inter-rater reliability, observability, exploratory factor analysis, inter-rater disagreements and differential item functioning. Results: A significant difference was found between the three groups (p<0.005), with experts scoring significantly better than intermediates (p<0.005) and intermediates scoring significantly better than novices (p=0.001). There was a strong positive correlation between the two raters' scores (r=0.79), and an inter-rater disagreement of more than one point in less than one-fifth of cases. Across all scenarios, 99.7% of skill categories and 84% of skill elements were observable. Factor analysis demonstrated appropriate grouping of skill elements. Inconsistencies in test performance across learner groups were shown specifically in the skill categories of situation awareness and decision making and prioritisation. Conclusion: We have demonstrated evidence for several aspects of validity of the Medi-StuNTS system when assessing medical students' NTS during immersive simulation. We can now begin to introduce this system into simulation-based education to maximise NTS training in this group. © Author(s) (or their employer(s)) 2021. No commercial re-use. See rights and permissions. Published by BMJ.
Background: The Medical Students' Non-Technical Skills (Medi-StuNTS) behavioural marker system (BMS) is the first BMS to be developed specifically for medical students to facilitate training in non-technical skills (NTS) within immersive simulated acute care scenarios. In order to begin implementing the tool in practice, validity evidence must be sought. We aimed to assess the validity of the Medi-StuNTS system with reference to Messick's contemporary validity framework. Methods: Two raters marked video-recorded performances of acute care simulation scenarios using the Medi-StuNTS system. Three groups were marked: third-year and fourth-year medical students (novices), final-year medical students (intermediates) and core medical trainees (experts). The scores were used to make assessments of relationships to the variable of clinical experience through expert-novice comparisons, inter-rater reliability, observability, exploratory factor analysis, inter-rater disagreements and differential item functioning. Results: A significant difference was found between the three groups (p<0.005), with experts scoring significantly better than intermediates (p<0.005) and intermediates scoring significantly better than novices (p=0.001). There was a strong positive correlation between the two raters' scores (r=0.79), and an inter-rater disagreement of more than one point in less than one-fifth of cases. Across all scenarios, 99.7% of skill categories and 84% of skill elements were observable. Factor analysis demonstrated appropriate grouping of skill elements. Inconsistencies in test performance across learner groups were shown specifically in the skill categories of situation awareness and decision making and prioritisation. Conclusion: We have demonstrated evidence for several aspects of validity of the Medi-StuNTS system when assessing medical students' NTS during immersive simulation. We can now begin to introduce this system into simulation-based education to maximise NTS training in this group. © Author(s) (or their employer(s)) 2021. No commercial re-use. See rights and permissions. Published by BMJ.
Entities: Chemical
Keywords:
behavioural marker systems; medical students; non-technical skills; simulation; validation
Year: 2020
PMID： 35521075 PMCID： PMC8936660 DOI： 10.1136/bmjstel-2019-000506
Source DB: PubMed Journal: BMJ Simul Technol Enhanc Learn ISSN： 2056-6697