Are 3D printed models acceptable in assessment?

BACKGROUND: Three-dimensional (3D) printed models are increasingly used in undergraduate anatomy teaching. However, their role and value in anatomy assessment remains under consideration. The aim of this study was to evaluate student and educator perspectives on acceptability of using novel 3D printed heart models for assessment.
METHODS: We used printed 3D models of the heart for first-year medical students, in small group teaching, formative assessment and revision at home. We adopted a mixed methods approach involving questionnaires, then focus groups to collect student and educator views. We used QSR Nvivo to manage thematic analysis of responses, carried out by student and educators, respectively.
FINDINGS: Overall, students 89% (n = 75/84) and educators 91% (n = 10/11) found the assessment acceptable. Thematic analysis of focus groups (n = 4 students, n = 5 educators) identified five key perceptions shared across student and educator groups: 3D models are the future, realism is valued, models appear feasible, consistent and provide a potential for a range of applications in assessment. DISCUSSION: There was agreement between educators and students that the use of 3D heart models was acceptable. Key recognised benefits include accessibility and consistency across settings, made more relevant in the current COVID-19 pandemic. We recommend integration of 3D models into teaching and assessment for educational alignment and careful selection of anatomy to model. Further research is required to explore the use of models in summative assessments.

BACKGROUND

Three‐dimensional (3D) printed models are gaining momentum in undergraduate anatomy education. They have been successfully incorporated into anatomy teaching, with the option of allowing students to take the models home. Proponents of the use of 3D model versus preserved anatomical specimens in teaching have cited key advantages, including lack of legal regulations, control over availability and integration into multimodal teaching for improved educational outcomes. ,

These advantages extend further to their use in anatomy assessment. As there are no legal restrictions on transport and use, this facilitates assessments taking place in a variety of hospital, university or home settings. The ease of access to readily available assessment tools is made more relevant in the context of the current COVID‐19 pandemic. Additionally, 3D models improve assessment reliability as all students are assessed with the identical model. Candidates can directly handle and annotate the model as part of the assessment.

Access to readily available assessment tools is made more relevant in the context of the current COVID‐19 pandemic.

Crucially, 3D models are anatomically representative of normal human anatomy, since high‐resolution models are derived from computed tomography (CT) and magnetic resonance imaging (MRI) of real people with normal findings. Previous studies have demonstrated the validity of 3D models in teaching anatomy. As defined by the Ottawa criteria for assessment, validity here means “the results of an assessment are appropriate for a particular purpose as demonstrated by a coherent body of evidence.” Given an aim of assessment is assessing student application of knowledge to identify realistic anatomical features, 3D models derived from normal scans represent a valid tool for that purpose.

While there is a growing use of 3D models in teaching anatomy, , studies examining the acceptability and feasibility of assessment using 3D models remain limited. The Ottawa criteria for good assessment defines acceptability where “stakeholders (including but not limited to faculty and students) find the assessment process and results to be credible” and feasibility, a key consideration inherently linked to this, where “the assessment is practical, realistic, and sensible, given the circumstances and context.”

CONTEXT

At University College London Medical School (UCLMS), first‐year medical students are assessed in an Objective Clinical and Practical Examination (OCaPE), a format similar to an Objective Structured Clinical Examination (OSCE). OCaPE assess integration of applied anatomical knowledge and feasibility is a particular concern. We considered 3D models as an alternative method of assessment, in part due to legal limitations on human tissues, which make their use in OCaPEs at multiple sites difficult. We were also concerned about the logistical challenges of implementing OCaPEs with human tissues within the timescales of the assessment cycle. Hence, feasibility is a key area of interest. Acceptability is also relevant, as we were concerned a change from assessment with human tissue to 3D models would not be acceptable to students and educators. These components are essential to a “good” assessment process.

At UCLMS, we produced 3D printed heart models from a database of STereoLithography file (STL) files, which describe surface geometry of a 3D object in computer‐aided‐design software for 3D printing. These files were obtained from a publicly available database of STL files derived from CT and MRI image files. This was achieved with support from the Anatomy Laboratory and Bartlett School of Architecture. We carefully selected and reviewed prototypes, consulted two members of the senior anatomy faculty to review anatomical features modelled. The finalised model is shown in Figure 1.

FIGURE 1

Image of the 3D printed heart anatomy model

As there a body of literature on 3D heart models in teaching undergraduate heart anatomy, it is key to connect our study of assessment methods with existing methods of teaching. Alignment of teaching and assessment methods is shown to have important educational benefits. We chose a heart model to investigate the acceptability of a 3D model for assessment as there is a paucity of cadavers without signs of previous coronary surgery and difficulty visualising the posterior aspect of the heart in 3D during cadaveric dissections. The features included in the 3D model correlate with learning objectives in the UCLMS anatomy curriculum, including key vessels such as the left anterior descending artery, left circumflex artery and right coronary artery.

The model was integrated into small group teaching of cardiac anatomy for first‐year medical students. The same model was used in a group viva‐style formative assessment for all first‐year students. The Anatomy and Assessment faculty both reviewed the assessment design, setting a format which required students to directly interact with the model to label anatomical features of the heart and answer relevant questions. Afterwards, we gave each student a model to take home for revision; however, due to the COVID‐19 pandemic, summative assessments did not include practical assessment of anatomy.

The model was integrated into small group teaching of cardiac anatomy for first‐year medical students.

Given the opportunities offered by 3D models in assessments, this study aims to evaluate the acceptability and feasibility of this novel assessment tool, from both student and educator perspectives.

METHODS

Aligning with the nature of an acceptability and feasibility study, we employ a pragmatic philosophy and a mixed methods approach. We used questionnaires to collect anonymous data conveniently, then semi‐structured focus groups to develop issues that emerged. We conducted semi‐structured interviews to strike a pragmatic balance between exploring participant perspectives while allowing a structure to confer reliability and validity.

Participants for the study were identified by their medical school year group. The entire cohort was taught using the 3D model and took the same formative assessment which included the 3D model. Two weeks after the assessment, following a lecture the whole year group were asked to voluntarily complete a live questionnaire. Six weeks later, a focus group was conducted via Microsoft Teams with those who expressed an interest in participating through the questionnaire. Similarly, we sent questionnaires to all educators involved in anatomy teaching or assessment using the 3D model and invited those who expressed interest to the focus group 4 months later. The development of the questionnaire was informed by Denscombe's The Good Research Guide, with educator questions quoting relevant technical terms based on the Ottawa criteria, while student questions contained non‐technical language. Questionnaire design was an iterative process of review by the assessment faculty and student investigators to ensure wording and purpose were clear. An educator investigator wrote both questionnaires, and student investigators edited the student questionnaire to clarify terminology and minimise medical education jargon. The finalised questionnaires are shown in Table 1. We reviewed response to ensure they aligned with the questions asked.

TABLE 1

Student and educator questionnaire

Student Questionnaire (MentiMeter – instant responses)	Educator Questionnaire (Typeform – via email)
1. Did you attend an anatomy lab teaching session involving scaled down 3D printed model hearts? ‐ Yes/No	1. What is your role in anatomy teaching? ‐ Academic ‐ Senior honorary demonstrator ‐ Near‐peer demonstrator ‐ CT2 surgery demonstrator ‐ Honorary demonstrator ‐ Senior near‐peer demonstrator ‐ Other ‐ No role
2. Did you attend a formative anatomy practical assessment and feedback session? ‐ Yes/No	2. What was your involvement in formative anatomy practical assessment? ‐ Assessment Design ‐ Assessing Students ‐ Feeding back to students ‐ Other
3. Was the 3D printed heart formative practical assessment acceptable to you? ‐ Yes/No	3. How would you rate the validity or coherence of the formative anatomy practical assessment involving the 3D printed heart? ‐ Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent)
4. Why did you think it was or wasn't acceptable? ‐ Free text	4. How would you rate the consistency or reproducibility of the formative anatomy practical assessment involving the 3D printed heart? ‐ Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent)
5. Has the 3D printed heart impacted your learning since then? ‐ Yes/No	5. How would you rate the equivalence of the formative anatomy practical assessment involving the 3D printed heart? ‐ Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent)
6. If Yes – Please describe how it has impacted your learning. If No – Why do you think it did not impact your learning? ‐ Free text	6. How would you rate the feasibility of the formative anatomy practical assessment involving the 3D printed heart? ‐ Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent)
7. What other uses could you see for 3D models? ‐ Free text	7. How would you rate the educational effect of the formative anatomy practical assessment involving the 3D printed heart? ‐ Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent)
8. Please rate how you feel about the following in relation to the use of 3D models in assessment: (Likert scale 1–5 where 1 = Strongly disagree. 5 = Strongly agree) They are fair They help my learning They should be used more	8. How would you rate the catalytic effect of the formative anatomy practical assessment involving the 3D printed heart? ‐ Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent)
9. Would you be interested in sharing your thoughts on 3D models in a short online focus group? If yes, please type your email: ‐ Free text	9. How would you rate the acceptability of the formative anatomy practical assessment involving the 3D printed heart? ‐ Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent)
10. Please give any other comments or feedback about the formative anatomy practical assessment using the 3D printed model of the heart ‐ Free text	10. Do you think it would be acceptable to have a similar station, using 3D models, as part of the summative assessments? ‐ Yes/No
	11. Why did you give this answer? ‐ Free text
	12. If you would consider participating in a short online focus group to explore the responses to the questionnaire, please include your email below: ‐ Free text

Focus group questions were developed guided by issues highlighted from respective group questionnaire responses. The time lapse between the assessment and focus groups was due to the COVID‐19 pandemic; educators were less available as they rapidly shifted to online teaching. Student and educator focus groups were led by student investigators and educator investigators, respectively, to encourage open discussion and improve validity. There were two student investigators (SWLY and NM) leading the student focus group and thematic analysis. SWLY has been a previous participant in focus groups. Educator CH provided student investigators with informal training and guidance ahead of their focus group and analysis, including an overview of thematic analysis, and mentorship throughout the process. During the student focus groups, CH remained silent but available for support. Supervision was provided by AS (head of UCLMS assessment faculty), GM and CH who have extensive years of medical education experience and training at master's level and above.

The research was carried out within normal educational requirements and with full informed consent. During the pandemic, ethical approval was not required under UCL guidelines for the aforementioned reasons and as the study was considered by the authors as low risk with no potential harm to the participants. All focus groups were audio‐recorded, anonymised and transcribed with written participant consent, as compliant with GDPR, and deleted after analysis. The participants also consented for the use of anonymous data in publications.

We thematically analysed interview transcripts with QSR NVivo and derived descriptive statistics for quantitative data. We generated initial codes from reading and re‐reading the interview transcripts. Themes were iterated independently by respective investigators, then discussed and reviewed by all authors. Where agreement could not be reached on defining and naming themes between investigators SWLY, NM and CH, the senior investigator AS made the final decision. Finally, the educator investigator merged the overarching themes identified in both groups. We acknowledge the reflexivity of the investigators involved in thematic analysis. The student investigators are both members of the university Anatomy Society, hence are inherently interested in how anatomy is assessed, and they have found the concept of learning from 3D models appealing. The educators have a stake in the success of implementing 3D models as a convenient method of delivering formative OCaPEs such that this may be used in further assessment cycles; however, they would also like to ensure this process is credible to stakeholders and feasible for assessments in the future.

FINDINGS

We collected 84 student responses (response rate 25%) and 11 educator responses (response rate 31%) to the questionnaires. When given a binary choice on whether 3D models are an acceptable assessment, students 89% (n = 75/84) and educators 91% (n = 10/11) found the assessment acceptable. Four students and five educators took part in the student and educator focus groups, respectively.

We extracted several key themes from thematic analysis of the qualitative data from student and educator focus groups and questionnaire free‐text responses, as shown in Figure 2.

FIGURE 2

Thematic map. This represents themes extracted from thematic analysis of educator and student focus groups

We found 3D models were an overall acceptable compromise for assessment. Illustrative examples are displayed in Table 2. “S” refers to student participants, and “E” to educators, the number being allocated by their order of contribution to the relevant focus group.

TABLE 2

Example quotes for using 3D models as an overall acceptable compromise for assessment

Overall acceptable compromise for assessment
E3: “I definitely think it's acceptable, and I think that what it comes down to for me is the compromise that we have to make with this new world of not being able to assess everyone with a spot test”	S1: “I thought it was very acceptable, I really liked it, and for the assessment I was like yeah I agree.”
E2: “I think this is a pretty good compromise, it's about as good as it gets”	S3: “Not all hearts have arteries and veins easy to identify, so having a model made this more acceptable”

This overall finding is supported by five main themes shared by educators and students, as well as student‐ and educator‐only themes, respectively. The shared themes extracted from student and educator groups are presented by theme in Table 3.

TABLE 3

Concordant themes between students and educators

3D Models are the future
Future use of 3D models	E2: “you know this imaging is now already going towards the 3D imaging in radiology … so when I saw that I really felt that this is the way forward”	S4: “I think it would be a nice addition, and I can see it being in an assessment in the future.”

E5: “That's really the case for using cadaver material is that you get that human variation side of it.”

E3: “us instructors are old fashioned enough to think that a spot test is the best way to assess this”

S3: “people were maybe slightly worried about it not being the real thing, … well you know obviously a real organ would be a much better thing to use.”

S2 “You kind of prefer … like want to do it in like a cadaver setting”

S1: “that was quite useful actually just to see kind of what a regular heart looks like and then testing that, as well as obviously testing the cadaver.”

E1: “It would allow to test many different aspects of anatomy in 3D, which would make it more inclusive what can be tested”

E4: “we could maybe produce a different summative model – even if it's the same structure … you use that one year – and maybe two years later you produce a different such model from a different CT scan, it would be very very different”

S1: “you could use the model … (to) express like differences between the right and the left and be like well you can see that the left one is bigger, the left ventricle is bigger, and you can ask questions from like that”

S3: “taking the same approach to maybe a different organ”

Equivalence

Comments on the difficulty of the assessment method were varied from students while no theme on this emerged from educators. Some expressed it was too easy; S2: “you can't make it any harder to identify stuff on a model.” Other students maintained it was of a similar difficulty to other formats; S1: “if you'd used it on the main heart of the cadaver or in a textbook it would also be in the same level of difficulty.”

Educators highlighted that the 3D model aligned assessment with teaching, as the same tool used in teaching was also used to assess knowledge, reinforcing key learning objectives. E2, “everyone did very well because … they had seen it, they had learned on it.”

The 3D model aligned assessment with teaching, as the same tool used in teaching was also used to assess knowledge.

DISCUSSION

Both quantitative and qualitative data demonstrate that the use of 3D models in formative assessment is acceptable overall, to students and educators at UCLMS. Notably, there was much agreement in perceptions of the value of these models. Both student and educator groups recognised the benefits of accessibility, consistency and equivalence in using the 3D model for assessment. They shared concerns over the trade‐off between resources and fidelity, when speaking of modifying design and printing the models. Their correlation of perceived benefits and concerns despite their respective positions in the subject is striking. However, on the following areas, their nuances in perspectives are explored.

Both student and educator groups recognised the benefits of accessibility, consistency and equivalence.

During the focus groups, educators focused on alignment and credited the 3D model with allowing constructive alignment of teaching activities and assessment strategies. Conversely, students spoke separately about learning and assessment. They commented that the model was a useful revision tool used alongside cadaveric dissection, and some felt the model served as a motivator for further learning. Many educators felt assessing with the model could result in narrowing of student knowledge of anatomy to one “normal” without account of human variations. However, this concern appears unfounded as students view the model as an “additional” assessment tool they can access any time or place. Hence, the model acts in conjunction with other multimodal assessment and learning strategies. In terms of suggestions for improvement, educators focused on design of the current model and greater integration of 3D models into the anatomy course while students explored other possible organs to model such as the lungs. Both groups were interested in the applications of the model to anatomy assessment; however, their viewpoints differ subtly. Students focused on its use in viva‐style questions directly applicable to themselves, such as using the model to ask multi‐step questions on the heart. Educators discussed the long‐term variations across the assessment cycle and the long‐term limitations.

The model acts in conjunction with other multimodal assessment and learning strategies.

There are a variety of factors which aided implementation of assessment with a 3D model in this context. For instance, we selected anatomy that may be difficult to view in situ in a cadaver, consulted anatomy experts and stakeholders, as well as facilitated the use in OCaPE assessment. This format examines integrated anatomy and clinical knowledge, in contrast to a traditional cadaveric spotter exam.

This study is limited by the low questionnaire response rates, involvement of only one first‐year cohort albeit in a large medical school, and evaluation of 3D models for only formative assessment. Due to the COVID‐19 pandemic, the availability of students and educators was reduced, impacting response rates and availability for focus groups. It is possible that students and educators may not recall the details of the assessment as optimally for depth of responses. However, the 6‐week delay for the students was important, as pragmatically, it would have been difficult to recruit students the weeks directly following the assessment. Also, students were given the models to take home and revise, giving a reminder of the assessment. Although not the focus of this study, this allowed exploration of whether assessment motivated further anatomy revision, which was only possible with adequate time for reflection.

Unfortunately, due to the pandemic, follow‐up summative examinations were cancelled. Future work will need to explore results following summative examinations. Formative assessments are designed to monitor learning and provide feedback whereas summative assessments evaluate the learning of students. For example, the comments relating to the assessment being too “easy” are unlikely to be a criticism from students in the higher stakes summative assessment, as opposed to in formative assessment where feedback is provided to drive further learning.

Despite these limitations, we can still draw valuable conclusions and implications for our future 3D modelling work. Overall, students and educators found the model an acceptable assessment tool, based on benefits gleaned from its consistency, equivalence and logistical accessibility. 3D models also open the possibility of online assessments in any setting. However, the model is limited as a smaller, unrealistic representation, as supported by previous literature. , Both groups expressed a preference for assessment using “real” anatomy. We recommend careful selection of anatomy and consultation of stakeholders when designing the model, particularly in decision‐making around feasibility (including scale and cost) to best align with learning outcomes.

Overall, students and educators found the model an acceptable assessment tool.

We recommend careful selection of anatomy and consultation of stakeholders when designing the model.

CONFLICT OF INTEREST

The authors have no conflict of interest to disclose.

ETHICS STATEMENT

Under the UCL guidance for research and ethical approval during the COVID‐19 pandemic, this study, which fell within normal educational requirements, was categorised as low risk and did not require ethical approval. Informed consent was obtained from all participants.