Construction of Ideal Remote Assessment Environments and the Impact of Self-Selected Music on Student Performance.

BACKGROUND: Remote online assessments require students to construct their own assessment environments, including selection of strategies (such as the use of music) to reduce stress. This study aimed to determine the impact of self-selected music on student performance during a remote online assessment and to identify factors important for constructing ideal assessment environments.
METHODS: Final year students were randomized to complete a voluntary remote online 2-hour care plan test. Those randomized to 'music' were required to play self-selected music during the assessment and those randomized to 'non-music' were asked not to play music. Prior to the assessment, perceived stress and resilience were measured. Performance between groups was compared and associations between stress, resilience, and performance determined. A post-survey identified music preferences/acceptability, and factors identified for ideal remote assessment environments.
RESULTS: A total of 79 students completed the study (n=40 music, n=39 non-music). The median assessment score in the music group was 90% (range 58 to 99%) and 88% (range 58 to 99%) in the non-music group (not significant). No associations were found between scores and perceived stress or resilience. The majority of students randomized to music (62.5%) found it helpful. Thirteen categories of factors were identified to contribute to an ideal remote assessment environment with the most common being lighting, location, quietness, distractions, and seating/set-up.
CONCLUSION: Findings support the notion that remote online assessment environments should not come as 'one size fits all' and many factors (including self-selected music) may influence a student's ability to perform to a high standard. © Individual authors.

INTRODUCTION

The study of learning environments has informed teaching and resulted in betterment of student achievement and attitudes.[1,2] Learning environments include the social, physical, psychological, and pedagogical contexts in which learning occurs.[3] Despite great advances in this field, little research has been conducted to better understand the influence of assessment environments on student success in demonstrating achievement of intended learning outcomes. Assessments are typically conducted in controlled environments with little consideration for student preferences or needs. Factors such as lighting, temperature, distractions, noise, and the ability to consume food/beverages and use the washroom are dictated by the institution, but could potentially influence student performance and satisfaction.[4,5] The increased recent focus on online education and the advent of remote online assessments during the COVID-19 pandemic has exposed the need for educators to better understand assessment environments and the role that specific factors may have in influencing student performance.[6]

Assessments, typically viewed as stressful events for students, require attention and focus often over long periods of time (e.g. 2-3 hours).[7] In traditional assessment environments (e.g. exam room, auditorium), students need to rely on internal coping mechanisms (e.g. breathing, positive inner dialogue) to relieve stress and complete the assessment.[8] In remote online assessment environments that are not invigilated, however, students are able to modify external factors that could influence stress levels, result in greater attention to assessments, and may improve student performance. The use of music, for example, has been shown to have positive effects in therapy; memory formation; relaxation; and stress/anxiety/sleep in stressed populations.[9-11] Research has also investigated the use of classical music on test anxiety and exam performance in college students.[12] Although no significant results were found by this study, its authors hypothesized that self-selected music (rather than forcing a specific genre such as classical) may have resulted in different results. Considering the impracticality of allowing self-selected music during controlled assessments, remote online assessments that are not invigilated provide a unique context to investigate the influence of music on student performance during assessment and to investigate whether the use of music helped or hindered those students with known high stress levels and/or low resilience.

Moving forward through the COVID-19 pandemic and beyond, it is likely that remote online assessments will remain as assessment options available to both in-person and online delivered programs.[13] As discussed above, educators need to better understand the factors that may influence student performance during these assessments, in order to best inform and support students when tasked to create their own assessment environments. Student support services can work with students struggling to optimize their environments by testing specific strategies that could be individualized according to their own preferences. Just as research has greatly advanced modification to learning environments to best support student achievement, there is new potential to do the same for assessments. This study aimed to determine the impact of self-selected music on student performance during a voluntary remote online assessment and to explore associations between performance and reported stress and resilience. An additional aim was to identify modifiable factors perceived by students to be important for optimizing remote online assessment environments.

METHODS

This was a randomized controlled trial paired with a cross-sectional survey of study participants. Ethical approval was obtained from the University of Otago (D20/297).

This study was conducted at the University of Otago at the School of Pharmacy in Dunedin, New Zealand. The School maintains a Bachelor of Pharmacy (BPharm) program (accredited by the Pharmacy Council of New Zealand) that enables graduates to proceed with a 1-year internship prior to registration as a pharmacist in New Zealand. In the final year of the program, students complete a capstone clinical course that targets consolidation of knowledge and skills prior to graduation. Assessments for this course include remote online assessments that require students to complete a patient-centered care plan for a given paper-based case. The remote online assessments were planned as such prior to the COVID-19 pandemic and will be maintained afterward. The care plan assessment requires students to identify the patient’s risk factors and signs and symptoms for the major presenting health-related problem, state the health-related problem, propose a patient-specific goal, list their desired outcomes from therapy, provide therapeutic (pharmacological and non-pharmacological) options to consider for the major health-related problem, state their decision for which option(s) to choose, justify their decision, outline a monitoring plan for both efficacy and safety, and provide patient education points pertinent to the success of the plan. The care plan is assessed according to a 5-point rubric (0 to 4) based on these components. Component are weighted as 3 points (risk factors, signs and symptoms), 4 points (health-related problem, patient-specific goal, desired outcomes, therapeutic options, and patient education points), and 8 points (decision, justification, and monitoring). The maximum number of points available was 50. The assessment used for this study was conducted according to these procedures, but was voluntary and not for credit. It occurred approximately one month prior to final examinations and therefore could have been considered a practice assessment for the care plan portion of the course assessments. No incentives were provided for participation, aside from the additional practice and rubric scoring. General feedback and answers were reviewed with the entire class (including those that did not participate) prior to the final examination.

All students enrolled in the 4th year of the BPharm program were eligible to participate. Recruitment occurred via email and posts on social media pages. Students were informed their participation was voluntary, would not influence their academic success in the program, and if they chose not to participate, they would have access to the study materials after completion of the study for exam preparation purposes. Recruitment occurred over a 1 week period. All students expressing interest to participate were provided with further information and consent was obtained.

Consenting participants were randomized to one of two groups: music or no music. Randomization occurred by assigning each participant a unique number that was put into an online randomization tool. This process was overseen by two investigators (JT and KW) to ensure groups were allocated as intended by the online tool. Participants were informed of their group allocation via email three days before the study day.

Participants completed two short surveys prior to the study day to measure baseline stress and resilience. Stress was measured using the Perceived Stress Scale and resilience was measured using the Brief Resilience Scale (BRS).[14,15] The Perceived Stress Scale measures participants’ perceived stress levels by having participants answer 10 questions targeting feelings and thoughts related to stress over 5-items of frequency from 0 = Never to 4 = Very Often.[14] The BRS measures resilience by having participants select their agreement or disagreement with 6 statements over 5 Likert items (Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree).[15] Each tool results in a total score for their respective measures. Surveys were administered using Google Forms. (Mountain View, CA, USA).

A patient case was developed by the senior investigator (KW). The case was of a patient with the major problem of chemotherapy-induced diarrhea requiring pharmacological and non-pharmacological management. The case was posted to an online sharing platform and released at the study time. Participants were instructed to download the case at that time and then upload a word document with their completed care plan after two hours. The Blackboard software (Blackboard Inc. Reston, VA, USA) allowed for anonymous grading and participants were instructed to not include identifying information within the document itself. Those randomized to the music group were instructed to begin playing any form of self-selected music at the beginning of the study and maintain playing music for the full two hours. Those randomized to no music were instructed to not listen to music at any time during the assessment. Any participant having technical difficulties during the assessment was instructed to contact the senior investigator (KW) by phone. No content related questions were answered, to mimic real assessment policies. Participants were allowed to access any resource (hardcopy or online) but were instructed not to complete the assessment with other participants and/or not communicate with any individual (aside from the senior investigator) at any time during the two hour assessment.

Participants completed a short survey at the conclusion of the assessment. This survey was designed by investigators to capture participants’ use and perception of music during the assessment. Questions elicited the genre of music listened to if randomized to music, the length of time music was played during the assessment, participants’ perceptions of the helpfulness or distracting nature of music when completing the assessment for those randomized to music, and an open-ended question that instructed all participants to describe their optimal assessment environment for remote online assessments.

Blinded care plans were graded by the senior investigator, who had eight years of experience grading similar assessments. Care plans were re-identified after completion of grading for matching to stress, resilience, and post-survey data. All quantitative data was input into SPSS v. 25 (IBM Corporation, Armonk, NY, USA). Normality of grades was tested using the Shapiro-Wilk test and data were determined to not be normally distributed (p<0.05). Non-parametric tests were used for all other analyses. Care plan grades and distributions were compared between the two randomized groups using the Mann Whitney U Test. Care plan grades from all participants were correlated with stress and resilience scores using Spearman’s rank correlation coefficient. A subgroup analysis of those randomized to each group was also conducted. Quantitative data from the post-survey were summarized descriptively.

Data from the open-ended question on the post-survey were extracted into Microsoft Excel (Redmond, WA, USA). One investigator (JT) coded each response according to the factors identified that depicted the participant’s ideal assessment environment and any qualifier for the factor (e.g. bright vs. dim lighting). Coding was reviewed by the senior investigator (KJ). Codes were combined into categories, where applicable and the proportion of participants identifying each factor as part of their ideal assessment environment was calculated. Investigators met in person to review categories and interpret themes. Discrepancies between investigators were resolved using discussion.

RESULTS

A total of 81 participants consented for the study and completed the pre-surveys. Two participants (no music) did not finish the assessment due to technical difficulties and therefore all data was removed from the analysis. Seventy nine participants completed all study components (40 randomized to music, 39 to no music). Care plan grades, perceived stress scores, and resilience scores are provided in Table 1. Stress and resilience scores were similar between groups. The median grade on the assessment was 88% (range, 58 to 99%). Those in the music group had higher median scores when compared to those in the non-music group (90 vs. 88%, respectively), but results were not statistically significant (p = .129). Correlation coefficients between assessment grades, stress, and resilience scores are also provided in Table 1 overall and for each group. No significant associations were observed between variables tested.

Table 1.

Results from Care Plan Assessment and Associations with Baseline Reported Stress and Resilience

	All	Music	Non-Music
	(n=79)	(n=40)	(n=39)
Care plan grade, median (range)	88% (58-99)	90% (58-99)*	88% (58-99)*
Perceived Stress Scale, mean (SD)	21.1 (3.1)	21.1 (3.3)	21.2 (2.9)
Correlation coefficient (r)	.044	.035	.058
Brief Resilience Scale, mean (SD)	20.8 (4.4)	20.6 (4.6)	21.0 (4.3)
Correlation coefficient (r)	-.067	-.146	.048

*p = .129 when care plan grades compared between music and non-music using Mann Whitney U Test

All students randomized to music stated they listened to music as instructed and all those randomized to no music stated they did not listen to music during the assessment. The majority of those randomized to music found it helpful (62.5%). Half of all participants did not think it was distracting (50%). A wide variety of music genres were self-selected during the assessment with 15 (37.5%) of participants in the music group stating more than one genre (Table 2). The most common genres were pop, soundtrack, and lo-fi.

Table 2.

Self-Selected Music Choices

Music Categories	Number (%) of Students (n=40)
Pop	16 (40.0)
Soundtrack	7 (17.5)
Lo-fi	6 (15.0)
Rap	5 (12.5)
Classical	5 (12.5)
RnB	3 (7.5)
Hip-hop	3 (7.5)
Drum and Bass	2 (5.0)
Rock	2 (5.0)
K-pop	2 (5.0)
J-pop	2 (5.0)
Country	2 (5.0)
Folk	2 (5.0)
Indie	2 (5.0)
Techno	1 (2.5)
J-rock	1 (2.5)
Alt-Rock	1 (2.5)
Jazz	1 (2.5)
Dance	1 (2.5)
Electronic	1 (2.5)

All students who completed the study provided answers to the open-ended question. The overarching theme identified for participants’ ideal assessment environments was that preferred factors are highly individualized (i.e. ‘one size will not fit all’). Thirteen categories of modifiable factors were identified and the proportion of times each was mentioned is provided in Table 3. The most common modifiable factors identified were lighting (59.5%), location (50.6%), and quietness (44.3%). Examples of each category are provided in Table 3.

Table 3.

Student Reported Factors Contributing to an Ideal Assessment Environment

	Student Number (%) (n=79)	Examples
Factor
Lighting	47 (59.5)	Bright lights, white lights
Location	40 (50.6)	Bedroom, bed, library
Quietness	35 (44.3)	Quiet spot, a quiet area, quiet but not dead silence
Distractions	32 (40.5)	No other people talking, no distractions
Seating/Set-up	27 (34.2)	Comfortable desk and chair, double monitor
Music	18 (22.8)	Big speaker to play familiar music, quiet music
Temperature	18 (22.8)	Not too hot or cold, warm temperature
Solo	15 (19.0)	In a room by myself, alone
Internet	10 (12.7)	Good internet, stable internet connection
Refreshments/toilet	7 (8.9)	Snacks, cold drink, close to a toilet
Airflow	5 (6.3)	Windows open for fresh air, good air flow
Tidy	4 (5.1)	No clutter, clean space/environment
Background Noise	2 (2.5)	Seldom bird singing, white noise

DISCUSSION

The purpose of this study was to determine the impact of self-selected music on student performance during a voluntary remote online assessment and to identify the modifiable factors perceived by students to be important for optimising remote online assessment environments. It was found that students randomised to the music group performed slightly better (higher average grade of 3%) but the finding was not statistically significant. No associations were found between students’ performance and stress or resilience. Aside from music, students identified many modifiable factors that contributed towards their ideal remote online assessment environments. Factors included location, set-up/seating, temperature, lighting, and others.

The key finding from this study was that ideal remote online assessment environments are highly individualized and this also includes the use of music during the assessment. Although no significant difference was found between music groups, this was not entirely unexpected. Assessment environment conditions, not unlike learning or study environments, do not likely come in a ‘one size fits all’ model and must be carefully constructed by the individual to meet their own preferences and needs. In other words, it might not be music itself that helps students, but students’ own preferences for strategies used to focus and maintain attention. Of course, the study may not have had enough power to detect a difference between groups, if a difference did indeed exist. Despite the lack of significance, results did show that the use of music did not disadvantage students and the majority of those randomized to music stated that music was helpful. The slight trend in increased performance and lower variance for those randomized to music may also suggest that listening to music during remote online assessments could be a strategy of use to some students and could be offered as suggested strategies for students struggling with optimizing their own remote assessment environments.

The findings obtained from this study align with other studies that have been conducted regarding music and assessments. Goldenberg et al. (2013) also showed no significant difference when college students were randomized to listening to classical music or no music.[12] Goldenberg et al. hypothesized that self-selection of music may have resulted in different findings but our study findings do not support this claim. As discussed above, results from these two studies show that music likely does not have a population effect on exam performance, but potential benefits to those individuals who find it helpful cannot be ruled out without further study. We also investigated if music may benefit those students who report greater stress or decreased resilience. Although other studies have shown beneficial effects of music for those with stress and anxiety, we did not observe any associations between performance and stress or resilience within the music group.[9-11] That being said, it cannot be ruled out that the use of music may benefit some of these students and further study is warranted.

One of the most interesting findings of this study was the identification of modifiable factors that students report to form their ideal assessment environments. Perhaps due to a lack of remote online assessments prior to the COVID-19 pandemic and an accepted norm of using controlled environments for assessment, there was very little guiding literature in this area and these results provide a foundation for future studies moving forward. The major theme interpreted was that ideal assessment environments are highly individualized, but common factors were identified that students may want to consider when planning their environment. The most common factors identified were noise (quiet), presence of distractions, and set-up/seating. Regarding noise (quiet), students should aim to decipher if they work best in areas where there is background noise or in quiet settings. This can be achieved by adjusting one’s location (e.g. sitting an assessment in one’s home, in a library, or in a cafe) or the use of background noise/headphones. Similarly, regarding the presence of distractions, students should aim to locate themselves in a setting, which reflects their own ideal level of distraction, as some students may function better in a location that is void of distractions whereas some may perform better with some ongoing background activity. Regarding set-up/seating, the results of the survey suggested that many students preferred a ‘tidy’ desk and required enough space to ‘lay out their supplies’. However, some students preferred to complete their assessments in bed, which further supports the idea of individualized ideal assessment environments. This notion of regulating one’s assessment environment is novel as this has not been previously possible due to the nature of controlled assessments in the past. Assessment coordinators and student support personnel should be aware of these factors to assist students in creating environments that will be most conducive for success, especially if remote assessments remain embedded within curricula.

This study adds to the current understanding of ideal assessment environments and has practical implications towards the learning and support programs of students, as well as implications for future research. From the findings of this study, it can be suggested that students should practice modifying their remote assessment environments according to the factors identified, in order to discover their own individualised preferences. Instructors should discuss these factors with students at the beginning of the semester to allow enough time for students to practice and determine what factors or range of factors work best for them to optimize focus and performance. This study also has implications for future research. As an emerging area for research, the influence of modifiable factors within an assessment environment on student performance should be further explored. Future studies should also attempt to identify students that may benefit from specific factors, such as the use of music to increase attention or focus on the assessment. Studies should also investigate student preferences with respect to traditional or self-selected assessment environments.

This study has limitations that must be addressed. Firstly, it was only conducted in one assessment at one institution and the sample size was limited by the number of students enrolled in the program. The study may have therefore been underpowered to determine any population-level differences in the use of music on performance. However, findings are still important and show that some students preferred the use of music during assessment. Secondly, some findings (e.g. music, location, distractions) would not be applicable to remote assessments requiring invigilation. Findings (aside from those relating to temperature, lighting, etc.) are therefore mostly specific for remote online assessments not requiring invigilation. Thirdly, results are likely not generalizable to assessments of greater cognitive load, such as exams requiring calculations or of greater complexity in nature. Finally, the assessment was voluntary and therefore may not be a true reflection of student preferences during real assessments.

CONCLUSIONS

This study investigated the impact of self-selected music on student performance during a voluntary remote online care plan assessment and identified modifiable factors that contribute towards an individualised ideal assessment environment. Findings do not support the use of self-selected music across an entire population for the purpose of increasing performance. Findings do support the notion, however, that similar to learning environments, assessment environments (when given the choice) should not come as ‘one size fits all’ and many factors (including self-selected music) may positively or negatively influence a student’s ability to perform to a high standard. Programs should therefore be aware of these factors and work with students to facilitate construction of their own ideal assessment environments prior to implementation of the assessment. Moving forward, the academy should also prioritize research in this area to better understand environmental factors important for student success during assessments, especially as remote online assessments become part of regular assessment procedures.

Funding/support: None

Conflicts of Interest: None