Project OPUS: Development and evaluation of an electronic platform for pain management education of medical undergraduates in resource-limited settings.

INTRODUCTION: Pain is a very frequent symptom that is reported by patients when they present to health professionals but remains undertreated or untreated, particularly in low-resource settings including Nigeria. Lack of training in pain management remains the most significant obstacle to pain treatment alongside an inadequate emphasis on pain education in undergraduate medical curricula, negatively impacting on subsequent care of patients. This study aimed to determine the effect of a 12-week structured e-Learning course on the knowledge of pain management among Nigerian undergraduate medical students.
METHODS: Prospective, multisite, pre-post study conducted across five medical colleges in Nigeria. Structured modules covering aspects of pain management were delivered on an e-Learning platform. Pre- and post-test self-assessments were carried out in the 12-week duration of the study. User experience questionnaires and qualitative interviews were conducted via instant messaging to evaluate user experiences of the platform. User experience data was analysed using the UEQ Data Analysis Tool and Framework Analysis.
RESULTS: A total of 216 of 659 eligible students completed all sections of the e-Learning course. Participant mean age was 23.52 years, with a slight female predominance (55.3%). Across all participants, an increase in median pre- and post-test scores occurred, from 40 to 60 (Z = 11.3, p<0.001, effect size = 1.3), suggestive of increased knowledge acquisition relating to pain management. Participants suggested e-Learning is a valuable approach to delivering pain education alongside identifying factors to address in future iterations.
CONCLUSION: e-Learning approaches to pain management education can enhance traditional learning methods and may increase students' knowledge. Future iterations of e-Learning approaches will need to consider facilitating the download of data and content for the platform to increase user uptake and engagement. The platform was piloted as an optional adjunct to existing curricula. Future efforts to advocate and support integration of e-Learning for pain education should be two-fold; both to include pain education in the curricula of medical colleges across Nigeria and the use of e-Learning approaches to enhance teaching where feasible.

Introduction

Pain is a very frequent symptom that is reported by patients when they present to health professionals [1]. It is defined by the International Association for the Study of Pain as “an unpleasant sensory and emotional experience associated with, or resembling that associated with, actual or potential tissue damage” [2]. For a multitude of reasons, it remains under-treated or untreated in low-middle income countries (LMICs) like Nigeria [3]. Common causes include poor knowledge and attitudes about pain relief, limited pain treatment facilities, restrictive government policies, high costs, socio-cultural challenges and problems with access and use of analgesic medications, especially opioids [4, 5]. This is despite research demonstrating improvement in healthcare providers knowledge of pain and its management usually results in a reduction in patients’ pain experience [6].

Lack of training in pain management remains the most significant and fundamental obstacle to effective pain management as an inadequate emphasis on pain education for medical undergraduates may eventually reflect in poor patient care practices following graduation [7-9]. Medical students often begin their education in Colleges of Medicine with little or no knowledge of pain and that situation, coupled with very few hours of pain training, results in their lack of confidence in assessing adult and paediatric patients with pain [10]. It is thought that existing prevalent negative attitudes of physicians toward patients with, for example, chronic non-cancer pain begins early in medical school [11]. In many developed countries, knowledge-based learning of pain is absent of emotional development and reflective capacity, hindering the ability of medical students to develop empathy [11].

Integration of pain education in undergraduate medical curricula varies across countries at all levels of development [12]. In Europe, 7% of medical schools lack any pain component in their curriculum [13]. However, in Canada around 92% of Canadian medical schools and 80% of medical schools in the United States have mandatory pain management content in the medical undergraduate curricula [14]. In the context of developing countries the gap in provision is starker. For example, in Nigeria, only three of over forty medical schools offer some form of pain management training. Lectures in pain management are not stand-alone courses, but tutorials embedded within the Anaesthesia clerkship; the latter having an average duration of three weeks and thus potentially leaving the medical students insufficiently exposed to pain management education. While there is a huge need to adapt existing medical school curricula to accommodate a new subspecialty curriculum around pain management, there is limited leadership for this at present in Nigeria. The medical curriculum in its present form is overloaded and would require foregoing existing elements of teaching to create capacity to include pain management training [15]. Therefore, in the near term, the authors consider the e-Learning approach to be a possible viable adjunct to support access to pain management education alongside the existing medical school curriculum.

An ever-evolving approach to delivery of training for those delivering healthcare is through utilising technology, often referred to as e-Learning [16]. e-Learning approaches have several synonyms which include computer-based learning, online learning, distance learning and web-based learning [17]. It has been defined as, ‘a learning process that involves the connection of digitally-conveyed content, system-based administrations and mentoring bolster’ [18]. Choosing to explore e-Learning approaches enables many advantages, including flexibility to learners’ needs and time commitment, opportunities for standardization of content, and built-in pre- and post-test evaluation capabilities [19, 20]. In addition, e-Learning provides content that may not be accommodated by routine classroom teaching while at the same time allowing students to learn at their own pace. There is good evidence that such online pain educational resources are effective at improving learner knowledge, and this has been demonstrated across a number of high-income country settings, including Canada, Finland, Germany, Italy and the USA [21, 22]. This is aligned with the general increase in the provision of online distance education worldwide [23]. However, e-Learning approaches to pain management have not been explored in the context of sub-Saharan Africa in countries such as Nigeria, where poor and costly internet access and irregular power supply may pose barriers. For example, no public Nigerian university, to the best of our knowledge, offered any online courses prior to the COVID-19 pandemic, forcing schools and higher education institutions to temporarily shut down. Online services in universities are often utilized for payment of school fees, staff salaries and to enable prospective students to attempt entrance exams or allow successful students’ register, but not as a mode for delivering teaching and learning [24]. This study sought to address this gap through the development and evaluation of an e-Learning approach to pain education, focusing specifically on pain knowledge, for medical undergraduates in Nigeria. It was hypothesised that delivery of pain management lectures using an e-Learning platform would lead to an increase in the pain knowledge of medical undergraduate students.

Methods

Context of system development

This work was undertaken as part of seed funding awarded by the International Association for the Study of Pain. The grant call sought to support initiatives for improving pain education and practice in developing countries. The remit of proposals was for one-year projects that were ready to begin within four months. The decision to develop a proposal focused on an online platform brought together an emerging collaboration on digital technologies between two of the manuscript authors (TO and MA), a previous collaboration between an IT company (InStrat Global Health Solutions (InStratGHS)) and a member of the research team (MA), and an established research partnership on the development of digital technologies for pain and palliative care research across sub-Saharan Africa (EN and MA).

Design

We adopted a research-led development process of an e-Learning platform followed by a pre-post study design to evaluate its use. The platform was designed for 5th year undergraduate medical students as participants (N = 659) and then deployed across five accredited Medical Colleges in Nigeria sites over a 12-week implementation period. Participation of students was voluntary, informed consent was obtained from each participant. The investigators first obtained ethical clearance from a local human investigations committee, the University of Nigeria College of Medicine Research Ethics Committee (COMREC; Protocol No: 079/09/2019). Further consent was obtained from local ethics committees for each of the other participating sites (Health Research Ethics Committee, College of Medicine, University of Lagos. Approval no: CMUL/HREC/09/19/616; Health Research Ethics Committee, Ministry of Health, Kano State. Reference no: MOH/Off/797/T.1/1818; Health Research Ethics Committee, College of Medicine, Enugu State University of Science and Technology. Reference no: ESUTHP/CMAC/RA/034/Vol1/218). The 12-week long program which was self-paced/asynchronous, was intended to help participants learn to define pain and identify types/forms of pain, understand the ethics of pain, learn the various pain assessment methods and be able to treat pain, particularly in special conditions.

Overview of platform development

The project team which was a multidisciplinary team led by a pain specialist (TO), a global palliative care researcher (MA), a global health researcher and person-centred care service development fellow of the African Palliative Care Association (EN) and InStratGHS, a mobile health technology company. Platform development followed the disciplined agile delivery (DAD) methodology, which is a formal structure used by software developers to guide health information technology system development from the initiation of ideas through implementation and eventual retirement [25]. The DAD methodology shares principles of approaches often used to develop interventions in health research, such as user-centred design [26] and participatory design [27], where the stakeholder, or end user of a technology or product (in this case, the medical student), is central to its design and development. Working within the DAD framework provided a clear development process for the system developers. It also provided clear time points for the research team, highlighting timepoints for curriculum development, initial assessment of prototypes of the platform by the research team, and agreement on content for pre- and post-test items included as part of the platform.

The DAD framework plans system development over four phases: inception, elaboration and construction, and transition. The inception phase of the project began with the team generating a working technical specification document, which outlined the planned components and functions that were initially deemed necessary for the online platform for pain education. During the subsequent elaboration and construction phases, the team developed a curriculum and supporting materials to form the content of the platform. The transition phase involved the implementation and evaluation of the platform across five accredited Medical Colleges in Nigeria. The research team adopted a mixed-methods approach, combining user engagement activities (e.g. needs assessment) to guide the development of the curriculum delivered via the online platform, quantitative assessment of changes resulting from use of the platform pre- and post-implementation, and then used qualitative interviews to understand the experience of students who used the system and the facilitators and barriers of use of the system. We outline the methodology used during the platform development and evaluation below, aligned with the stages of the DAD framework.

The inception phase

This phase began by the research team undertaking a review of materials to shape the scope of the curriculum. This involved systematic searching of literature across Medline, Embase and PsycInfo databases using keywords such as pain education, undergraduate, medical education, and low-resource setting. Information on existing undergraduate pain curricula was sought from eight potential implementation sites of the platform. Details sought on pain education included the form of pain education provision, hours spent teaching pain, pain topics covered as well as methods of teaching and assessment. Alongside mapping out existing pain education, we received input from experts on the availability of materials to inform the development of the platform. This included consultations with the African Palliative Care Association which is a pan-African non-governmental organization working to promote and support the integration of palliative care, including pain and symptom management, into health systems across Africa, and the identification of open access materials from several pain websites that could guide the development of platform content.

Alongside consultation on the content of the platform, discussions took place between the research team and InStratGHS about the functionality of their existing platforms. Relying on the requirements of the project early in the development enabled InStratGHS to better understand the needs of the project and identify a suitable solution for deploying the pain education curriculum. The company selected VTR Mobile, the proprietary mobile learning platform of InStratGHS, for the project, which is a self-paced learning platform that can be used both online and offline on an Android smart phone, tablet computer, or laptop/desktop computer. It features a mobile application which allows users to view training contents offline where there is no internet connectivity or to save data. VTR Mobile supports multi-media training content and its test taking features allowed results to be generated in real time while its backend provided real-time user participation statistics and test results.

The construction and elaboration phases

This phase involved the creation of the pain module content and learning materials by the project team. We utilised the e-learning systems’ theoretical framework to guide the design of the platform, comprising three components: people, technologies, and services [28]. The feasibility, practicalities and relevance of the different components were discussed. From consultation with sites during the Inception phase, feedback suggested that it would not be possible to incorporate the online platform into the existing undergraduate curriculum. The platform would have to supplement existing teaching activities and provide a standalone resource for students to access. Furthermore, the fifth year of study was chosen as the target group for the platform, being mid-way through students’ clinical studies with adequate familiarity with clinical entities that require pain management. In Nigeria, the 4th year is the beginning of the clinical training during which time students are introduced to medicine and surgery. By the 5th year, the curriculum includes paediatrics, obstetrics and gynaecology as well as community medicine. This is followed by clinical training in medicine and surgery in the 6th and final year of study. The majority of the project team are experienced teaching staff delivering training to medical undergraduates in the regions of Nigeria included in the study so were able to ensure the content and curriculum were developed appropriately for 5th year medical students.

Discussions across the research team, technology developers and university sites guided both the technologies and services components of the platform. The pedagogical models and instructional strategies were informed by the supplementary approach required for the platform. An open learning model was adopted, enabling flexibility and inclusivity of access to the platform content. A combination of instructional strategies was incorporated into the presentation of content, including contextualising instruction, activating and assessing learner outcomes and presenting and cueing content. Key resources used for developing the content identified during the inception phase included the detailed International Association for the Study of Pain (IASP) pain curriculum for medical graduates which was first developed in 1988 [29]. Inputs were also made from Beating Pain [30], a pocket text book on pain designed for self-directed reading that teaches the ‘total pain’ concept, the multidisciplinary approach to pain management as well as care of all aspects of pain developed by the African Palliative Care Association.

The review of existing resources was led by the research team (TO, NI) with expertise in the teaching and clinical practice of pain management in Nigeria. The research team developed a provisional pain education curriculum for undergraduate medical students informed by activities conducted during the inception phase, including feasibility of delivery within existing curricula (from consultation with potential implementation sites) and availability of resources from literature searching. The research team determined that, in the absence of specific pain education curricula, the structure should align where possible with the IASP pain curriculum for medical graduates [29]. An initial curriculum was developed, aligned to the IASP pain curriculum for medical graduates, which was supplemented by content specific to the sub-Saharan Africa from Beating Pain [30]. Examples of supplementary content included, for example, factors that lead to women in Arica being more likely to suffer pain than men, context-specific pain and symptom outcome measures, and pharmacological management. The initial curriculum was shared with a panel of four experts in the field of pain management, palliative care, paediatrics and geriatrics drawn from different parts of the African continent, who were independent of the project team. Comments were requested from the panel of experts on framing the content for undergraduate medical education, outlining the essential content for inclusion, and the optimal structure of modules. A first round of feedback was received on the overall curriculum, its content and overview of modules. A second round involved detailed feedback on the wording of module content alongside accompanying video and media materials. The final curriculum for pain education to be delivered using the online platform was agreed following two iterations of feedback from the expert panel. An overview of the curriculum is provided in Table 1 below and comprised six modules.

Table 1

Overview of the pain education curriculum of the online platform.

Module title	Overview of Module
Module 1; Multidimensional nature of pain	• Definition of pain and classification of pain
	• Consequences of untreated pain.
Module 2: Neuroanatomy, Neurophysiology and Neuropharmacology of Pain	• The neuroanatomy and neurophysiology of pain
	• Classification of analgesics and their pharmacological targets
Module 3: Psychology of pain	• Psychological aspects of pain management
	• Pain-related beliefs and illness behaviours of patients with chronic pain
	• Cultural differences in pain meanings and treatment approaches
Module 4: Pain assessment	• Barriers to pain management
	• Popular pain myths
	• History-taking and measurement of pain in children and adults
	• Pain tools
	• Pain documentation
	• Uses and value of pain apps
Module 5: Treatment of pain	• Forms of pain treatment
	• WHO analgesic ladder
	• Analgesic treatment principles
	• Non-pharmacologic treatment of pain
Module 6: Pain in Special situations	• Principles of pain control
	• Special pain situations: Pain emergency, Postoperative pain, Pain syndromes, Labour pain, Cancer pain
	• Substance abuse and pain management

Following the development of the curriculum, supporting written and video materials were developed by the research team and shared with InStratGHS. A prototype was developed and full access provided to the research team for review. Following feedback, InStratGHS provided a second and final iteration of the online platform for implementation across sites in the transition phase. Technical specifications of the components of the final online version of the platform is outlined in Table 2 below.

Table 2

Overview of the technical specification of the online platform.

Technical aspect	Details of online platform
File Format	Documents: PDFs; Videos: MP4
Data storage location	Encrypted cloud server on Amazon Web Services (AWS)
Data export format	.txt or.csv files
Maximum File Size	65MB
Total download requirements across all modules	750MB
Required User registration information	First Name, Last Name
Testing Format	Multiple Choice Options
Pass Threshold	80%

The transition phase

In this phase, the platform was deployed across multiple sites (N = 5). Eligible sites were medical schools with full accreditation status from the National Universities Commission (NUC) and the Medical and Dental Council of Nigeria (MDCN), whose Provosts or Deans had given consent to the study and who had obtained ethical clearance for the training from their respective institutions. Each site was assigned a site coordinator from faculty staff to provide local oversight and coordination of the project (JO, KO, DM, MG, AN, SO, CI, NU). The site coordinator also served as an intermediary between the class and the project team. Members of the InStratGHS team provided technical support where required at each site. Participating sites did not commence the training simultaneously but commenced after ethical clearance was given. Each site coordinator and dedicated InStratGHS staff, together with the Principal Investigator (TO), held an initial face-to-face meeting and demonstration to introduce and familiarize the students with the rationale for the platform and provide an overview of the included modules. At each site, the online platform (see Fig 1) was presented to students as a supplementary resource, independent of their progression to the final year and independent of the students’ continuous assessment scores. The training was launched 12 hours following a site demonstration at each study site after which participants received personalized login details (username and password) alongside details for accessing the platform via mobile device or laptop. For the duration of the study, the site coordinators and InStratGHS staff were in constant communication with students, engaging in active discussions with them via the WhatsApp groups created to address any technical or other queries that arose.

Fig 1

Screenshots of the interface for the VTR mobile app (a = login screen; b = pre-test notification; c = module list; d = example of written content) and the web application (e = login screen; f = module list; g = written content; h = video content).

The deployment of the e-Learning platform in this study adopted an asynchronous, self-learning approach, omitting a function that enables messaging between teachers and students. The platform functions included some interactive elements, such as the selection of the courses by platform users and options to view or skip any course, alongside the ability to take and retake the tests. The content, written content and videos, was presented as static material that students could access and view, but there were no interactive elements to the content itself. Each module comprised of an introductory video summarizing the module content, text-based materials such as written documents and slides, and videos of pain experts and pain procedures related to the module. Video content was in two forms; compulsory videos and optional videos. References to relevant reading materials and websites were also placed at the end of each module to encourage the participants to engage in further reading.

Data collection

Prior to accessing the modules, each student was required to complete a pre-test survey to assess their baseline knowledge of pain and its management. The pre-test comprised of twenty single-correct option, multiple choice questions, each allotted a mark of 5 points with no bench-mark total score. Modules were then completed sequentially through the module numbers. At the end of each module, students were required to complete an end-of-module assessment, comprising of five multiple choice based on content from the module, with a pass mark of 80 percent set by consensus, (i.e. at least 4 correct questions out of 5) to qualify for progression to the next module. If the participant’s test score on an end-of-module assessment was below 80, he/she had the option to repeat the quiz as many times as they deemed necessary to achieve a pass mark score of 80 and above in order to progress to the next module. On completion of all six modules, a post-test and the User Experience Questionnaire (UEQ) were completed once online and submitted. The post-test replicated the same items as those presented during the pre-test and both were used to assess changes in pain knowledge. Completion of the post-test after 12 weeks was a requirement for accessing/obtaining a certificate of completion and this was provided to students by the respective site coordinators. The certificate of completion was accredited and issued by the African Palliative Care Association.

The User Experience Questionnaire (UEQ) [31] contains 26-items (adjective pairs) and is designed to evaluate the user experience of a new product in situations where the product has no previous user experience evaluation. The questionnaire assesses a product for its pragmatic (task-oriented) and hedonic (non-task oriented) qualities. Each item is rated as a 7-point Likert scale and answers are scaled from -3 (fully agree with negative term) to +3 (fully agree with positive term) and 0 referring to a neutral answer. The 26 items align with six high-level scales linked to the attractiveness, perspicuity, efficiency, dependability, stimulation and novelty of a product [32]. The general impression the participants had about the Project OPUS training platform was measured by the attractiveness scale. Perspicuity refers to ease or difficulty encountered in using the platform while efficiency measured user interface look and dependability measured the participants’ expectation of the online platform. The level of excitement and interest the participants had while using the platform was measured by the stimulation scale while the novelty scale measured innovativeness and creativity of the platform.

All data entered on the InStratGHS VTR Mobile platform was stored on its secure encrypted cloud server on Amazon Web Services (AWS). Access to this data is possible only with assigned Administrative login and passwords. Data from the pre-test, end of module assessments, post-test and UEQ as well as user engagement data (registration date, course progress and test taken, test scores, and error logs) were all captured via the InStratGHS VTR Mobile platform and stored on AWS. An InStratGHS Administrator was responsible for maintaining the password-protected access to the reporting portal that was hosted on the secure web server. Error logs were generated for those students who required technical support. All study data were exported from the InStratGHS system into Microsoft Excel for ease of transfer to appropriate statistical software packages. The downloaded data was reviewed for quality assurance and securely shared with the research team for further review and analysis.

Following the completion of all modules, site coordinators at all sites contacted students to invite them to participate in an online group discussion on perceptions about e-Learning which was held utilizing WhatsApp. Two sites, BUK and ESUT, reflected sites that had relatively high levels and low levels of engagement on the online module respectively. Participants who had used the e-Learning and those that had not participated in the online learning were invited to participate. A topic guide was used to direct discussions, focusing on the reasons for uptake and non-use of the e-Learning platform, its role in supporting pain education, and for those who used the platform, experiences in their use of the platform.

Data analysis

Data included pre-test and post-test scores, module test scores, user experience questionnaire data, and transcripts from semi-structured interviews. To describe the characteristics of students and different sites in the study population, relevant descriptive statistics (frequencies, means and standard deviations as appropriate) were produced. The normality of distribution of the data was assessed using the Shapiro-Wilk test. Comparison of the pre-and post-test scores for all participants was undertaken using the Wilcoxon sign-ranked test with effect size calculated at 95% confidence interval while the Kruskal-Wallis test used to compare pre- and post-test scores across the five participating sites.

User experience data was analyzed using the UEQ Data Analysis Tool® version 7, which is freely available from the UEQ homepage (http://www.ueq-online.org). The tool calculates the scale means and the mean and standard deviation per item. Values between -0.8 and 0.8 represent a more or less neutral evaluation of the corresponding scale, values > 0.8 represent a positive evaluation and values < -0.8 represent a negative evaluation. The tool groups the 26 items to create scores for 6 domains of attractiveness, perspicuity, efficiency, dependability, stimulation and novelty. Means scores were calculated for each domain. The tool also produces a benchmark graph comparing the product against a benchmark dataset of scores from ≈ 250 product evaluations using the UEQ including business applications, development tools and web shops and services [32]. As part of understanding user experience, technical reports were classified into higher-level categories of issues experienced with login, downloading content, video content and app installation, and the frequencies of occurrence calculated.

Following completion of the focus group interviews, data was extracted from the WhatsApp Messenger groups after download from site coordinator’s phone and turned inserted into Microsoft Word documents to create interview transcripts. The analysis was led by a member of the team (MJA). Interview transcripts were analysed using Framework Analysis [33] to draw out key themes from the data. The Framework Analysis process involved five key stages: (1) Familiarisation—getting an overview of the issues raised during the interviews; (2) Identifying a thematic framework—making notes on the key issues discussed; (3) Indexing—applying the thematic framework to the data; (4) Charting—moving data from individual interviews and putting sections into the framework; (5) Mapping and interpretation—the researcher attempts to make sense of the data and interpret the key themes and issues discussed.

Results

The cohort of Nigerian medical students eligible for this program was 659. Of the 659 invited participants, 326 commenced the training (see Fig 2), with a total of 219 students completing all sections of the e-Learning programme, providing a completion rate of 33.2%. The mean age of the students was 23.52 years, with a slight female predominance (55.3%). There was wide variation in levels of completion across sites from invitation to completion of all modules on the e-Learning platform, ranging from 4.7% to 74.8%.

Fig 2

Summary of participation across different stages of study completion.

Pre- and post-test scores and level of change

The modules scores for both pre- and post-test were found not to be normally distributed. The median pre-test score of all participants was 40.0 (IQR = 20.0), increasing to 60.0 (IQR = 25.0) following completion of six modules on the e-Learning platform. A Wilcoxon signed-rank test outlined that differences between pre- and post-test scores were significant (Z = 11.3, p<0.001). For all sites, there was a marked improvement in the post-test as evidenced by statistically significant increases in median scores differences and related effect sizes (Table 3).

Table 3

Wilcoxon test to explore effect of module on student’s pain management knowledge.

Variables	n	Mean (SD)	Median (IQR)	Z-stat	Effect Size (95%CI)	NR PR Ties
Pre-test Overall	219	41.0(13.3)	40.00(20.00)	-11.3**	1.3(-0.2 to 2.7)	19 185 15
Post-test Overall	219	63.2(16.7)	60.00(25.00)
Pre-test BUK	89	40.7(15.7)	35.00(15.00)	-7.4**	1.4(-2.5 to 5.4)	9 75 5
Post-test BUK	89	68.7(20.5)	65.00(37.50)
Pre-test ESUTH	8	38.1(10.0)	35.00(17.50)	-2.4*	2.2(0.2 to 4.2)	0 7 1
Post-test ESUTH	8	60.0(12.0)	60.00(21.25)
Pre-test NAU	9	45.0(13.0)	50.00(15.00)	-2.4*	1.5 (-1.1 to 4.0)	1 7 1
Post-test NAU	9	64.4(15.5)	60.00(32.50)
Pre-test UNILAG	93	41.9(11.8)	40.00(20.00)	-7.0**	1.4 (-0.9 to 3.6)	9 77 7
Post-test UNILAG	93	58.9(12.5)	60.00(15.00)
Pre-test UNN	20	36.8(9.1)	37.50(13.75)	-3.8**	2.5(0.9 to 4.1)	0 19 1
Post-test UNN	20	59.0(8.1)	60.00(15.00)

Z = Wilcoxon-Ranked test; BUK = Bayero University, Kano; ESUT = Enugu State University of Science and Technology, Enugu; NAU = Nnamdi Azikiwe University, Awka; UNILAG = University of Lagos; UNN = University of Nigeria Nsukka; PR = Positive Ranks, NR = Negative Ranks;

* p<0.05;

** p<0.001.

There was no statistically significant difference in the median of the pre-test scores across all the schools (p = 0.26) (Table 4) suggesting a homogenous composition of baseline pain knowledge. There were however statistically significant differences in post-test score between the different sites, [χ2(4) = 10.7, p = 0.03]. In addition, the highest post-test scores across all sites occurred in Module 6 while the lowest post-test scores were seen in Module 1 (Table 5).

Table 4

Kruskal-Wallis test to compare pre- and post-test scores among schools.

	Pre-test Score				Post-test Score
Schools	Median (IQR)	Median rank test score	χ2*	p-value	Median (IQR)	Median rank test score	χ2*	p-value
BUK	35.0(15.0)	103.5	5.3	0.26	65.0(37.5)	126.3	10.7	0.03
ESUT	35.0(17.5)	97.6			60.0(21.3)	103.6
NAU	50.0(15.0)	132.2			60.0(32.5)	113.5
UNILAG	40.0(20.0)	118.2			60.0(15.0)	97.4
UNN	37.0(13.8)	95.0			60.0(15.0)	97.2

χ2* = Kruskal-Wallis test, BUK = Bayero University, Kano; = Enugu State University of Science and Technology, Enugu; = Nnamdi Azikiwe University, Awka; = University of Lagos; = University of Nigeria, Nsukka.

Table 5

Median post-test scores in various modules across schools.

	Median (IQR)
Module	BUK	ESUTH	NAU	UNILAG	UNN	OVERALL
1: Multidimensional nature of pain	80.0(20.0)	70.0(20.0)	60.0(0.0)	80.0(20.0)	80.0(0.20)	80.0(20.0)
2: Neuroanatomy, Neurophysiology and Neuropharmacology of Pain	80.0(40.0)	100.0(20.0)	100.0(20.0)	80.0(20.0)	80.0(35.0)	80.0(20.0)
3: Psychology of pain	100.0(20.0)	100.0(20.0)	80.0(30.0)	80.0(20.0)	80.0(20.0)	80.0(20.0)
4: Pain assessment	100.0(20.0)	80.0(0.0)	80.0(30.0)	80.0(0.0)	80.0(35.0)	80.0(0.0)
5: Treatment of pain	100.0(20.0)	80.0(0.0)	80.0(30.0)	80.0(0.0)	80.0(35.0)	80.0(20.0)
6: Pain in Special situations	100.0(20.0)	100.0(0.0)	100.0(0.0)	80.0(20.0)	80.0(20.0)	100.0(0.20)

BUK = Bayero University, Kano; ESUT = Enugu State University Science and Technology, Enugu; NAU = Nnamdi Azikiwe University, Awka; UNILAG = University of Lagos; UNN = University of Nigeria, Nsukka; Module 1 = Multidimensional nature of pain; Module 2 = Neuroanatomy, Neurophysiology and Neuropharmacology of Pain; Module 3 = Psychology of pain; Module 4 = Pain assessment; Module 5 = Treatment of pain; Module 6 = Pain in special conditions.

User experience

In total, user experience data was obtained for 46 participants across the five sites. For those who completed the e-Learning programme and provided user experience data, the overall user experience rating was positive. In particular, the platform was rated positively for the attractiveness (mean score = 1.699) and stimulation domains (mean score = 1.750) (Fig 3).

Fig 3

Mean scores of 6 composite domains with the output generated using the UEQ Data Analysis Tool® version 7.

The specific responses to individual items provided by participants are shown in Fig 4. Overall, across items, responses were positive for most participants. Items where the e-Learning platform received the most negative responses for over half of respondents was where respondents reported the system as being “slow” and “unpredictable”.

Fig 4

Distribution of participants’ answers across the 7-point scale are presented for each of the 26 items in the user experience questionnaire.

Colour coding varies based on the positive or negative attributes chosen to reflect user perspectives of the e-Learning platform. Colour-coding varied across the 7-point scale from 1 (dark red) suggesting a negative rating against the item (e.g. very conservative or very unfriendly) to 7 (dark green) suggesting a positive rating against the item (e.g. very innovative, very friendly).

When plotted against benchmarking data (presented in Fig 5) from comparative online tools and web applications, the e-Learning platform was rated above average and good for the majority of domains.

Fig 5

Quality benchmark graph for Project OPUS training module.

Errors logged also highlighted technical reports generated by the e-Learning platform (see Table 6). Technical error logs showed problems with login to be the most frequent issues encountered, with errors also reported relating to downloading and installing the app, and accessing its content. Most error were associated with access to a stable internet connection and using the most up-to-date version of the app.

Table 6

Overview of IT errors recorded across five sites.

Error category	Technical issues	Number of students	Solution
Login	No login details received	29	Sent login details
	Login failure due to inputting wrong login ID	11	Correct ID shared
	login failure due to app issues	9	Reinstall latest app version
Download	Download interrupted due to poor network	12	Update app to latest version and redownload with stronger internet connection
	Cannot proceed to the next module	8	Update app to latest version
Video	Videos not opening due to app issues	10	Update app to latest version
	Videos not opening because they were not fully downloaded	5	Redownload and wait for the successful download notification
App installation	Error when updating app	10	Sent link to manually update
	Unable to install mobile app	4	Share link directly via Google Drive or using the web application

Analysis of focus group interviews yielded three themes as presented in Table 7: i) Acceptability and engagement with the e-Learning platform; ii) Perceived value of e-Learning approaches for pain; and iii) Recommendations for how engagement and the platform might be developed.

Table 7

Findings from the Framework Analysis of focus group interviews.

Theme	Summary
Acceptability and engagement with the e-Learning platform	The e-Learning platform was seen favourably by both those who interacted with the online platform and those that did not. Of those who used and engaged with the platform, it was reported that the platform was useful in “creating an avenue for me to learn via my phone any day and anytime” (BUK student 1). The ability to access and complete tasks on the platform was valued across participants. Despite a lack of engagement from some participants, most saw value in increasing teaching and content around pain and felt it would be an informative and useful platform, “…because it helps us understand the physiology of pain, which is very useful for clinical practice” (ESUT student 1).
	Of those that engaged with the platform, multiple drivers for its use were cited across students. These included those with a “quest for knowledge” (ESUT student 2), alongside those wanting to gain knowledge to inform their clinical practice, “…to be able to do something and save the life of the patient” (BUK student 2) and act accordingly in the “…face of an emergency and a patient in pain” (BUK student 3).
	Both barriers and facilitators to engagement were noted for those who did not engage with the platform. For most participants, barriers to use were education-related (e.g. competing deadlines and existing high levels of work across the course), technology-related (e.g. limited phone network signal, lack and expense of data, difficulty downloading the phone application), and personal (e.g. family health, finding the platform exhausting and requiring concentration). A small number of participants were deterred from using the platform due to issues downloading data that were caused by either their mobile phone network coverage or the platform itself. Facilitators to use of the platform noted by participants included individual drive and determination to undertake the training online, perceived cost benefit of online platform when compared to relative expense of buying pain textbooks, and some participants noting offers held by their mobile phone provider that enabled them to allocate more data to accessing and it “…was helpful in downloading all the section materials” (ESUT student 3).
Perceived value of e-Learning approaches for pain	The appraisal of content by those who engaged with the platform highlighted its importance in recapping on “…some basics in physiology and biochemistry” (BUK student 1). However, content was also welcome that taught participants about non-pharmacological approaches to pain management (“…many more methods of pain management other than drugs” (ESUT student 1)). Furthermore, participants noted improvements in overcoming confusion around pain management, including that it “…greatly improved my understanding and cleared multiple misconceptions that are not really taught in a classroom” (BUK student 4).
	In terms of the content covered, there was interest in particular for content covered towards the beginning on the modules, focusing on pain physiology, different types of pain, and definitions surrounding pain and its management. The platform was acknowledged as addressing a broad curriculum, “ranging from the types and causes of pain to the different approaches in its management” (ESUT student 4). Furthermore, the content on opioids and end of life care were noted as very important too, which helps to address “a lot of misconceptions about abuse of opioids. And there’s very little end of life pain management provided in this area of the world” (BUK student 5).
	The perception of the platform from peers of those who engaged with the platform indicated that many saw it as an opportunity to acquire new knowledge and it was viewed positively by most peers. The platform was reported as being seen as usable, informative and interesting by peers. However, for some participants, peers reported indifference and lack of interest. For those who did not engage with the platform, some alongside their peers reported that they did not see a need for the platform, viewed it as unnecessary and that the process of completing all activities on the platform took a long time.
Recommendations for how engagement and the platform might be developed	Participants who engaged and did not engage with the platform provided multiple suggestions for how the platform and future e-Learning platforms might be implemented in the context of undergraduate medical education in Nigeria. In terms of integration with the existing curriculum, participants felt that it could be included as part of surgical training, as internal medicine, or as “…a broad aspect of pharmacology or medicine” (ESUT student 3). Suggestions were provided for how engagement might be increased in future iterations or deployments of the platform. Some participants felt that creating wider awareness of the platform would be advantageous, although many felt that the platform would “need to be made compulsory” (UNN student 1) to increase engagement. In terms of platform content, participants suggested that “reducing the number of video tutorials may make it a bit cheaper as regards data usage”, or that content should be made more concise. Incentives were also suggested by participants, including “…making the app offline, cutting down on the many modalities” (ESUT student 4), providing “…free Wi-Fi strong and accessible by all students” (LUTH student 1), potentially providing a “small income because some get discouraged they have to use lots of data without getting paid” (BUK student 2), or providing participants with more time to access and complete content on the platform.

Discussion

To our knowledge, this is the first study to explore the potential of an online approach to improve the knowledge of pain and its management in medical students studying at colleges of medicine located in an African LMIC. The e-Learning course significantly increased the knowledge of pain management across all five participating sites with student users rated the e-Learning platform positively for both its pragmatic (task-oriented) and hedonic (non-task oriented) qualities. A lack of capacity within the medical curricula meant the e-Learning platform was provided as an optional activity for students, with uptake from one third of all students who were invited to participate. For those that engaged with the platform, e-Learning was highlighted as a valuable approach to delivering pain education. Factors highlighted as requiring consideration for future development included arranging access points for downloading content at no personal cost to the user, and integration within existing curricula, such as surgical training or internal medicine.

Improved pain management knowledge through e-Learning achieved within this study aligns with previous online programmes evaluated in developed country settings for pain management [21, 22]. Pain beliefs and attitudes are formed during medical education [11]. The e-Learning platform developed within this project provides a resource that could help to address a gap in pain education of undergraduate medical students in Nigeria. This pilot project represented a novel effort to bridge the gap created by the absence of a standardized pain curricula in Nigeria. Due to compact curricula across the participating sites, we were not able to integrate the e-Learning platform into ongoing teaching activities. Having demonstrated its ability to improve pain knowledge across participating students, the next crucial phase of development rests on integration within colleges of medicine. This will require an increase in leadership to drive advocacy for adaptation of existing medical school curricula to accommodate a new subspecialty curriculum around pain management. However, e-Learning in medical education is a means to an end, rather than the end in itself [34], and a broader focus on developing the institutional readiness in human and infrastructural resources needed to ensure adoption and sustainability of the platform is now essential. If integration can be achieved, the e-Learning platform could be a tool for health system strengthening to better prepare medical students to support the unmet needs of those with pain in Nigeria and the wider sub-Saharan Africa region [35]. The e-Learning platform developed in this study was supported by a private technology company with expertise in eLearning and an existing track record of supporting health systems strengthening initiatives in Nigeria. The platform has previously been adopted by the government in Nigeria for health worker training in both child and maternal health [36] and disease outbreaks such as Ebola [37]. The next steps of development will require further consideration to determine how to balance access to content and its wider rollout with no or minimal cost to the end user, alongside ensuring alignment to frameworks aiming to enhance the effectiveness of e-learning as an educational tool to increase the quantity and quality of medical education programs [38].

E-Learning offers multiple benefits in the context of developing countries, including flexible learning, time efficiency, reducing costs of printing and paper-based materials, easily modified and updated content, standardization of course content and delivery, the ability to deliver teaching at a distance, and scalability [38]. Despite being optional, a third of all participants completed and sought to engage with the e-Learning platform for pain education, with those who engaged reported many of the beneficial attributes of e-Learning approaches. For those who did not participate, there was a sense of not seeing value, having the time, or the interest in using the platform, alongside experiencing issues with technology; similar experiences have been reported elsewhere [39]. Perceived usefulness of e-Learning can have a strong impact on students’ e-Learning intention [40]. The need to advocate for pain education as part of the curriculum is likely to be a crucial component to support further e-Learning approaches, ensuring greater perceived utility of an e-Learning platform that would complement teaching. Furthermore, strategies to improve student engagement in online courses are being developed which may be embedded into future iterations of the platform, including active learning assignments (e.g. discussion boards) which serve to engage students with course content and their peers’ course [41].

The study sought to engage students in an online approach to learning but equally adopted an emerging method to capture views and perspectives on the content and structure of the e-Learning platform. Using WhatsApp as a means of conducting focus group interviews is still an evolving research area, providing participants with easier access to interviews, greater freedom to talk about sensitive issues, the ability to express themselves via text and the possibility of exercising greater control over the interview [42]. Participants were able to freely express positive and negative opinions alongside indifference regarding the platform, whilst providing useful insights into drivers and barriers to engagement. However, as has been highlighted in previous research, WhatsApp group chat may also reduce the quantity and richness of conversation when compared to focus conducted in person [43]. This is an emerging research tool that worked well for undergraduate medical student participants, but may require further innovation and adaptation to evolve alongside phone-based and in-person qualitative research approaches.

This study has limitations. The one-group pre- and post-test design which was used to evaluate changes to pain knowledge has been criticised for its vulnerability to internal validity threats, such as maturation, history, and testing [44]. However, the lack of pain education within the existing curriculum and restricted time between the pre- and post-test measures reduced the impact of possible alternative explanations for the observed differences. Our study limited its focus to pain knowledge, meaning we are unable to determine any influence on skills and attitudes relating to pain management. Participation in the study was voluntary so may have introduced selection bias in the study population, recruiting those most engaged with study. With coverage of one third of all invited participants, this may have led to inflated improvements. However, when considering this at level of site, 74% of those invited at BUK participated, with this site reflecting the largest marked improvement in the post-test as evidenced by mean scores differences. Although, due to low completion rates across some sites the subgroup analyses may have been limited in its ability to detect significant changes in pain knowledge across sites. A further limitation to the study was the budget. This was a pilot study funded through a pump-priming grant. We were unable to support wider costs incurred during delivery, such as data usage for downloading content by students. This may have deterred participation and will be factored in to future iterations, including exploring support that can be provided within institutions to facilitate content downloads. Access to content must be ensured in an equitable way to avoid exacerbating any divide driven by an ability to cover costs of data.

Conclusion

We outline the development and evaluation of an e-Learning approach to address deficits in existing provision of medical undergraduate training. The e-Learning platform led to improvements in participants’ pain knowledge and provided a good user experience; positive outcomes for a target population in which pain beliefs and attitudes are formed. We sought to provide a detailed overview of our development and evaluation process to address criticism of existing research which has failed to provide sufficient detail to support transferability or inform future e-Learning approaches. Future iterations and implementation of the e-Learning platform will be heavily reliant on institutional support, to accommodate pain education within medical school curricula and ensure adequate infrastructure to enable equitable access to content (e.g. facilitating downloading of content). e-Learning has the potential to fundamentally change and shape health systems and the quality of health education. By utilising this approach for pain education in Nigeria, the quality and quantity of health care delivery and access for those with unmet pain needs could be improved.

Individual-level pain knowledge scores.

Pain knowledge scores attained by students at pre-test, following completion of modules and at post-test.

(PDF)

Click here for additional data file.

11 Sep 2020

PONE-D-20-19766

Project OPUS: Development and evaluation of an electronic platform for pain management education of medical undergraduates in resource-limited settings

PLOS ONE

Dear Dr. Allsop,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The recommendation is to revise the manuscript as follows (see for details the enclosed reviewers' comments):

- mention in the limitations of the study that only knowledge acquisition was examined without considering skills and attitude

- show is there is any feedback about comparison of App with other platforms

- clarify if there is any Interactive element in the e-learning system

- clarify if the e-learning system can be used only for self learning or student interaction could be allowed with the teacher and/or with the other students

- use latest IASP 2020 pain definition

- mention other similar courses on pain in developed countries – UK, Europe, Australia, Canada

- discuss implications that VTR Mobile is a proprietary item – or mention under limitations

- correct the citation style per journal requirement

- discuss with more detail differences between completers and non-completers

- write more concisely some areas and summarise the development in a flow diagram

-specify guiding pedagogical theories of online/e-learning that supported the development of the curriculum.

Please submit your revised manuscript by Oct 26 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Filomena Papa

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Competing Interests section:

"Okey Okuzu is Founder and CEO, and Chinelo Oduche a Project Manager, for InStrat Global Health Solutions, the company providing the software on which the e-Learning platform was developed in collaboration with the research team. All other authors declare no conflict of interest."

We note that one or more of the authors are employed by a commercial company: Instrat Global Health Solutions.

2.1. Please provide an amended Funding Statement declaring this commercial affiliation, as well as a statement regarding the Role of Funders in your study. If the funding organization did not play a role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript and only provided financial support in the form of authors' salaries and/or research materials, please review your statements relating to the author contributions, and ensure you have specifically and accurately indicated the role(s) that these authors had in your study. You can update author roles in the Author Contributions section of the online submission form.

Please also include the following statement within your amended Funding Statement.

“The funder provided support in the form of salaries for authors [insert relevant initials], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section.”

If your commercial affiliation did play a role in your study, please state and explain this role within your updated Funding Statement.

2.2. Please also provide an updated Competing Interests Statement declaring this commercial affiliation along with any other relevant declarations relating to employment, consultancy, patents, products in development, or marketed products, etc.

Within your Competing Interests Statement, please confirm that this commercial affiliation does not alter your adherence to all PLOS ONE policies on sharing data and materials by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests) . If this adherence statement is not accurate and  there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

Please include both an updated Funding Statement and Competing Interests Statement in your cover letter. We will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

3. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is a very good and much-needed study, with robust methodology and clearly presented findings. However, there are a number of relatively minor comments.

1. Only 1/3rd of the eligible students completed the course – difference between completers and non-completers? Selection bias potential. Has been briefly discussed under limitations, but a comparison would help.

2. Was it at appropriate stage for the students to grasp? What is the medical education system there? It would help the international reader.

3. Knowledge acquisition – what about skills and attitude? Should be mentioned as a limitation.

4. App vs. other platforms? Any feedback?

5. Was there any Interactive element? To what extent?

6. Pain definition has been revised recently – use latest IASP definition 2020.

7. Please mention other similar courses on pain in developed countries – UK, Europe, Australia, Canada.

8. VTR Mobile – proprietary item – concerns about dependence on this particular proprietary item? Needs discussion or mention under limitations.

9. Any reason why pass threshold was set at 70% (Table 1)? Page 13 last sentence says 80%. Please clarify.

10. Table 5: cite only the median with IQR data, not the mean and SD data.

11. Please correct the citation style per Journal requirement (number of authors before et al. can be used).

Reviewer #2: Thank you for submitting this paper for consideration with PLOS one. This is an interesting study that was funded by a leading international body, the International Association for the Study of Pain. The development and evaluative research are carefully described and the study uses mixed and multi-methods to evaluate a pain curriculum and mobile platform for use in Nigeria. Some areas of the paper could be more concisely written which is difficult with a complex project and data. Some specific comments are below that I hope are helpful in refining the paper.

A structured abstract that provides a good summary of the project

Introduction

A well written and interesting introduction that provides a strong rationale for the project.

First line describes pain as a very frequent complaint. Please consider revising this as representing pain as a complaint suggests that it is minor, a symptom rather than a complex condition or experience that can have a major impact on people. Also, this emphasizes the reporting of pain and there are many who cannot express pain or are reluctant to do so.

IASP definition of pain has recently been updated and needs to be reflected in the paper

Overview of platform development – HIT acronym needs explaining/writing out

This paper describes the detailed development of the platform and curriculum which is one of its strengths. Anyone developing something similar would find this information helpful. However, some areas could be more concisely written and the development could be summarised in a flow diagram to reduce the word count.

As well as the review to determine the pain-related content, were there any guiding pedagogical theories of online/e-learning that supported the development of the curriculum?

Research tools: Knowledge survey –what was covered in this? Pain Beliefs Survey – reference and details needed. Whether reliability and validity testing was conducted on these is needed

Page 15 - In this sentence, change done to assessed ‘The normality of distribution of the data was done…’ and later in the same paragraph – undertaken is another option

Results section

Page 16, third line- should this be completion rate rather than retention rate? And again two lines later. Not sure retention is the correct term

Table 3 states that 8 people completed the pre and post test from the ESUTH site but in Figure 2, this number is 5. Please can this be corrected with whichever is correct.

Table 7 presents the qualitative analysis but it is unclear why a table has been used to present nearly 900 words of text. Consider revising.

Discussion

A strong discussion. Some acknowledgement of the limitations around the smaller sub-groups (sites) and statistical analysis would be important

Conclusion

First line needs revision as it raises much broader issues that are not the focus of this paper, particularly opioid abuse/misuse. The line reads ‘Inadequate pain education will perpetuate existing public health problems of undertreated, pain, untreated pain and opioid abuse

Thank you for the opportunity to review the paper and good luck with the next stages of development for the project.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Sukanya Mitra

Reviewer #2: Yes: Dr Emma Briggs

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

21 Sep 2020

Please see attached our cover letter and 'Response to Reviewers' document outlining our point-by-point response.

Submitted filename: Response to Reviewers.doc

Click here for additional data file.

11 Nov 2020

PONE-D-20-19766R1

Project OPUS: Development and evaluation of an electronic platform for pain management education of medical undergraduates in resource-limited settings

PLOS ONE

Dear Dr. Allsop,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

It is recommended to carefully revise the manuscript including modifications suggested by reviewer 3, correcting typos (e. g. in section data analysis "pre-and post-test", page 8 "pre and post-test", "user-centered design[25]", "retirement.[24]") and eliminating discrepancies in notation (e .g. page 18 "Pre and post test scores " and "The pre-test, post-test...", Table 5 "Median Posttest").

Please submit your revised manuscript by Dec 26 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Filomena Papa

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: All the comments have been responded to my satisfaction. Corrections and changes have been made where possible and needed, and mentioned as limitations where needed but not possible. This version appears acceptable to me now. Congratulations on a very important work!

Reviewer #3: This original research paper evaluated the implementation of a voluntarily 12-week e-learning course on pain education across five medical schools in Nigeria, with positive outcomes in regards to implementation of the method of e-learning and increased knowledge on pain. The advantages and limitations of this study have been nicely discussed. I want to congratulate the authors on their nice work on an important topic.

I have minor suggestions for revision:

1. The whole reference list needs to be revised due to errors and mixups. As an example, see refs 28 and 33.

2. In the abstract, result section, perhaps it would be nice to state the collective pre- and post test scores across all medical schools in numbers and not only mention the good effect sizes. If word count is an issue I suggest removing (for instance) the mention of what statistical test used to compare pre- and post scores.

3. Methods. Very precisely described in regards to the platform. However, I would like some more information on how the selection of different sections from the IASP curriculum was made. Information on why the whole curriculum was not used and why additional resources were used to form the teaching material for this course. I think you should also add a reference to the specific curriculum outline used on the IASP website (if any).

4. On page 14, Method section; it is stated that the students were able to retake the tests. I recommend stating if this relates to the tests taken after completing each section (or if it also includes the pre/post tests.)

5. The pain beliefs survey mentioned in the methods; as far as I can see these results are not presented in this paper and the reference to its validation is not made on the students from this study, so why is this mentioned in the methods? Please clarify.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Sukanya Mitra

Reviewer #3: Yes: Linda Rankin

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

17 Nov 2020

Please see attached our cover letter and 'Response to Reviewers' document outlining our point-by-point response.

Submitted filename: Response to Reviewers.doc

Click here for additional data file.

24 Nov 2020

Project OPUS: Development and evaluation of an electronic platform for pain management education of medical undergraduates in resource-limited settings

PONE-D-20-19766R2

Dear Dr. Allsop,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Filomena Papa

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: No further comments. All the comments have been responded to

my satisfaction. Corrections and changes

have been made where possible and needed,

and mentioned as limitations where needed

but not possible. This version appears

acceptable to me now. Congratulations on a

very important work!

Reviewer #3: All my comments and recommendations have been met by the authors. I therefore recommend for this paper to be accepted.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Prof. Sukanya Mitra

Reviewer #3: Yes: Linda Rankin

26 Nov 2020

PONE-D-20-19766R2

Project OPUS: Development and evaluation of an electronic platform for pain management education of medical undergraduates in resource-limited settings

Dear Dr. Allsop:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Filomena Papa

Academic Editor

PLOS ONE