Continuous research monitoring improves the quality of research conduct and compliance among research trainees: internal evaluation of a monitoring programme.

Background: Research site monitoring (RSM) is an effective way to ensure compliance with Good Clinical Practice (GCP). However, RSM is not offered to trainees (investigators) at African Institutions routinely. The Makerere University/Uganda Virus Research Institute Centre of Excellence in Infection and Immunity Research and Training (MUII-Plus) introduced internal monitoring to promote the quality of trainees' research projects. Here, we share our monitoring model, experiences and achievements, and challenges encountered.
Methods: We analysed investigators' project reports from monitoring visits undertaken from April 2017 to December 2019. Monitors followed a standard checklist to review investigator site files and record forms, and toured site facilities. We planned four monitoring visits for each trainee: one at site initiation, two interim, and a closeout monitoring visit. A team of two monitors conducted the visits.
Results: We monitored 25 out of the 26 research projects in progress between April 2017 and December 2019. Compliance with protocols, standard operating procedures, GCP, and GCLP improved with each monitoring visit. Median (IQR) compliance rate was 43% (31%, 44%) at site initiation visit for different monitoring items, 70% (54%, 90%) at the 1st interim monitoring visit, 100% (92%, 100%) at 2nd interim monitoring visit and all projects achieved 100% compliance at site closeout.  All investigators had good work ethics and practice, and appropriate facilities. Initially, some investigators' files lacked essential documents, and informed consent processes needed to be improved. We realized that non-compliant investigators had not received prior training in GCP/GCLP, so we offered them this training. Conclusions: Routine monitoring helps identify non-compliance early and improves the quality of research. We recommend continuous internal monitoring for all research studies. Investigators conducting research involving human subjects should receive GCP/GCLP training before commencing their projects. Institutional higher degrees and research ethics committees should enforce this as a requirement for project approvals. Copyright:

Background

Research site monitoring (RSM) is a systematic process that involves the close supervision of an investigator to ensure that all research activities are implemented according to the approved study protocols and good clinical practice (GCP).

All studies that involve humans subjects must be reviewed ethically and scientifically before their start [1– 5] and monitored as per international human research regulatory guidelines [4]. Research Ethics Committees are critical in giving independent corrective review of proposed studies to ensure that the dignity and wellbeing of potential participants are fully protected [1]. These committees review study tools such as consent and data collection forms and laboratory and data analysis protocols, to ensure that they align with GCP and Good Clinical Laboratory Practice (GCLP) guidelines [4, 6].

Research site initiation procedures and routine monitoring of ethically approved human studies are essential to ensure that all investigators are qualified and competent to undertake the proposed work, required study facilities and tools are available, participants’ rights and safety are protected during data collection, and data is collected accurately to produce reliable results [4]. Additionally, continuous monitoring prevents research fraud, minimizes un-ethical practices, enables early detection of protocol deviations, and ensures rightful and effective dissemination of research results [4, 6, 7].

The Makerere University/Uganda Virus Research Institute Centre of Excellence in Infection and Immunity Research and Training (MUII-Plus) is a program under the African Academy of Sciences DELTAS Initiative, whose goal is to promote scientific quality and to train future research leaders for excellence ( www.muii.org.ug). The MUII-Plus umbrella supports trainees (investigators) including undergraduates, postgraduates, post-doctoral fellows, and emerging research leaders.

At the start of the MUII-Plus programme, we realised that a number of trainee investigators had limited knowledge of procedures governing research and how to conduct their projects correctly. To equip the investigators with the necessary skills and promote scientific quality, MUII-Plus launched routine monitoring of research sites and activities for all their investigators in April 2017.

In this paper we present a model for internal monitoring of trainee investigators’ research projects that we have found achievable and effective in a local academic research setting. We believe this model can be adopted by other training programmes to benefit and support the progress of their investigators.

Methods

Site monitoring processes

Routine monitoring of research projects for all MUII-Plus investigators commenced in April 2017 to date. This involves internal monitors reviewing and evaluating investigators’ research sites and projects based on a standard checklist ( Table 1). In this study, we report findings for monitoring done between April 2017 and December 2019.

Table 1.

Items included in the MUII-Plus trainee’s monitoring checklist.

Reviewed documents
1.   Institutional Review Board & Uganda National Council for Science and Technology approval/favourable opinion notification
2.   List of members of Ethic Committee
3.   Administrative letter from the study site (e.g. hospital; if applicable)
4.   Signed approved protocol (and all amendments)
5.   Stamped consent/assent forms & all translations (including translation certificate)
6.   Subject recruitment material e.g. briefing/information slides, participant handouts, adverts for subject recruitment such as radio, TV & other media adverts
7.   Blank copies of Case Report Forms (CRFs), source documents, lab request forms, master Serious Adverse Event form, protocol deviation form, screening log, enrolment log, reimbursement form etc.
8.   Study financial agreement (put note to file if this is filed elsewhere)
9.   Insurance statement for research related injury (if applicable)
10.  Study staff training records e.g. protocol training, Standard Operating Procedure (SOP) training, source document training, CRF & electronic (e)CRF training records,
11.  Updated signed Curriculum Vitae for each study staff
12.  Certificate of qualifications
13.  Updated signed Job Descriptions
14.  GCP/HSP certificate
15.  GCLP for lab personnel in addition to the above certificate
16.  Annual Practice Licenses (APL) (where applicable)
17.  Study monitoring plan
18.  Site monitoring log
19.  Site Initiation Visit (SIV) agenda
20.  Site Initiation Visit Report
21.  Interim monitoring agenda
22.  Interim monitoring report
23.  Close out monitoring agenda
24.  Close out monitoring report
25.  Delegation of Duties (DoD) Log
26.  Site staff contact details list
27.  Study quality management plan
28.  Participant flow chart
29.  Communication flow chart
30.  Lab accreditation certificate if applicable
31.  Laboratory analytical plan if applicable
32.  Material Transfer Agreement if applicable
33.  Study specific SOPs
34.  MUII-Plus engagement plan
35.  MUII-Plus award letter/acceptance letter
36.  Meeting minutes with supervisor and study team
37.  Gantt chart
Inspection of facilities
1.   Adequate facilities for all study related procedures
2.   Site has received all supplies required to conduct the study
3.   Adequate facilities for storage of samples

Four monitoring visits were planned for each research project; site initiation (SIV), two interim (IMV), and a closeout monitoring visit (CMV). A team of two (MA and SC) conducted the monitoring visits. MA, a registered midwife, worked as a research nurse for nine years, then trained as a clinical trial monitor in 2011 under the East African Consortium for Clinical Research (EACCR), and was certified as a Clinical Research Associate (CRA) in 2017 by African Clinical Research Organisation (ACRO); she is experienced in monitoring observational studies and clinical trials. MA was assisted by SC, a registered nurse with a 15-year experience; SC also trained as a Clinical Trial Monitor under EACCR.

For each new study, the monitors and investigator (trainee) discussed, planned, and shared a list of essential documents to be reviewed at least a week before the first monitoring referred to as the site initiation visit (SIV). Once the SIV date was confirmed, the monitors sent a monitoring agenda to the investigator before the visit.

Site initiation visit (SIV)

The SIV was to establish research sites and facilities to ensure investigators had all the necessary approvals, qualified and skilled staff, data collection tools and documents, and laboratory materials to implement the proposed research project.

During this visit, the investigator was asked to share and explain his or her project proposal, clinical, laboratory, or pharmacy procedures as applicable, and data management plan. Similarly, the monitors informed investigators about the purpose of the monitoring, the monitors’ and investigator’s responsibilities, informed consent procedures, and good documentation practices. Additionally, the monitors critically verified the investigator’s site file (ISF) which comprised of academic documents, approved protocols, valid practicing licenses, GCP, and GCLP certificates for staff as applicable ( Table 1). In case of any queries, the investigator was given time to address them and a SIV follow-up visit was done for corrective action before the project commenced.

Interim monitoring visit (IMV)

The IMV followed the SIV intending to review the progress of the commenced project. First, the monitor checked whether the investigator screened and enrolled participants, collected, documented, and managed data as described in the approved standard operating procedures (SOPs) and protocols. For data management, the monitor verified that the completed data collection forms or source data matched that entered in the database and backed up routinely. Second, the research site was toured to ensure adequate and proper use of research materials, procedure rooms, and storage facilities for specimens and samples, documents and drugs as applicable. The IMV was concluded with a discussion on the key issues identified and the investigator was advised on the appropriate action to address the issues, as the investigator awaited a detailed visit report.

Study closeout monitoring visit (CMV)

The CMV was performed at the end of the research project when all study participants’ visits and follow-up were complete and all data collected as required. Here, the monitor revisited the ISF, consent forms, data collection forms, and databases, to ensure that all were complete. In addition, the monitor and investigator planned for proper storage of study documents and samples to enable easy retrieval for future use.

Monitoring report

After each visit, a written report was shared with the investigator and his or her supervisors for review and signing. Then the monitor co-signed the final report and shared it with the investigator, MUII-Plus programme centre manager and director. In case of any critical findings that could not be resolved between the trainee and monitor, the director or centre manager would have meetings and discuss the way forward with the investigator.

Model used to review the monitoring reports

To assess compliance of MUII-Plus investigators to Good Clinical Practice, the monitoring team considered six elements: (1) regulatory documents, (2) informed consent process and documentation, (3) protocol adherence and Source data verification (SDV), (4) study-related training, (5) working practices and (6) tour of project site facilities ( Table 2).

Table 2.

General Monitoring Activities conducted for all the four visits.

Item	Essential document for review	Observations
	1.  Regulatory documents	Approved protocols	Availabilities of study related documents
		Informed consent / assent forms and wavier of consent if applicable
		Approval letters from Research Ethics Committees (REC)
		Case Report Forms
		Annual Practice Licenses (APL)
Curriculum Vitae and academic documents
2.  Informed consent documentation and participant status	All the screened and enrolled participants have signed and dated copy of current approved ICFs prior to any study-related procedures being conducted	Observe the process of obtaining informed consent forms
	Investigators maintain logs of screened and enrolled participants within the study
	Storage consent forms available for all samples stored in the freezers or waiver of consent if applicable
	Amount of reimbursement approved in consent forms given to participants and documented
3.  Protocol adherence and Source Data Verification (SDV)	Source documents and other study records are accurate, complete, and up-to-date, and check the accuracy and completeness of the case report form entries	Observe protocol deviation
4.  Study related training	Protocol and SOP training records, source documents/case report form training	Ability to perform as trained
	Updated GCP/GCLP certificates	Training certificates
	Protection of Human Research Participants (PHRP) Certificates
5.  Working practices	Availabilities of SOPs and delegation and responsibility log	SOPs at work station
	Minutes of meetings with Supervisors and study team if applicable	Frequency of meetings
6.  Tour of project site facilities	Clinic room, Laboratory process area and data management area and pharmacy facility	Adequate facilities for study related procedures and storage of records and study drugs
	Study reagents and materials	Site has received all supplies required to conduct the study
	Storage facilities for specimens collected and study drug if applicable	Adequate facilities for storage of samples and study drug if applicable

Regulatory documents are guidelines that the monitor uses to keep the investigator within the legal and ethical boundaries during their research projects, and assess the research conduct and quality of data generated. These included approved protocols and consent documents, data collection forms, curriculum vitae, academic documents and others, as described in ( Table 2).

Obtaining informed consent from participants is very important in the ethical process of human research. This process requires that the investigator respects and protects the rights of the participants by thoroughly explaining the research objectives and expected requirements from participants before obtaining their consent. All participants sign and date on the consent form as proof of their consent to enroll in the research. After this, a copy of the form is shared with the participant. Throughout the project, the investigator and participant maintain information exchange, and the participants reserve the right to withdraw their consent.

Protocol adherence and source data verification requires that the investigator adheres to the approved protocols to ensure data generated and captured is accurate and complete.

An investigator and their staff must undergo thorough training on different aspects of the proposed research project, including GCP/GCLP guidelines and SOPs, so that they are competent in their work. Often, members are awarded certificates on completion of the trainings which they put on file.

Evaluation of working practices involves assessment of teamwork and coordination between research investigators and staff for effective communication and implementation of the research project. For example, tracking the number of times trainee investigators meet their supervisors and checking whether meeting minutes are on file. All these aspects were evaluated based on whether documentation was present at each site visit.

Data extraction and analysis

We extracted data on components monitored from the approved and signed off monitoring reports from each visit. The data was entered into an excel spreadsheet with each variable representing an item in Table 2. For each project, a score of one (1) was assigned to each item if its documentation/facility was present and zero (0) otherwise. An average score was obtained and converted into a percentage compliance for every visit. We used Stata version 15.0 (StataCorp, College Station, USA) for analysis.

Ethics and consent

This report describes the findings of an internal evaluation undertaken to support learning, following the implementation of internal monitoring to enhance the quality of work undertaken by research trainees. The work was reviewed by the Research Ethics Committee of the Uganda Virus Research and a determination of “non-research”, waiving the requirement for ethical review and approval, was made. All the investigators gave written permission for the reports on their work to be used for this evaluation and publication.

Results

We reviewed documents and reports for masters, PhD, and post-doctoral fellows’ projects running between April 2017 and December 2019. During this period, there were 26 research projects, and we monitored 25 (96.2%) of these. Of the monitored studies, 18 underwent a site initiation visit (SIV), 12 underwent SIV follow-up, 14 had the first interim monitoring visit (IMV), 5 had a second IMV, and 8 had a closeout monitoring visit (CMV) by the time of analysis. Some studies did not have all monitoring visits because they started earlier than the monitoring programme ( Figure 1).

Figure 1.

Projects monitored at each monitoring visit.

Regulatory documents

During the SIV, 43% of the projects were compliant based on regulatory documents. The compliance was lower than expected because investigators had not obtained project or protocol approvals from the different Research Ethics Committees at the time of analysis. One study lacked regulatory documents on file, and it was hard to determine whether it had valid approvals and was compliant in other administrative aspects. At this time, none of the projects that planned to ship biological samples had obtained the material transfer agreements (MTA) required. We also observed poor documentation practice: for instance, many investigator files did not have a table of contents, and it was difficult for the monitor to identify and access filed records quickly.

However, the compliance improved to 77% at the time of the SIV follow-up visit. There was an improvement of 92% and 100% during the second interim and final closeout visits, respectively ( Table 3 [8]).

Table 3.

Performance of investigators at each monitoring visit.

	Mean percent (%) compliance of investigators
	Site Initiation Visit (SIV)	SIV Follow-up Visit	1 st Interim Monitoring Visits (IMV1)	2 nd Interim Monitoring Visit (IMV2)	Close-out Monitoring Visit (CMV)
Number of investigators assessed	n=18	n=12	n=14	n=05	n=08
1.  Regulatory documents	43	77	70	92	100
2.  Informed consent documentation and participant status	44	75	93	100	100
3.  Study related training	31	75	54	90	100
4.  Working practices	28	50	50	100	100
5.  Tour of project site facilities	81	92	90	100	100
Average compliance across all domains	45	73	71	96	100

Informed consent process and documentation

There was 44% compliance with the informed consent process and documentation at SIV. Sometimes essential documents such as the informed consent forms were still being developed or under consideration by the ethics committees. Compliance levels increased during the following monitoring visits, 75% at SIV follow-up and 93% at first IMV. At these visits a few projects had incomplete or missing consent forms and in some cases research staff had signed as witness for participants (contrary to good practice).

By the second IMV and CMV, all projects (100%) were compliant with complete consent forms and documentation ( Table 3).

Protocol adherence and source data verification (SDV)

Overall, the majority of the investigators adhered to their research protocols and standard operating procedures, and data collected was accurate and complete.

Study-related training

Only 31% of investigators had evidence of study-related trainings at the SIV because most of them had not received training on GCP/GCLP guidelines and SOPs. On subsequent monitoring, 75% and 54% of investigators and their staff had been trained and certified at the SIV follow-up and first IMV, respectively. Towards the last monitoring visits, we achieved 100% compliance ( Table 3).

Working practices

Under working practices, compliance was 28% at the SIV and improved to 50% at both the SIV follow-up and first IMV. Some investigators had held and documented study-related meetings regularly. A few had no meetings at all at SIV. During the second IMV and CMV, full compliance of (100%) were recorded ( Table 3).

Tour of project site facilities

While carrying out the tour of project site facilities, the monitors focused on the clinic room, laboratory process area, data management area, study reagents and materials availability, storage facilities for specimens collected, and study drugs.

The majority of the research sites had facilities that were adequate to conduct the studies. The facilities complied with the minimum standards described in Table 2 at 81% during the site initiation visit and above 90% for the subsequent monitoring visits ( Table 3). However, we noted congestion at participant recruitment stations.

Challenges encountered by monitors

The monitors encountered logistical delays from investigators in confirming appointments for monitoring, reviewing and giving feedback on monitoring reports, and addressing monitoring issues raised.

Discussion

We have presented an internal evaluation of the MUII-Plus research monitoring programme. Our findings show that many trainee investigators, and their research teams, needed training in good clinical research practice – to an extent that we had not recognised at the start of our programme. Through internal monitoring, we recognised the needs of investigators and trained them, which improved their compliance with the research guidelines. We believe that these findings highlight a critical training need, and we present a monitoring model that could contribute to advancing research excellence across Africa.

The reviewed reports emphasized the need for investigators to pay close attention to the regulatory requirements, especially ethical approvals for their research projects, and the monitors to carry out a pre-site assessment visit to minimize non-compliance observed during the site initiation visits. Our findings reflect experience across the continent: in one example, only 9.8% of student dissertations on HIV across universities in Cameroon [9] documented ethical approvals. There is a need to address the lack of knowledge in both students and their mentors about principles guiding human research, and requirements for documentation of approval processes.

Informed consent is an aspect of ethical human research that needs keen attention. A study done in Uganda between 2007 and 2010 showed that 36% of research sites violated the informed consent process [7]. We found that, at first, the informed consent process was not adequately practiced by some investigator trainees in the MUII-plus programme: some projects had incomplete consent forms, project staff signed as witnesses for participants, signed copies were not given to participants, and occasionally forms were missing. However, our continuous monitoring showed improved compliance up to 100% at the second IMV and CMV. The marked improvement observed in our study implies that consent processes during investigators’ projects can be improved by prior training and sensitization of investigators and their study teams and frequent monitoring.

During the site initiation visit, compliance in terms of providing research teams with study-related training and adopting good working practices, such as regular team meetings, was low, 31% and 28%, respectively. Here, GCP/GCLP certificates and protocol training logs for team members, and minutes for supervision or team meetings, were lacking. This low compliance was because the trainee investigators lacked knowledge on the kind of team trainings they were supposed to undertake. This prompted the MUII-Plus programme to fully fund face-to-face GCP/GCLP training for all investigators and their teams in 2017 and 2018. Following the first training, all investigators undertake a refresher online GCP/GCLP training, such as the course hosted by the Global Health Training Centre (GHTC) [10], every two years. Investigators must learn the importance of providing protocol and SOP training for their team before research work commences, and be involved in team-building activities and regular team meetings to maintain effective communication and implementation during the research activities.

Site facilities for our trainees were found to be relatively adequate concerning clinic rooms, laboratory process areas and data management areas, study reagents and materials availability, and storage facilities for specimens collected and study drugs. Not surprisingly, given the setting of busy African hospitals and clinics, a good number of investigators faced the challenge of congestion at recruitment locations, due to limited space.

Reviews of protocol adherence and source data verification were reassuring: the data collected was generally accurate, and complete.

During the CMV, we observed that among some studies that collected samples the investigator lacked a proper plan for longer-term sample and document storage. This is a significant challenge that needs to be faced by African institutions for their trainees and research teams.

Undertaking research as a post-graduate student or post-doctoral researcher is a challenging process with many competing demands on trainees’ time. This must have contributed to the challenges faced by monitors in scheduling their work. Institutional buy-in and a research culture that supports quality and rigour in compliance with human subjects research guidelines is needed to support an effective internal monitoring programme.

Recommendations

Through the MUII-Plus programme monitoring, we have learnt the importance of inducting and training investigators and their teams on GCP/GCLP guidelines, the informed consent process, and protocols before the research activities begin. We urge that Universities and research institutes across Uganda and Africa prioritise these trainings to staff and students before allowing them to embark on any human research project. Institutional research ethics committees should enforce GCP/GCLP training as a requirement for project approval.

Conclusions

The MUII-Plus programme’s monitoring model has improved the confidence and quality of the research output of the investigators tremendously. Routine site monitoring is a successful tool to identify gaps in research training and implementation, and improve the quality of research. Research site monitoring should be introduced and implemented across research institutions in Africa.

Data availability

Underlying data

LSHTM Data Compass: Internal monitoring within MUII-plus for research capacity development. https://doi.org/10.17037/DATA.00001938 [8]

This project contains the following underlying data:

Project_monitoring_data_XLSX.xlsx (A dataset containing data provided by 25 projects for an internal monitoring evaluation of the MUII-plus research programme)

Data are available under the terms of the Creative Commons Attribution 3.0 Unported license (CC-BY 3.0).

Main comments:

This paper has the potential to be a valuable contribution to the discussion of how best to support the training of new clinical trial investigators. It details a programme of monitoring of the studies of trainee Investigators by the Makerere University/Uganda Virus Research Institute Centre of Excellence in Infection and Immunity Research and Training (MUII-Plus), giving details of monitoring findings across the programme and proposing it as a potential, practical, model for similar institutes. The results and discussion highlight several findings that should be of concern to all those involved in Investigator training, and highlight the utility of monitoring not only to the research studies themselves, but also to wider investigator training.

However, I think that it would benefit from some careful clarification of its background and aims sections (and abstract) to allow potential readers to appreciate the subject matter that it deals with. At present the title and above named sections do not accurately convey the nature of the paper. As it is, the reader only really realises the full scope of the paper part way through the Methods or even into the Results section. This is a shame, as the results section provides strong evidence of the need to provide experienced monitors, independent of the research project, to inspect new Investigator’s first research studies, and provide guidance and training at the appropriate point to ensure regulatory compliance, patient safety and wellbeing, and study integrity. The authors also point to some extremely important findings regarding the training of new Investigators (especially in GCP/ GCLP) before they even embark on a research study with human subjects. Slight changes to the abstract and background would much better set the scene for these important observations.

I also have some concerns over the simplicity of the analysis, which is very simple, and perhaps overly basic. Scoring was binary, with a point awarded for each item if a document or facility was present (zero if not). The presence of a point therefore does not necessarily indicate that the quality of that document or facility was adequate, or at least, the reader is not told that the scoring took into account quality. This leaves the question of whether, say, a point would have been awarded if the delegation of duties log had been present but not up-to-date, or a monitoring plan in place but no evidence of it being followed. I would therefore like to see an improved manuscript that explains in more detail the scoring, and includes a good discussion of the limitations of this approach.

Overall, the paper proposes a useful approach to supporting new Investigators with their first clinical studies, and highlights some important gaps in Investigator training. However, several key issues with the current manuscript need to be resolved.

Other issues:

The terminology used can be a little confusing especially at the beginning of the paper when the reader is trying to work out what it is about. This is possibly in part because of the difficulty of finding terminology to fit in a field in which “monitoring” already has such heavily used but varied meanings. Here, monitoring is carried out by those external to the actual study (I’ll call them programme monitors hereafter) to assist in the training of new Investigators, and to ensure that their research studies are being carried out appropriately. The use of “internal monitoring” is therefore somewhat confusing. I would suggest that either the terminology is changed to something more appropriate, or a clear definition is set out early in the paper e.g. “ To equip the investigators with the necessary skills and promote scientific quality, MUII-Plus launched routine monitoring of all investigator’s research sites and activities in April 2017, hereafter termed “internal monitoring”. (Note the slight rewording for clarity!).

On a similar note, the use of the phrase “continuous monitoring” could lead to confusion with real-time monitoring methods such as central monitoring of the trial database, so I would suggest removing the word “continuous” or changing it to “periodic”.

Also, the first few paragraphs seem to confuse/conflate checks by Ethics Committees with site monitoring. I would like to see this rewritten to ensure that what is being stated is clear and appropriately referenced. I was surprised to see “and monitored as per international human research regulatory guidelines” being referenced with the Operational guidelines for ethics committees that review biomedical research, which discusses ethical review. I was also surprised not to see ICH GCP E6 (R2) only being referenced by way of the course suggestion at the end of the paper (reference 10): it has a lot to say on monitoring!

I assumed from the mention of GCP that the research studies being monitored were clinical trials, but this is not explicitly stated. It would be useful to state in the Background which types of studies the Investigators were running e.g. clinical trials, single-site, intervention type.

The background section states that the paper will present a model for “internal monitoring” but I think it should also explicitly state that the paper will detail the findings from testing this model out in the MUII-Plus Programme. Similarly, the Methods section should discuss in more detail how the model was evaluated across the programme: i.e. this section shouldn’t just describe the model itself but how the results will be assessed across the whole programme to determine the value of this kind of monitoring to training programmes such as MUII-Plus. (See also my comments on the point system that was used, above).

Table 1 indicates that the programme monitors’ checklist included checks of the monitoring documents for each study but there is no mention of checks of monitoring in Table 2 nor in the Results section. It would have been helpful to see some discussion of the monitoring strategies being employed for the studies by the Investigators themselves, who was carrying them out and what they broadly entailed, and any findings from the programme monitors. As the Background states, monitoring is an essential part of human studies so the emission is a big one in a paper that is basically about monitoring.

The percentages quoted in the Results section can be a little misleading as sometimes they are obtained from few data points. It would be helpful to give a sense of the number of sites involved for each percentage e.g. “There was 44% compliance with the informed consent process and documentation at SIV” could become “There was 44% (8/18 studies) compliance with the informed consent process and documentation at SIV”. For that matter, it is not always clear what is being discussed. For example, in the Regulatory Documents sub-section there is the statement: “During the SIV, 43% of the projects were compliant based on regulatory documents”, but if that meant 8/18 studies that had the SIV, how is the percentage 43% and not 44%? It leaves the reader wondering whether they know all they need to know about how compliance was assessed. Describing nominators and denominators may help.

I would have liked to see a limitations section in the discussion, and details of any future plans: is MUII-Plus continuing the monitoring programme and will they continue to evaluate it? If so, what might they do differently?

I think Figure 1 is useful but would suggest that any good figure should include all the information needed to interpret it. I therefore suggest that any acronyms are explained either in the Figure itself or the Figure label.

I am pleased to see that the data has been shared online, and that a user guide has been supplied. I do think that it would be helpful to explain a few more of the variables in the user guide however, e.g. those that include acronyms such as APL.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Clinical Trial Conduct Methodology; Monitoring; Data Sharing

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

I agree with prior reviewers that the terminology of Research site monitoring (and its definition) implies that it is the supervisory work of investigators; however this activity seems to be the institution of an institution's internal quality monitoring focusing on trainee research projects at different stages. This is a great workforce development and research compliance activity, it would be highly instructional and important to advancing trainee research careers.

Also, this is a method of providing capacity building to your research enterprise and the establishment of institution-wide institutional quality compliance monitoring. Were there other monitoring groups conducting monitoring (e.g., industry sponsors, government sponsors, etc.) or was this the only monitoring taking place at your institution?

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

My areas of research is in clinical research core competencies, workforce development, clinical research training and education, GCPs

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

The authors present a very interesting project, aimed at increasing the quality in the execution of clinical trials. However, I have some major considerations:

cit n 3 refers to GCP R1 but R2 is is the current version.

What types of studies was monitoring applied to? Interventional, observation, industry sponsored, academics?

In case of application to industry sponsored studies, were there any differences to the monitoring normally performed by the promoter?

Table 2: the observations are too general and not adequate to describe the real result of the monitoring. What was observed? What kind and degree of deviations?

Figure 1 is difficult to read, it is recommended to modify it.

To consider the preparation of the PIs adequate, was only the possession of a GCP training taken into account? Perhaps it is not enough, a continuous training program should be considered.

Challenge encountered by monitor: how often? Detail this.

It is not clear what types of indicators and criteria were considered. Furthermore, the description of any hypothetical corrective and preventive actions is missing. With what purpose was the project started? Just observation?

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Not applicable

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Clinical research methodology and management; clinical research training programs

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

The Makerere University/Uganda Virus Research Institute Centre of Excellence in Infection and Immunity Research and Training (MUII-Plus) introduced internal monitoring between April 2017 and December 2019 to promote scientific quality specifically for research projects led by trainees and early to mid-career researchers. This involved up to 4 pre-determined monitoring visits across 25 out of 26 projects during the study period. They report an improvement in the quality of the data and GCP compliance and now propose a model for other research institutes to potentially adopt/adapt in order to promote quality research. Overall, the group share data on a practical implementable approach to improving compliance with GCP in clinical research for trainee/early researchers in LMIC settings. They audit this approach over 2 years and report their findings. The authors may want to consider the following items:

There isn’t a clear description of the differences between research site monitoring (RSM) and internal monitoring and these seem to be used interchangeably. Further definition of both would be useful including who conducts RSM vs internal monitoring and the different aims of the 2 processes. The abstract refers to all investigators having good research ethics and practice but doesn’t define how this was measured. The term continuous research monitoring isn’t clearly defined and may be confused with continuous monitoring processes such as central monitoring suggesting real-time dynamic monitoring processes. We would suggest the authors simply use the regular term, research monitoring, rather than continuous research monitoring for clarity. Details on the nature of each study (clinical trial, observational study, length of the study, sample size, IMP or registered products etc.) would be useful to contextualise the applicability of this approach to different study designs of varying complexity. The utility of this approach may vary depending on these factors. It may be worth commenting on other factors that may have affected the monitoring process; any changes in the regulatory landscape in the period that the studies were evaluated, e.g. new submission requirements, how many protocols had amendments to their procedures, staff turnover etc.

A clear description of what scoring criteria were used to quantify rates of compliance for each monitored item isn’t available. The evaluation and comments on the effect of monitoring should be interpreted cautiously as the required SIV, 2 IMVs and CMV were not conducted across all the studies with a relatively small sample size. Figure 1 requires further clarification. The outcome of those due a visit (IMV or close out) isn’t clear, did these visits occur? For those that didn’t have sequential visits i.e. had 1 rather than 2 IMVs, it is not clear why this was the case. Table 3 could be made clearer. It is unclear if the number of investigators assessed reflect the number of studies monitored, this should be clarified.

It’s unclear if there was a monitoring plan for each study independent of the 4 visits. The authors may want to consider/discuss how this monitoring approach compares/interacts with other monitoring approaches particularly in LMIC settings (e.g. risk-based approach, central, remote monitoring etc.) and benefits and disadvantage of this approach versus others.

It would have been useful if there was a key or breakdown of the scoring for compliance as per the six domains (table 2), noting that in table 3 there were on results on 5 domains.

Do the authors have observational data on studies that had not been monitored using this approach throughout their lifecycle according to the criteria in table 1 and 2 for comparison? This would strengthen any findings.

Overall, proposes a pragmatic approach to trainees/early researchers improving compliance to GCP and the quality of their research outputs using the proposed monitoring tools but there are several minor clarifications that could benefit from further details as suggested above.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Clinical research, Clinical trials, Monitoring

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.