Introducing quality measures in an academic research consortium: Lessons and recommendation from implementing an ad hoc quality management system for organ model research: Lessons and recommendation from implementing an ad hoc quality management system for organ model research.

Lessons from implementing quality control systems in an academic research consortium to improve Good Scientific Practice and reproducibility.

Low reproducibility rates within biomedical research negatively impact productivity and translation. One promising approach to enhance the transfer of robust results from preclinical research into clinically relevant and transferable data is the systematic implementation of quality measures in daily laboratory routines.

Although many universities expect their scientists to adhere to GSPs, they often neither systematically support, nor monitor the quality of their research activities.

Today's fast‐evolving research environment needs effective quality measures to ensure reproducibility and data integrity (Macleod et al, 2014; Begley et al, 2015; Begley & Ioannidis, 2015; Baker, 2016). Academic research institutions and laboratories may be as committed to good scientific practices (GSPs) as their counterparts in the biotech and pharmaceutical industry but operate largely without clearly defined standards (Bespalov et al, 2021; Emmerich et al, 2021). Although many universities expect their scientists to adhere to GSPs, they often neither systematically support, nor monitor the quality of their research activities. Peer review of publications is still regarded as the primary validation of quality control in academic research. However, reviewers only assess work after it has been performed—often over years—and interventions in the experimental process are thus no longer possible.

The reasons for the lack of dedicated quality management (QM) implementations in academic laboratories include an anticipated overload of regulatory tasks that could negatively affect productivity, concerns about the loss of scientific freedom, and importantly, limited resources in academia and academic funding schemes.

Scientists in academia often need greater flexibility

A pragmatic and science‐driven quality management in preclinical research would support innovation and ensure data integrity and an economic use of resources. The recently implemented quality system by the EQIPD (European Quality in Preclinical Data) consortium is specifically designed to support evidence‐based, rigorous research practices at universities and research institutes and in the private sector. The EQIPD quality system entails 18 core requirements that need to be implemented in preclinical biomedical research projects (Bespalov et al, 2021).

However, scientists in academia often need greater flexibility for implementing systematic quality measures. This requires a modular quality assurance system, where application and usage can be tailored to the individual research phase of the project. Such a modular quality system was developed by the team around PREMIER (Productiveness and Robustness through Modular Improvement of Experimental Research) (Dirnagl et al, 2018; Kurreck et al, 2020). The PREMIER quality system provides practical elements for ensuring quality standards founded on data integrity principles and good research practices to enhance scientific value and research reproducibility in academic research.

A pragmatic and science‐driven quality management in preclinical research would support innovation and ensure data integrity and an economic use of resources.

Here, we describe the ad hoc implementation of quality measures in an academic research consortium inspired by the PREMIER Quality System. We present strategies and tools for quality assurance and monitoring, but also discuss bottlenecks and challenges encountered during the process.

Implementing quality control in a research consortium

The global COVID‐19 pandemic motivated several laboratories across Germany to join forces to address questions concerning SARS‐CoV‐2 tropism and virulence using targeted infection of 3D‐human organ models (Adhikary et al, 2021). The formed consortium, Netzwerk Universitätsmedizin—NUM—Organo‐Strat, was funded by the Federal Ministry of Education and Research (BMBF) in Germany in 2020/21. 3D‐organ models are a promising new technology that provides unique opportunities to study human diseases and treatment designs. However, proving the reliability and robustness of these novel approaches will be crucial for their broad acceptance and successful application in translational biomedical research (Clevers, 2016; Kim et al, 2020). Common quality standards will smooth the way for organ model systems to gain broad acceptance in biomedical research as disease models and alternatives to certain animal models.

Common quality standards will smooth the way for organ model systems to gain broad acceptance in biomedical research as disease models and alternatives to certain animal models.

To ensure the generation of reliable and reproducible data in the consortium, a QM team was formed to safeguard quality standards for infection studies on organ models across the participating laboratories. This ad hoc QM team comprised three scientists with biomedical, GLP, biostatistics and bioinformatics research backgrounds. As the consortium combined groups from various biological disciplines—cell biologists, infectiologists, virologists, and pathologists—tailored quality measures had to be developed to accommodate individual laboratory workflows, experimental pipelines, and staff structures, while simultaneously guaranteeing a low implementation threshold. Additionally, the quality measures had to be fast and easy to implement due to the urgency of the COVID‐19‐related research performed in the consortium (London & Kimmelman, 2020).

The initial task of the QM team was to build communication structures and connect the scientists of the consortium. Here, online meeting platforms turned out to be very helpful as they enabled simultaneous interactive exchanges with multiple scientists in different locations and laboratories.

Each laboratory worked with a specific 3D‐organ model that presented individual challenges and limitations, such as the life span of the model, its culturing conditions, or different ways of infection. The QM team supported the generation of template protocols and defined standards for method‐specific materials, kits or devices—for instance, TaqMan probes for qPCR, primary antibodies for Western Blots or immunofluorescence staining—to make results comparable across laboratories. It required intense dialog to balance the need for standardization and scientific rigor against the urgency of the projects and specific workflows in each laboratory. The developed template protocols served as guidelines for conducting experiments with jointly agreed standards for each method. Experimenters stated all deviations from the template in their individual protocols.

As shown in Fig 1, various analyses were performed on the same organ model sample by different laboratories. These sample transfers were identified as essential steps prone to errors, as, for instance, virologists had little experience with human organ models and their delicate culturing conditions. Thus, permanent communication and rigorous experimental designs with appropriate controls were established with all collaborating groups. The organization of an uninterrupted workflow accompanied by the supply of related protocols for each set of experiments had to be assured by the QM team to avoid delays or impacts on sample quality.

Figure 1

Overview of the workflow in the Organo‐Strat consortium. Organ models for lung, liver, kidney, heart, brain, gastrointestinal tract, urogenital tract, skin, and adipose tissue were generated, and d0 analyses (d0 = uninfected samples) for the basic expression of SARS‐CoV‐2 entry factors performed (green) were performed. Organ models were transferred to biosafety level 3 laboratories for infection and analyses of viral replication (pink). Post‐infection analyses and additional measurements, assays, and readouts were conducted by specialized service laboratories (blue) (e.g., RNA sequencing; proteomics = AQUA—absolute protein quantification mass spectrometry; imaging: immunohistochemistry and in situ hybridization).

Just get started – but where?

Three quality tools were implemented as ad hoc QM measures for protocol generation/harmonization and for data and knowledge exchange: Sharepoint®—a web‐based collaborative platform by Microsoft; a self‐developed tool for progress monitoring and documentation, called SAMPLE TRACKER that is connected to a data management platform; and audits as a measure to overview and assess network activities (Fig 2).

Figure 2

Overview of the implemented platforms and tools: Sharepoint®, as an information and data exchange platform made protocols and additional documents accessible to all consortium members. The SAMPLE TRACKER as documentation platform, where each individual sample/batch of organ models was documented (sample ID) and connected with respective results and protocols to the SODAR data management platform. Audits as measure to track the progress and adherence to established workflows and quality standards.

Sharepoint® was the fastest solution to implement as information and data exchange platform and was useful in rapidly increasing the transparency in the consortium. To further extend the functionality of the infrastructure, future steps could involve the implementation of a MediaWiki platform (e.g., developed by PREMIER; Kurreck et al, 2020) and introduce an electronic lab notebook (ELN) in all groups (Table 1) (Dirnagl & Przesdzing, 2016). Unfortunately, pandemic time constraints and different access regulations at universities did not allow for the ad hoc introduction of such tools to laboratories.

Table 1

List of recommended tools to support the implementation of quality measures in a preclinical/academic research environment.

Tool	Advantages	Source
EQIPD quality system	A lean quality system of non‐regulated preclinical research at universities, companies, research institutes, providing a set of 18 core requirements and offering tools for their implementation	Bespalov et al, 2021 https://doi.org/10.7554/eLife.63294
PREMIER quality system	A modular quality system with low implementation threshold for academic biomedical research groups, departments, institutions, providing modules with minimum requirements that can be adapted to the needs, resources and stages of the projects	https://premier‐qms.org/
PREMIER Wiki	Platform to transparently share, store and further develop knowledge withing a department/laboratory/institute.	https://demo.premier‐qms.org
OMERO	OMERO is client‐server software for managing, visualizing and analysing microscopy images and associated metadata.	https://www.openmicroscopy.org/index.html
Audits	Audits have the potential to improve preclinical biomedical research in academia. This study listed specific recommendations regarding their benefits and provided practical resources for their implementation (e.g., study design and audit templates, audit workflow)	Kurreck et al, 2020 https://doi.org/10.1371/journal.pone.0240719

Whereas Sharepoint® can be used to upload small data files, large raw data sets and sensitive patient meta‐data should be handled by a data management group and more specialized repositories (Nieminen et al, 2020). For example, large image data sets should be deposited on server platforms such as OMERO (Open Microscopy Environment Remote Objects) that enable analysis and sharing in a centralized, protected, and standardized manner (Allan et al, 2012).

In order to monitor progress and standardize documentation, we created a coded Excel workbook (called “SAMPLE TRACKER”) that was easy to introduce to all the relevant staff in the laboratories and did not require a long implementation phase compared with other data management tools or LIMS (Laboratory Inventory Management System). The SAMPLE TRACKER allowed the tracking of samples during transportation and processing in the consortium (Fig 3). Each set of organ model samples received a unique sample ID that was included in every related protocol and raw data file. The unique sample IDs enabled systematic data organization and retrieval by the data management team on the SODAR (system for omics data access and retrieval) platform (Nieminen et al, 2020).

Figure 3

Example of requested information per experiment in SAMPLE TRACKER. Sample arrival: Samples were shipped to a laboratory for a specific experiment or sent out to another laboratory for additional testing afterward. Shipment: All sample transfers had to include a specification sheet. All experiments needed to follow a detailed protocol/work instruction, which had to be deposited on Sharepoint® and referred to in the SAMPLE TRACKER by providing the link to the protocol's location.

An online auditing process was implemented as an additional quality‐assuring measure. Audits reveal insights into projects and the adherence to quality standards on a regular basis (Kurreck et al, 2020). The initial round of audits between the QM team and each group aimed to clarify individual and group responsibilities, assess the status quo of the research project, and determine the level of implementation for predefined workflows and transparent documentation. Each audit was finalized with an audit report, including action plans and responsibilities, which was shared with all participants. The QM team stressed that audits in this academic research environment are not meant as a policing instrument but rather as a systematic communication tool to prevent operational blindness and offer an outside review of ongoing research.

Owing to the ongoing pandemic, it was not possible to conduct on‐site visits or peer audits by network partners in each other's laboratories. Peer audits are a special form of external validation and an efficient tool to solicit external feedback and foster professional exchange of ongoing projects. They are very effective in improving methods or processes (Kurreck et al, 2020).

For one year, the QM team laid the groundwork for implementing quality tools in the consortium. Although the implemented tools are surely expandable, none of them would have been even remotely effective without the key for any effective quality system: communication. Given the shortness of the funding period, the introduced methods were developed by scientists for scientists and had to be easily adaptable. The consortium members valued the presence of a QM team and, according to the final survey, profited mostly from the implemented communication structures as well as the access to standardized protocol templates. In fact, detailed experimental documentation is an important challenge for human organ model research to harmonize and standardize protocols and increase transparency and reproducibility.

Recommendations for academic consortia that plan to implement a Quality System

Based on our experience, here are some recommendations for academic consortia that plan to implement a Quality System. First, QM does not come without costs, so allocate sufficient and qualified personnel to this task. Ideally, plan and structure ahead, so that you can hit the ground running once the consortium starts its work. Our ad hoc approach was frequently lagging behind the requirements of the consortium. Second, the QM team should involve qualified people with a project‐focused expertise and a detailed understanding for quality‐assuring measures in the respective field of research. Depending on the size and duration of the consortium, personnel responsible for the QM does not necessarily need to be professional quality managers, but can also be well‐trained scientists willing to take over the responsibility.

Third, conduct a survey on current practices and make an inventory of already existing documentation solutions—for instance, electronic lab notebooks or protocol platforms. Find a compromise on tools that is feasible for all members of the consortium. Here, open‐source solutions can be cost‐effective but sometimes come with an additional administrative overhead. However, getting used to a new tool in addition to novel QM workflows may hinder rapid uptake by lab personnel who need a practical solution without a steep learning curve. Fourth, balance sustainability versus implementation speed. A 2‐year consortium can be more flexible and accommodate less sophisticated, but easily implementable tools than a permanent research infrastructure. Also, expect to fill gaps with custom‐made tools like the SAMPLE TRACKER as not all commercial tools will perfectly fit the purpose for each consortium.

Finally, closely communicate with the people who directly work at the bench to include them in the QM development process and signal appreciation of their work (Fig 4). The close interaction with the lab‐based scientists and staff might reveal challenges in the operational processes that are potentially difficult to identify for principal investigators. Overall, quality measures in an academic research environment have to evolve over time and need the continuous contribution and willingness from the scientists, as well as a patient but persistent QM team.

Figure 4

Steps to consider before and while implementing a Quality System in an academic research consortium.

Author contributions

Maren Hülsemann: Conceptualization; data curation; supervision; validation; investigation; visualization; methodology; writing – original draft; project administration; writing – review and editing. Janine Wiebach: Conceptualization; data curation; software; validation; methodology. Natascha Ingrid Drude: Conceptualization; methodology. Silke Kniffert: Methodology. Laura Behm: Project administration. Katja Hönzke: Validation; methodology. Morris Baumgardt: Validation; methodology. Stefan Hippenstiel: Funding acquisition; project administration; writing – review and editing. Andreas C Hocke: Conceptualization; funding acquisition; project administration; writing – review and editing. Ulrich Dirnagl: Funding acquisition; writing – review and editing. Ulf Tölch: Conceptualization; funding acquisition; writing – review and editing.

Disclosure statement and competing interests

ND is an external consultant and animal welfare officer at Medizinisches Kompetenzzentrum c/o HCx Consulting GmbH, Wendisch Rietz, Brandenburg, Germany.