Identifying research questions for HIV, tuberculosis, tuberculosis-HIV, malaria, and neglected tropical diseases through the World Health Organization guideline development process: a retrospective analysis, 2008-2018.

OBJECTIVES: World Health Organization (WHO) guidelines for health programmes and healthcare delivery are the foundation of its technical leadership in public health and essential to decision-making globally. A key function of guideline development is to identify areas in which further evidence is needed because filling these gaps will lead to future improvements in population health. The objective of this study was to examine the knowledge gaps and research questions for addressing those gaps generated through the WHO guideline development process, with the goal of informing future strategies for improving and strengthening the guideline development process. STUDY
DESIGN: We did a systematic, retrospective analysis of research questions identified in the published guidelines.
METHODS: We analyzed guidelines published between January 1, 2008, and December 31, 2018, by the Communicable Diseases Cluster in five disease areas: tuberculosis (TB), HIV, malaria, TB-HIV, and neglected tropical diseases (NTDs). Research questions were extracted independently by two researchers. We analyzed the distribution of research questions by disease and by topic category and did a qualitative assessment of optimum practice for research question generation during the guideline development process.
RESULTS: A total of 48 guidelines were included: 26 on HIV, 1 on malaria, 11 on TB, 5 on TB/HIV, and 5 on NTDs. Overall, 36 (75%) guidelines encompassed a total of 360 explicit research questions; the remainder did not contain specific research questions. The number of research questions that focused on TB was 49, TB/HIV was 38, HIV was 250, and NTDs was 23. The number of research questions that focused on diagnosis was 43 (11.9%) of 360, prevention was 62 (17.2%), treatment was 103 (28.6%), good practice was 12 (3.3%), service delivery was 86 (23.8%), and other areas was 54 (15%). Research questions were often not formulated in a specific or actionable way and were hard to identify in the guideline. Examples of good practice identified by the review team involved the generation of specific and narrowly defined research questions, with accompanying recommendations for appropriate study design.
CONCLUSIONS: The WHO must strengthen its approach to identifying and presenting research questions during the guideline development process. Ensuring access to research questions is a key next step in adding value to the guideline development process.

Introduction

One of the most important normative roles of the World Health Organization (WHO) is to develop guidelines for health programmes to support best practice in healthcare delivery. Producing robust guidelines is essential to inform decisions regarding diagnosis, management, and treatment, in support of evidence-based approaches to the prevention and control of diseases.1, 2, 3, 4 WHO guidelines aim to promote the achievement of the Sustainable Development Goals and access to universal health coverage and reflect the core WHO value of the ‘right to health.’,

The WHO and other national and international guideline development groups strive to ensure that their guidelines meet the highest international standards and are impactful at the country level. In 2007, the WHO Guidelines Review Committee (GRC) was established to oversee the processes and methods used to develop WHO guidelines and to ensure the quality of all published guidelines. The GRC re-established a set of guideline development standards and adopted the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach in formulating evidence-based recommendations. The guideline development process involves carrying out systematic reviews of the evidence for each of the key questions underpinning recommendations in a guideline, with assessment of the quality or certainty of the body of evidence, and the explicit and transparent formulation of recommendations based on the balance of benefits and harms of an intervention and other important considerations such as acceptability, resource use, and effects on equity. In addition, guideline development groups should formulate research questions needed to address identified gaps in knowledge.,

There has been significant improvement within the WHO in developing public health guidelines., However, there has been little emphasis on the opportunity provided by the guideline development process to identify, formulate, and compile relevant research questions that address knowledge gaps. This approach has been promoted for informing the development of a public health research agenda for the WHO., Since 2014, the WHO Handbook for Guideline Development has included the following advice: ‘When gaps in the evidence are such that significant uncertainty exists with respect to the balance of an intervention's benefits and harms, such knowledge gaps should be described and questions and methods for addressing the gaps should be suggested.’ Answering the research questions identified through the guideline development process fills knowledge gaps directly relevant to programmes and contributes to improved delivery of interventions and better health. Systematically compiling and disseminating the research questions identified through the WHO guideline development process can therefore help maximize public health relevance of future research.,,,

For a selected set of WHO guidelines, i.e., those developed by the WHO Communicable Diseases Cluster on tuberculosis (TB), HIV, malaria, TB-HIV, and neglected tropical diseases (NTDs) between 2008 and 2018, we therefore assessed the extent to which the guideline development process identifies research questions that address knowledge gaps. The objective of this study was to examine the research questions generated through the WHO guideline development process with the goal of informing future strategies for improving and collating these questions into an open-access online directory.

Methods

Inclusion and exclusion criteria

We did a systematic, retrospective analysis of research questions contained in all WHO guidelines approved by the GRC and published between January 1, 2008, and December 31, 2018, by the CDS at WHO headquarters in Geneva, Switzerland. This unit produces guidelines on TB, HIV, malaria, TB-HIV, and NTDs. A research question was defined as an answerable or actionable enquiry generated through the guideline development process describing an identified knowledge gap or where it was explicitly stated in the guideline to be a research question.

The GRC Secretariat provided a database containing all WHO guidelines published during the relevant time period. From this database, we identified guidelines related to the five disease areas of interest (TB, HIV, malaria, TB-HIV, and NTDs). The most recent guidelines were used when multiple guidelines were available on the same topic.

Data extraction

Research questions were extracted from the published guideline documents independently and in duplicate by J.H. and S.H. This involved a systematic search of the guideline for the following terms: research, research questions, research gaps, research needs, research priorities, knowledge gaps, outstanding research, quality of evidence, and implications of research. Research questions were extracted verbatim into an Excel file and assigned to the relevant disease area. Where research questions were present in paragraphs of text pertaining to research gaps or research questions, we disaggregated the text into separate research questions, wherever possible.

Analysis and validation

We categorized research questions into six broad areas: diagnosis, prevention, treatment, specific procedural/operational needs to establish good practice, service delivery, and ‘other.’ Once all data had been extracted from the guidelines and categorized, we analyzed the number of guidelines for each disease and the distribution of research questions by disease grouping and by topic area. We did a qualitative assessment of optimum approaches for defining actionable research questions, which involved two researchers doing an in-depth reading of all included guidelines to explore areas of good and bad practice in the generation of research questions and knowledge gaps during the guideline development process. The researchers took detailed notes during the process, which were discussed during a face-to-face review team meeting to agree on optimum approaches, to inform the guideline development process going forward.

Because we planned to use the identified research questions to populate an open-access online directory, identified research questions underwent an internal validation process by senior WHO technical staff with responsibility for each of the five disease areas under study to assess which research questions were still relevant to the current disease context. An Excel (Microsoft, Redmond, WA, USA) spreadsheet of research questions identified from the guidelines was sent via email to each of the staff members who then coordinated a discussion within their department to assess which research questions were still relevant. Irrelevant and outdated questions were removed.

Results

Distribution of guidelines and research questions by disease

A total of 48 guidelines were included in total (2008–2018), including 26 on HIV, 1 on malaria, 11 on TB, 5 on TB-HIV, and 5 on NTDs (see Fig. 1). Among the 48 guidelines reviewed, 30 (62.5%) were developed before the updated guidance on identifying research questions in 2014.

Fig. 1

Included guidelines by disease area. NTD = neglected tropical disease; TB = tuberculosis.

There was considerable heterogeneity across the guidelines in terms of research questions generated, with some disease areas showing a higher emphasis than others on generating a set of defined research questions as part of the guideline development process (Fig. 2). Of the 48 guidelines, 36 (75%) encompassed explicit research questions, including 360 research questions in total: HIV, 250 (69.4%); TB, 49 (13.6%); TB-HIV, 38 (10.6%); NTDs, 23 (6.4%), and malaria, 0 (Fig. 2). Only one guideline was identified for malaria, which did not explicitly state any research questions. Rarely did the guideline development groups propose an appropriate study design to accompany a defined research question.

Fig. 2

Research questions by disease area (%). NTD = neglected tropical disease; TB = tuberculosis.

Distribution of research questions by category

Of the 360 research questions, the focus was on diagnosis in 43 (11.9%), prevention in 62 (17.2%), treatment in 103 (28.6%), good practice in 12 (3.3%), service delivery in 86 (23.8), and ‘other’ in 54 (15%) questions.

There was variation in the emphasis of questions generated by research area across the disease categories. Among the 250 research questions on HIV, the most commonly reported were those on treatment (n = 82), followed by service delivery (n = 64). Among the 49 research questions on TB, those on treatment were also most frequently reported (n = 16). The main focus of the 38 TB/HIV research questions was on service delivery (n = 14), followed by prevention (n = 10). The main focus of the 23 research questions on NTDs was on prevention (n = 15).

Validation of research questions

Table 1 shows the number of validated research questions. The full data set of extracted research questions is available as Supplementary Information. The key reasons cited by Disease Leads as to why research questions were removed from the list of identified research questions include the following:

Table 1

Included guidelines and research questions after validation.

Disease area (number of guidelines and research questions)

HIV: 7 guidelines; 107 research questions

TB: 8 guidelines; 63 research questions

TB-HIV: 3 guidelines; 27 research questions

Malaria: 0

NTDs: 3 guidelines; 20 research questions

NTD = neglected tropical disease; TB = tuberculosis.

The guideline from which the research question was extracted is no longer valid.

Research questions were reframed and incorporated into a newer guideline.

The research question is now obsolete or no longer relevant.

The research question is not well formulated.

Qualitative assessment of optimum approaches

We found that research questions were commonly dispersed across the guideline in various sections, making it difficult for the reader to clearly see the research gaps generated by the guideline development process. Research questions were often not formulated in a specific or actionable way, with interventions not specified, study design not defined, and research questions too broad.

In many cases, guideline development groups did not specify explicit research questions or knowledge gaps but rather opted for paragraphs of interconnected text containing a broad discussion on research gaps, which makes it difficult for the reader to clearly identify the research questions. In guidelines published after the 2014 guidance—in which guideline development groups (GDGs) were specifically asked to address the issue of research question generation—we found that guideline development groups began to generate a defined section of ‘research questions,’ ‘research gaps,’ or ‘research priorities.’

We noted a number of good examples of research questions in the cohort of guidelines that we examined, with specific and narrowly defined questions, accompanied by recommendations regarding study design. Examples include the following:

“Large RCTs are needed to compare the effectiveness of topical amorolfine and butenafine in order to establish an alternative to oral treatments for toenail infections, in both HIV-infected and the general population.”

“Field evaluations of commercially available point-of-care technologies are needed to confirm the accuracy of results and the strategic placement of this technology within national programmes.”

The ‘Consolidated and Updated Guidelines on the Programmatic Management of Latent Tuberculosis Infection’ published in 2018 was highlighted by the review team as an example of good practice in research question generation. The guideline concludes with the research questions based on existing knowledge gaps, to support the improvement of quality of care (Table 2), with recommended study designs stated.

Table 2

Research questions extracted from a WHO guideline: an example of good practice in research question generation.

Evidence on the risks of a number of at-risk populations for progression from LTBI to active disease will be crucial for determining the potential benefits of LTBI treatment and for designing appropriate public health interventions. In particular, strong evidence from clinical trials is lacking for the following groups: patients with diabetes, people with harmful use of alcohol, tobacco smokers, underweight people, people exposed to silica, patients receiving steroid treatment, patients with rheumatological conditions, indigenous populations and cancer patients.

Evidence is required on differential harm and the acceptability of testing and treatment for LTBI in specific risk groups, including socially adverse events such as stigmatization.

Defining the best algorithm for ruling out active TB: Operational and clinical studies should be conducted to exclude active TB before preventive treatment is given. The performance and feasibility of the algorithms proposed in these guidelines should be assessed. In particular, few data are available on children and pregnant women. Strategies to save cost and improve feasibility (e.g. use of mobile chest radiography) should also be explored.

The performance of LTBI tests should be evaluated in various at-risk populations, such as the best way of using the available tools (e.g. combination or sequential use of TST and IGRA) in each at-risk population.

Research to find shorter, better-tolerated treatment regimens than those currently recommended is a priority.

Studies of efficacy and adverse events in certain risk groups (e.g. people who use drugs, people with alcohol use disorder and elderly people) are essential. In particular, there are no or very limited data on the use of rifapentine in children <2 years and pregnant women. Studies should be conducted of the pharmacokinetics of interactions between rifamycin-containing regimens and other drugs, particularly antiretroviral drugs.

The durability of protection by preventive treatment should be evaluated in settings in which TB is endemic, including the efficacy of repeated courses of preventive treatment.

Monitoring of adverse events: Prospective randomized studies are required to determine the incremental benefits of routine monitoring of liver enzyme levels over education and clinical observation alone for preventing severe clinical adverse events, with stratification of the evidence by at-risk population.

Risk of drug resistance following LTBI treatment: Programme-based surveillance systems and clinical studies are needed to monitor the risk for bacterial resistance to the drugs used in LTBI treatment. Particular consideration should be given to rifamycin-containing regimens because of the dearth of data.

Adherence to and completion of treatment: Carefully designed studies, including RCTs, are required to generate evidence on the effectiveness of context specific interventions for enhancing adherence and completion of treatment. The studies should include specific risk groups, depending on the available resources and the health system infrastructure. Use of “digital health” to improve adherence is an important area. Further research is required on the effectiveness of self administration of the 3-month regimen of weekly rifapentine plus isoniazid.

Although a number of studies of the cost-effectiveness of TB preventive treatment are available, their wide heterogeneity obviates a comprehensive appraisal of the cost-effectiveness of LTBI management stratified by population group and type of intervention. Direct measurement of cost-effectiveness in certain settings and populations would allow extension of the LTBI strategy at national or local level.

Preventive treatment for contacts of people with MDR-TB: RCTs with adequate power are urgently needed to update the recommendation on preventive treatment for contacts of people with MDR-TB. Trials should be performed with both adult and paediatric populations and with at-risk populations such as people living with HIV. The composition, dosage and duration of preventive treatment regimens for MDR-TB should be optimized, and the potential role of newer drugs with good sterilization properties should be investigated. The effectiveness and safety of preventive treatment for contacts of people with MDR-TB should be evaluated in operational conditions. Further evidence on the risk of contacts of people with MDR-TB for progression to active TB will be important for understanding the benefits of preventive treatment.

Epidemiological research should be conducted to determine the burden of LTBI in various geographical settings and risk groups and as a basis for nationally and locally tailored interventions, including integrated community based approaches. Research is also needed on service delivery models to ensure that patients are properly managed including the provision of additional interventions for tobacco smokers, illicit drug users, and people with harmful use of alcohol. Household implementation models could improve the effectiveness and efficiency of delivery of interventions. Tools should be developed and assessed to facilitate monitoring and evaluation of programmatic management of LTBI.

WHO = World Health Organization; LTBI = latent tuberculosis infection; TB = tuberculosis; IGRA = interferon gamma release assay; TB = tuberculosis; TST = tuberculin skin tests; RCT = randomised controlled trial; MDR-TB= multidrug-resistant TB.

Discussion

The cohort of guidelines on infectious diseases that we assessed varied considerably in the extent to which they identified research questions as part of the guideline development process. Of the included guidelines, 75% contained explicit research questions, most frequently focusing on disease treatment. A relevant study design accompanying the research questions was rarely proposed. The better examples involved the generation of specific and narrowly defined research questions, in its own defined section of the guideline that is easily accessible to the reader, with accompanying recommendations for appropriate study design.

This analysis provides evidence of the lack of a systematic approach in identifying research questions during the guideline development process, which is relevant to the WHO's guideline development groups and other organizations generating guidelines in the field of public health. Explicit guidance on how to identify knowledge gaps and actionable research questions and to present them in WHO guidelines would add value to each guideline and to the setting of evidence-informed public health research agendas. This guidance could build on existing work on the generation of research agendas through systematic reviews.

Guidance is needed on when in the guideline process, developers should start thinking about research questions and how reseach question formulation can be better integrated into the guideline development process. Consideration must be given to what expertise is needed to identify and formulate optimal questions and to the approaches that may be useful for subsequent prioritization among these research questions.

There were limitations identified with respect to this review. Primarily, the review team may have missed regional guidelines or research questions within these guidelines. However working directly with disease leads for each disease means that this would have been unlikely. We are not aware of any other organizations involved in guideline development that have analyzed and assessed their approach to research question generation through the guideline development process. Nor were we able to identify any published or gray literature from other organizations on how to generate research questions. Organizations such as the Guidelines International Network (https://www.g-i-n.net/) may be well placed to strengthen approaches in generating research questions and highlighting evidence gaps during guideline development.

This review has generated some key new considerations to inform the standardized and systematic identification and compilation of research questions for guideline development in the future, which may be relevant to other health guideline development groups. There should be sufficient expertise in research among members of the guideline group to help generate research questions. In addition, research questions are more easily accessible to guideline end users if they are short and clearly defined and in a defined section of the published guideline.

Opportunities for the WHO to ensure the research questions identified through its guideline development process are made more widely available, including the compilation of an online directory of research questions hosted by the WHO Global Observatory on Health R&D and presentation on the WHO website where guidelines are published. Work is currently underway to disseminate research questions using these fora.

Conclusions

This analysis shows the variable extent to which the WHO guideline development process identifies research questions. The results indicate the need for the WHO to strengthen its guideline development process by systematically identifying and compiling research questions that address key knowledge gaps. Such an approach will facilitate the formulation of relevant and impactful research agendas that will ultimately help to improve health programmes and achieve the Sustainable Development Goals for health.

Author statements

Ethical approval

None sought.

Funding

The study was funded by the Special Programme for Research and Training in Tropical Diseases (TDR).

Competing interests

S.L.N. was the Secretary of the World Health Organization (WHO) Guidelines Review Committee, until 2020 and in that position, she was responsible for supporting and overseeing the methods used and standards implemented for WHO guidelines, including for many of the guidelines included in the study cohort. She was the lead author of WHO Handbook for Guideline Development (2nd edition, 2014). All other authors report no competing interests.

Availability of data sets

The data set supporting the conclusions of this article is included as an additional supplementary file.

Author contributions

D.M., N.F., and S.L.N. conceived the idea for this study. S.H. led the investigation and data analysis, with input from L.B.N. and J.H. G.B., A.F.G., N.G., and M.Z. validated the data extraction. S.H. and D.M. drafted the manuscript, and all authors provided input and approved the final manuscript.