Using the NIH Research, Condition and Disease Categorization Database for research advocacy: Schizophrenia research at NIMH as an example.

In 2008 the National Institutes of Health established the Research, Condition and Disease Categorization Database (RCDC) that reports the amount spent by NIH institutes for each disease. Its goal is to allow the public "to know how the NIH spends their tax dollars," but it has been little used. The RCDC for 2018 was used to assess 428 schizophrenia-related research projects funded by the National Institute of Mental Health. Three senior psychiatrists independently rated each on its likelihood ("likely", "possible", "very unlikely") of improving the symptoms and/or quality of life for individuals with schizophrenia within 20 years. At least one reviewer rated 386 (90%), and all three reviewers rated 302 (71%), of the research projects as very unlikely to provide clinical improvement within 20 years. Reviewer agreement for the "very unlikely" category was good; for the "possible" category was intermediate; and for the "likely" category was poor. At least one reviewer rated 30 (7%) of the research projects as likely to provide clinical improvement within 20 years. The cost of the 30 projects was 5.5% of the total NIMH schizophrenia-related portfolio or 0.6% of the total NIMH budget. Study results confirm previous 2016 criticisms that the NIMH schizophrenia-related research portfolio disproportionately underfunds clinical research that might help people currently affected. Although the results are preliminary, since the RCDC database has not previously been used in this manner and because of the subjective nature of the assessment, the database would appear to be a useful tool for disease advocates who wish to ascertain how NIH spends its public funds.

Introduction

The National Institutes of Health is the largest source of funding for medical research in the world. As such, it is a major source of hope for individuals afflicted with specific diseases and their associated advocacy groups. In 1998, the Institute of Medicine (IOM) issued a report entitled “Scientific Opportunities and Public Needs” noting that there were problems regarding the public’s perception of how research funds were allocated at NIH [1]. Specifically, it said that some believed “that NIH cares more about curiosity than cure, more about fundamental science than clinical application”. The IOM report highlighted the need for public input into NIH using “a formal mechanism through which the public can inform the priority setting”. One of the recommendations of the report was that “NIH should improve the quality and analysis of its data on funding by disease”. Similar discussions subsequently took place among members of Congress, and in 2006, as part of the NIH Reauthorization Act, Congress mandated that “the Director of NIH shall establish an electronic system to uniformly code research grants and activities…The electronic system shall be searchable by a variety of codes, such as the type of research grant, the research entity managing the grant, and the public health area of interest” [2].

The result of this congressional mandate was the NIH Research, Condition, and Disease Categorization (RCDC) system, a publicly available online computerized database, updated annually, that reports the amount being spent by each NIH institute for each disease category. The RCDC includes extramural grants and contracts as well as intramural research projects for all 27 NIH institutes and centers. The database includes the name and institution of the principal investigator; the project’s funding history; an abstract summarizing the research; and a statement regarding the project`s public health relevance. The explicit intent of Congress was to make such information available to the public. As stated on the RCDC website: “The American people want to know how the NIH spends their tax dollars. The RCDC process categorizes the NIH research projects funded with those tax dollars” [3].

The RCDC database is thus a potentially rich research resource that could be useful for disease advocacy groups. Surprisingly, the database appears to be little known or used. A Medline search of articles related to the RCDC identified only two articles, on statistics [4] and disease burden [5]. Another article used the RCDC database without naming it to ascertain NIH research expenditures for cystic fibrosis and sickle cell disease [6]. The present report illustrates how the RCDC can be used by advocates to evaluate NIH research for particular diseases. We used schizophrenia research funded by the NIMH to illustrate how this information can inform the public about funding priorities.

Schizophrenia is one of the nation’s most important diseases. Its one-year prevalence in the United States, as reported by the National Institute of Mental Health (NIMH) to Congress in 1993, is 1.5% among adults 18 and over and 1.2% among children 9 to 17 [7]. These numbers were based on the Epidemiologic Catchment Area study, the last in depth prevalence study carried out in the U.S. Based upon the estimated 2018 population of 327.2 million, this translates into 4.25 million Americans affected by this disease. Schizophrenia is also a major contributor to the problems of homelessness and the overcrowding of jails and prisons. The most recent estimate of the annual cost of schizophrenia in the United States is $155.7 billion [8].

In recent years, concerns have been raised regarding the relevance of schizophrenia research carried out by NIMH. An editorial published in the British Journal of Psychiatry in 2016, authored by 20 current or former members of the NIMH National Advisory Mental Health Council, including one of the present authors (Dr. Rush), noted that NIMH research had become increasingly focused on basic research, especially genetics and neural circuits, instead of more clinical research that might help people currently afflicted. They therefore called “for an increase in public discussion of how to appropriate funding resources across mental health research domains” [9].

Others have expressed similar concerns. Dr. Steven Hyman, a former NIMH Director (1996–2001), recently claimed that “no new drug targets or therapeutic mechanisms of real significance have been developed for more than four decades” [10]. Dr. Thomas Insel, another former NIMH Director (2002 to 2015), observed that despite spending $20 billion “on the neuroscience and genetics of mental disorders… I don’t think we moved the needle in reducing suicide, reducing hospitalization, [or] improving recovery for the tens of millions of people who have mental illness” [11].

Two advocacy groups in the United States focus specifically on serious mental illness, especially schizophrenia: the Treatment Advocacy Center in Arlington, Virginia (www.treatmentadvocacycenter.org) and Mental Illness Policy Org in New York City (www.mentalillnesspolicy.org). Other advocacy groups such as The National Alliance on Mental Illness (NAMI) and Mental Health America focus on mental disorders more broadly but do not specifically target schizophrenia. Based on the reports of NIMH’s apparent failure to carry out sufficient clinical research (9–11), the authors undertook discussions with the two schizophrenia-focused advocacy groups regarding how to assess the NIMH research portfolio.

Methods

The following research was not based on a formal protocol but rather evolved from the discussions cited above. One of the authors (reviewer one) became aware of the RCDC database and proceeded to do a rating of the projects to determine the feasibility of such research. Reviewer one then recruited reviewers two and three to independently do a similar rating, masked to the results of the other two assessors. All three reviewers had an association with one or both of the advocacy groups and shared a belief that NIMH should support a balanced portfolio between basic and clinical research, acknowledging that both were necessary. Each of the reviewers had had extensive clinical experience with patients with schizophrenia as well as extensive research experience, including having been funded by NIMH for research on schizophrenia and/or bipolar disorder. Dr. Rush served as both a member or Chair of three different research study sections at NIMH, as a member of the NIMH National Advisory Mental Health Council and as coauthor of the 2016 consensus statement referenced above [9].

Each reviewer was asked to estimate the relative likelihood (likely, possible, very unlikely) that the completion of the research project would improve the symptoms and/or quality of life for persons with schizophrenia during the next twenty years. The terms “likely”, “possible”, and “very unlikely” were not formally defined. Instead the reviewers relied on general medical knowledge regarding how long it has taken basic brain research in such areas as genetics and neural circuits to be translated into clinically useful knowledge. The twenty-year period was chosen based on the fact that the median age of all Americans is 38 and individuals with schizophrenia have a 15–25 year shortened life expectancy [12]. It was assumed that individuals affected with schizophrenia should have a reasonable chance of profiting from ongoing NIMH research during their remaining lifetime.

The public RCDC database was accessed for 2018, the most recent year for which final data was available at the time of the study. See accompanying box for details on accessing this system. In the RCDC system, disease relevance is ascertained by a computerized analysis of all funded proposals.

How to access the NIH schizophrenia–related research grants:

Go to: https://report.nih.gov/categorical_spending.aspx.

Scroll down the table until you find schizophrenia.

Go to the 2018 column which has $248 million;

Click on the $248 million. This will give you a listing of all 542 NIH schizophrenia–related grants for 2018.

For details of any given grant click on the grant number. This will provide you with the project information, including an abstract, a statement on public health relevance, and the grant’s funding history.

For 2018 the RCDC database listed 542 new and ongoing NIH-funded projects as being schizophrenia-related, which included 428 funded by NIMH and 114 funded by other NIH institutes. Of the latter, the largest number (n = 26) were basic brain studies funded by the National Institute of Neurological Disorders and Stroke (NINDS). Other examples of such grants and contracts included studies of smoking cessation in schizophrenia funded by the National Institute on Drug Abuse and the development of new Positron Emission Tomography (PET) ligands funded by the National Institute of Biomedical Imaging and Bioengineering.

The results of the independent ratings were summarized qualitatively and were also tested for inter-rater reliability using SAS software, version 9.4, to calculate the intraclass correlation coefficient for Fliess kappa. In reporting these results the Standard Reporting Qualitative Research (SRQR) guidelines were also used [13].

Results

The NIMH schizophrenia-related research portfolio for 2018 consisted of 428 projects costing a total of $201 million, which represented 11.5% of NIMH’s total budget of $1.755 billion for 2018. It included 418 extramural grants and contracts costing $176 million or an average of $421,000 per project per year. It also included $25 million for schizophrenia-related intramural projects which included eight separate laboratory projects, $3.9 million for the Office of the Intramural Scientific Director and $10.2 million for “NIMH space activation, maintenance and improvement”. The information available in the abstracts on the RCDC database was sufficient for assessing most research projects but less so for the intramural projects or for research centers when multiple projects were summarized in a single abstract.

The average duration of funding for the 428 NIMH research projects in 2018 was four years (range 1–31 years). Thirty-eight extramural projects had been funded for 10 years or more and 11 of these for 20 years or more. The longest funded projects were two training grants funded respectively for 31 and 28 years along with a genetics study and a study of sensory processing, each funded for 27 years. Among the intramural projects a study of neuroimaging had been funded for 26 years.

Geographically, the 418 extramural awards went to 107 institutions. There was a significant clustering of awards with seven institutions receiving a total of 120 awards. The University of Pittsburgh received the most awards with 30, followed by Johns Hopkins University with 20. Columbia University, the University of Maryland, Icahn School of Medicine at Mount Sinai, and UCLA each received 15 or more. Only five awards were made to non-American principal investigators—four Canadians and one Swedish investigator—although a few others had foreign components administered by an American principal investigator.

Table 1 and Fig 1 summarize the assessments of the three individual reviewers regarding the likelihood of the research projects improving the symptoms and/or quality of life for persons with schizophrenia during the next 20 years. At least one reviewer rated 90% (386/428) of the research projects as being very unlikely to improve the symptoms and/or quality of life for persons with schizophrenia during the next 20 years, and all three reviewers agreed on this 71% (302/428) of the time.

Table 1

Likelihood of research projects (n = 428) improving the symptoms and/or quality of life for persons with schizophrenia within 20 years.

	Very unlikely	Possible	Likely
Reviewer 1	337 (79%)	80 (19%)	11 (2%)
Reviewer 2	341 (80%)	67(16%)	20(4%)
Reviewer 3	366 (86%)	56(13%)	6(1%)

Fig 1

Venn diagrams to indicate relationship amongst reviews’ ratings of the research projects.

Similarly, at least one reviewer rated 29% (126/428) of the research projects as possibly able to improve the symptoms and/or quality of life for persons with schizophrenia during the next 20 years. When the categories of “possible” and “likely” are combined, all three reviewers unanimously agreed that only ten percent (43/428) of the research projects held such promise. Overall, the intraclass correlation coefficient revealed a moderate level of agreement among the three raters (Fliess kappa = 0.473). Agreement was highest for those rated “very unlikely” (0.570); intermediate for those rated as “possible” (0.427); and poor for those rated as “likely” (0.165). This is also reflected in Fig 1.

Taken together, the individual reviewers selected a total of 30 projects (30/428) or 7% as being likely to improve the symptoms and/or quality of life for persons with schizophrenia within the next twenty years. Seven of these thirty were selected by two of the reviewers but none was selected by all three. For twenty-one of the thirty, all reviewers agreed that the project was either possible or likely. Examples of projects selected as likely included attempts to improve social function, medication adherence, symptoms such as auditory hallucinations, and clozapine usage; Table 2 lists all thirty research projects. The total cost of the thirty projects selected by one or more reviewer as likely to improve the symptoms and/or quality of life for persons with schizophrenia was $11,120,544; this was 5.5% of the schizophrenia-related research portfolio or 0.6%of the total NIMH budget for 2018. Note also that the average cost of a research project rated as likely was $371,000; this was significantly less than the $421,000 average cost for all extramural schizophrenia-related projects.

Table 2

The 30 research projects selected by at least one of the reviewers as likely to improve the symptoms and/or quality of life for persons with schizophrenia within 20 years.

CSC OnDemand: An Innovative Online Learning Platform for Implementing Coordinated Specialty Care	4R44MH111283-02
A transdiagnostic sleep and circadian treatment to improve community SMI outcomes	5R01MH105513-04
Testing Effectiveness of a Peer Led intervention to Enhance Community Integration	5R01MH102230-04
Imaging Neuroinflammation in Clinical high risk and Schizophrenia	5R01MH100043-05
Enhancing Social Functioning in Schizophrenia through Scalable Mobile Technology	5R21MH111501-02
Targeting Stress Reactivity in Schizophrenia: Integrated Coping Awareness Therapy	5R33MH100250-05
Dimensional outcomes and neural circuitry associated with psychosis risk	1R01MH112584-01A1
Creating Live Interactions to Mitigate Barriers (CLIMB): A Mobile Intervention to Improve Social Functioning in People With Schizophrenia	1R43MH114765-01A1
Longitudinal Mediation Analysis to Identify Effective Intervention Components in Clustered Trial of RAISE-ETP (Recovery after an Initial Schizophrenia Episode-Early Treatment Program)	5R03MH112053-02
Effectiveness of a Mobile Texting Intervention for People with Serious Mental Illness	5R56MH109554-02
Using mHealth to optimize pharmacotherapy regimens	1P50MH115843-01
Administrative Core for the following three research projects:	1P50MH115842-01
Adapting an evidenced-based weight management intervention and testing strategies to increase implementation in community mental health programs	1P50MH115842-01
Promoting evidenced-based tobacco smoking cessation treatment in community mental health clinics	1P50MH115842-01
Using an innovative quality improvement process to increase delivery of evidenced-based CVD risk factor care in community mental health organizations	1P50MH115842-01
Levetiracetamin in First Episode Psychosis	5R61MH112833-02
Texting for Relapse Prevention: Improving outcomes for people with schizophrenia	5R34MH108781-03
A Trial of "Opening Doors to Recovery" for Persons with Serious Mental Illnesses	5R01MH101307-06
Peer Support and Mobile Technology Targeting Cardiometabolic Risk Reduction in Young Adults with SMI	5R01MH110965-03
Real-time fMRI Neurofeedback as a Tool to Mitigate Auditory Hallucinations in Patients with Schizophrenia	1R61MH113751-01A1
Neural Biomarkers of Clozapine Response	5K23MH110661-04
Using Medicaid data to advance care for people with schizophrenia at risk for HIV (Medicaid-DASH)	5R01MH112420-02
Biomarker and Safety Study of Clozapine in Benign Ethnic Neutropenia	5R01MH102215-04
Trajectories of treatment response as window into the heterogeneity of psychosis: a longitudinal multimodal imaging study in medication-naieve first episode psychosis patients	1R01MH113800-01A1
Trial of Integrated Smoking Cessation, Exercise, and Weight Management in SMI	5R01MH104553-05
Targeting Auditory Hallucinations with Alternating Current Stimulation	5R33MH105574-04
ASSESSMENT OF MEDHERENT MEDICATION MANAGEMENT DEVICE AND ADHERENCE PLATFORM	1R44MH116765-01
Early Stage Identification and Engagement to Reduce Duration of Untreated Psychosis (EaSIE)	5R34MH115463-02
Molecular pathways of the kynurenine system in the neuroimmunology and psychophysiology of schizophrenia.	1R21MH117512-01
CRCNS: Collaboration toward an experimentally validated multiscale model of rTMS	1R01MH118930-01

Discussion

In keeping with the intent of Congress, the RCDC public database was used to assess the schizophrenia-related research projects funded by NIMH in 2018. Three experienced reviewers independently but unanimously agreed that 71% of the research projects were very unlikely to improve the symptoms and/or quality of life for individuals with schizophrenia over the next 20 years. The reviewers also unanimously agreed that only 10% of the research projects held any possibility for such improvement. One or more reviewers selected 30 out of the 428 total research projects as actually likely to lead to such improvement but no project was so rated by all three reviewers. The results of the study would appear to confirm the opinion of 20 current or former members of the National Advisory Mental Health Council that the NIMH research portfolio is disproportionately focused on basic research instead of clinical research that might help people currently affected [9].

The information generated by this study will be useful to the schizophrenia-related advocacy groups that co-authored this study as well as to other advocacy groups and individuals with an interest in this research. The advocacy groups will encourage their followers to contact NIMH and their representatives in Congress. From past experience we have found that advocates for individuals with schizophrenia and other serious mental illnesses feel very strongly about the paucity of research on these diseases. When provided with specific information, they will often contact their congressional representatives as well as writing letters to newspapers. A follow up study using the RCDC database looking at NIMH’s funding over time is already underway. The present study also demonstrates the feasibility of using the RCDC database to generate information of value to other disease advocacy groups. The authors plan to send the current study to the major disease advocacy groups in the United States to make them aware of this resource and its potential for influencing future research funding decisions by NIH. This is consistent with the intent of Congress in creating this database.

In assessing the NIMH schizophrenia-associated research portfolio, the reviewers were also impressed by the research areas being ignored. Foremost among these were attempts to find better treatments for individuals with schizophrenia. Among the 428 research projects there was just one treatment trial using a pharmacological agent, the anticonvulsant levetiracetamin, to treat individuals with schizophrenia. Another trial used amphetamine to enhance the effects of cognitive therapy and a third trial used a gluten-free diet. Two additional treatment trials used forms of transcranial stimulation in attempts to improve the symptoms of schizophrenia. These five trials together, representing 1.2% of the 428 research projects, were the only such trials.

We contend that NIMH should invest much greater resources to find better treatments for schizophrenia, including testing off label and potentially repurposed medications and other compounds for which there is no intellectual property protection, including prebiotics and probiotics. A major program should also be undertaken to find medications effective against schizophrenia’s negative symptoms. Cost-benefit studies should be undertaken for clozapine, the most effective anti-psychotic but markedly underutilized in the U.S. Head to head efficacy studies for various long-acting injectable antipsychotic would also be very useful.

Associated with the paucity of treatment trials for schizophrenia in the NIMH research portfolio was the absence of clinically useful studies on the optimal use of antipsychotic drugs. Twenty years ago, NIMH funded the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE), which provided clinicians with useful information regarding the selection of antipsychotic medication. It was arguably the most important schizophrenia research funded by NIMH in the last two decades. Nothing similar has been subsequently funded by NIMH despite many important and outstanding questions regarding which antipsychotic or combination is likely to be effective for specific subgroups of patients. Pharmaceutical companies are reluctant to fund head-to-head antipsychotic studies so unless NIMH does so such studies will not be done. This situation is similar to the need for head-to-head drug trials for such disorders as hypertension and rheumatoid arthritis.

NIMH should also invest resources in improving the lives of individuals with schizophrenia who are homeless or incarcerated. For example, it should study the efficacy of assisted outpatient treatment (AOT) in reducing homelessness as suggested in preliminary studies. For individuals with schizophrenia who are incarcerated for major crimes, NIMH should study the relative effectiveness of conditional release, Forensic Assertive Community Treatment (FACT) teams, and Psychiatric Security Review Boards in reducing recidivism.

Another research area conspicuously absent from the 428 NIMH research studies was the epidemiology of schizophrenia, its treatment or treatment outcomes; there was not a single epidemiological study. This stands in marked contrast to Europe where epidemiological studies are generating useful hypotheses regarding schizophrenia`s etiology, including marked differences in the incidence of schizophrenia in different parts of Europe [14] and in the incidence of schizophrenia among various immigrant groups [15,16]. The last major epidemiological study of schizophrenia funded by NIMH was the 1980s Epidemiologic Catchment Area (ECA) study; one finding from this study was a very low lifetime prevalence of schizophrenia among Mexican Americans in Los Angeles, a finding that was never followed up [17].

In summary, the main strengths of this project are the use of a novel NIH database and its demonstration as a disease advocacy tool; the use of experienced psychiatric reviewers; and the collaboration between the reviewers and the advocacy groups. The study’s main limitation is the subjective nature of the review criteria.

Recommendations

The 1998 Institute of Medicine report on NIH research suggested the need for “a formal mechanism through which the public can inform the priority setting” [1]. The RCDC database provides this mechanism. Accordingly, the authors suggest the following.

Publicize the availability and encourage the use of the RCDC database for research advocacy groups for all medical and psychiatric disorders.

At least 50% of NIMH schizophrenia-related projects should be classifiable by advocacy groups as possible or likely to improve the symptoms and/or quality of life for individuals with this disease in less than 20 years.

Put a much greater funding priority on developing better neurobiological or psychological treatments for schizophrenia and/or enhance their delivery in clinical care settings. Include especially the many off-label medications and plant-derived compounds that may be useful in treating schizophrenia and for which there is insufficient intellectual property protection for these compounds to be tested by industry.

Request that the National Academy of Medicine convene a review group to examine the NIMH intramural research program and assess both its mission and its cost effectiveness. In 2018, according to the RCDC database, it included eight schizophrenia-related research projects at a cost of $25 million.

Incentivize public and private community healthcare systems to conduct research to address the clinical needs of patients and providers in order to bring basic/translational research results into practice more rapidly.

In conclusion, the NIMH schizophrenia research portfolio is disproportionately weighted toward basic research that promises better treatments in the distant future. One is reminded of the advice given in similar circumstances in 1971 by a leading cancer researcher: “We cannot afford to sit and wait for the promise of tomorrow so long as stepwise progress can be made with the tools at hand today” [18].

NIMH schizophrenia related research projects for 2018: 1 = likely; 2 = possible; 3 = unlikely.

(XLSX)

Click here for additional data file.

Reporting checklist for qualitative study.

(DOCX)

Click here for additional data file.

15 Jul 2020

PONE-D-20-13102

Using the NIH Research, Condition and Disease Categorization Database for research advocacy: Schizophrenia research at NIMH as an example

PLOS ONE

Dear Dr. Torrey,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

First, I would like to thank the two authors for their important comment. Both suggest that the papers has merits and I agree.

I have however some additional comments that need to be addressed :

- First : the manuscript must mention explicitly :

. If there was a protocol YES / NO ;

. If it was registered a priori (e.g. on the Open Science Framework) ;

. Data must be submitted as a supplementary material ;

. The paper should be reported using the adequate reporting guideline (and a specific grid should be filled and provided): the method section must be expanded to allow for reproduction of the study ;

. A limitation of the study is the subjectivity in the assessment and the fact that the outcome used in this study had no prior validation : the exploratory nature must be explicit in both the limitation section and the abstract to avoid any spin and to avoid any over-interpretation of the results ;

. I suggest drawing a figure using Venn Diagram (or an alluvial plot) to show the agreement among the 3 reviewers. I acknowledge that it can be challenging to draw such a figure, but this is all the more important ;

. I also suggest that the full list of proposals listed as likely is dysplayed in a table in the main manuscript ;

. The same in a web appendix for studies listed as possible and for those listed as unlikely ;

Please submit your revised manuscript by Aug 28 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have a patent relating to material pertinent to this article.

a. Please provide an amended statement of Competing Interests to declare this patent (with details including name and number), along with any other relevant declarations relating to employment, consultancy, patents, products in development or modified products etc.

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” If there are restrictions on sharing of data and/or materials, please state these.

Please note that we cannot proceed with consideration of your article until this information has been declared.

b. This information should be included in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This is an interesting paper on an important topic for advocacy organisations and consumer participants. I would like to have seen more emphasis on the public perspective in the paper and how advocacy groups had informed the design of the study, how they could influence the development and use of the RCDC given the conclusions of the authors. It was not clear whether this review was attempting to demonstrate that the database was not functional, or the research recorded in it was not of sufficient value. The paper appears to be arguing two perspectives, firstly that the RCDC could be a useful tool and secondly that the schizophrenia research within it is not providing value for money and is not of sufficient efficacy to lead to improvements in patient outcomes. At present the focus of the paper seems to be a little more on the authors area of interest i.e. the relevance of the schizophrenia research funded by the NIH, than the stated focus of the study which is the use of the database for research advocacy. It would be useful if the authors could state their intentions more clearly. They need to be clearer about the messaging and intent of the study. The key message, which is implied at the beginning and the end of the paper, seems to be that the public needs to inform research priorities and that the RCDC could be a valuable tool in facilitating this.

The intended purpose of creating the RCDC database was not entirely clear from the paper. It appears to be simply to provide a source of information for the public. Was it also the intention to engage the public and/or advocacy groups in the prioritisation of research topics? If the primary intention of the RCDC database was to enable public access, and presumably increase the degree of transparency of NIH funding decisions, it would be interesting to know if the authors considered including advocacy organisations as reviewers in the study or if there is any intention for them to carry out a similar review following this study to evaluate the value of the database in that context. The paper refers to how the RCDC might be a rich research resource for advocacy groups but does not explain how those groups might use it. For example, would or can they use it to identify unanswered research questions (in a similar way to the James Lind Alliance), to identify ongoing or completed research, or to identify potential partners for research? How does this then inform the public about funded research?

The outline of the methodology would benefit from greater detail. The database was accessed for data for 2018; however, it is not clear if the study was reviewing projects approved and funded in 2018 or all the projects that were ongoing in 2018 but may have started prior to that date. This is important information in the context of the discussion in the results section relating to the clustering of awards and whether there may have been an historical shift in the location of awards or changes in the types of projects funded. It would also be helpful to understand what criteria the three assessors used for their ratings of ‘likely’, ‘possible’ or ‘very unlikely’. For example:

- Was it based on their individual experience and expertise, knowledge of clinical decision making or did they develop a shared assessment framework?

- How was the decision to have three assessors arrived at? Did this give the study sufficient power?

- How did the reviewers ensure any potential bias towards the topic areas was excluded?

- The reviews seem to be based on an assessment of project abstracts, were they sufficiently detailed to make an assessment of impact over a period of 20years?

- Why was the period of 20 years selected as he optimum period for the demonstration of clinical improvement?

Specific comments relating to the content of the article:

- The term ‘allegations’ in the abstract seems inappropriate and unnecessarily adversarial. It could be expressed as ‘criticisms’. The authors also need to substantiate their statement to identify which stakeholder(s) made the ‘allegations’.

- The final sentence of the abstract needs to be qualified as it is a very broad statement which reads as the personal opinion of the authors rather than something substantiated by their study. Did the study show that ‘the RCDC database is an excellent tool’ and should be used more frequently, or is this the authors view?

- The statement on page 4 that refers to the database being little known or used, cites academic articles that reference the RCDC. Presumably the articles do not report data about the number of public individuals who search the database or use the information from it as a lobbying tool for their disease area? What is the level of public access to the database? Advocacy groups, who are users of research, are less likely to publish academic articles or lead research therefore their use of the database may not be referenced in past papers.

- Page 4 refers to the prevalence of schizophrenia in the US; how does this compare with the prevalence of other disease areas and the cost to the economy?

- Some abbreviations and acronyms are not explained e.g. NAMI, PET.

Without a more detailed description and rational for the review process, the paper is weakened. It is important to have this detail if the study is to be replicated in other disease areas as suggested by the authors. Occasionally the paper seems to reflect the personal perspective of the authors in relation to the types of research that receive funding. To provide a more rounded perspective, it would be useful to balance this with the types of research clinicians report as important and the research topics of importance to advocacy groups and patients. It is not clear how, or if, patients inform priority setting although this was a stated intention of the I of M report on NIH research. I would like to see more emphasis on the public perspective, and how advocacy groups can inform the development and use of the RCDC, within the paper.

Reviewer #2: This article reports on the percentage and associated expenditures of the research portfolio funded by the USA’s National Institute of Mental Health in the area of schizophrenia and provides an assessment of the extent of the portfolio that is likely to provide clinical improvement to individuals with the disorder over the next 20 years. The article is very well written, balanced, and timely. It has numerous strengths, including a succinct but informative review of the push by advocacy organizations for public accountability in Congressionally funded mental health research, the public-health significance of schizophrenia, and the NIMH research portfolio for 2018. Other strengths include the use of a multi-rater system for coding the likelihood of clinical impact, estimation of an intra-class correlation coefficient among raters, discussion of the areas of research being ignored, and a cogent set of recommendations. The ms. also has a few minor weaknesses, which would strengthen the article if addressed:

• P.4, lines 77-78: Some readers may not understand why the ms. goes back to 1993 for prevalence data on schizophrenia in the USA. Aren’t more recent data available, they may ask? One advantage of these data is that they were provided by the NIMH itself to Congress, thereby making them the “official” rates of the organization. In addition, as the ms. points out, more recent community-based epidemiological data on schizophrenia in the USA are limited, due in part to methodological difficulties and lack of funding interest. The ms. might benefit from a sentence that clarifies why 27-year-old data are being used to report the national prevalence of the disorder.

• P.8, lines 152-154: The ms. usefully notes the methodological limitation of more sparse descriptive information on research studies conducted in the intramural program and research centers. However, it does not state how this limitation was addressed, if at all. One possibility is to provide percentages of the NIMH budget spent on schizophrenia research separately by type of research program (i.e., extramural, intramural, research center).

• The ms. might be strengthened with a few more examples of the kind of schizophrenia research the NIMH could fund (and proactively request) that is likely to be of more immediate benefit. Some examples are mentioned (e.g., new medications and psychological treatments) but more information and additional examples would make the article more compelling by noting the potential benefits that are not being reaped. Otherwise, some readers may think the NIMH is following its policy because there is little to study that could be more immediately useful.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr Virginia Minogue

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

3 Aug 2020

Dear Dr. Naudet, Dr. Minogue and Reviewer #2,

Thank you for your useful comments and critiques. We have done the following:

• We have stated explicitly that there was not a protocol.

• The master file, including all 428 research projects, by title and number along with the grades of the three reviewers are being submitted as supplementary material. To make the order of reviewers consistent between supplementary material and the manuscript, we reversed the numbers between reviewers 2 and 3 in Table 1.

• We have included a reporting checklist, along with mention in the methods section, a reference, and a specific grid.

• We have re-written the methods section to make explicit the exploratory nature of the study.

• We have also included this in the sections on limitations at the end of the paper.

• We have included three Venn diagrams to show the agreement among the three reviewers for the three questions.

• We have added Table 2 to include the full list of the 30 proposals selected as “likely”.

• The supplementary material now includes all studies in the same order as they are listed in the RCDC database. Thus individuals can ascertain the rating for all studies listed as “likely”, “possible”, or “very unlikely”.

• The Venn Diagrams are being submitted as separate figure files.

• The Title and Numbers of the two patents that were included pertain to depression and thus are not relevant to our study on schizophrenia. We have added the suggested statement.

Dr. Minogue

• In re-writing the methods section, we clarified the intent and development of the study, including the use of the advocacy groups in its development.

• The intended purpose of the RCDC database was to make the information available to the public and to shift the lobbying for funding from members of Congress to the NIH institutes. The directors of the advocacy groups were included as authors.

• We have added a section in the discussion regarding how the study will be used by the advocacy groups. Unfortunately we do not have an organization similar to the James Lind Alliance as far as I know. The main relationship between the NIH institutes and most disease advocacy groups is the former asking the latter to ask Congress to give them more money; thus what we are doing is very different.

• The methods section has been enlarged to provide more details on the development of the study. We have added “new and ongoing” to indicate that the projects from the past have been included.

• We have clarified that the ratings of “likely”, “possible”, and “very unlikely” were subjective based on individual experience and expertise. We did not attempt to standardize the definitions.

• We decided to use three assessors as a guess, based on the fact that we had 428 research projects. We had no idea when we started how much agreement we would have and were pleasantly surprised to find that we had good agreement.

• We added a sentence to clarify that our shared bias was that NIMH should be doing both basic and clinical research.

• Most of the project abstracts were sufficiently detailed to make a reasonable assessment of what they are likely to produce over the next twenty years.

• We have added an explanation for why twenty years was selected as the optimum period.

• In the abstract “criticisms” has been substituted for “allegations”.

• The final sentence in the abstract has been modified.

• Based on my discussion with the NIH official who was in charge of the RCDC until he recently retired, I think it is very unlikely that the database was used by any disease advocacy groups.

• The prevalence of schizophrenia in the US is probably a little above average by world standards. In most studies, it has been said to be one of the costliest diseases.

• We have spelled out NAMI and PET.

• We have significantly enlarged the description for the review process.

• We have enlarged the discussion of research projects that would be clinically useful that NIMH should be doing.

Reviewer #2

• We have clarified why we are using the 1983 prevalence data.

• Although the descriptions of some of the intramural research projects was sparse, the reviewers had independent information on many of them. For example, two of us had worked with researchers doing the imaging studies many years ago. Thus we felt comfortable in our ratings. The percentage of NIMH funds going to intramural projects is stated at the beginning of the results section.

• We added additional examples of the kind of schizophrenia research that NIMH should be funding.

Submitted filename: Response to Reviewers PLOS ONE.docx

Click here for additional data file.

27 Aug 2020

PONE-D-20-13102R1

Using the NIH Research, Condition and Disease Categorization Database for research advocacy: Schizophrenia research at NIMH as an example

PLOS ONE

Dear Dr. Torrey,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

First, I would like to thank here the two reviewers who helped me in reaching a decision.

I have some additional comments :

- In the abstract, please make it more clear that the conclusions are only exploratory ;

- Please add in the abstract a few words about the limitations, to avoid any spin / and a few words about the agreement ;

- By the may the agreement was not good for projects labeled as "Likely usefull" (see the Venn Diagram). Please avoid any spin about this and state it explicitely (it is not satisfying to see that you wrote that the agreement was good, especially for the unlikely category / one could say the agreement was not so good, especially for the likely category). Please edit both the text and the abstract.

- I tend to think that a Kappa would be more easy to interpret and indeed more appropriated to judge about inter-rater reliability than a Cronbach's alpha. Please change / or in case you disagree / please give me more reasons to think that you are right.

Please submit your revised manuscript by Oct 11 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Dr Virginia Minogue

Reviewer #2: Yes: Roberto Lewis-Fernandez, MD

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

10 Sep 2020

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

September 10, 2020

Dear Dr. Naudet,

Thank you for the further review of our manuscript. We have done the following.

• Clarified in the abstract that the conclusions are only exploratory;

• Clarified in the abstract the limitations of the research, including the limitations of the agreement among the raters;

• Stated explicitly in the text that the interrater agreement was good for the “very unlikely” research projects, but only fair for the “possible” projects and poor for the “likely” projects. This should also be clear to readers from looking at the Venn diagrams.

Regarding the proper statistical test for assessing the interrater reliability, my colleagues (who understand this issue much better than I do) believe that the Cronbach's alpha is the more appropriate test because it is a measure of internal consistency and scale reliability. What we are testing is scale reliability, not a diagnostic test or x-ray, etc. Cronbach’s is most commonly used for scale reliability and that seems consistent with what we are doing. Cohen`s kappa, by contrast, is usually used when you have just two raters. However if you prefer that we use a kappa statistic we can look for an alternative to the Cohen’s kappa.

I am also sad to report that one of our authors, DJ Jaffe, died unexpectedly on August 23. I have so indicated on the title page.

E. Fuller Torrey, M.D

Associate Director for Research

Stanley Medical Research Institute

Kensington, MD

Submitted filename: Response to reviewers.docx

Click here for additional data file.

11 Sep 2020

PONE-D-20-13102R2

Using the NIH Research, Condition and Disease Categorization Database for research advocacy: Schizophrenia research at NIMH as an example

PLOS ONE

Dear Dr. Torrey,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I'm really sad to learn that one of your co-authors died unexpectedly. I send you my sincere condolences.

Thank you for the changes you made, the better is better now. But I'm still not convinced about the use of Cronbach' alpha in this occasion. You are not working on a multiple item scale. There are actually alternative to Cohen's Kappa such as Fleiss' kapa for instance, that can handle multiple rater. Please ask the help of a statistician to choose the approach that fits better your needs.

Please submit your revised manuscript by Oct 26 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

6 Oct 2020

RE: Manuscript PONE-D-20-13102R2

Using the NIH Research, Condition and Disease Categorization Database for research advocacy: Schizophrenia research at NIMH as an example

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor. DATE: October 5, 2020

PLOS ONE

Dear Dr. Naudet,

Following up your letter of September 11, we have consulted with Thomas Carmody PhD, a Professor in the Department of Population and Data Sciences at the University of Texas Southwestern. After reviewing our manuscript and some of the literature on interrater reliability, he concluded that both Fliess kappa and Cronbach alpha have been used in situations analogous to ours. He sent a couple of references justifying the use of the latter. However, he also said that the kappa “is more widely used for interrater agreement”, whereas alpha “is most often used to measure internal consistency of scales”. We, therefore, decided to use the Fliess kappa which he calculated for us:

Fleiss kappa for each level of rating and overall

Rating Kappa Error z Prob > z

1 0.16513 0.027907 5.9171 <.0001

2 0.42658 0.027907 15.2858 <.0001

3 0.56954 0.027907 20.4083 <.0001

overall 0.47261 0.024540 19.2588 <.0001

We have made the appropriate changes to the text.

Thank you for your assistance.

Please let us know if you need additional information.

E. Fuller Torrey MD

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

8 Oct 2020

Using the NIH Research, Condition and Disease Categorization Database for research advocacy: Schizophrenia research at NIMH as an example

PONE-D-20-13102R3

Dear Dr. Torrey,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Thank you for all the edits that have improved the quality of the paper in my opinion.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Florian Naudet, M.D., M.P.H., Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

13 Oct 2020

PONE-D-20-13102R3

Using the NIH Research, Condition and Disease Categorization Database for research advocacy: Schizophrenia research at NIMH as an example

Dear Dr. Torrey:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Pr. Florian Naudet

Academic Editor

PLOS ONE