Are trials of psychological and psychosocial interventions for schizophrenia and psychosis included in the NICE guidelines pragmatic? A systematic review.

INTRODUCTION: The NICE clinical guidelines on psychosocial interventions for the treatment of schizophrenia and psychosis in adults are based on the results of randomized controlled trials (RCTs), which may not be studies with a pragmatic design, leading to uncertainty on applicability or recommendations to everyday clinical practice. AIM: To assess the level of pragmatism of the evidence used to develop the NICE guideline for psychosocial interventions in psychoses.
MATERIAL AND METHODS: We conducted a systematic and critical appraisal of RCTs used to develop the 'psychological therapy and psychosocial interventions' section of the NICE guideline on the treatment and management of psychosis and schizophrenia in adults, published in 2014. For each study we assessed pragmatism using the pragmatic-explanatory continuum indicator summary-2 (PRECIS-2) and the Cochrane risk of bias tool. The mean score of PRECIS-2, averaging across nine domains, was calculated to describe the level of pragmatism of each individual study.
RESULTS: A total of 143 studies were included in the analysis. Based on the PRECIS-2 tool, 16.8% were explanatory, 33.6% pragmatic, and 49.7% were rated in an intermediate category. Compared to explanatory studies, pragmatic studies showed a lower risk of bias. Additionally, pragmatism did not significantly improve over time, and no associations were found between pragmatism and a number of trial characteristics. However, studies with a UK leading investigator had the highest mean score of pragmatism. Cognitive behavioural therapy (CBT), art therapy, family intervention, psychoeducation, and adherence therapy, showed the higher average pragmatism scores.
CONCLUSIONS: Two third of studies used to produce NICE recommendations on psychosocial interventions for the treatment of schizophrenia and psychosis in adults are based on studies that did not employ a pragmatic design.

Introduction

For many years randomized controlled trials (RCTs) have been recognised as the most robust methodology for evaluating the effects of interventions in mental health care and in many other fields of medicine [1]. However, questions have been raised about the generalizability of findings from RCTs to patients treated “in the real world” [2]. This concern is related to the fact that most RCTs are optimised to determine efficacy of interventions in absolute terms, i.e. they are explanatory rather than pragmatic. Pragmatic randomised trials are usually undertaken to help support a decision on whether an intervention should be delivered in a real-world setting. Explanatory randomised trials, by contrast, are undertaken to test whether an intervention is effective under ideal circumstances. There is no simple threshold to determine whether a trial is explanatory or pragmatic, and there are few purely explanatory or pragmatic trials. In most cases, trials include both explanatory and pragmatic characteristics, suggesting a continuum, rather than a dichotomy, between these two polarities [3].

The methodological explanatory rigour of conventional RCT design is seen as unsatisfactory for studying situations of high treatment complexity [4], such as those involved in psychosocial interventions within routine health care [5]. As a consequence, there is general agreement that pragmatic trials should be conducted to show the real-world effectiveness of interventions [6], and thus answer questions of interest to patients, clinicians and policy makers [7].

In the UK, based on the results of RCTs, the National Institute for Health and Care Excellence [8] produces clinical guidelines for health and social care. In 2014 it published the updated guideline on psychological and psychosocial interventions in the treatment of psychosis and schizophrenia in adults [8], which is currently used to guide clinical and policy decisions[9]. A way to understand the applicability to everyday clinical practice of this NICE guideline, the level of pragmatism within the included evidence needs to be assessed. This is especially pertinent for mental health care where the gap between evidence and real-world clinical practice is particularly wide for the treatment of severe mental disorders [10].

In 2009 Thorpe and colleagues [11] defined a tool to help researchers design trials based on whether the purpose of the trial was broadly pragmatic or explanatory: the pragmatic–explanatory continuum indicator summary (PRECIS). In 2015, a new version of PRECIS was developed, improved, and validated with the help of over 80 international trialists, clinicians, and policymakers (PRECIS-2) [3, 12, 13]. Despite being initially developed to inform the design phase of trials, the PRECIS-2 tool has also been applied retrospectively to critically appraise existing trial evidence [14-16].

The aim of this appraisal was therefore to use the PRECIS-2 tool to assess the level of pragmatism of clinical trials testing psychological and psychosocial interventions as included in the 2014 NICE guideline for treatment and management of psychosis and schizophrenia. The appraisal of trials will assist with understanding the evidence-practice gap on psychosocial interventions. Moreover, it aims to identify determinants that affect the pragmatism of studies.

Materials and methods

Protocol and registration

The protocol of this work was registered on PROSPERO (CRD42016050116).

Eligibility criteria

We included all randomized control trials used to develop the ‘psychological therapy and psychosocial interventions’ section of the NICE guideline on the treatment and management of psychosis and schizophrenia in adults, published in 2014 [8]. Participants included in these trials were 16 years old or older. This section includes trial focused on the following interventions: adherence therapy, art therapies [e.g. music therapy, body psychotherapy), social skills Training (SST), Cognitive Behavioural Therapy (CBT), Family Intervention (FI), Cognitive Remediation (CR), Psychoeducation, Counselling and supportive therapy (CST) and psychodynamic therapy.

Data collection process

Data extraction was carried out by a one researcher (CG). Two additional researchers (ST, FM) independently assessed the data extracted, the pragmatism and the quality of studies. Any disagreement was discussed with additional authors (VJB, CB, SP). Standardised data collection forms were used to extract data. The information extracted from each study included: year of publication, journal name, country of the lead investigator, sample size, age and diagnosis of participants, type of experimental intervention, results of the study and length of follow-up.

Pragmatism was assessed using the pragmatic–explanatory continuum indicator summary-2 (PRECIS-2). This tool has nine domains (eligibility criteria, recruitment, setting, organization, flexibility (delivery), flexibility (adherence), follow-up, primary outcome, and primary analysis). Each domain can be scored using a 5-point Likert scale in which 1 means very explanatory, 2 rather explanatory, 3 equally pragmatic and explanatory, 4 rather pragmatic and 5 very pragmatic [3]. PRECIS-2 was selected for its reliability and validity [12]. Where it was not possible to give a score to a domain due to a lack of information, the domain was rated as “very explanatory”, using a conservative approach, oriented to not overestimate pragmatism.

The risk of bias of each study was assessed using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions for risk of bias [17].

Statistical analysis

Descriptive statistics (number and percentage of each possible value) were calculated for the following variables: year of publication (published before 1995, between 1995 and 2005 and after 2005); type of journal where the primary reference was published (psychiatric or not psychiatric); multi-centricity (unclear, multi-centric, not multi-centric); country of the lead investigator (UK, Rest of Europe, North America, Asia and Middle East, Australia and unclear); sample size (<50, 50–100, >100 participants, unclear); length of follow-up (post treatment/ end of the intervention; <3 months, 3–6 months, >6 months, unclear); results (intervention significantly better, intervention not better i.e. control better and not significant difference and unclear); intention-to-treat (ITT) analysis (ITT, not ITT and unclear); diagnosis of participants (only schizophrenia, schizophrenia and other psychoses, unclear); substance abuse in eligibility criteria (primary diagnosis of abuse or dependence excluded, secondary diagnosis excluded, users excluded, participants with abuse or dependence not excluded, unclear).

The mean score of PRECIS-2, averaging across nine domains, was calculated to describe the level of pragmatism of each individual study. Studies were then clustered into three groups according to their score: explanatory studies (score< = 2.5), intermediate (score strictly between 2.5 and 3.5) and pragmatic studies (score > = 3.5). These cut-offs where selected to discriminate between studies whose items expressed central values in the explanatory-pragmatic continuum, and those clearly leaning to one side.

Descriptive statistics were reported both for the average pragmatism score (as divided into the categories described above) and for each item on the pragmatism tool, both as continuous and categorical variables, using the cut-off scores described in PRECIS-2 [3]: 1–2 points for explanatory studies, 3 equally pragmatic and explanatory, 4–5 for pragmatic studies.

Additional descriptive and inferential analyses assessed whether pragmatism was associated with selected trial characteristics. First, the mean value of pragmatism for each intervention type was calculated, together with the related confidence intervals. We then analysed the association between pragmatism and risk of bias. For each dimension of risk of bias as defined by Higgins and colleagues [17], the percentage of studies classified respectively as “low risk”, “high risk” and “unclear” was calculated for each value of pragmatism as a categorical variable. Then, domains of the risk of bias were dichotomized (low risk vs high risk or unclear) and the significance of their association with pragmatism was assessed through Fisher’s exact test. Finally, as secondary analyses, we studied whether pragmatism was associated with year of publication, type of journal where the study was published, country of the study’s lead investigator (as grouped into: UK, Rest of Europe, North America, other), sample size and trial results. We categorized these characteristics as described above. We considered pragmatism both as a continuous variable (by performing ANOVA tests) and as a categorical one (by performing Chi-squared tests, or Fisher’s exact test, when appropriate).

Results

Characteristics of included studies

A total of 143 studies were included in the appraisal, i.e. all studies used to develop recommendations described in the NICE guideline for schizophrenia (Fig 1). Included studies assessed the following interventions: adherence therapy, art therapies (e.g. music therapy, body psychotherapy), social skills Training (SST), Cognitive Behavioural Therapy (CBT), Family Intervention (FI), Cognitive Remediation (CR), Psychoeducation (PE), Counselling and supportive therapy (CST) and psychodynamic therapy. Comparators were standard care, wait list control, pharmacological treatment, and other psychological therapies (details and number of studies for each type of intervention are reported in Fig 1).

Fig 1

PRISMA flow chart.

Table 1 presents the main study characteristics.

Table 1

Characteristics of included studies.

	N of studies	%
Year of publication:
Before 1995	33	23.1
1995–2005	72	26.6
>2005	38	50.3
Type of journal
Psychiatric	128	89.5
Not psychiatric	15	10.5
Multicentricity
Unclear	47	32.9
Multicentre	43	30.1
Not multicentre	53	37.1
Country
Unclear	37	25.9
UK	26	18.2
North America	38	26.6
Rest of Europe	21	14.7
Asia and Middle East	17	11.9
Australia	4	2.8
Sample size
Unclear	11	7.7
<50	53	37.1
50–100	55	38.5
>100	24	16.8
Length of FOLLOW UP
Unclear	13	9.1
Post treatment	48	33.6
<3 months	18	12.6
3 months-6months	30	21.0
>6 months	34	23.8
Results
Unclear	14	9.8
Intervention sign. Better	89	62.2
Intervention not better	40	28.0
ITT analysis
Unclear	13	9.1
Yes	51	35.7
No	79	55.2
Diagnosis
Unclear	39	27.3
Schizophrenia	55	38.5
Schizophrenia and other psychosis	49	34.3
Substance abuse in eligibility criteria
Unclear	69	48.3
Primary diagnosis excluded (abuse/dependence)	14	9.8
Secondary diagnosis excluded	26	18.2
Users excluded	12	8.4
Not excluded	22	15.4

Fifty-five studies enrolled only participants with a primary diagnosis of schizophrenia, while forty-nine included participants with related disorders. Only fourteen studies included patients with substance dependence/abuse as a secondary diagnosis. Diagnostic criteria were mainly based on the Diagnostic and Statistical Manual of Mental Disorders Third edition revised or Fourth Edition (DSM-IIIR or IV) or ICD-10. Participants were aged between 18 and 65 years old in the majority of studies. One third of studies was multi-centric; one third were single site studies with classification of the remaining studies unclear. The country of the lead investigator was the UK for 26 studies (18%), North America for 37 studies (of which 35 (27%) USA and 2 Canada), Europe (excluding UK) in 21 studies, other countries in 17 studies, and Australia in four studies (Table 1).

The mean study sample size was 79.0 participants (SD 6.2, CI 66.7 to 91.2). There were 48 studies with a follow-up only at post-treatment, 18 short-term studies with a follow up between 2 and 12 weeks after the end of intervention, 30 medium-term studies with a duration between 13 weeks and 6 months and 34 long-term studies with a follow-up duration of more than 6 months (Table 1).

Pragmatism of studies

Descriptive statistics

After comparing scores assigned by two researchers, we found an agreement and marked every domain of the PRECIS-2 with a score for each study. We then calculated and average score for each study. Based on these scores we clustered studies in three categories: 24 (16.8%) were classified as explanatory studies, 71 (49.7%) as intermediate studies and 48 (33.6%) as pragmatic studies. Details of average scores and categories for each study are reported in supplementary material (S1 Appendix).

Moreover, we explored each PRECIS-2 domain: Table 2 presents the distribution of studies in the explanatory, intermediate, or pragmatic categories for each item of the PRECIS-2 tool.

Table 2

Number of studies with an explanatory, intermediate and pragmatic domain.

Mean scores and standard deviations for each domain of the PRECIS-2 tool.

	Explanatory (n, %)	Intermediate (n, %)	Pragmatic (n, %)	Mean score (SD)
1. Eligibility	61 42.66	28 19.58	54 37.76	3.01 (1.24)
2. Recruitment	61 42.66	18 12.59	64 44.76	2.87 (1.54)
3. Setting	53 37.06	20 13.99	70 48.95	3.27 (1.48)
4. Organization	34 23.78	41 28.67	68 47.55	3.34 (1.23)
5. Flexibility (Delivery),	41 28.67	36 25.17	66 46.15	3.31 (1.26)
6. Flexibility (Adherence)	90 62.94	12 8.39	41 28.67	2.38 (1.58)
7. Follow-Up	47 32.87	25 17.48	71 49.65	3.26 (1.43)
8. Primary Outcome	23 16.08	42 29.37	78 54.55	3.57 (1.05)
9. Primary Analysis	40 27.97	34 23.78	69 48.25	3.48 (1.48)

Overall, based on mean scores, the domains rated as most pragmatic were “primary outcome”, “primary analysis” and “organization”, whereas, “flexibility adherence” and “recruitment” were the most explanatory. Fig 2 shows the average score for each domain on the PRECIS-2 wheel for all included studies. This diagram conveys how pragmatic versus explanatory a trial is by the distance of the marks on each domain from the centroid: the further away from the centre, the more pragmatic the trial is on that domain.

Fig 2

Cumulative PRECIS Wheel.

Pragmatism was additionally studied by type of intervention. Fig 3 shows that CBT, art therapy, family intervention, psychoeducation, and adherence therapy, have the higher average pragmatism scores and this difference was statistically significant (p<0.001).

Fig 3

Mean precis score by intervention (p<0.001).

Ad T = Adherence Therapy (N = 4); PSYCHOED = psychoeducation (N = 17); CBT = Cognitive Behavioural Therapy (N = 33); FI = Family Intervention (N = 31); ART = Art Therapies (N = 7); CR = Cognitive Remediation (N = 22); CST = Counselling and Supportive Therapy (N = 7), SST = Social Skills Training (N = 18); PSYCHODINAM = Psychodynamic and Psychoanalytic Therapies (N = 3)

We assessed the risk of bias for each study and then we reported it in a graphic representation of the quality of studies by pragmatism (Fig 4).

Fig 4

Risk of bias of studies by PRECIS category.

From the top to the bottom: risk of bias of explanatory, pragmatic and intermediate studies respectively.

Pragmatic studies show a lower risk of bias in all RoB categories, compared to explanatory and intermediate studies, with the exception of reporting bias. Intermediate studies show an intermediate risk of bias compared to the other two categories. Nevertheless, these differences were statistically significant only for the two categories of selection bias (p< 0.01 in both cases) (More details in the supplementary materials, S1 Table and S1 Fig).

Relationship between pragmatism and other variables

Performing additional descriptive and inferential analyses to assess whether pragmatism was associated with selected trial characteristics we found that pragmatism did not significantly change over time as measured by the year of publication (Table 3).

Table 3

Analyses of associations.

	Pragmatism mean, SD	Difference: p-value
Sample size		0.294
<50	3.1, 0.68
50–100	3.2, 0.66
>100	3.2, 0.64
Year of publication		0.237
before 1995	3.0, 0.62
1995–2005	3.2, 0.75
after 2005	3.2, 0.51
Results		0.284
Intervention sign. better	3.1, 0.75
Intervention not better	3.2, 0.65
Country		<0.001
UK	3.7, 0.58
North America	2.9, 0.54
Europe	3.2, 0.67
Others	3.2, 0.67

Similarly, we did not find any significant association between pragmatism and sample size or study results. The analysis evaluating the association between pragmatism and type of journal was not conducted as only 15 out of 143 trials were published in a “non-psychiatric journal”. Studies with a UK lead investigator had the highest mean score of pragmatism with a significant association (p<0.001) (Table 3). By replicating the analyses using pragmatism as a three-level categorical variable, the same findings were obtained (more details are reported in table A and B in S2 Table).

Discussion

Main findings

This systematic and critical appraisal showed that the 143 psychosocial and psychological trials included in the NICE guideline for schizophrenia and psychosis differed on the extent to which they were rated as pragmatic versus explanatory and consequently on the level of applicability of their results to clinical practice. The average PRECIS-2 score of all studies was between explanatory and intermediate on the continuum.

Across all studies, some domains of the PRECIS-2 tool were rated as more pragmatic then others on average. The PRECIS-2 domains rated as most pragmatic were “primary outcome”, “primary analysis” and “organisation”, whereas those that were most explanatory were “flexibility-delivery”, “flexibility-adherence” and “recruitment”. This finding may be just partially explained by the lack of adequate reporting for these domains. These results are consistent with those of Loudon and colleagues [12] who, while testing the discriminant validity and interrater reliability of PRECIS-2, found that these two domains were not statistically better discriminants than chance, as both were poorly described in the trial protocols. Another possible explanation is that it may be easier to be pragmatic for some domains than for others, as pointed out by Johnson and colleagues in their work [18].

Another finding was that trials had different levels of pragmatism depending on the intervention being delivered. Those focusing on art therapy, adherence therapy, psychoeducation, CBT and family intervention had on average a more pragmatic score, whereas psychodynamic and psychoanalytic therapies, social skills training, cognitive remediation, counselling and supportive therapy had a more explanatory score. This result could be related to the type of intervention itself, as adherence and art therapy, CBT, psychoeducation and family intervention are very flexible interventions, that can be delivered also by non-medical or non-psychological staff (nurses, social workers, other therapists). These results appear to be highly reliable due to the high number of trials with CBT, psychoeducation and family intervention (33, 17 and 31, respectively). This result suggests that the latter interventions may be easier to implement in clinical practice, with implications for policy makers.

The secondary aim of this systematic appraisal was to assess if there is an association between the level of pragmatism on PRECIS-2 and characteristics of the studies. Firstly, explanatory domains did not relate to study quality. By contrast, pragmatic trials had a higher quality than explanatory trials, showing a lower risk of selection bias. This finding is particularly interesting, as it has been suggested that explanatory trials pursue internal validity (quality) at the cost of external validity (pragmatism) [19]. This review does not support this view. We found that pragmatic trials place a premium on external validity while maintaining internal validity, and this is consistent with other previous findings [18, 19].

Reporting bias was the only type of bias in which the risk was lower in explanatory than pragmatic trials. It may be possible that, as explanatory trials choose very specific outcomes at the study design stage in order to assess the efficacy of interventions, they report those outcomes more often than pragmatic trials in order to confirm their initial hypotheses.

As the discussion about the relevance of pragmatism in mental health clinical trials has been raised decades ago and it has been growing progressively [7], we expected to find a trend showing increasing pragmatism over the years. However, the analysis failed to demonstrate this. It appears that pragmatism has not increased from the ‘80s until 2009, when the psychological and psychosocial chapter of the 2014 guideline was updated.

The only statistically significant association found was between pragmatism and country of the study’s lead investigator; UK psychosocial RCTs were on average more pragmatic than studies conducted in other countries. In the UK clinical trials tend to be based in “real world” mental health services that are mostly community based centres with a psychosocial approach. Conversely, in the USA the majority of studies are conducted in high quality academic centres that can be very different from usual care services in staff, resources or type of patients involved.

Strengths and limits

To our knowledge, this is the first systematic review that analysed the pragmatism of trials included in a guideline for mental health care. So far, most authors have used the PRECIS-2 tool to assess pragmatism at the stage of the trial protocol development, to ensure the design reflects their intended purpose. Within other fields of medicine there are a few examples of PRECIS-2 being used to appraise single studies or studies included in reviews retrospectively [16, 20]. This is relevant as the pragmatism of trials included in the guideline may be an indicator of the overall pragmatism of the guideline itself, i.e. of the applicability of evidence.

These findings may be interpreted with caution because the psychosocial and psychological treatment chapter of the 2014 NICE guideline for psychosis and schizophrenia had not been updated with research evidence published up to 2014 but only to 2009. Since then, more RCTs have been published, and changes in the evidence-base may have been found with new trials potentially being more pragmatic, as interest around this topic has increased in the last 10 years. Despite this issue, the findings of this review remain relevant as current clinical practice in the UK is based on the evidence included in the NICE guideline analysed.

Another limitation was the paucity of details reported in the study reports. This meant that accurate assessments of what actually happened within the trial were difficult, especially in the domains of recruitment and adherence. Instead we assessed the impact of study quality as measured by the Risk of Bias. Finally, although PRECIS-2 is a validated and comprehensive tool, there were difficulties applying it to heterogeneous trials. Due to disagreements with the second and third reviewers, further definition of more detailed “standardized” criteria to ensure internal consistency was developed. A number of researchers from different backgrounds (from both research and clinical practice) and countries were involved in the discussion as suggested by the authors of the PRECIS-2 tool [12].

Conclusions

Overall, two third of studies used to produce NICE recommendations on psychosocial interventions for the treatment of schizophrenia and psychosis in adults are based on studies that did not employ a pragmatic design.

We would encourage a discussion around weighting and interpreting evidence based on its position on the pragmatism-explanatory continuum when considering new evidence for the next round of updates. As these guidelines currently influence clinical and policy decisions in the UK and in other European countries the results should be considered by clinicians, commissioners, and future research.

Moreover, we have identified several areas of major deficiency in terms of reporting quality and pragmatism including random sequence generation, allocation concealment, explicit outlining of study implementation, blinding, and description of patient recruitment and adherence to therapy. This is in line with the findings of other authors who raised concerns about the methodology employed in the development of NICE guidelines [21, 22]. Recently Dal-Re [14] recommended the inclusion of PRECIS-2 evaluation in trial papers and protocols as this could be useful both to design studies that are more pragmatic and to reduce reporting bias. Based on our findings we would add that future trial reports should improve their comprehensiveness more generally and thereby aid appraisals of quality and pragmatism by using the checklist developed as part of the extension to the CONSORT statement [23, 24]. Further research that is designed for the purpose of pragmatic outcomes and comprehensively reported research is needed in the field of psychosocial intervention for schizophrenia and psychoses to improve the applicability of NICE guidelines within mental health care.

References of included studies.

(DOCX)

Click here for additional data file.

Table mean score and category of included studies.

(DOCX)

Click here for additional data file.

Table A relationship between pragmatism and risk of bias, Table B relationship between pragmatism and other variables.

(DOCX)

Click here for additional data file.

Risk of bias of pragmatic studies.

S1B FIG Risk of bias of intermediate studies. S1C FIG Risk of bias of explanatory studies.

(DOCX)

Click here for additional data file.

24 Jul 2019

PONE-D-19-17507

Are trials of psychological and psychosocial interventions for schizophrenia and psychosis included in the NICE guidelines pragmatic?

PLOS ONE

Dear Dr. CHIARA GASTALDON,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

ACADEMIC EDITOR: Although it is of interest, the reviewers have raised a number of points which we believe major modifications are necessary to improve the manuscript, taking into account the reviewers' remarks.

==============================

We would appreciate receiving your revised manuscript by Sep 07 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Wisit Cheungpasitporn, MD, FACP

University of Mississippi Medical Center

Twitter: @wisit661 Email: wcheungpasitporn@gmail.com

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

3. At this time we request that you update the title of this manuscript to identify it as a systematic review, as indicated in the PRISMA checklist.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: General aspects

It is a review paper in which he seeks arguments for a treatment guide for schizophrenia and other psychoses. There was a careful inclusion criterion of statistical studies and care, although of basic statistics, the results have discussed aspects that gives support to the conclusion.

Here there are some questions:

1)      The study presents the results of scientific research. Despite being a systematic review study, the presentation of the results is adequate, being a study of the evaluation of NICE pragmatism from previously published works. This point is important for using the guide.

2. Results reported have not been published elsewhere. Although being a review article, the results were organized in an appropriate way to think about the validity or not of the NICE.

3. A basic statistic was used, but the way the authors clustered the data was adequate.

4. Conclusions are presented in an appropriate fashion and are supported by the data.

5. Regarding the references of the 143 studies, will these studies be in supplemental material?

6. There are two initials that don’t have meanings.

7. It was observed that the author does not explain the meaning of two initials – see pg 7.

Reviewer #2: Some revision of the English language is needed. There are some parts of the paper where it is quite difficult to make sense of some sentences. English edit will help to improve the quality of the manuscript. To mention as several as below:

“These cut-offs where selected to distinguish” ???where

“more pragmatic then others” ???then

“other diagnosis” is not correct in grammar

“the risk of bias dimensions were dichotomized” is not correct in grammar.

“both pragmatism” is not correct in grammar

“evidences” is not correct. It is uncountable noun.

“may be interpret” is not correct in grammar.

“involved to stimulate” is not correct in grammar.

Reviewer #3: Definition of pragmatism is unclear. Authors assumed that most people are well versed in the definition and topic of pragmatism in psychiatry. It will be easier for people to understand the message of this paper if definitions of important keywords such as pragmatism and explanatory are well defined.

Authors mentioned that cut off point for studies included was 2009 and NICE was published in 2014, therefore more studies should have been included. However, to honor the protocol of the NICE guidelines, only studies up to 2009 were taken. Therefore it is not a good suggestion to highlight NICE should take more recent studies, as that would violate their protocol. Furthermore, it takes time from the studies taken within the cut off period to the production of a guideline. Furthermore, the authors did say that no statistical significance found between years of publication with pragmatism (page 14 of manuscript).

In page 6 of the manuscript, authors mentioned PRECIS-2 was developed in 2015. NICE guideline was published in 2014, therefore it is not possible to use PRECIS-2 as a measure of pragmatism in including relevant studies for this guidelines, prior to its publication. However it is a good idea to use these criteria retrospectively to examine the level of pragmatism.

Their strength and limitations were addressed.

As for conclusion, in my opinion it is acceptable to say “2/3 of studies used to produce NICE recommendations on psychosocial interventions for the treatment of schizophrenia and psychosis in adults are based on studies that did not employ a pragmatic design”. But to say “This poses serious concerns to the applicability of NICE recommendations” is a rather strong statement as we are aware that studies involving psychosocial treatments are already very scarce in the current picture. Even if these studies were not as pragmatic as indicated by the PRECIS-2, they are still important and relevant in production of the NICE 2014 guideline, which uses an evident-based medicine approach. Perhaps suggestion to use PRECIS-2 as a measure of pragmatism for future studies before including them in future guidelines suggestion is sufficient, without needing to point out the threat that it poses with regards to NICE’s applicability.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Anuska Irene de Alencar

Reviewer #2: No

Reviewer #3: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Pone D 19 17507 review.pdf

Click here for additional data file.

12 Aug 2019

JOURNAL REQUIREMENTS:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming.

Author: thank you, we changed the file according to the requirements

2. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Author: thank you, we included what you asked

3. At this time we request that you update the title of this manuscript to identify it as a systematic review, as indicated in the PRISMA checklist.

Author: thank you, we changed the title as required by the PRISMA STATEMENT

RESPONSE TO REVIEWERS

Reviewer #1: General aspects

It is a review paper in which he seeks arguments for a treatment guide for schizophrenia and other psychoses. There was a careful inclusion criterion of statistical studies and care, although of basic statistics, the results have discussed aspects that gives support to the conclusion.

Author: thank you, no revision needed

Here there are some questions:

1) The study presents the results of scientific research. Despite being a systematic review study, the presentation of the results is adequate, being a study of the evaluation of NICE pragmatism from previously published works. This point is important for using the guide.

Author: no revision needed

2. Results reported have not been published elsewhere. Although being a review article, the results were organized in an appropriate way to think about the validity or not of the NICE.

Author: no revision needed

3. A basic statistic was used, but the way the authors clustered the data was adequate.

Author: no revision needed

4. Conclusions are presented in an appropriate fashion and are supported by the data.

Author: no revision needed

5. Regarding the references of the 143 studies, will these studies be in supplemental material?

Author: yes studies are already available in supplementary material

6. There are two initials that don’t have meanings.

Author: Corrected

7. It was observed that the author does not explain the meaning of two initials – see pg 7.

Author: Corrected

Reviewer #2: Some revision of the English language is needed. There are some parts of the paper where it is quite difficult to make sense of some sentences. English edit will help to improve the quality of the manuscript. To mention as several as below:

“These cut-offs where selected to distinguish” ???where

“more pragmatic then others” ???then

“other diagnosis” is not correct in grammar

“the risk of bias dimensions were dichotomized” is not correct in grammar.

“both pragmatism” is not correct in grammar

“evidences” is not correct. It is uncountable noun.

“may be interpret” is not correct in grammar.

“involved to stimulate” is not correct in grammar.

Author: revision of the English language was done, thank you.

Reviewer #3: Definition of pragmatism is unclear. Authors assumed that most people are well versed in the definition and topic of pragmatism in psychiatry. It will be easier for people to understand the message of this paper if definitions of important keywords such as pragmatism and explanatory are well defined.

author: ok, thank you. A definition of pragmatism was added. The new text reads as follows: “Pragmatic randomised trials are usually undertaken to help support a decision on whether an intervention should be delivered in a real-world setting. Explanatory randomised trials, by contrast, are undertaken to test whether an intervention is effective under ideal circumstances. There is no simple threshold to determine whether a trial is explanatory or pragmatic, and there are few purely explanatory or pragmatic trials. In most cases, trials include both explanatory and pragmatic characteristics, suggesting a continuum, rather than a dichotomy, between these two polarities (3).”

Authors mentioned that cut off point for studies included was 2009 and NICE was published in 2014, therefore more studies should have been included. However, to honor the protocol of the NICE guidelines, only studies up to 2009 were taken. Therefore it is not a good suggestion to highlight NICE should take more recent studies, as that would violate their protocol. Furthermore, it takes time from the studies taken within the cut off period to the production of a guideline. Furthermore, the authors did say that no statistical significance found between years of publication with pragmatism (page 14 of manuscript).

Author: Actually we are not suggesting that NICE should have taken more recent studies in that guideline, but that NICE should update the guideline itself. The guideline is based on evidence published up to 2009. We think that it would be important to update the guideline including newer studies, considering that this field is rapidly evolving.

In page 6 of the manuscript, authors mentioned PRECIS-2 was developed in 2015. NICE guideline was published in 2014, therefore it is not possible to use PRECIS-2 as a measure of pragmatism in including relevant studies for this guidelines, prior to its publication. However it is a good idea to use these criteria retrospectively to examine the level of pragmatism.

Author: As reported by the referee, and as mentioned in the paper, we retrospectively examined the level of pragmatism of those trials using PRECIS-2.

Their strength and limitations were addressed.

Author: thank you, no revision needed

As for conclusion, in my opinion it is acceptable to say “2/3 of studies used to produce NICE recommendations on psychosocial interventions for the treatment of schizophrenia and psychosis in adults are based on studies that did not employ a pragmatic design”. But to say “This poses serious concerns to the applicability of NICE recommendations” is a rather strong statement as we are aware that studies involving psychosocial treatments are already very scarce in the current picture. Even if these studies were not as pragmatic as indicated by the PRECIS-2, they are still important and relevant in production of the NICE 2014 guideline, which uses an evident-based medicine approach. Perhaps suggestion to use PRECIS-2 as a measure of pragmatism for future studies before including them in future guidelines suggestion is sufficient, without needing to point out the threat that it poses with regards to NICE’s applicability.

Author: We believe the referee is right. The sentence “This poses serious concerns to the applicability of NICE recommendations” was deleted from the conclusions of the abstract and main text.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

10 Sep 2019

[EXSCINDED]

Are trials of psychological and psychosocial interventions for schizophrenia and psychosis included in the NICE guidelines pragmatic? A systematic review

PONE-D-19-17507R1

Dear Dr. CHIARA GASTALDON,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Wisit Cheungpasitporn, MD, FACP

University of Mississippi Medical Center

Twitter: @wisit661 Email: wcheungpasitporn@gmail.com

Academic Editor

PLOS ONE

Additional Editor Comments:

I want to commend the authors on their superb efforts to revise the manuscript according to all reviewers’ suggestions. The quality of the manuscript has improved substantially.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: It is a review paper in which he seeks arguments for a treatment guide for schizophrenia and other psychoses. There was a careful inclusion criterion of statistical studies and care, although of basic statistics, the results have discussed aspects that gives support to the conclusion.

Reviewer #2: the authors have addressed the raised issues, no further comment. all comments were appropriate and welcome accept as is

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Anuska Irene de Alencar

Reviewer #2: No

13 Sep 2019

PONE-D-19-17507R1

Are trials of psychological and psychosocial interventions for schizophrenia and psychosis included in the NICE guidelines pragmatic? A systematic review

Dear Dr. Gastaldon:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Wisit Cheungpasitporn

Academic Editor

PLOS ONE