Patient and public involvement in the early stages of clinical trial development: a systematic cohort investigation.

BACKGROUND: Randomised controlled trials (RCTs) are considered particularly likely to benefit from patient and public involvement (PPI). Decisions made by professional researchers at the outset may go on to have a significant impact on the potential for PPI contributions.
OBJECTIVE: To increase knowledge of PPI within the early development of RCTs by systematically describing the reported level, nature and acceptability of proposed PPI to the funders.
METHODS: Documentation from the outline application process for all RCTs that received funding from the Health Technology Assessment (HTA) Programme 2006-2010 was requested. For each application, data were extracted on trial characteristics, references to PPI in the development of the outline application and funding Board feedback, and plans for PPI in the full application and after the trial was funded.
RESULTS: 110 applications were eligible with outline applications available for 90 (82%). The cohort covered a wide range of interventions and conditions. 54% (49/90) provided some information about PPI. 26 (28.9%) indicated PPI within the development of the outline application itself; 32 (35.6%) planned involvement in the full application and 43 (48%) once the trial was funded. Recruitment at diagnosis and surgical interventions were less likely to describe PPI. Blinded trials and trials in which participants may receive placebo only, more frequently described PPI activity. The HTA commissioning Board feedback rarely referred to PPI.
CONCLUSIONS: Incorporation of PPI within the development of the outline application or specification of plans for future involvement was low. Funder requests for applicants to provide information on PPI and justification for its absence should be welcomed but further research is needed to identify the impact of this on its contributions to research. Comments on PPI by reviewers should be directional rather than state that an increase is required. Challenges facing applicants in initiating PPI prior to funding need to be addressed. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

This is the first study to systematically investigate patient and public involvement (PPI) early in the development of funding applications and across a wide range of trials.

Presents results of a large unselected cohort addressing concerns of selective reporting of case studies within the literature.

Is the first study to consider clinical and trial characteristics in relation to the model of PPI adopted.

Cohort is limited to a single funder covering all randomised controlled trials receiving funding between 2006 and 2010.

Documentation could not be made available for unsuccessful applications. However, PPI was not mandatory within the period of the cohort, and that on its own would not have led to a decision that the outline application should not progress.

As the cohort was identified by receipt of funding during the period rather than the year that the outline application was submitted for consideration, fewer trials were available in the extremes of the cohort.

Changes to funder requirements for PPI have changed and this may impact generalisability.

Background

Public involvement in research is described as research being carried out ‘with’ or ‘by’ members of the public rather than ‘to’, ‘about’ or ‘for’ them (http://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/, last accessed 5 September 2013). It has been suggested that clinical trials are particularly likely to benefit from patient and public involvement (PPI).1 2 PPI can start at various stages of a trial and influence many aspects. Increased recognition that patients and public are stakeholders in research has led to increasing calls that they be represented within that research process. The UK Department of Health (2006) Best Research for Best Health guidelines3 state that PPI must be included in all stages of the research process including priority setting, defining research outcomes, selecting research methodology, patient recruitment, interpretation of findings and dissemination of results. Many funding bodies now request researchers to provide evidence of PPI when submitting their proposals.4 The National Institute of Health Research (NIHR) Health Technology Assessment (HTA) programme encourages applicants to consider whether the ‘scientific quality, feasibility or practicality of the proposal can be improved by patient and public involvement’ (http://www.hta.ac.uk/public/).

Staniszewska et al5 discussed the importance of PPI being implemented at the design stage of studies to optimise the impact and relevance of research. During the design stages of a trial many decisions are made that determine the relevance and conduct of the proposed research: the precise specification of the research question, including the outcomes to be measured; visit schedules; methods of data collection; recruitment and consent procedures. Fudge et al6 suggest that decisions made by professional researchers at the outset of a study have a cumulative and significant influence on the potential for PPI to impact on a study and that involvement is more difficult to achieve once studies are under way. It is therefore important to consider PPI activity in the early stages of the research process. However, little is known about how or when researchers incorporate PPI or what impact may stem from that involvement. Concerns have been expressed that the existing literature is selectively reported with many reports aiming to ‘make the case’ or ‘convince the sceptics’ about PPI.7 Furthermore, as much reporting has involved single case studies, generalisability of the PPI literature is limited.7

The aim of this paper is to increase the knowledge of PPI within the early stage design and development of clinical trials by systematically studying an unselected cohort of outline applications that were in receipt of funding from the HTA programme between 2006 and 2010. Specific objectives are to identify whether and how PPI is described within the early development of a grant application for funding; examine how PPI contributions within the development of the outline application were reviewed by the HTA Board; describe applicants’ PPI plans should their application be successful; and describe variations in PPI in relation to time of funding and trial characteristics.

Methods

We examined a cohort of randomised controlled trials (RCTs) that received funding from the NIHR HTA) programme in the UK between 2006 and 2010. During the period of the cohort the HTA programme encouraged PPI, and allowed for resources to support PPI activity to be requested; however, PPI was not mandatory. Inclusion of a PPI contributor on the Trial Steering Committee (TSC) was recommended. The HTA programme has a two-stage application process, comprising an outline application followed by a full application for those successful at the outline stage (http://www.hta.ac.uk/funding/clinicaltrials/index.shtml, accessed 30 April 2012). We requested all documentation relevant to the outline applications from the HTA. This comprised the commissioning briefs (if applicable) in which the topic of research driven by funders rather than applicants, outline applications and the minutes of the relevant Board meetings at which the outline applications were considered and which contained feedback for the applicant. Prior to the release of these documents the NIHR HTA contacted the chief investigators of the trials involved informing them that their names were released as these were readily available within the public domain. We signed a confidentiality agreement and the NIHR HTA redacted sensitive information regarding budget information and names of co-applicants (not available within the public domain) before releasing the documentation to us.

Once the trial was funded we established a PPI advisory group of five members with experience of providing PPI in RCTs. Members were recruited through an open and transparent process. Advertisements for members were placed on various PPI websites and applicants who expressed an interest were sent an application pack which included a detailed remit and role description. Applicants were then short listed and invited to take part in an informal interview over the phone, and had the opportunity to ask further questions about the role of the group. To assist in extracting and analysing the information on PPI within the outline applications an access database was developed by CG. The advisory group commented on the data extraction tool and made recommendations for changes to variable descriptors and coding categories. All suggestions were discussed in a face-to-face meeting and agreement reached on their incorporation within the project.

Data were extracted to characterise the cohort and to describe PPI activity within the application process and plans for involvement once trial funding was secured. Trial characteristics linked to trial complexity or thought to be barriers or facilitators to recruitment8 were also extracted. In brief, the extracted data included characteristics of the trial design and setting, proposed start date, disease or condition under study, type of intervention, participant characteristics, recruitment setting and any text that described or was relevant to PPI. Data were extracted by three reviewers (LD, JP and CG). The extracted text was anonymised by replacing any identifying details with a general term in brackets [term], or using [...] to indicate removed text (see online supplementary table S1).

Text extracts were examined to determine whether PPI had been described and if so whether PPI had been undertaken during the development of the outline application (OL), was planned to occur during the development of the full application (F) and/or was planned to occur after a trial was funded (T). The text was also examined to determine the role of the PPI contributor's input. This was categorised as: managerial (M); responsive (R) and membership of Trial Steering Committee (TSC). We categorised PPI contributors as managerial if they were described as co-applicants or involved in the management of the trial. We categorised PPI contributors as having a responsive role if descriptions of their input were largely confined or targeted towards a particular aspects of the application or trial, or if PPI contributions were on an ‘as required’ basis. Descriptions of PPI in the outline applications were often limited, and we were therefore required to devise coding ‘rules’ (based on our knowledge of grant application development and clinical trial implementation processes) in order to categorise the descriptions. The codes were developed by CG and LD after reading the PPI descriptions and then reviewed by JP. CG and LD independently categorised the PPI descriptions. Disagreements between CG and LD were discussed and agreement reached. All categorisations were cross checked by JP. The coding rules (box 1) and the classifications (see online supplementary table S1) were reviewed by the PPI advisory group and no changes were requested.

Coding rules for categorising stage of involvement

If a PPI contributor is described as a member of the research team or ‘lay member’ categorise them as inputting across all future stages of the study.

The design of the study is determined within the full application stage so if a PPI contributor is described as inputting into the design of the study categorise their input as starting no later than at development of the full application.

If a PPI contributor's role is confined to Trial Steering Committee membership (which is usually agreed by the funders and follows the funding decision) categorise their input as starting after the full application regardless of the tense of the sentences describing their involvement.

Coding rules for categorising role

If a PPI contributor's role is described as managerial or as a co-applicant or referred to as a part of the team categorise their role as managerial.

If a PPI contributor's role is confined to a panel or an advisory group categorise their contribution as responsive.

If a PPI contributor's role is limited to a specific aspect of the trial then categorise their input responsive.

The cohort was identified as RCTs that actively received funding from the NIHR HTA programme between 2006 and 2010. In considering trends over time, the year the outline application was submitted was used.

Specific conditions were selected to consider in more detail. These were identified either by their strong history of PPI or the establishment of an NIHR Clinical Research Networks for a specific condition (http://www.nihr.ac.uk/infrastructure/Pages/infrastructure_clinical_research_networks.aspx, last accessed 5 September 2013). The categories were mental health, pregnancy and childbirth, HIV/AIDS, cancer, stroke, paediatrics, diabetes and dementias and neurodegenerative diseases.

Categorical data were summarised using descriptive statistics with numbers and percentages. χ2 Tests or Fishers exact tests were used as appropriate.

Ethical approval was obtained from the University of Liverpool Institutional Ethics Board and the study was funded by INVOLVE and managed by NIHR HS&DR.

Results

Between 2006 and 2010, 111 RCTs received funding from the NIHR HTA Programme. Of these 110 were required to submit an outline application and were eligible for inclusion within the cohort. The dates of submission of the associated outline applications were between 2003 and 2008. Copies of the outline applications were available for 90 (82%) and comments from the Board on the outline application were available for 77 (70%). There were five trials in which there was a disagreement between data extractors, each of which related to the presence/absence of PPI text and each were resolved by referral to the application form.

Table 1 provides a summary of the conditions and interventions. Trials were funded across a wide range of clinical conditions, most of which were for long-term conditions, with the most common area of study being trials in mental health (17%). The majority of trials (82.2%) were aimed at treatment of a condition with 14.4% being prevention trials. The trials used a wide range of interventions. Over a third investigated a medicinal product and just under a fifth each evaluated behavioural interventions (18.9%), surgical techniques (17.8%) and devices (16.7%).

Table 1

Summary of conditions and interventions

	N=90Number of trialsn (%)
Long-term condition	51 (56.7)
Rare condition	2 (2.2)
Conditions expected to reduce lifespan	27 (30.0)
General shortening	14/27 (51.9)
Rapid mortality	13/27 (48.2)
Conditions under study*
Mental health	15 (16.7)
Heart disease/condition	3 (3.3)
Haematology/phlebology	7 (7.8)
Infections	7 (7.8)
Musculoskeletal	6 (6.7)
Cancer	3 (3.3)
Renal	4 (4.4)
Childbirth and pregnancy	2 (2.2)
Obstetrics and gynaecology	2 (2.2)
Addiction	4 (4.4)
Dermatology	2 (2.2)
Gastroenterology	6 (6.7)
Diabetes	2 (2.2)
Obesity/nutrition	3 (3.3)
Falls in the elderly	3 (3.3)
Sleep disorders	2 (2.2)
Stroke	4 (4.4)
Dental	2 (2.2)
Degenerative neurological disorders	4 (4.4)
Respiratory	4 (4.4)
Other	5 (5.6)
Aim of intervention
Treatment	74 (82.2)
Prevention	13 (14.4)
Diagnostic	3 (3.3)
Nature of interventions used*
Drug	31 (34.4)
Behavioural	17 (18.9)
Device	15 (16.7)
Surgery	16 (17.8)
Comparison with conservative management (n=16)	7 (43.8)
Physical, eg, exercise	11 (12.2)
Educational	10 (11.1)
Community care	5 (5.6)
Other	4 (4.4)
Commissioned brief†	45 (50.0)

*Categories not mutually exclusive.

†Topic of research identified by the funders for which collaborative applications are invited as opposed to an open call in which the applicants suggest the topic of research.

Table 2 describes the characteristics of trial participants and features of the trial designs. Over three-quarters of the trials recruited adults only, with paediatric trials accounting for 16.7% of the cohort. The majority of the trials were not gender-specific and approximately a quarter recruited participants at the time of diagnosis. Trial recruitment was most commonly conducted within secondary care (61%). Just under a quarter of the trials involved blinding of the treating clinical team or the participants. Just over 15% of trials used a placebo with all participants receiving an active intervention in one-third of these, indicating the use of a double-dummy design. For example, in a double-dummy trial comparing two treatments A and B trial participants would receive either treatment A and a placebo of treatment B, or a placebo of treatments A and B.

Table 2

Trial participant and design characteristics

Trial participant and design characteristics (N=90)	n (%)
Age group
Adults only	69 (76.7)
Paediatrics only	15 (16.7)
Adults and paediatrics	5 (5.6)
Unclear	1 (1)
Gender
Female	6 (6.7)
Male	0 (0)
Male and female	84 (93.3)
Recruiting newly diagnosed patients	24 (26.7)
Trial recruitment setting*
Secondary	68 (61)
Primary	53 (58.9)
Community	12(13.3)
Emergency	8 (8.9)
Tertiary	6 (6.7)
Social care	6 (6.7)
Blinded trial†	43 (47.8)
Clinician blind	15/43 (34.9)
Participant blind	19/43 (44.2)
Trial involves a placebo	14 (15.6)
If a placebo is involved, do all participants get an active intervention?	5/14 (35.7)

*Categories not mutually exclusive.

†Twenty-three trials in which blinding only related to the outcome assessor.

Of the 90 outline applications for which documentation was available 49 (49/90=54%) provided some information on PPI. Extracts from each application along with the classifications on the stage and role of PPI are provided in online supplementary table S1. Table 3 summarises the classifications across the trials. The first row of table 3 shows that there were 19 applications in which the text provided within the outline described PPI occurring at all three stages (within the outline application, full application and once the trial was funded). Of these 19 applications the role was managerial (M) at each stage in 13. In the remaining six there was variation in the roles across the stages, or it was unclear when a statement indicated that PPI would occur but no detail was provided to allow classification. An example of a description that we were unable to categorise is:

Table 3

Summary of stages of PPI by stage and role

Stage			Number of outline applicationsN (%) n=90	Role			Number of outline applications	Comments
Outline (O)	Full (F)	Trial (T)		O	F	T
Y	Y	Y	19 (21.3)	M	M	M	13	2 had multiple approaches alongside M in the main trial (TSC; TSC plus R)
				R	M	M	1
				R	R	R	2
				R	U	U	1
				U	R	R	1
				U	U	U	1
Y	Y	NS	2 (2.2)	R	R	–	1
				O	R	–	1	O=described as informal contacts
Y	NS	Y	2 (2.2)	R	–	O	1	O=scale development
				R	–	TSC	1	Consulted in pilot study
Y	NS	NS	3 (3.4)	O	–	–	3	O=clinical studies group; survey; Service user Forum
NS	Y	Y	8 (9.0)	–	M	M	3	In 1 also TSC & R in trial
				–	R	R	2	In 1 TSC too
				–	R	M	1
				–	U	U	2
NS	Y	NS	1 (1.1)	–	R	–	1
NS	NS	Y	12 (13.5)	–	–	M	3
				–	–	R	1
				–	–	TSC	7	2 above also listed TSC alongside higher order approach
				–	–	O	1	Piloted and then refined with users
NS	NS	NS	40 (44.9)	–	–	–	40
NS	U	U	2 (2.2)	–	U	U	1
				–	M	M	1	Unclear when PPI initiated but when it starts it is at M

M, managerial; NS, none specified; O, other; PPI, patient and public involvement; R, responsive; TSC, member of Trial Steering Committee; U, unclear; Y, yes.

Investigators have, and will continue to, collaborate with service users.

Twenty-six applications (26/90=28.9%) specified a role of PPI within the development of the outline application. In 13 this was managerial, on a responsive basis in seven, unclear in two and ‘other’ approaches used in four (eg, a patient survey or pilot feedback). Within the ‘other’ approaches it was difficult to determine conclusively whether this was PPI or whether they were examples of data collection aimed at ascertaining the public opinion. In the three applications that specified use of a survey the extent of the distribution of the survey was unclear in two. PPI was planned to occur within the full application for 32 trials (35.6%). In 18 this was managerial, responsive in 9 and unclear in 5. Forty-three (48%) applications indicated that PPI was planned after the trial was funded. This was as: managerial in 22; responsive in 6; a member of the TSC in 8; unclear in 5 and other in 2. Table 4 provides a summary of the specification of PPI within the outline applications by the disease area. The numbers of outline applications by year with and without details of PPI are displayed in figure 1, with figure 2 showing the percentage of applications with PPI. Figure 1 shows a general trend for increasing the number of funded applications; however, the proportion of those containing PPI fluctuates, ranging from approximately one half to two-thirds (figure 2), with the exception of 2003, for which only one application was available.

Table 4

PPI specified within the outline application by disease area

Disease/condition category	PPI details in outline text		Totals	p Value
	Yes	No
Pregnancy and childbirth	1 (50.0)	1 (50.0)	2 (2.2)	0.51*
Cancer	1 (33.3)	2 (66.7)	3 (3.3)
Stroke	3 (75.0)	1 (25.0)	4 (4.4)
Mental health	8 (61.5)	5 (38.5)	13 (14.4)
Paediatrics†	8 (53.3)	7 (46.7)	15 (16.7)
Degenerative neurological disorders	4 (100.0)	0 (0.0)	4 (4.4)
Other	24 (49.0)	25 (51.0)	49 (54.4)
Total	49 (54.4)	41 (45.6)	90 (100.0)

*Fisher's exact test.

†Paediatrics cuts across specific topics. For example, a trial may be in a mental health disorder affecting young people. All such trials were classed as being ‘paediatric’. To avoid double counting two trials in diabetes, three in mental health, and one in neurological were classified within the paediatric category only.

PPI, patient and public involvement.

Figure 1

Number of outline applications by the year in which the application was made (PPI, patient and public involvement).

Figure 2

Percentage of outline applications containing patient and public involvement (PPI) details by the year in which the application was made. The number of trials included within each year is indicated at the top of each bar.

Figure 3 and table 5 provide the data by year for specific conditions. Both of the diabetes trials were in children and were coded as paediatric, and there were no HIV/AIDS trials within the cohort. Figure 3 suggests declining rates of PPI in paediatric and mental health, with other areas including the general other category demonstrating an increase over time; however, as shown in table 5, the numbers in some categories were small.

Figure 3

Cumulative percentages of outline applications by disease/condition (PPI, patient and public involvement).

Table 5

Number of cumulative applications by disease/condition category

	Cumulative number of applications by year
	2003	2004	2005	2006	2007	2008
Mental health	1	4	4	6	10	13
Cancer			1	1	3	3
Paediatrics			5	8	11	15
Degenerative neurological				1	2	4
Pregnancy and childbirth					1	2
Stroke				1	1	4
Other			3	16	36	49

Table 6 shows the associations between trial design characteristics and characteristics of the condition under study, with consideration of PPI within the outline application.

Table 6

PPI specified within the outline application by condition and trial characteristics

Intervention	PPI details in the outline application		TotalN=90 n (%)	p Value
	Yes	No
	n (%)	n (%)
Aim of intervention
Treatment	40 (54.1)	34 (45.9)	74 (82.2)	0.66*
Prevention	8 (61.5)	5 (38.5)	41 (45.6)
Diagnostic	1 (33.3)	2 (66.7)	3 (3.3)
Nature of intervention†
Drug	16 (51.6)	15 (48.4)	31 (34.4)	0.70
Device	7 (46.7)	8 (53.3)	15 (16.7)	0.51
Surgery	4 (25.0)	12 (75.0)	16 (17.8)	0.01
Education	6 (60.0)	4 (40.0)	10 (11.1)	0.75*
Behavioural	11 (64.7)	6 (35.3)	17 (18.9)	0.35
Physical	8 (72.7)	3 (27.3)	11 (12.2)	0.19
Setting†
Primary care	10 (47.6)	11 (52.4)	21 (23.3)	0.47
Secondary care	30 (56.6)	23 (43.4)	53 (58.9)	0.62
Emergency care	4 (50.0)	4 (50.0)	8 (8.9)	1.00*
Community	6 (50.0)	6 (50.0)	12 (13.3)	0.77
Social care	4 (66.7)	2 (33.3)	6 (6.7)	0.68*
Tertiary	4 (66.7)	2 (33.3)	6 (6.7)	0.68*
Blinding
Yes‡	30 (69.8)	13 (30.2)	43 (47.8)	0.01
No	19 (40.4)	28 (59.6)	47 (52.2)
Involved a placebo
No placebo	39 (51.3)	37 (48.7)	76(84.4)	0.17
Placebo§	10 (71.4)	4 (28.6)	14 (15.6)
Received an active intervention
Received Placebo only¶	8 (88.9)	1 (11.1)	9 (10.0)	0.04*
Received an active intervention	41 (50.6)	40 (49.4)	81 (90.0)
Recruitment at diagnosis
Yes	8 (33.3)	16 (66.7)	24 (26.7)	0.02
No	41 (62.1)	25 (37.9)	66 (73.3)
Long-term condition
Yes	31 (60.8)	20 (39.22)	51 (56.7)	0.17
No	17 (47.2)	19 (52.8)	36 (40)
Impact of condition on life expectancy
None	36 (57.1)	27 (42.9)	63 (70.0)	0.72
General shortening	7 (50.0)	7 (50.0)	14 (15.6)
Rapid mortality	6 (46.2)	7 (53.9)	13 (14.4)

*Fishers exact test.

†Categories are not mutually exclusive.

‡Includes 23 trials in which only the outcome assessor was blind.

§Five trials used a double-dummy technique.

¶Groups receiving placebo did not receive an active intervention.

PPI, patient and public involvement.

Of the intervention aims, numbers were too small to draw conclusions about involvement in diagnostic studies, but prevention trials more frequently described PPI than intervention trials. Trials involving educational, behavioural or physical interventions were more likely to have provided details of PPI in the outline application than trials involving drugs or devices. Surgical trials were significantly less likely to have provided details of PPI.

The settings for trial recruitment did not appear to impact on specification of PPI, while recruiting participants at the point of diagnosis was associated with less PPI.

Forty-three trials were described as being blind and these trials were associated with increased frequency of describing PPI. Of these trials 23 involved blinding of the outcome assessor only. Only 14 trials involved a placebo, and of these, participants in 5 trials all received an active intervention with the placebo used as a double dummy. The allocation of a placebo only to one arm of the trial was significantly associated with greater frequency of PPI detail.

Only two trials were in rare conditions. Both of these trials provided detail of PPI within the outline application. While there appeared to be some increase in describing PPI when the condition was long term there did not appear to be any influence based on impact of life expectancy.

Board comments on outline applications

NIHR HTA Board minutes were available for 77 (77/110 70%) outline applications. Only nine (9/77 12%) Board minutes gave feedback on PPI representation and two of these did so indirectly (table 7). One comment was supportive of the PPI described in the outline, which involved a PPI contributor as a co-applicant.

Table 7

Board comments on outline applications relating to patient and public involvement (PPI)

Unique identifier	Year outline application submitted	Text from Board minutes for outline applications
28	2007	The applicants should consider involvement of disadvantaged groups.
65	2006	There was no clear service user involvement and this needs to be addressed.
70	2006	The Board would be pleased to see letters of support from appropriate PCTS and patient groups that the trial is feasible. The Board wish to see patient and public involvement in any full proposal.
34	2008	The Board noted it was a well designed study that has received input from patients.
92	2007	Ethical aspects including acceptability to parents must be fully considered.
39	2008	Consideration should be given to increasing service user involvement.
42	2008	Patient representation is required.
98	2007	An explanation and demonstration of acceptability to parents must be included.
105	2007	The application would benefit from strong patient and public involvement.

The corresponding outline applications were also available for 64 (84%) of the 77 applications for which Board minutes were available. Of these applications 25 (25/64=39%) gave no information about PPI.

Eight of the nine sets of Board comments that were made about PPI expressed the need for applicants to increase PPI. Six contained no detail about PPI within the application. Of these six, two were drug trials, three were exercise interventions and one was comparing an invasive with a non-invasive intervention. Two were recruiting participants with addiction: smoking and alcohol; two were recruiting elderly participants, one was recruiting infants, and one was recruiting participants with a long-term chronic debilitating condition.

In the two outline applications which had given some details on PPI, comments from the Board were to further increase PPI. In one application the PPI contributors were employed within an NIHR condition-specific network in roles relating to PPI. These individuals also had relevant experience as carers or patients of the condition. In the other application the PPI contributor was a co-applicant with an unrelated clinical background (midwife) and was the mother of a child with the condition being studied.

Discussion

Summary

This is the first study to systematically investigate whether and how PPI is described within the earliest stages of trial development across a cohort of studies. The cohort covers a variety of clinical conditions and interventions, and this study is the first to examine associations between trial design, intervention and condition characteristics and applicants’ plans for PPI. Key findings are summarised in box 2. PPI was described in only half of the applications at the outline stage, with only a quarter indicating that PPI contributions were enlisted at the outline application stage itself. There was some evidence to suggest that the further the trial deviates from routine clinical practice the more likely the application is to describe PPI, and PPI was particularly frequent in applications for blinded trials or trials allocating participants to placebo only. This may indicate the beginning of a risk-based approach to PPI. However, applications for trials recruiting at the point of diagnosis or trials with surgical interventions were less likely to describe PPI. For trials recruiting at the time of diagnosis this may reflect the difficulty of identifying PPI representatives close to the time of their diagnosis, while for surgical trials this may reflect the lower frequency and therefore evolution of research practices within their RCTs.

What is already known on this topic

There is an increased awareness of the need for patient/public involvement in research

There is an expectation from funders of research that patient and public involvement (PPI) should be incorporated within that research

There is selective reporting of PPI within the published literature.

What this study adds

A systematic analysis of PPI in a cohort of randomised controlled trials receiving funding from the National Institute of Health Research Health Technology Assessment programme between 2006 and 2010

Only half of applications consider PPI within the early stages of the development of the research, with only a quarter having involvement within the development of the outline application itself

There is evidence that particular conditions, and design considerations, impact on whether PPI is likely to be considered within the early stages of development

There is insufficient consideration of PPI at the early stages by funding Boards.

There were some indications that planned PPI contributions would increase as the application progressed from outline to full application through to funded trial. However, the proportion of applications describing PPI did not appear to increase across the years of the cohort.

Strengths and weaknesses of the study

This study used a cohort of trials funded by the NIHR HTA programme. To be eligible for inclusion within the cohort, applicants were required to have received funding for the trial between 2006 and 2010. Documentation could not be made available for unsuccessful applications. Consequently this study does not link details of PPI within the application to the successful award of the grant. However, while the HTA Programme encouraged PPI during this period, via its guidance notes to applicants and web information, and allowed it to be budgeted for, PPI was not mandatory within the period of the cohort, and that on its own would not have led to a decision that the outline application should not progress. As the cohort was identified by receipt of funding during the period rather than on the year that the outline application was submitted for consideration, fewer trials were available in the extremes of the cohort, but this approach was a consequence of the archiving system. The key limitation is the historical nature of this cohort. The period of the cohort should be considered when generalising the findings to the present day, due to revisions to the guidance for applicants and application forms in relation to PPI. From 2012 HTA boards have started to include PPI membership, and the standard NIHR application form has been introduced, with revised sections requiring applicants to clearly define their PPI involvement with guidance notes to applicants clearly stating that there is now an ‘expectation’ for ‘active involvement’ of patients and the public in the research it supports (http://www.nets.nihr.ac.uk/__data/assets/pdf_file/0005/67685/OutlineGuidanceNotesSAF.pdf, last accessed 28 April 2014). In addition, attitudes towards PPI may have changed and this may influence how early researchers engage with PPI contributors. However, further research is needed to consider this and its potential for impact.

It has been argued that researchers use a tokenistic approach to patient involvement and the level of detail on PPI plans within this cohort may lend some strength to that argument. In order to summarise the details of PPI across the cohort some means of categorising descriptions of PPI activity was necessary. The lack of detail provided within the applications required clear coding rules to be established; however, these were clearly documented. Establishing coding rules was considered reasonable, as within the peer review of an outline application similar assumptions would need to be made by reviewers considering the PPI descriptions provided.

The results should be interpreted with some caution, owing to the small numbers within some categories, and this caution applies to the results of the statistical tests, which were not powered to detect differences and should be considered as exploratory.

Strengths and weaknesses in relation to other studies

There are many examples or accounts of PPI within the published literature focusing on particular trials or topic areas9–12 and concerns have been raised about selective reporting.1 This is the first study to systematically investigate PPI early in the development of funding applications and across a wide range of trials. A previous paper on PPI within clinical trial units13 (CTUs) concluded that involvement appears to be growing. While the majority of applications within the current cohort included collaborations with CTUs, we found no clear trend of PPI activity increasing within this time. A standard operating procedure (SOP) for PPI in CTUs has been recently developed; however, its focus is on PPI after funding has been awarded.14 The SOP suggests that 1% of the research budget be allocated to PPI; however, other more detailed costing models are available.15

If PPI is accepted as the best practice with early involvement, as it provides the greatest opportunity to maximise its contribution, then implementation of PPI and its descriptions within outline applications need to be increased and improved with justifications and methods for the approaches taken. It is noteworthy that no application within the cohort examined provided this level of information. The NIHR HTA programme has made changes to the outline application form, which now requests details of PPI and justification if it is absent. This may lead to increased descriptions but may not be associated with better involvement: a view which has some support from the current guidance, which states, “Whilst patient and public involvement (PPI) may not always be needed for all types of research, it is always relevant for HTA trials. Many PPI sections on the application are unconvincing to our consumer referees and Board members.” (http://www.nets.nihr.ac.uk/funding/hta-researcher-led/Preparing-a-full-application-for-the-Clinical-Trials-and-Evaluation-Board.pdf, last accessed 20 November 2013).

Descriptions should specify how PPI has been incorporated within the development of the outline application and the plans for involvement should the application progress. However, the research community needs to consider the challenges facing researchers in developing PPI at the early design stage. In particular, the often short timeframes within which outline applications have to be developed and PPI contributors identified, and the lack of funding to support this work prior to the grant award, are major obstacles to enhancing PPI at an early stage in trial development. In England, a PPI bursary scheme has recently been launched by some of the NIHR Research Design Service (RDS), but is limited to those in receipt of advice from the RDS and has been shown to be beneficial.16 17 There are few other alternatives to funding development activity at the pre-award stage.15

Although funding body policies support PPI, this support was not usually evident in the HTA Board feedback to applicants at the outline stage. This may be due to the difficulties in assessing PPI given the lack of detail provided within applications, or indicate that at the outline stage the weight of the decision about progression is on the perceived relevance of the question and design. When feedback about PPI was given this did not provide any guidance on how it should be addressed. Statements about the need to improve PPI should be supported with guidance on what PPI contributions would be appropriate within the trial being considered. Peer reviewers and Board members who are asked to comment on PPI should be supported in doing so. Adoption of critical appraisal guidelines may be beneficial in achieving this.18 From 2012 NIHR HTA funding Boards include a PPI representative. The impact of this on the frequency of comments and their content is not yet known.

Our findings also raise questions about what sort of experience qualifies (or disqualifies) individuals to be PPI contributors, as identified within the comments from the Board. Regardless of whether they have relevant personal experience, should individuals with professional clinical or research experience become PPI contributors? Does an individual's experience in a professional capacity somehow weaken their ability to adopt a patient or carer perspective, or does a professional background strengthen a person's ability to contribute to the research? Current NIHR HTA guidance states that “To achieve the aim of bringing fresh eyes to the work of the HTA programme a patient or member of the public should not normally be a health practitioner, manager or researcher” (http://www.hta.ac.uk/PPIguidance/section1_2.shtml, last accessed 26 September 2013).

There is also confusion about whether it is acceptable for individuals who have a PPI role in a professional capacity as well as relevant personal experience to provide PPI. Concerns have been previously raised about PPI representatives becoming professionalised,19 and evidence is needed on whether this is a legitimate concern. This should be rationalised against the fact that PPI representatives are generally individuals who are selected by researchers based on their attributes, and their ability to ‘represent’ the wider population of interest may be questioned, however, this may not impact their ability to communicate their unique ‘lay’ perspective.20

Future research

The lack of detail on PPI at the early stages of research development needs to be addressed. There is a keen interest in determining whether the perceived benefits of PPI lead to tangible benefits for research. The activities of PPI contributors need to be tracked to provide insights on when and what type of involvement may be linked to impact. Further research tracing the evolution of PPI within this cohort is being undertaken.