Development and testing of a medline search filter for identifying patient and public involvement in health research.

BACKGROUND: Research involving the public as partners often proves difficult to locate due to the variations in terms used to describe public involvement, and inability of medical databases to index this concept effectively.
OBJECTIVE: To design a search filter to identify literature where patient and public involvement (PPI) was used in health research.
METHODS: A reference standard of 172 PPI papers was formed. The references were divided into a development set and a test set. Search terms were identified from common words, phrases and synonyms in the development set. These terms were combined as a search strategy for medline via OvidSP, which was then tested for sensitivity against the test set. The resultant search filter was then assessed for sensitivity, specificity and precision using a previously published systematic review.
RESULTS: The search filter was found to be highly sensitive 98.5% in initial testing. When tested against results generated by a 'real-life' systematic review, the filter had a specificity of 81%. However, sensitivity dropped to 58%. Adjustments to the population group of terms increased the sensitivity to 73%.
CONCLUSION: The PPI filter designed for medline via OvidSP could aid information specialists and researchers trying to find literature specific to PPI.

Patient and public involvement in health research has become more prevalent as reflected in the growth in literature on this topic.

Literature on public involvement in health research is difficult to locate due to lack of effective subject headings in databases and inconsistent terminology.

Information specialists could use a search filter to locate relevant PPI literature effectively and thus help systematic reviewers investigating the topic or researchers wishing to use PPI in their own research.

The search filter designed here was tested, adjusted for sensitivity and specificity, and offers an effective way of locating this literature.

Background

The use of PPI (patient and public involvement) in health care research is important because it leads to research that is more relevant to patients and the people who care for them.1 Furthermore, the process of involvement encourages an approach which is more inclusive of multiple perspectives about what research should be carried out, what forms of health care are worth investigating and which health outcomes are important.2 One of the key principles of the National Institute for Health and Care Excellence (NICE) approach to PPI in the UK is that lay people, and organisations representing their interests, should have opportunities to contribute to developing NICE guidance, advice and quality standards, and support their implementation.3 In the United States, the Agency for Health care Research and Quality in the US states that the involvement of stakeholders from the beginning improves research results and ‘helps to ensure that findings are more relevant to users’ distinct concerns and have applications in real‐world situations.4 In Europe, the U‐BIOPRED (Unbiased Biomarkers in Prediction of Respiratory Disease Outcomes) project, funded by the Innovative Medicines Initiative and involving patients throughout, identified five key principles for the success of patient engagement in health research including: early involvement; deep involvement; feedback from patients on the project process; inclusion of patients in dissemination; helping patients convey their own story.5

In research, the importance of PPI is becoming widely recognised, with some funders making it a requirement to obtain funding, for example the National Institute for Health Research (NIHR) for patient benefit funding stream. The NIHR website6 states ‘We expect researchers to actively involve the public in their research. We also involve members of the public in reviewing funding applications and as members of decision‐making committees and panels, making recommendations about research funding’. Internationally, the Cochrane Consumer Network has the primary aim of getting health care consumers involved in the production of Cochrane systematic reviews.7, 8

As PPI in research expands, so does the body of literature about PPI methods, or the reporting of its use in research. The INVOLVE PPI library9 holds two papers from 1995 and 35 from 2014. A search using the PubMed data mining tool PubReMiner for the three words ‘patient, public, involvement’ returns 26 records from 1990, 60 from 2000 and 504 from 2015. Despite this rapid expansion of the literature about PPI, or reporting PPI in research,10 locating this literature is problematic. PPI generally does not form the main focus of the research; it is often reported inconsistently or poorly,11, 12 and the terms used to describe it are varied and contested.8, 13 Furthermore, PPI is a concept which is not precisely defined in the health literature and can have different meanings and values for particular groups.14 For the purposes of this work, the definition as used by the advisory group INVOLVE, which aims to advance PPI in research, is adopted. INVOLVE defines patient involvement as ‘that which is done with, or by, patients and the public, rather than to, for, or about them’.9

The broad definition of PPI used in the literature means that searching for relevant studies poses a number of challenges. Firstly, the terminology is very varied, examples being patient involvement; public participation; public involvement; community engagement; consumer involvement. Secondly, few databases have a controlled term (e.g. Medical Subject Heading or MeSH) for the concept of PPI and, for those that do, there is either no scope note (explanation of what is indexed under the term) available or the scope note is not specific enough. The databases medline, psycinfo and ASSIA all have controlled terms for patient participation. However, only medline has a scope note on its use: ‘Patient involvement in the decision‐making process in matters pertaining to health’, which pertains more to engagement with the individual's own health rather than involvement in health research. Searching using this term alone returns over 18 000 results, covering general communication with the public, participant adherence and patient reported outcomes, as well as PPI as defined by INVOLVE. There are currently no validated search filters for identifying literature on PPI, and the controlled terms assigned to this area on the main databases are either non‐existent or largely impractical. Therefore, the PPI filter designed in this study could be a valuable tool to researchers seeking this type of information.

Search filters

Search filters (also known as hedges) are individual validated search strategies that are designed to retrieve specific types of evidence from bibliographic databases.15 They act as a filter by facilitating the non‐retrieval of irrelevant references while preserving the capture of relevant references. Filters are designed to be either highly sensitive (measured by the proportion of all relevant records correctly retrieved) or highly precise (number of relevant references retrieved as a proportion of the total number of references returned).16 Most search filters focus on methods and are developed to retrieve studies with a particular design17, 18 although topic‐focused filters are becoming more popular.16, 19, 20, 21, 22

Objectives

The purpose of this study was firstly to develop a search filter which would identify either research where PPI was used or where methods of using PPI in research are described, and secondly to test the filter for sensitivity, specificity, precision and ‘number needed to read’ (the number of records needed to screen in order to find one that is relevant) against a body of relevant literature and a ‘real‐life’ systematic review.

Method

The search filter checklist as devised by Glanville et al.23 and Jenkins’ list of criteria for evaluating methodological search filters24 were used as guides for constructing this filter.

Formation of the reference standard

Development of the reference standard was informed by reviewing other filter development literature. The most common method for forming a reference standard was by hand‐searching journals16, 20, 21, 22, 25, 26 although other methods were also used, such as citation chasing or using included references from systematic reviews,16, 20 literature from standalone databases27 or combinations of all methods.28 To identify literature for this reference standard, the libraries of the Peninsula CLAHRC Patient and Public Involvement team at the University of Exeter Medical School and the INVOLVE website9 were searched initially for relevant studies. The reference lists of relevant systematic reviews were then searched for further studies. The journals Health Expectations and International Journal of Consumer Studies were found to be the most common titles for PPI literature identified, so these were then hand‐searched for relevant papers from 2011 to 2013. Two information specialists then independently screened all literature found for inclusion in the reference standard. References were included if they described the use of PPI in health research or provided details of methods for using PPI in health research. They were excluded if the involvement did not meet the INVOLVE criteria, for example if the involvement did not go beyond making decisions about their own health care or patient involvement was only at the level of being a subject of a research study. Disagreements were resolved by discussion.

All papers in the reference standard were analysed for inclusion on three databases (medline, embase and psycinfo) to determine which would be the most appropriate database for the filter. As medline contained more of the references than the other databases it was agreed that the filter should be developed for this database. The records were then divided randomly between two sets: a development set and a test set. Any additional records that were identified during the development process were added to the test set. A chart showing the formation of the reference standard is shown in Fig. 1.

Figure 1

Formation of the reference standard

Term selection

The unique reference numbers for all records in the development set were identified in medline and used to produce a single set of results that could be combined easily with the search terms developed for the filter. Terms and phrases were selected by scrutinising the title and abstracts of the papers in the development set and by examining the MeSH (Medical Subject Headings) thesaurus in medline for relevant terms. Each phrase and MeSH term was individually tested for sensitivity using the development set of references. Terms describing participation (e.g. involvement, engagement and partnership) were placed in various combinations with terms for the consumers (e.g. patients, public, users) and also tested for sensitivity against the development set of references.

The phrases and headings that were found to be the most sensitive were then tested in various combinations until 100% sensitivity was achieved against the development set. The sensitivity of the final combination was then examined against the test set. Tests were run in January 2014.

Testing the filter

The unique reference numbers for the papers in the test set were identified and used to form a single set of results. This was combined with the PPI search filter using the Boolean command AND to establish the sensitivity of the filter against the test set.

Specificity, precision and NNR (number needed to read) were tested using a ‘real‐life’ systematic review carried out by Bailey et al.29 which explored involving children with disabilities. The review used for the second part of testing was chosen because it was carried out in‐house, which meant access to complete and accurate figures for numbers of records that were retrieved, screened and included. Papers included in the review were checked for inclusion on medline. The original search was rerun on medline via OvidSP in January 2015, and new results were screened by two reviewers (AB & KB) for inclusion. The results from the original systematic review search and the new results formed a second test set. The SR search was run again in March 2015 with the filter attached. Results were compared to the original results and the second test set in order to establish specificity, precision, NNR and to retest sensitivity. Definitions used were taken from Terwee et al.30 and are as follows:

Sensitivity: The number of relevant records in the reference standard retrieved by the search filter as a proportion of the total number of relevant records in the reference standard.

Precision: The number of relevant records retrieved as a proportion of the total number of records retrieved.

Specificity: The number of records that are not relevant and are not retrieved as a proportion of the total number of records that are not relevant.

Number needed to read: the number of records that need to be read to identify one relevant record.

Results

74% (n = 127) of the records from the reference standard were indexed on medline (38% were on embase and 30% on psycinfo). The filter was therefore developed for medline using the platform OvidSP. From the 127 records found on medline, 59 were randomly selected for filter development, and the remaining 68 were used for testing the filter.

Two MeSH terms were identified that covered the inclusion criteria: ‘consumer participation’ and ‘patient participation’. Other related phrases, for example ‘public user involvement’ or ‘service user involvement’, cross‐referenced back to these headings when searching MeSH terms. Although ‘patient participation’ was a subheading of ‘consumer participation’, the sensitivity of both headings was tested against the development set to establish whether both were required in the filter. As standalone headings, they retrieved references from the development set that were unique from each other and therefore both needed to be included (alternatively, exploding the term ‘consumer participation’ would generate the same results as there were no other subheadings assigned to this term). Table 1 shows the sensitivity of the search terms and MeSH headings identified during development.

Table 1

Most frequent terms and phrases identified in the development set

Search terms	Sensitivity number of key records (% retrieved)	Total number of records retrieved
MeSH (Medical Subject Headings)
Consumer participation	34 (58)	14 548
Patient participation	17 (29)	18 396
Search phrases (in the title and abstract fields)
Public involvement	11 (19)	344
Consumer involvement	10 (17)	221
User involvement	9 (15)	431
Participatory research	7 (12)	1981
Patient participation	4 (7)	1346
Patient involvement	3 (5)	1084
Public participation	3 (5)	461
Consumer participation	2 (3)	233
Public engagement	2 (3)	274
User participation	2 (3)	131
Community engagement	1 (1)	654
Community participation	1 (1)	2125
Community involvement	1 (1)	1166

Terms describing participation (e.g. involvement, engagement and partnership) were placed in various combinations with terms for the consumers (e.g. patients, public, users). The terms ‘involvement’ and ‘participation’ were more commonly used than ‘engagement’ or ‘partners/partnership’, but population terms (e.g. public or patient or user) were more varied throughout the development set papers. Combining ‘involvement’ with the population terms (and the words ‘health’ or ‘research’) retrieved the most key records (see Table 2).

Table 2

Records retrieved using combinations of search terms

Combinations of search terms	Number of key records (% retrieved)	Number retrieved in total
Participat* AND (patient* or lay or people or public or consumer* or user* or citizen*) AND (health or research)	29 (49)	49 379
Involv* AND (patient* or lay or people or public or consumer* or user* or citizen*) AND (health or research)	44 (75)	73 863
Engag* AND (patient* or lay or people or public or consumer* or user* or citizen*) AND (health or research)	16 (27)	15 010
(Partners or partnership) and (health or research)	9 (15)	27 929

The final combination of terms as formed during the development stage is shown in Fig. 2. The filter found 67/68 records from the test set (sensitivity 98.5%) during initial testing and demonstrated high specificity (85%) when tested against systematic review results.

Figure 2

Initial patient and public involvement search filter for medline via OvidSp (high specificity)

The sets for testing specificity, precision, NNR and further sensitivity testing, using a published systematic review27 were derived as shown in Table 3.

Table 3

Sets used for additional testing

Set	Description	Number of hits
N1	SR search run on medline via OvidSp	1485
N2	Relevant medline records from previous search and update	26
N3	medline search with added filter	287
N4	Relevant records from N3	15
N5	Irrelevant records not retrieved	1187

The results of the calculations for the stage 2 testing (prior to and following adjustments being made to the search terms) are shown in Table 4.

Table 4

Results from stage 2 testing

	Calculation	Initial results	Results after adjustments made
Sensitivity	N4/N2	0.58	0.85
Specificity	N5/N1‐N2	0.81	0.47
Precision	N4/N3	0.052	0.025
NNR	N3/N4	19.1	39.4

The filter was found to have a sensitivity of 58% and specificity of 85% when tested against the systematic review. Adding more terms raised the sensitivity to 85% (specificity 47%). The adjusted filter for high sensitivity is shown in Fig. 3.

Figure 3

High sensitivity version of the patient and public involvement search filter for medline via OvidSp

Discussion

This study described the development of a sensitive filter for medline via OvidSP to identify PPI literature in health research. The filter was successful in identifying 67/68 records in a test set with a specificity of 98.5% but failed to pick up one reference from the test set.31 The PPI group in this study was specifically referred to as mental health service users, but the population in this case would have been captured by the term ‘user’ in the filter. Both the terms ‘health’ and ‘research’ were present in the abstract, so the missing element was in the activity group of terms (involvement etc.). Simply adding ‘conducting’ to the terms reflecting involvement would have captured this final reference. Nevertheless, the finding highlighted the fact that the population could be described more in terms of their condition (e.g. people with diabetes, or adults with mental health issues), rather than more generally as patients, public or service users. It is possible therefore that the filter could miss key papers and although none like this were found to be omitted here, further development and testing on a large body of literature should be carried out if studies are identified as being missed for this reason. The filter would in these cases be more effective if terms specific to the population were added.

The specificity results were better than anticipated (85%). However, when tested with the search strategy used for this review, sensitivity of the filter dropped considerably to 58%. One of the reasons for this drop was that the population of interest in the study was children, which meant that several of the relevant studies used the terms ‘children’ or parents’ for the population of interest, rather than ‘public’ or patient’. Adding ‘child* or parent*’ to the population terms increased the sensitivity to 73%. Additional terms such as ‘interviews’, ‘questionnaires’ and ‘self‐report’ further increased the sensitivity while not affecting the results from the initial stage of testing. Finally, adding terms for participation (e.g. consulting and collaborating) increased the sensitivity of the filter to 85%.

In all tests, precision was low. Specificity is a more accurate measure of effectiveness as it takes into account irrelevant records that were not retrieved, whereas precision only measures the proportion of relevant records in the total retrieved. Where topics such as PPI in research form a small proportion of studies in any topic area, there will always be a difficulty in designing a search that pinpoints them. It is hoped that this work will go some way to alleviating this issue, for this particular topic.

One major problem with locating PPI literature is that PPI in studies often form a relatively minor part of the methods and therefore will not be reported in the title or abstract fields. Furthermore, research has highlighted that the quality of PPI methods reporting is poor11 which could render this literature even harder to find. The filter created here is only partly able to cope with this problem using subject headings (MeSH); however, these rely on an indexing process that is not fully comprehensive and cannot be relied upon given the irrelevance of many of the records that are indexed under these headings. In addition, most databases are currently unable to run free text searches across the full text of papers, so unless relevant terms appear in the title or abstract fields, or the indexing process identifies the concept of PPI being included in a study, relevant papers will be missed. As the importance of PPI becomes more widely recognised across all fields, it is hoped that the indexing of this concept onto databases becomes more efficient, reflecting PPI better in the subject headings assigned to published studies and therefore allowing the retrieval of papers where PPI is not the primary subject.

It is important to note that this filter has been designed for medline via OvidSP only. Where the focus of a study is not predominantly medical, for example nursing or allied health professions, the filter will need to be adjusted for other databases such as cinahl.

A further issue identified during this study was the capture of literature concerning self‐management, adherence to treatment and patient empowerment, but specifically with regard to managing their own health care. The filter does not provide a solution for isolating PPI in research literature from the literature on these similarly termed, but vastly different subjects.

This study addressed the challenges of locating PPI literature for the purposes of health research. There are many other areas of health care where PPI is of paramount importance, such as in policy making or in participating in decisions about individual health. Although studies like this were not filtered out, it might be the case that future research could develop a filter that would be more effective for locating PPI studies in this different context.

Conclusion

PPI literature is hard to locate due to it often not forming the main concept of a study, being poorly reported within research, and poor indexing of studies and assignment of subject headings to publications. It is also difficult to isolate from literature on self‐management of health and disease. The development of the filter in this study should aid with identifying literature where PPI was used in specified areas of health research. The filter would be most effective in searches for PPI in general populations and should be amended to include specific populations of interest, if required. Although this filter will help to locate PPI literature, the issue of locating research that is hidden away in the body of the text will remain until indexing of PPI literature on medical databases is improved, or until full text searches can be carried out comprehensively within the major databases. This study will be of interest to both librarians and other information professionals who design and run literature searches in this field, researchers that specialise in PPI research and those working in the health sector seeking studies involving PPI in research.

Funding

This study presents independent research funded by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) for the South West Peninsula. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health in England.