Creating a database of internet-based clinical trials to support a public-led research programme: A descriptive analysis.

BACKGROUND: Online trials are rapidly growing in number, offering potential benefits but also methodological, ethical and social challenges. The International Network for Knowledge on Well-being (ThinkWell™) aims to increase public and patient participation in the prioritisation, design and conduct of research through the use of technologies.
OBJECTIVE: We aim to provide a baseline understanding of the online trial environment, determining how many trials have used internet-based technologies; how they have been used; and how use has developed over time.
METHODS: We searched a range of bibliographic databases to March 2015, with no date limits, supplemented by citation searching and references provided by experts in the field. Results were screened against inclusion and exclusion criteria, and included studies mapped against a number of key dimensions, with key themes developed iteratively throughout the process.
RESULTS: We identified 1992 internet-based trials to March 2015. The number of reported studies increased substantially over the study timeframe. The largest number of trials were conducted in the USA (49.7%), followed by The Netherlands (10.2%); Australia (8.5%); the United Kingdom (5.8%); Sweden (4.6%); Canada (4%); and Germany (2.6%). South Korea (1.5%) has the highest number of reported trials for other continents. There is a predominance of interventions addressing core public health challenges including obesity (8.6%), smoking cessation (5.9%), alcohol abuse (7.7%) and physical activity (10.2%); in mental health issues such as depression (10.9%) and anxiety (5.6%); and conditions where self-management (16.6%) or monitoring (8.1%) is a major feature of care.
CONCLUSIONS: The results confirm an increase in the use of the internet in trials. Key themes have emerged from the analysis and further research will be undertaken in order to investigate how the data can be used to improve trial design and recruitment, and to build an open access resource to support the public-led research agenda.

Introduction

Online clinical trials are rapidly growing in number. This novel method for assessing the effectiveness of health interventions offers tantalising potential benefits to researchers, patients and society, such as: reducing the cost of studies; reducing the marginal cost of recruiting additional participants, enabling larger sample sizes and reducing random error; improving the representativeness of trials by enabling the involvement of participants who tend to be neglected by current research methods; improving recruitment to trials; improving stakeholder involvement in trials; cheaper treatments; permitting easier long-term follow-up of participants; allowing novel questions to be addressed that would not have been logistically possible to test in traditional trials; and improved dissemination of results. However, the online method also brings potential threats such as: the ease of setting up trials leading to poor quality assurance; ease of recruitment leading to large attrition rates and uninterpretable results; contamination between arms due to social media or communication between participants; the technological challenges of doing research online leading to failed trials; and exposure to hacking and potential breaches of participant confidentiality. It is important, therefore, to evaluate this new technology to identify the actual advantages and disadvantages of online clinical trials in practice and to learn what factors can help maximise benefits and minimise risks.

ThinkWell

The International Network for Knowledge on Well-being (ThinkWell™)[1] is a not-for-profit organisation set up by a group of consumers, researchers, health service users and clinicians in 2006 with the mission of using modern communication technologies to enable the public and patients (a) to make health decisions informed by the best available research evidence and (b) to participate in health research on the issues that affect the day-to-day decisions of individuals, involving the public in the prioritisation, design and conduct of research. ThinkWell’s focus is on interventions that people can choose to do for themselves, without the involvement of a health care professional (such as exclusion diets, online psychological therapies, lifestyle choices, dietary supplements, etc.), and the organisation is committed to an evidence-based approach and stakeholder partnership throughout the research process. The type of intervention ThinkWell is interested in (self-management) and the fact that all research is designed and undertaken in partnership with the public and patients means that online trials are a particularly appropriate methodology to address proven uncertainties in this area. Therefore, it was important for ThinkWell to learn from best practice in online health trials regarding how to achieve meaningful stakeholder collaboration to inform the methodological, safety, and ethical considerations specific to the online environment. However, when ThinkWell started work in this area there was little published evidence about these issues. We decided to begin by looking at existing online trials to learn about their advantages and disadvantages; how stakeholders had been involved; and identify good and bad practice in conduct and reporting. A first step was to systematically identify as many relevant online clinical trials as efficiently as we could. This led to the creation of the Online Randomised Controlled Trials of Health Information Database (ORCHID). This paper describes the rationale and methods of creating ORCHID.

Rationale for public-led trials

Why involve the public and patients as partners in health research?

Measures to improve the health of citizens by enabling them to make better-informed decisions about their health and well-being have been supported by national public health initiatives, media campaigns, the development of patient education resources, and tools to support shared decision making. An increase in the use of internet and mobile technologies as a channel for health-related information and advice has led to the development of computer-supported decision aids[2-4] and other self-management initiatives,[5] in tandem with the informal explosion of social media applications,[6] through forums and discussion groups,[7] and communication activities designed to promote behaviour change.[8] Pew internet research states that 35% of adults have searched online to find health information,[9] and in the UK government policies to tackle issues such as smoking cessation and obesity, self-management of long-term conditions, and improvements in mental health have been implemented through national digital initiatives such as Change4Life[10] and NHS Choices.[11]

Internet technologies may provide an efficient and cost-effective mechanism for the delivery of good-quality health information, public health-focused interventions, and health and social care transactions. The rapid rise in the use of smart phones, tablets and an increase in the accessibility of mobile internet technologies has led to the promotion of a ‘digital first’ strategy for the Department of Health in England,[12,13] which has the stated aim of making all public-facing transactions in health and social care digital by default. However, there are concerns about a lack of research into the impact of digital technologies in terms of harms;[14,15] on issues relating to digital skills;[16] self-diagnosis;[17] and on health inequalities.[18-20] Mobile technologies may provide an opportunity to support community engagement,[21] and although systematic reviews report that community engagement initiatives can impact positively on policy and practice, on health, and on public health outcomes, there is a lack of strong evidence on whether there is a beneficial effect on health inequalities, comparative effectiveness between models, or on cost-effectiveness.[22] More high-quality syntheses or studies are needed to inform reliable policy and practice about whether the increasing use of internet-based technologies has the potential to improve health equity.

In addition to the increased awareness of the need to promote better-informed and shared decision making, there has also been an acknowledgement of the need for a corresponding increase in patient and public involvement (PPI) in policy making, planning and service delivery,[23-26] health and social care guidance,[27] patient safety,[28] and importantly, in the research process itself.[29] Lack of meaningful stakeholder involvement has the potential to add to the waste identified in the research process by,[30] and represents a lost opportunity to harness new technologies for, socially led research.[31]

It has been suggested that without full involvement at all stages of the research process, much of the research activity that is conducted will not answer the questions that are most important to patients and the public,[32-35] and that a more participative research process will lead to better quality and more relevant research output. A range of policies and programmes have been implemented with the aim of improving PPI in health-related research, although there are concerns that this has been largely driven by a researcher-led agenda.[36] Chalmers[37] suggests that most health-related research is commissioned by policy makers and designed mainly by researchers, and often only involves patients and future participants after its initiation. Failure to involve potential participants in the prioritisation and design of research may partially explain why trials fail to recruit sufficient numbers or encounter a lack of cooperation from those who enrol patients at the point of care. The use of social networking sites to increase public involvement in health research and practice has been increasing, and guidance has been published to aid researchers;[38] however, further exploration of the full potential is required. Studies are often observational in design and lack the capacity to fully explore the interactional potential of the media.

Randomised controlled clinical trials are the current methodology of choice for testing the effectiveness of interventions because they minimise potential biases (for example, pre-existing differences in the groups being compared). However, there are known limitations and problems associated with their conduct, particularly in the area of public health and well-being. Problems include failures to recruit sufficient sample sizes, the costs associated with running large trials, problems in follow-up, and relevance.[39,40] This may impact on the ability to provide answers to the most important questions for patients, the public and clinicians, but also on the successful involvement of patients and the public in trial design and conduct,[41] with a lack of engagement contributing to concerns about poor reporting, publication bias and the suppression of information about adverse events and harms.[42] This lack of transparency undermines the confidence placed in reported findings[43-45] as well as providing another barrier to increased public involvement. These concerns have led to a number of initiatives to improve the quality of research evidence through synthesis in systematic reviews, better reporting standards[46] and a commitment to make patients aware of research in the NHS Constitution in England,[47] although the failure to register clinical trials remains a matter of serious concern.[48]

Consideration of these issues has led to specific initiatives that aim to harness the development of new technologies as a mechanism to both improve the useful outcomes of clinical research, built on the principles of involvement and through the development of public-led research.

While a growing interest in the use of internet technologies in general within clinical trials has been identified from an early stage in the development of the medium,[49,50] studies using internet technologies vary in the aspect and degree to which these tools are employed at different stages of the trial process, including recruitment, gaining consent, data collection, and the dissemination of results. As the adoption of internet-based technologies in clinical trials has increased rapidly, it has led to unstructured approaches to reporting and indexing, which in turn leads to problems in the retrieval of relevant studies, and in the assessment of quality and reliability. Murray[51] identified a set of issues concerning the use of the internet in clinical trials, and has suggested that online trials share common issues with traditional trials, such as recruitment and retention, randomisation, data quality and fidelity, but face additional challenges related to their technological characteristics, including spamming, data theft, and ‘cybersquatting’.

In a rapidly changing technical environment, the problems associated with the ability to be able to replicate interventions in reports of clinical trials[52] may be an even greater challenge.[53] Internet technologies have the potential to be used in all stages of trial conduct; however, scoping searches reveal that the most common application in the reports of clinical trials is in the delivery of the intervention or comparison, such as in studies investigating substance misuse;[54,55] smoking cessation;[56] weight loss;[57] and in the use of cognitive behavioural therapies.[58] The growth of this type of trial is reflected through an increasing number of published systematic reviews of these types of studies,[59-62] and also in the appearance of national guidance.[63] However, there are currently no agreed standards for the description or indexing of these concepts in individual studies, or in systematic reviews.

Given the lack of a robust evidence base, it was decided that a more systematic approach to the identification of uncertainties in relation to internet-based clinical trials would be useful for the emerging ThinkWell programme and for other researchers in the field of health research. UK DUETS[64,65] was established in 2007 in order to identify, collate and publish uncertainties about the effects of treatments. Information in UK DUETs not only helps health professionals identify treatment uncertainties quickly, but can also help those responsible for promoting and supporting research focus on important gaps in knowledge and the unmet information needs of patients and clinicians. A project to adapt the UK DUETs methodology to investigate the most important questions in relation to the use of internet-based technologies in health and well-being research was piloted, and analysis of the initial results showed that a number of important questions could benefit from further research.[66] These included areas such as data security, data collection, self-report issues, and technological solutions, but also raised broader questions concerning ethical, methodological and qualitative perspectives. It was unclear how many clinical trials had already been conducted using internet-based approaches, and to what extent they had been used in specific stages or elements of the trial process. We set out to consider the epidemiology of internet-based clinical trials, with the aim of establishing some core baselines and a better understanding of current activities, through a project to identify, classify, record and analyse reports of trials.

The ORCHID project

The ORCHID project was proposed to create a repository of internet-based clinical trials in health and well-being topics, to provide a baseline understanding of the nature of the internet-based clinical trial environment. We propose to use ORCHID to mine for trends in trial topics and types of interventions, to identify methods used in online trials, to explore how and when PPI is used, as well as any other emerging trends in the development of online trials.

The initial project aimed to address the following questions:

How many randomised controlled trials have used internet-based technologies as part of their trial processes?

In what stages of the trial process have internet-based technologies been used and how might these be characterised?

How has the use of internet-based technologies in randomised controlled trials developed over time?

The mapping process aimed to establish the relevant concepts and the size of the literature, and will feed into the process of establishing priorities for research in this area alongside further development of the ThinkWell uncertainties harvesting project.

Methods

Definitions and scope

One of the major impediments in identifying internet-based clinical trials is related to the lack of a consistent taxonomy, and the wide range of terms used in both the reporting of the trial and the quality of indexing of trials in bibliographical databases. For the purposes of this project, the following definitions were used for key concepts.

Internet-based

The ‘internet’ concept is defined in the Oxford English Dictionary as “a global computer network providing a variety of information and communication facilities, consisting of interconnected networks using standardized communication protocols”. For the purposes of this research, the term ‘internet-based’ was used predominantly to refer to applications which use the world wide web as a primary means of communication or data entry, including the use of mobile technologies where these are used in combination with the internet.

Randomised controlled trials

The first phase of the database development focused on identifying a core set of reports of internet-based randomised controlled trials. To be considered for inclusion in the initial development phase, reports of studies needed to include reference to a randomisation process in the abstract, to be indexed with a relevant publication type, or to have ‘randomised controlled trial’ included in a declarative title. No assessment of trial quality was undertaken at this stage.

Health and well-being

The potential range and scope of this topic is extremely large, and the concept of ‘health’ is subject to a range of different interpretations and definitions. The focus for the initial database development was therefore limited to reports of trials relating to interventions in diagnosable illnesses or disorders, including non-communicable diseases and conditions. ICD10 categories were used to provide additional guidance for the inclusion or exclusion of borderline topics, for example in the area of addiction. For the term ‘well-being’, there is even less consensus concerning how this should be defined, particularly in the boundaries between the concepts of ‘mental health’ and ‘well-being’. The recent government strategy for mental health No health without mental health; a cross-government mental health outcomes strategy for people of all ages, 2011[67] provides the following definitions:Factors affecting adult health, such as sexual and reproductive health and drug misuse, were included, but only where there was a defined health care, or public health, intervention. In order to manage the scope of the initial phase of development, other social determining factors such as housing, transport and planning, or broader environmental health; issues around ageing; domestic violence; or accidental injury were excluded.

Good or positive mental health is more than the absence or management of mental health problems; it is the foundations for well-being and effective functioning both for individuals and for their communities.

Well-being is a positive state of mind and body, feeling safe and able to cope, with a sense of connection with people, communities and the wider environment

Trial participants

Trials would be included if the participants were patients or members of the public. Trials dealing exclusively with health care professionals would be identified but not included for analysis at this stage.

Identifying and selecting studies

Search methods

A broad and comprehensive search strategy was developed, and a wide range of sources used in order to retrieve relevant studies. The following sources were searched to identify relevant studies: MEDLINE; EMBASE; PsycINFO; CINAHL; The Cochrane Controlled Trials Register (CCTR); Physiotherapy Evidence Database (PEDro); OT Seeker; ERIC; and LILACS. The search strategies are listed in Appendix 1.

Given the wide range of databases searched, their different functionalities and tools, the search strategy was adapted for each source. For health-related databases the ‘internet’ concepts were combined with a methodological filter for randomised controlled trials. For non-health-related databases the ‘internet’ concepts were combined with health or well-being terms, as well as any relevant randomised controlled trial concepts or methodological filters if available.

It was determined that although the preference was for conducting a comprehensive search in a wide range of sources, i.e. not just MEDLINE and EMBASE, in some smaller databases it would be more pragmatic and cost effective to use a single term or less sensitive approach, if a more precise or tested filter were not available. This would be supplemented by hand searching of journals, personal contacts, and citation and pearl searching of key papers. No specific date limits were applied to the search, due to a lack of clarity as to when trials first started to use the internet as a tool, and also due to the variable rate of development of different internet technologies.

Health and well-being research and randomised controlled trials are international in scope and relevance, and therefore no language limits were applied. However, reports of research in languages other than English were only included if an English abstract was available.

Search terms were derived from MeSH, EMBASE, and other relevant taxonomies; from core articles identified from scoping searches; and by experts in the field. A process of text word analysis was undertaken, and the identified relevant thesaurus and free-text terms, and combinations of terms, tested where possible. As previously reported, there is a very wide variation both in terminology, definition and application of internet-based technologies within the research environment, resulting in a complex set of activities which are not well controlled in terms of taxonomy or indexing.

Internet-based technologies

The term ‘internet’ itself represents a poorly controlled concept, and as use of the technology itself has increased over time, the term has become almost ubiquitously used in study reports. For example, the terms “web” and “online” are frequently retrieved as text words due to statements such as “we searched the web for xx”. Some of the terms that were tested retrieved very large numbers of irrelevant results, for example, using “e-” as a prefix to terms such as “e-research” retrieves examples such as “Vitamin E”. Where possible, controlled vocabularies were used; for instance, the MeSH term “internet” was created in 1999, with the previous indexing term of “computer communication networks” becoming a broader term. All subheadings of “internet/” were included to increase sensitivity.

Relevant search filters for randomised controlled trials were identified for MEDLINE, EMBASE, PsycINFO and CINAHL. In order to ensure that a systematic approach was taken to the choice of filter, a literature search was undertaken to identify what was known about specific filter characteristics and performance.[68-71] Poor and inconsistent indexing within and between databases made it difficult to compare the performance of specific terms and filters. This meant that the choice of methodological filter to identify reports of randomised controlled trials would have a high impact on the number of retrieved records. Test searches were run to estimate likely outcomes which resulted in poor performance in terms of the precision of the tested filters, and the very broad and sensitive nature of the other concept terms of the search led to the potential retrieval of a very large number of results. Therefore although the preference was to use a highly sensitive search strategy in order to increase sensitivity, a pragmatic approach had to be taken to achieve maximum balance between sensitivity and precision.

The initial database searches were run in January 2011, with further update searches completed in September 2013, and January 2015. The combined searches identified 21,813 results, which were downloaded to EndNote, and then transferred to the SENTE software package for screening and indexing. The results were then screened using the inclusion and exclusion criteria. Results were initially ordered by year of publication, to identify how the number had increased over time.

Selection of studies

Stages of the trial process

The database was developed to identify at which stage of the trial process internet technologies have been used. This would include use of the internet to gain informed consent, self-enrolment, the delivery of the intervention, data collection and analysis, and outcome measurement. Given that the use of internet tools for trial registration, randomisation, publicity and promotion, or dissemination has become standard practice in most clinical trials, reports of studies where the only reference to the use of the internet is related to these concepts were excluded.

Iterative classification and labelling

The retrieved studies were then indexed to record their core characteristics, including the country the trial took place in, the conditions being investigated, and the interventions being tested. Initially studies were indexed using the terms applied by the source database, such as MeSH. Given that the source databases use a range of different controlled vocabularies, and the lack of consistency of indexing within databases, a pragmatic, hybrid system was developed. This entailed a two-stage process, where the condition and intervention indexing terms were extracted from the relevant field in each study report, using the classification label provided by the source database. This was then mapped against the existing set of codes, and an iterative process. If a relevant code was already in the structure, this was applied. If no relevant code was available, a new code was created using the terminology from the study report in question. When the indexing was completed, the results were scanned for consistency and re-labelled where appropriate. As it was envisaged that detailed analysis within specific condition or intervention areas would be undertaken by topic experts as part of secondary projects, no further work on the classification was undertaken at this stage.

Results

The current number of identified internet-based randomised controlled trials in the database as of January 2015 is 1992.

Identified studies

Date of publication

The number of reports of internet-based randomised controlled trials retrieved and added to the database by year of publication increased as expected over time, with the majority published between 2007 and 2015 (Figure 1).

Figure 1.

Cumulative growth in internet-based RCTs.

Geography

The largest number of trials reported as using internet technologies were conducted in the USA (49.7%), followed by The Netherlands (10.2%), Australia (8.5%), the United Kingdom (5.8%), Sweden (4.6%), Canada (4%), and Germany (2.6%). South Korea (1.5%) has the highest number of reported trials for other continents.

Topic

The topics most often studied were those reporting interventions relating to non-communicable conditions such as obesity, smoking and substance abuse, depression, anxiety and phobias. The second largest group contained conditions where self-management is already a major feature of clinical care, for example in diabetes and cardiovascular conditions. The fastest growing fields of interest in the period between 2010 and 2014 were in treatments for depressive disorders and obesity (Table 1).

Table 1.

Highest ranked topics or condition concepts.

	No of trials	% of included trials
Topic
depression	218	10.9
obesity	171	8.6
cancer	162	8.1
alcohol abuse	154	7.7
diabetes	133	6.7
cardiovascular diseases	131	6.6
anxiety	111	5.6
stress	79	4
phobias	49	2.5

Intervention

The highest numbers of concepts indexed for study interventions related closely to the nature of the conditions, as reported previously; for example, behaviour change therapies, such as cognitive behavioural therapy for anxiety and depression; smoking cessation interventions, and interventions to increase physical activity (Table 1). The ranked concepts reflect the most specific terms as indexed by the source database, and are therefore an indication of the intervention and not grouped by permuted index (Table 2).

Table 2.

Highest ranked intervention concepts as indexed.

	No of trials	% of included trials
Intervention
Psychotherapy	674	33.8
Behaviour change	613	30.8
Education	398	20
Cognitive behavioural therapy	376	18.9
Health promotion	335	16.8
Self-management	331	16.6
Communication	214	10.7
Physical activity	204	10.2
Health information/literacy	167	8.4
Monitoring	162	8.1
Social support	157	7.9
Feedback	156	7.8
Weight loss	155	7.8
Web-based care	135
Risk assessment	130
Smoking cessation	117
Counselling	109
Nutrition	105

Discussion

A number of key themes have emerged from the initial analysis of the study reports that have been included in ORCHID. These themes reflect the rapid growth of the use of internet technologies in health and well-being research, but also raise issues which require further investigation.

Location of trials

It is clear from the analysis of the trial location mapping that a high percentage of internet-based trials were conducted in a small number of countries. This may reflect existing publication bias in the scientific literature, and in the selection of journals indexed in bibliographic databases, previously found in studies of country and language of publication, such as in MEDLINE, where in a 60-year period the USA (22.2%) and United Kingdom (12.5%) produced most publications reporting randomised controlled trials.[72] Of 468,191 journal articles added to MEDLINE in 2000, 67.9% were initiated in the ‘Anglo’ countries.[73] This is confirmed in some respects from the ORCHID data, although we have observed a trend where a small number of key centres of excellence have developed, with expertise in the conduct of specific types of internet-based clinical trials. This may reflect the interest and publications of one research group or researcher, who publish multiple studies on either the same set of trials or sequences of trials on similar interventions, and may skew the position of that country in the overall table.

Expertise in the conduct and evaluation of internet-based clinical trials has developed in small pockets, and this learning could be shared to benefit others in terms of the methodological and economic advantages. Improvements in the ability to recruit and retain trial participants could help in a number of environments, including large-scale public health trials, and in particular in the availability and conduct of research in under-developed countries. At this moment in time research using these technologies tends to favour the diseases and conditions of high-income countries.

Type of topic and intervention

There is clearly a predominance of results in two main topic areas. These are those relating to modifiable risk factors for preventable causes of death and illness that impact on public health priorities, such as tobacco, obesity and lack of physical activity, and alcohol and substance abuse; and mental health concerns such as depressive disorders, anxiety and phobias.

It is not surprising, then, that the most common interventions in the database are those related to existing treatment options for these conditions, and where the delivery of the intervention transfers more easily to an internet-based mode of delivery, such as those for weight loss and smoking cessation programmes, talking therapies and physical activity interventions. It is not known how much the nature of the technology may pre-dispose to a particular choice of intervention.

To improve the relevance of treatment options being tested, future research in partnership with patients and the public could be based on uncertainties and the most important questions for patients, carers and practitioners. This could ensure that the technological options under consideration are driven by research need rather than by researcher preference. It is not clear whether the predominant topic areas reflect the most relevant use for internet-based technologies, or whether those particular researchers bring their interest in the technology to bear on their subject through a preference or interest in that technology.

Stage of trial

The majority of internet applications represented in ORCHID are used in the delivery of the intervention. Some 669 studies were indexed as reporting use of the internet for data collection, and 387 indexed as self-reporting. A large number of the most popular interventions use the internet to enable participants to self-report data, but there is not yet a body of trials using this in non-behavioural trials. Sixty-four studies were indexed as reporting use of the internet for data analysis or follow-up. Very few of the studies identified use internet technologies in other stages of the trial process. Some 176 studies were retrieved that reported on the methodological issues related to internet-based trials. These were not entered into the database, but were incorporated into work to conduct a methodological systematic review. Further work will be undertaken by ThinkWell researchers to investigate specific internet applications within these trials, and to develop more specific taxonomies.

Limitations

The large number of results from the US, UK and Australia, and therefore the predominance in English-speaking, high-income countries, may be due to publication bias in terms of reported studies, journal indexing and choice of databases searched. The issue of resource allocation must be considered as other countries may experience bandwidth limitations, limited access to the technology required for online trials or reduced access to the internet or internet-wired devices. Other considerations such as research capacity, governance, online and data security, and technological capacity will also need to be in place to fully exploit internet-based trial potential. ThinkWell is actively seeking partnerships with researchers in low-income countries in order to collaborate and promote knowledge transfer.

A major challenge of the project was to develop and pilot a set of search terms, phrases, and combinations in order to retrieve references to, or about, internet-based clinical controlled trials. Indexing in some non-medical databases is poor and uncontrolled, leading to retrieval problems and the need for excessive amounts of screening and filtering due to poor reporting of studies and abstracts. Although comprehensive search strategies and filters were developed as part of this project, the changing nature of the terminology, which is in line with the pace of socio-technical change, requires an exponential list of free-text search terms in the absence of recognised controlled vocabularies and ontologies.

Given the need to retrieve as many relevant studies as possible, the search strategy needed to be highly sensitive, but when combined with the poorly controlled terms for the ‘internet’ concept, this led to extremely large numbers of results with very poor precision. Because of this, the range and scope of databases used was restricted, and made it difficult to determine the impact of not using validated filters.

A number of different approaches to the description of terms were identified, including the development of taxonomies on behaviour change,[74] regular updates of MeSH,[75] the work of individual review groups or researchers, and organisations such as WHO[76] and the National Institute of Health and Care Excellence (NICE).[77] Work is underway to improve the reporting of interventions,[78] and this research on the identification of internet-based trials could feed into the better co-ordination of approaches to taxonomies and labelling. Standardisation in indexing methodology could complement transparency in research and lead to more effective searching and reporting of internet-based clinical trials research.

Definition of concept of ‘health and well-being’

Health and well-being is a very wide concept. Using the ICD categories was a pragmatic way of limiting the results, for instance in relation to the concept of addiction, which was restricted to drug/alcohol but excluded internet addiction, as the primary focus of the project was on health care and public health settings. This may have resulted in missed trials.

Indexing and coding

Similar limitations were experienced in the terms used for coding and indexing included studies. A search of the literature revealed a number of potential alternative systems for the coding of studies, particularly in the area of behaviour change interventions. Grimshaw et al.[79] draw attention to four Cochrane Reviews that cover interactive health communication applications; interventions to enhance medication adherence; contracts; and new methods of communication. Within the original cited reviews, specific elements of existing controlled terms, consistent with MeSH and other major taxonomies, are used in different combinations, including social support, decision support, and behaviour change support. Haynes et al.[80] identify a range of interventions used within trials for medication adherence, including instruction, counselling, family interventions, mentoring, etc. This approach, although ensuring consistency with the original decisions made by indexers, does not reflect consistency across all the studies, and therefore the categories tabled may include a range of intervention headings and classifications that are not comparable.

The mapping relied on the terms used by authors to report their studies, and on the indexing terms chosen by the host database. No quality appraisal of individual studies was made, other than that of determining whether the study was a randomised controlled trial by looking for reports of randomisation.

Conclusions

There is clearly a need to develop communities of practice to share learning and expertise in internet-based clinical trials, and to ensure that the public-led aspects embedded in the nature of the technologies is harnessed. This will be one of the major aims of the ThinkWell and PLOT-IT initiatives. The original purpose for creating ORCHID was as an internal resource for ThinkWell; however, as interest from other researchers was high, we decided to continue to add results after the initial searches in 2011 and to publish the details of ORCHID’s methods and findings to date. Interest from external research teams has led to collaboration on other projects, including a text-mining project.

To increase public engagement and to inform future internet-based clinical trials, the ThinkWell community will embed participatory processes in the future development of the database, the PLOT-IT platform[81] and the harvesting and prioritisation of research priorities. A core group of citizens is being recruited to help plan this process, and carry out a social network analysis. A two-stage systematic review is being undertaken to map what is known about internet-based clinical trials, and to synthesise specific aspects of the known uncertainties, including questions about ethics, methodology, and qualitative perspectives.[82] Qualitative work is also underway to explore both researcher and public experience in internet-based clinical trials, which will help to identify additional learning and support tools for all those involved in this area.

ORCHID has the potential to be a useful resource for researchers to mine data from the included studies, and will also be expanded to include a repository of what is known about the conduct of internet-based trials. Future development is planned to enable open access to this resource, to support the development of a community of practice which any researcher or member of the public can use. It is currently used as a resource to identify key terms for search strategies, and to further determine trends in this rapidly developing area. Regular updates are projected, and additional sources will be explored to increase comprehensive coverage, and to enable ThinkWell to expand the progression of public-led research.

Textbox 1.

Inclusion and exclusion criteria.

All studies that meet the following criteria were included:

• randomised controlled trials using internet-based technologies in the trial process

• studies using mobile technologies where there is also internet-based activity

• studies investigating health research, public health research topics and settings

• studies that include educational or behavioural interventions of health and well-being topics

• Studies involving patients or members of the public

The following studies were excluded:

• Studies using mobile telecommunications technologies exclusively, with no internet-based content

• Studies investigating interventions in social care or educational settings, where the main topic of investigation is not health related, or where clinical diagnoses are not included in ICD10

• Studies where participants were health professionals or students