Pilot trials in thrombosis: Purpose and pitfalls.

Randomized controlled trials provide important evidence to guide clinical practice. These full-scale trials are expensive, time consuming and many are never successfully completed. Well conducted pilot studies help with full-scale trial design, assessment and optimization of feasibility, and can avoid the waste of resources associated with starting a full-scale trial that will not succeed. They also provide an opportunity for capacity growth and mentorship of new investigators. It is important to appreciate that the usual goal of a pilot trial is assessment of feasibility and refinement of trial design rather than to gain preliminary evidence of efficacy. Indeed, using event rates from a pilot trial to calculate sample sizes can be misleading in therapeutic trials. Misconceptions exist that pilot trials are just "small trials," are easy to perform, and are not worthy of publication. While, in the past, many pilot trials were poorly conducted and not followed by a full-scale trial, by following the recommendations in the "CONSORT 2010 statement: extension to randomized pilot and feasibility trials," high-quality pilot trials can be performed and reported that will greatly improve the chances of successfully completing a practice-changing trial. We propose that pilot trials are a valuable investment and describe the TRIM-Line pilot trial (NCT03506815), a pilot study assessing the feasibility of a randomized controlled trial investigating primary thromboprophylaxis with rivaroxaban in patients with malignancy and central venous catheters, as an illustrative example of how a pilot trial in the area of thrombosis should be designed.

Well‐conducted pilot trials inform full‐scale randomized controlled trials.

Pilot trials are an efficient and effective way to support early career investigators.

The purpose and common misconceptions and pitfalls of pilot trials are discussed.

The TRIM‐Line pilot trial is used as an illustrative example in this tutorial.

INTRODUCTION

Rigorously conducted randomized controlled trials (RCTs) provide important evidence to guide clinical and policy decisions about medical interventions. Phase III studies, often enrolling thousands of patients, provide high quality evidence about the efficacy and safety of a drug or device,1 and are needed before regulatory agencies, clinicians, guideline developers, and policymakers confidently adopt or reject a health intervention. Phase III randomized trials need to be well designed and implemented to yield valid conclusions.2 Conversely, those that are poorly designed and implemented are inefficient and wasteful, and may be unethical and jeopardize patient safety.

Conducting full‐scale clinical trials is an expensive and time‐consuming process. In the current research environment, with increased recognition of research waste3 and growing accountability and competition for research funding, granting agencies look favorably on pilot data demonstrating the feasibility of recruitment, procedures and methods, and field testing the logistical aspects of a large trial on a smaller scale.

What is a pilot trial?

A pilot trial is a small study conducted to help design and assess the feasibility of doing a larger, full‐scale trial. They also investigate whether the methods and procedures are able to obtain the data that is needed to answer the question that will be addressed in the larger study.4

Currently, there is a lack of consistency in the terms used to refer to a study aimed at assessing the feasibility of a large phase III RCT (Box 1) and this topic is the focus of a detailed report and conceptual framework.5 In this framework, feasibility studies include: randomized pilot studies, in which the future RCT is conducted on a smaller scale (“piloted”); nonrandomized pilot studies in which the intervention and other study processes are tested, and feasibility studies that develop processes that will be used in a future full‐scale trial (eg, gathering information through interviews or questionnaires) but do not pilot the processes or interventions of the future trial. Eldridge et al. conclude that the interchangeable use of various terms seems acceptable to use for a randomized study with a primary aim of assessing the feasibility of a future full‐scale RCT6; they recommend that in the title and abstract, the study be clearly identified by using the words pilot or feasibility, along with the terms randomized and trial. This paper focuses on randomized pilot studies and we refer to them as pilot trials.

There are many misconceptions about pilot trials. As highlighted by Thabane et al., studies performed as a research project by a trainee, or on a small scale at a single center because of lack of funding, should not be referred to as “pilot studies” if their aim is not to inform larger‐scale studies.4 This misuse of terminology could account for the fact that only 50% of pilot studies identified in a systematic review reported the intention of future work, and only a small minority (<10%) were actually followed by a large study.7

In this paper we describe the conception, design, and initial stages of implementation of a pilot trial supported by the Canadian Venous Thromboembolism Clinical Trials and Outcomes Research (CanVECTOR) Network (http://www.canvector.ca). Our network was funded in 2015 by the Canadian Institutes of Health Research (CIHR) though a targeted competition that recognized a need to “… improve efficiency of clinical trials through the conduct of pilot studies (a prerequisite for large clinical trials that will ultimately prevent multi‐year delays and increase clinical trials efficiency); and enrich [the VTE] community with national training, mentoring, and career development programs (including multi‐site programs) based on the most recent best practices.” In its first 2 years CanVECTOR has supported six pilot trials led by thrombosis fellows or early career investigators; one of these, The TRIM‐Line Pilot Trial (NCT03506815), a pilot study assessing the feasibility of a randomized controlled trial investigating primary thromboprophylaxis with rivaroxaban in patients with malignancy and central venous catheters, is profiled in this paper as an example of a well‐designed pilot study.

The TRIM‐Line pilot trial

The TRIM‐Line RCT arose from a perceived knowledge gap in the prevention of upper extremity deep vein thrombosis (UEDVT). The Division of Hematology in Ottawa is asked to help manage many patients with UEDVT secondary to cancer and indwelling central venous catheters (peripherally inserted central catheters [PICC] or ports). After seeing many of these patients, a hematologist completing a thrombosis research fellowship (RI), wondered whether line‐associated thrombosis in this high‐risk population could be prevented. With his supervisor (MC) he formulated a research question: in adult patients with cancer and central venous catheters, does low‐dose rivaroxaban (10 mg daily) safely and effectively prevent line‐associated UEDVT compared to no treatment.

A search of ClinicalTrials.gov and the literature was conducted to confirm there were no published or in‐progress RCTs addressing this research question. Before attempting to answer this question in an RCT, the lead investigators considered it important to determine if recruitment to the full‐scale prophylaxis trial was feasible, given the vulnerable patient population and uncertainty about the availability and receptiveness of eligible patients at clinical sites of varying capacities. An application for funding was submitted to the CanVECTOR network which had recently launched a pilot trials competition intended to support teams that included: a trainee, early career investigator, established investigator, patient partner, and research methodologist. The application was awarded funding, thereby enabling a clinical question encountered during practice to become a pilot trial led by a fellow with mentorship from faculty.

METHODS

Objectives and outcomes

As described above, a defining feature of pilot trials is that the primary aim is to determine feasibility of a large full‐scale trial. Consequently, regardless of the full‐scale trial's design or methods, the objectives of the pilot trial are focused on feasibility and the methodology of the pilot trial is tailored to these feasibility‐focused objectives. A common error is for the objectives of a pilot trial to be either unclear or stated as being identical to the large RCT.8 In Table 1 we provide examples of the types of objectives that are suitable for pilot trials and full‐scale trials.

Table 1

Examples of objectives for pilot trials and full‐scale randomized controlled trials

Pilot trial objectives	Randomized controlled trial objectives
To evaluate or measure	To evaluate or measure
Recruitment Ability to recruit and retain study participants (access to patients, eligibility, center capacity and willingness) Patient acceptance of the intervention and study procedures (informed consent)	Efficacy Treatment effect: VTE or recurrent VTE Dose response
Intervention Feasibility of the intervention Compliance/adherence with the intervention Crossovers to another study intervention	Safety Major/clinically relevant non‐major bleeding Other adverse events, side effects, toxicity Deaths
Evaluation and Follow‐up Time requirements (participants and staff) Availability of equipment and resources Compliance with study procedures, evaluations and follow‐up schedule	Patient‐Centered/Oriented Outcomes Quality of life Symptoms control Post‐thrombotic syndrome Satisfaction Preferences
Data Capture and Management Data capture procedures (field testing for clarity, consistency, and applicability across sites and international borders) Quality and completeness of data captured Data management mechanisms	Economics Costs Cost effectiveness
Trial Logistics and Management Research personnel time requirements Ability to recruit clinical centers Time required/barriers to obtain site approvals (ethics and contracts)

VTE, venous thromboembolism.

Sample size determination

Sample size for a pilot trial is based on the pilot trial's objectives, outcomes, and analysis plan, depending on whether the primary outcome is a proportion, binary, ordinal, or continuous outcome. In contrast, for phase III full‐scale trials, the sample size is calculated based on the desired statistical power, accepted type 1 error (alpha risk), expected effect size (minimal clinically important difference), and the variability of the outcome measure.

From an audit of sample sizes in pilot and feasibility studies registered in the UK Clinical Research Network database, authors conclude that even though sample size calculations are not a requirement for pilot trials, all pilot trials should have a sample size justification which states the rationale for the target sample size.9 Further, they stress that due to the high level of uncertainty in a pilot study, the target sample size should be considered a preliminary figure. In the examples of sample size rationale statements provided in the CONSORT extension to randomized pilot and feasibility trials, this uncertainty is reflected in the use of the words “estimated” and “aimed for.”6

It is tempting to follow one of the “rules of thumb” that have been suggested for pilot trials, such as using a minimum sample size of 12 per group, 50 per group, or corresponding to at least 3% of the sample size of the full‐scale trial.9 Thabane et al. suggest using a confidence interval approach to determine sample size when the objective of the study is to assess feasibility as reflected by a proportion (eg, proportion of screened patients who are eligible and consenting).4 The pilot study should then include enough patients to ensure that the lower bound of the 95% confidence interval around the estimate of this proportion exceeds the preset value for feasibility.4 While these rules are based on statistical theory and modelling, it is essential that investigators are conscious of the situation (eg, type of outcome and analysis) for which the rule is intended and not apply a rule indiscriminately.

TRIM‐Line sample size justification

The CanVECTOR network is comprised of three types of clinical centers. High‐ and low‐volume experienced sites, with established research infrastructure and a strong track record of recruitment into thrombosis studies, and emerging sites which are new to the network, have limited research support infrastructure, and whose recruitment capacity remains untested. Due to the uncertainty in the recruitment potential at all three types of centers for the full‐scale TRIM‐Line Trial, we decided to pilot the study in one “experienced high volume,” one “experienced low volume,” and one “emerging” site. The number recruited per center per month (the primary outcome) will inform the number and mix of clinical centers for the full‐scale trial, the duration of recruitment, and the resources/funding required. We have completed preliminary sample size calculations for the full‐scale TRIM‐Line RCT: based on published event rates of venous thromboembolism in cancer patients with central venous catheters10 of 6.8% in the untreated and 3.7% in the treated population, sufficient data for estimation of sample size of the full‐scale trial was available. Using an alpha of 0.05, with 80% power and after accounting for an expected 10% loss to follow‐up rate, the required sample size was calculated to be 1892.

We aim at testing our capacity to enrol 50 patients between the three sites within 6 months; we estimate that we will enroll 4 to 6 per month in the “high‐volume experienced” site, and 1 to 2 in both the “low‐volume experienced” and the “emerging” sites (success criterion). If we observe average recruitment at the three types of centers, respectively, at 5, 1.5, and 1.5 patients per month, this will result in a sample size of approximately 50 patients in the TRIM‐Line Pilot Trial. Given the number and type of sites in the CanVECTOR network, the observed recruitment rate in the pilot trial will allow us to model the recruitment for the full‐scale RCT and determine if it is feasible to conduct the trial through the CanVECTOR network (Table 2).

Table 2

Modelling of minimum, average and maximum pilot trial enrollment per month and potential capacity of the CanVECTOR network to support enrollment of 1892 participants in the full trial

	Characteristics of clinical research center	Average pilot trial participant enrollment per month	Number of recruiting sites for the full RCT	Potential total participant enrollment per month for the full RCT	Total months of recruitment for the full RCT	Potential total participant enrollment in total months of recruitment	Estimated loss to follow‐up	Total enrolment accounting for loss to follow‐up
Scenario 1: Minimum recruitment targets are met in the pilot trial	Experienced high volume	4	6	24	44.5	1068	−10%	961
	Experienced low volume	1	9	9	44.5	401	−10%	360
	Emerging	1	14	14	44.5	623	−10%	561
	Totals	6	29	47		2091.5		1882
Scenario 2: Average recruitment targets are met in the pilot trial	Experienced high volume	5	6	30	32.5	975	−10%	878
	Experienced low volume	1.5	9	13.5	32.5	439	−10%	395
	Emerging	1.5	14	21	32.5	683	−10%	614
	Totals	8	29	64.5		2096		1887
Scenario 3: Maximum recruitment targets are met in the pilot trial	Experienced high volume	6	6	36	25.5	918	−10%	826
	Experienced low volume	2	9	18	25.5	459	−10%	413
	Emerging	2	14	28	25.5	714	−10%	643
	Totals	10	29	82		2091		1882

Statistical analysis

Similar to phase III trials, pilot studies should have clearly identified outcomes, corresponding to their feasibility objectives. The analysis should focus on these objectives, and results interpretation should be based on prespecified criteria for success. These criteria will lead to the determination of whether the full‐scale RCT is feasible, may be feasible with modification, or is not feasible. Although the outcome measures that will be used for the full‐scale RCT may be looked at, they should not be the primary focus. By their nature, pilot trials will be underpowered to compare the primary outcome of the full trial. Further, if the intention from the outset is to pool the data from patients enrolled in the pilot trial with data from patients in the full‐scale RCT, this should be specified in the pilot trial's protocol and should include data merging and analysis plans, along with statistical consequences for the main trial. Pilot trial results are worth publishing as they inform the research community on the feasibility (or lack of) of the intended RCT, identify feasibility barriers, and may suggest solutions to these obstacles.

The TRIM‐Line pilot trial's methods

The design of the TRIM‐Line pilot trial is summarized and contrasted with the design of the full trial in Figure 1. In the figure, elements that are common to the pilot trial and the full trial are shown in the center, those that are unique to the pilot trial are on the left and the design of the full trial is on the right. As previously noted, the primary goal of the TRIM‐Line pilot trial was to assess recruitment feasibility (the number of patients recruited per site per month). Secondary goals were to assess: the time and personnel resources required to recruit and follow each patient; consent rates, losses to follow up; study drug adherence (good adherence being defined as having taken 80% or more of medication); and data collection and data management procedures and tools. The outcomes for the full trial are listed in Figure 1.

Figure 1

Design for TRIM‐Line pilot trial and TRIM‐Line full‐scale trial

In Table 2, three possible enrollment scenarios from the pilot trial are extrapolated to the full‐scale trial, using all sites of the CanVECTOR network. Based on the primary outcome of enrollment rate/month, the results of the pilot trial will be interpreted as unfeasible if the minimum enrollment targets are not met; that is, if over 6 months of recruitment, an average of four patients per month in the experienced high‐volume site and an average of one patient per month in the low‐volume and emerging sites cannot be achieved. If enrollment meets the prespecified target of five patients per month, on average, in the high‐volume site and 1.5 patients in the other sites, the full‐scale trial will be considered feasible. The observed enrollment rate will assist the investigators in determining the timeframe and number of sites required for the full‐scale trial. The model assumes all sites of the CanVECTOR network would join the study, which is somewhat optimistic. The duration of recruitment will need to be extended if fewer clinical sites participate or, alternatively, recruitment could be extended to international sites through established collaborations with the International Network of Venous Thromboembolism Clinical Research Networks (INVENT).11

Approvals

While a pilot trial will enroll fewer patients than the full‐scale trial, planning for a pilot trial is almost as demanding as planning to initiate the full‐scale trial. In the case of the TRIM‐Line pilot trial, due to the intervention with rivaroxaban outside of its approved indications, obtaining approval by local ethics boards and oversight by Health Canada was necessary. The regulatory requirements in our jurisdiction for the pilot trial and the full‐scale trial are identical. Having three pilot trial clinical centers in different provinces required adaptation of the consent form in consideration of different laws, privacy regulations, funding mechanisms, and translation requirements. For the TRIM‐Line pilot trial, it took 12 months to finalize the protocol and consent form and apply for regulatory and ethics approvals. In our experience, more than 12 months has been required for early career investigators to start a regulated randomized controlled pilot trial.

DISCUSSION

The CanVECTOR network is supportive of pilot trials and considers them to be valuable. This commitment to funding pilot trials is based on the belief that pilot trials prevent research waste through identifying barriers to full‐scale trial feasibility at modest expense. In addition to the formal analyses in a pilot trial, there's an opportunity for qualitative observations that can be taken into consideration when planning the full‐scale trial, such as feedback from colleagues and patients, noted barriers or delays to start‐up or recruitment, and trends in recruitment such as slow initial recruitment that increases over the first months before reaching a steady state.

Funding pilot trials can also be a means of supporting early career investigators by providing them with both funds and mentorship to gain experience and establish a track record of successful clinical investigation. This, in turn, will increase their competitiveness when applying for grants for large RCTs. The first pilot trial supported by the CanVECTOR network is a success story: an early career investigator conducted the COBRRA pilot trial (NCT02559856; Comparison of Bleeding Risk Between Rivaroxaban and Apixaban for the Treatment of Acute Venous Thromboembolism) which demonstrated feasibility of recruitment and study procedures at four Canadian sites. This feasibility data contributed to her success at securing funding from the Canadian Institutes of Health Research to conduct the full‐scale trial (NCT03266783), with a sample size of 2760.

Importance of mentorship and a team approach

Mentorship is crucial to the success of clinician scientists during their training and early research career12 (Box 2). Mentorship has been found to increase research productivity, increase the confidence of mentees and provide opportunities for professional networking.13 Conducting a pilot trial is complex and requires knowledge of research methodology, best practices, regulatory and ethics requirements, negotiation of clinical trial agreements with subsites, establishment of data capture mechanisms, development of study databases, and biostatistical collaborations. Since no one person is an expert in all fields, support from colleagues in one's own and in other clinical centers is critical for research success. Mentors can contribute knowledge, support valuable relationships with peers, and provide resources to early career investigators that would otherwise be unavailable. CanVECTOR supports mentorship through a formal program linking senior and early career investigators, and facilitates the sharing of examples (eg, successful funding application, pilot trial protocol, consent form) or tools (eg, data capture form, study start‐up checklist) with less experienced researchers.

Budget considerations for a pilot trial

A common misconception is that a lack of resources for a large multicenter trial justifies doing a pilot trial.4 However, many of the costs and infrastructure requirements of a full‐scale trial are common to a pilot trial, including salary for personnel (eg, to prepare the protocol, consent form, data capture/case report forms, research ethics board application, application(s) for regulatory approval, pilot trial registration such as clinicaltrials.gov), and annual administrative fees (eg, hospital pharmacy, laboratory, and diagnostic imaging department). Certain expenses such as placebo manufacturing entail a large upfront charge for setup of the production line and low per‐unit costs thereafter. Pilot trials are unsuited to take advantage of the economy of scale, thus increasing the per‐patient cost of enrollment and making it difficult to absorb the high costs associated with placebo manufacturing, packaging to ensure blinding, and distribution.

Costs of a pilot trial will be less than for the full‐scale trial because there are fewer clinical centers and participants, less data to monitor and clean, analyses are usually descriptive and do not require complicated analysis, only a subset of study outcomes may be collected, and follow‐up time may be reduced. The typical budget of the six pilot trials funded by CanVECTOR is approximately $50 000 Canadian, and ranges between $39 000 and $134 000, depending on the study requirements.

If the sampling frame and methodologies are the same for the pilot trial and the full‐scale trial, data from the pilot can be used as part of the full‐scale trial, provided the decision and plan for combining the data has been prespecified in the protocol and is not influenced by knowledge of the pilot trial's findings.4 If no major design changes are made, increased efficiency through reduced start‐up times and costs may be realized from the wealth of information gathered from the pilot trial. Care should be applied, however, since there may be statistical consequences, such as reduced power, if the outcome data from the pilot have been analyzed before the end of the full‐scale trial. In addition, if there is a long time gap between the pilot trial and the full‐scale trial, as can sometimes occur while obtaining funding, there may have been shifts in practice over time which could affect the results.

TRIM‐Line budget

The TRIM‐Line pilot trial was granted funding of $52 223 from the CanVECTOR Pilot Trials Competition. For the TRIM‐Line pilot, the original design was placebo‐controlled and double‐blind. Upon discussion with drug companies and placebo manufacturers, the cost of producing, blinding, storing, and distributing placebo was found to be unfeasible for this pilot. Therefore, the decision was made to change the design to an open‐label pilot trial. This would not affect the number eligible but could have some effect on the number of people that consent. Ultimately, if an open‐label trial was not feasible, it would be unlikely that the full‐scale double‐blind trial with higher complexity and expense would be successful. This compromise in design is a limitation of this pilot and prevents data from patients enrolled in the pilot trial to be included in the full‐scale trial.

Publishing a pilot trial

Early reviews found that the quality of published pilot trials was poor21 and journal editors cited lack of rigor as a reason that they did not publish pilot trial results.7 However, even if a pilot trial finds that a full‐scale trial is nonfeasible, this is an important finding that should be published so that other investigators may benefit from this information. The BioMed Central journal Pilot and Feasibility Studies publishes both the protocols and results of pilot trials and is committed to ensuring that the results of all well‐conducted, peer‐reviewed, pilot and feasibility studies are published, regardless of outcome or significance of findings.

The goals and focus of reporting for pilot trial results differs from full‐scale RCTs, thus motivating the “CONSORT 2010 statement: extension to randomized pilot and feasibility studies.”6 Particularly, there is a greater focus on the recruitment process, including initial screening to identify subjects who meet inclusion criteria, assessment of exclusions in those who meet inclusion criteria, and rate and reasons that study‐eligible patients do not consent to participate. In the extended statement, less emphasis is placed on describing procedures such as randomization, allocation concealment, and blinding, unless the objectives of the pilot trial specifically relate to these topics. However, providing the registration number for the pilot trial in a trial registry, information about where the pilot trial protocol can be accessed, and confirming that the pilot trial has received ethics approval are all recommended. The rationale for registration of a pilot trial is the same as that of the full‐scale trial: it will inform other researchers of work in progress, thus preventing duplication, and can improve accountability for reporting results of the pilot trial by allowing others to more easily assess for selective outcome reporting or changes in the design or outcomes. The CONSORT 2010 extension to randomized pilot and feasibility trials includes a side‐by‐side comparison of the checklist of items to include when reporting an RCT and a pilot trial (their Table 2).6

In summary, a positive pilot trial can provide strong support for a granting agency to fund a full‐scale trial, but a negative pilot trial is also important and should be published so that it can lead to changes in the design of an RCT or prevent scarce resources from being wasted.

ACKNOWLEDGMENTS

R. Ikesaka is supported by a CanVECTOR fellowship award; the TRIM‐Line pilot trial is funded by the CanVECTOR Network; the Network receives grant funding from the Canadian Institutes of Health Research (Funding Reference: CDT‐142654). G. Le Gal holds a ‘CP has Heart’ Clinician Scientist Cardiovascular Award from the Heart and Stroke Foundation of Ontario, and the Department of Medicine Chair on Diagnosis of Venous Thromboembolism at the University of Ottawa. C. Kearon is supported by an Investigator Award from the Heart and Stroke Foundation of Canada and the Jack Hirsh Professorship in Thromboembolism. M. Carrier holds a Tier 2 Department of Medicine Research Chair in Cancer and Venous Thromboembolism from the University of Ottawa.

RELATIONSHIP DISCLOSURES

Dr. Ikesaka and Ms. Langlois report salary support from the CanVECTOR Network, during the conduct of the study. Dr. Ikesaka reports grant support from the CanVECTOR network during the conduct of the study. Dr. Carrier reports other from Leo Pharma, other from Bristol‐Myers Squibb, other from Bayer, other from Octapharma, other from Sanofi Aventis, other from Pfizer, other from Boehringer Ingelheim, all are outside the submitted work. Dr. Le Gal reports other from Portola Pharmaceuticals, other from Boehringer‐Ingelheim, other from Pfizer, other from Bristol‐Myers Squibb, other from LEO Pharma, other from Daiichi Sankyo, other from Bayer, other from Sanofi, other from bioMérieux, all are outside the submitted work. Dr. Kearon reports other from the Heart and Stroke Foundation of Canada, other from Jack Hirsh Professorship in Thromboembolism, during the conduct of the study.

AUTHOR CONTRIBUTIONS

R. Ikesaka wrote the first draft of the manuscript and designed the TRIM‐Line pilot trial. N. Langlois and G. Le Gal contributed sections. N. Langlois, G. Le Gal, C. Kearon oversee the clinical trials platform and pilot trials competition of the CanVECTOR network and critically reviewed the manuscript. M. Carrier supervised R. Ikesaka in the design and application for funding for the TRIM‐Line pilot trial and critically reviewed the manuscript.

Feasibility trial

Feasibility RCT

Feasibility study

Pilot trial

Pilot RCT

Pilot study

Randomized pilot trial

Randomized feasibility study

Vanguard trials

Higher likelihood to continue in a research career14, 15

Increased confidence and preparedness to pursue a research career16

Greater research productivity and grant success13, 17, 18, 19

Additional opportunities for networking and collaboration16

Better overall satisfaction13, 18, 20