Reporting practices of pharmacodynamic studies involving invasive research procedures in cancer trials.

BACKGROUND: Tumour biopsy for pharmacodynamic (PD) study is increasingly common in early-phase cancer trials. As they are non-diagnostic, the ethical justification for such procedures rests on their knowledge value. On the premise that knowledge value is related to reporting practices and outcome diversity, we assessed in a sample of recent invasive PD studies within cancer trials.
METHODS: We assessed reporting practices and outcomes for PD studies in a convenience sample of cancer trials published from 2000 to 2010 that employed invasive, non-diagnostic tissue procurement. Extracted data were used to measure outcome reporting in individual trials. Using a reporting scale we developed for exploratory purposes, we tested whether reporting varied with study characteristics, such as funding source or drug novelty.
RESULTS: Reporting varied widely within and across studies. Some practices were sporadically reported, including results of all planned tests (78% trials reporting), use of blinded histopathological assessment (43% trials reporting), biopsy dimensions (38% trials reporting), and description of patient flow through PD analysis (62%). Pharmacodynamic analysis as a primary end point and mandatory biopsy had statistically significant positive relationships with overall quality of reporting. A preponderance of positive results (61% of the studies described positive PD results) suggests possible publication bias.
CONCLUSION: Our results highlight the need for PD-reporting guidelines, and suggest several avenues for improving the risk/benefit for studies involving invasive, non-diagnostic tissue procurement.

Biopsy for pharmacodynamic (PD) and biomarker analysis is increasingly common in early-phase cancer trials (Twelves, 2006; Goulart ). In principle, PD end points can provide evidence of target effects for a drug, and support decision making for subsequent trials (Workman, 2003; Sarker ; Sarker and Workman, 2007; Tan ). However, many PD studies require invasive procedures like tumour biopsy.

Studies find that many patients are willing to undergo research biopsy (Seah ) and that ethics review committees and oncologists may overestimate patient anxiety associated with biopsies (Agulnik ). In one study, overall and major complication rates for tumour biopsies were 5.2% and 0.8%, respectively (Overman ). However, the majority of patients describe their biopsies as being painful (Agulnik ) and other studies indicate that 10% of patients receiving one common procedure – breast tumour biopsy – report moderate-to-severe pain (a more extended discussion of tumour biopsy risk and burden is available at Brown et al (2008); Hemmer ; Kimmelman ). As biopsies often have no value for subjects in terms of clinical management, their ethical justification rests on an expectation that their performance will be redeemed by the value of the knowledge accrued (Olson ).

Given that the burdens of such procedures are well understood, debates concerning their application revolve around conflicting views about the scientific utility of PD evidence. Some commentators question whether research biopsies return sufficient knowledge to justify their risks (Dowlati ; Parulekar and Eisenhauer, 2004; Davis ; Goulart ; Ratain and Glassman, 2007). Such critics describe research biopsies as ‘taking without giving in return' and an ‘expensive distraction' (Helft and Daugherty, 2006; Ratain and Glassman, 2007; Olson ). One critic argues, ‘given that biomarker support of mechanism, or lack thereof, has not contributed to go/no-go decisions in practice, sponsors should reconsider the value of including any biomarker evaluations in phase I oncological studies' (Ratain and Glassman, 2007). Others insist that the procedures are safe and feasible, and stress the importance of gathering mechanistic evidence in drug development; defenders point to examples where enrichment trials involving biopsy enabled rapid translation of cancer strategies (Kelloff and Sigman, 2005; Agulnik ; Cannistra, 2007; Brown ; Peppercorn ).

Such debates are hampered by a paucity of systematic evidence concerning the knowledge value of PD studies. In part, this reflects the fact that there are no widely accepted measures of knowledge value. In this report, we sought to highlight measures that could improve the risk/knowledge value of tumour biopsies and associated PD analyses. In particular, we measured two objective proxies of knowledge value: reporting practices and outcome diversity. In order for ‘knowledge value' to accrue, scientific findings must be reported in sufficient detail to permit readers to form or update beliefs. They must also enable others to reproduce findings in studies addressing similar questions. We measured the extent to which publications reported on study elements that were viewed as important in similar studies – those involving tumour prognostic biomarkers. The second proxy builds on the premise that a population of studies is more informative when it reflects a diversity of outcomes for tested hypotheses. Pharmacodynamic studies generally set out to test well-formulated hypotheses about specified target effects. Finding that in a population of PD studies, hypothesised target effects are almost always confirmed suggests either publication bias, limited information gain (as outcomes were predicted in advance of the PD study), or both. Our studies highlight the potential value of reporting standards for PD studies in cancer.

Materials and methods

Our primary objective was to describe the reporting practices in a convenience sample of recent invasive PD studies embedded within cancer trials. Our secondary objectives were to measure diversity of study outcomes and to identify characteristics of studies that correlate with better reporting.

Sample

Our study utilised a convenience sample of studies involving tumour biopsy. To capture a sample of studies that involved PD analyses and invasive tissue procurement while excluding the very large volume of studies involving minimally invasive collection (for example, venipuncture), we devised a search strategy that was highly specific. Briefly, we used keywords like ‘biopsy' and ‘pharmacodynamic' to search PubMed for articles published from 2000 to 2010 (inclusive) reporting on the use of invasive, non-diagnostic tissue procurement in cancer trials. We excluded articles where (a) non-diagnostic status of tissue procurement was ambiguous; (b) biopsy was not performed; (c) trials did not involve cancer patients; or (d) tissue procurement was minimally invasive (for example, venipuncture). Our search methods are described in greater detail elsewhere (Freeman and Kimmelman, 2012). After an initial screening by title and abstract, eligibility was confirmed using the full report.

Extraction elements

We developed a data extraction form for assessing study reporting and outcomes. Our form (Appendix 1) covered three domains: (1) study characteristics (for example, the year of publication, phase of trial, drug identity); (2) PD study practices and reporting (for example, description of assays, patient flow through study, use of blinded analysis); and (3) study outcomes (for example, confirmation status of PD hypotheses, author conclusions).

Elements within the second domain were adapted from REMARK criteria and supplemented with items described in Eisenhauer ; McShane . Extraction elements and coding conventions were initially developed by JK, and then discussed, refined, and approved by JD and JGM. After piloting extraction against 15 studies, we refined our form and coding criteria.

Extraction

All articles were extracted using paper forms by two reviewers (GF and JK) blinded to the other's extractions (but not author identities). We interpreted the absence of an affirmative practice statement as the absence of that practice (that is, studies not reporting blinded assessment were coded as not having implemented blinded outcome assessment). Studies were classified as implementing mandatory biopsy when explicitly stated in the report or when tissue samples were collected from all subjects. Data from extractions were entered into an Excel spreadsheet for analysis. Cohen's κ-inter-rater agreement was calculated; values exceeded 0.8, which we considered ‘good agreement' (Fleiss, 1981; Toulmonde ). Disagreements were resolved through discussion.

Reporting score

We developed a reporting score (RS) in order to explore the range of reporting quality, and to enable a series of tests concerning relationships between study characteristics and reporting. Our score was modelled after those used for prognostic tumour biomarker studies and randomized trials (Lai ; Kyzas ; Rios ; Toulmonde ) and was developed through discussions with all authors. It consisted of eight reporting domains: (1) goal and hypothesis; (2) subject eligibility; (3) specimen characteristics; (4) assay protocol; (5) statistics; (6) subject flow; (7) results; (8) discussion. Domains contained one or more evenly weighted reporting variables. Reporting on any item within a domain would result in a fractional score and each domain had a potential score of one. Scores in each domain were summed to calculate an overall RS for each study.

Outcome reporting

Studies were assessed along three outcome categories. The first was results of hypothesis tests. Results were coded as positive where a treatment caused hypothesised changes in targets (that is, an increase in apoptosis assessed by TUNEL staining with a proapoptotic drug) and negative where hypotheses failed confirmation (but were not necessarily disconfirmed). As most studies tested many markers, we coded each report according to whether some, all, or no tested hypotheses were positive. The second outcome category was discussion of results in light of hypotheses. Studies were scored as ‘positive' when discussions indicated that PD results were consistent with the predicted molecular effects of the agent. Discussions were coded as ambiguous where they gave no clear indication as to whether PD supported the predicted effect of the agent, and were coded as negative where they suggested PD did not support the predicted molecular effects. The third outcome assessed was discussion of results in light of future study planning. Studies were coded as informative where PD results (whether themselves positive or negative) were said to inform planning of future studies. Discussions were coded as uninformative where they gave no clear indication of how PD results related to future investigations.

In a post hoc analysis, we studied the effect of industry funding on PD outcome reporting, focusing on the proportion of positive assay results and the discussion of those results in light of hypotheses and planning for future studies. Fisher's exact test of independence was used to calculate significance (McDonald, 2009).

Statistics

As this was an exploratory study, we used a convenience sample of PD studies rather than a prospectively determined sample size. We tested a priori-formulated hypotheses of correlation between RS and the following seven variables: (1) the year of publication, (2) public funding, (3) journal impact factor, (4) separate publication for PD results, (5) use of a non-novel test drug, (6) mandatory biopsy; and (7) author assessment of the trial outcome (negative outcome defined as studies recommending that further trials of the investigational agent should not be undertaken). Significance of relationships was tested using one-way ANOVA with SPSS software. We defined significance as P⩽0.05. We did not correct for multiple comparisons.

Results

Our PubMed search produced a sample of 68 eligible articles reporting results from early-phase cancer trials utilising non-diagnostic biopsy for PD analysis (flow of articles is described in Figure 1; see Appendix 2 for an inventory of studies). Table 1 displays the characteristics of the trials in our sample; Table 2 reports biopsy characteristics within our sample. Ten studies in our sample (15%) actively reported safety events related to biopsy; of these, one reported a single adverse event at or above grade 3.

Figure 1

Diagram of flow of the published articles selection process.

Table 1

Characteristics of early-phase cancer trials included in sample (n=68)

Location corresponding author
North America	64.7%
Europe	33.8%
Australia	1.5%
Number of trials by time period
2000–2005	36.8%
2006–2010	63.2%
Trial goalsa
Safety	52.9%
Dose	38.6%
Biologic effect/efficacy	42.6%
Pharmacodynamics (PD)	76.5%
Pharmacokinetics	16.2%
PD as the primary end point	41%
Trial phase
0	3%
1	53%
1/2	6%
2	31%
Not specified	7%
Trial agent characteristics
Single agent	66%
Novel agentb	53%
Patients
Average number of patients per trial	39
Range	7–270
Total number of patients enrolled across all trials	2644
Funding sourcea
Industry	38%
Foundation	31%
Government	51%
Trial outcomea
Dose identified	38.2%
Mechanism demonstrated	29.4%
Clinical activity demonstrated	23.5%
Further trials inappropriate	35.3%
Other	26.5%

Characteristics of the parent studies.

Trials reported multiple goals, outcomes, and funding sources. Percentages do not necessarily add up to 100%.

Defined as not FDA approved at time of study.

Table 2

Characteristics of early-phase cancer trials included in sample (n=68)

Biopsy overview
Average number of research biopsies collected per trial	57.3
Range	1–942
Average number of biopsies per patient	2.3
Range	1–9.5
Total number of biopsies collected across all trials	3781
Mandatory biopsya
Yes	36%
No/unclear	64%
Biopsy locationa
Skin	46%
Breast	17%
Head and neck	17%
Liver	15%
GI tract	15%
Lung	9%
Bone marrow	5%
Ovary	5%
Other	17%
Procurement methoda
Core needle	41%
Punch	31%
Fine needle	16%
Surgical excision	14%
Percutaneous/trucut	14%
Endoscopic	12%
Other	12%
Purpose of pharmacodynamic (PD) investigationa
Dose escalation guide	6%
Marker identification	21%
Find recommended dose	13%
Proof-of-concept	66%
Mechanism, effect on tissue or function	65%
Biodistribution	6%
Relate mechanism to response	26%
Unclear or not stated	24%
Other	13%

Characteristics of the invasive tissue procurement procedures.

Refers to the percentage of studies. Trials reported multiple biopsy locations and purposes of PD investigation. Percentages will not add up to 100%.

Our sample captured a total of 2644 patients receiving invasive non-diagnostic biopsies. Although reporting of patient flow through PD studies was poor, we recorded author explanations for discrepancies between patients approached for biopsy, samples collected, and samples analysed. The most common reason for discrepancy was insufficient quality or quantity of sample for analysis (84%), followed by patient refusal (19%) and medical contraindication for biopsy (19%). Missed samples (3%) were because of patient death.

We calculated the RS for each article in our sample. The RS range had a score centred around 5.5 (Figure 2). Some variables, like description of causal pathway and biopsy location, were consistently reported (Table 3). However, there was broad variation within specific domains in the RS. A fifth of articles did not report results for all PD analyses performed; 57% did not report the status of blinding for pathological analysis and 62% did not provide information about the dimensions of the biopsy sample.

Figure 2

Distribution of RSs for sample of early-phase cancer trials utilising biopsy for PD study.

Table 3

Reporting score (RS) outcomes (n=68)

Reporting domain	Weighted reporting variable (weight)a	% Trials	95% confidence interval
Goal and hypothesis
	Stated goal clearly (0.5)	76	66–87
	Description molecular causal pathway (0.5)	97	93–100
Subject eligibility
	Patient eligibility for biopsy described (1)	62	50–73
Specimen characteristics
	Biopsy dimensions (0.25)	38	27–50
	Method of procurement (0.25)	72	61–83
	Location (0.25)	96	91–100
	Description of specimen processing (0.25)	69	58–80
Assay protocol
	Protocol described or reference provided (0.2)	59	47–71
	Identity of person conducting assay/analysis given (0.2)b	32	21–43
	Description of controls (0.2)	49	37–60
	Scoring or quantitation protocols described (0.2)	77	66–87
	Blinded outcome assessment described (0.2)	43	32–54
Statistics
	Statistical justification sample size, significance (1)	56	44–68
Subject flow
	Number of biopsies collected and analysed reported (0.5)	62	50–73
	Explanation for disparity or unaccounted samples (0.5)	59	46–73
Results reporting
	Results shown for all tested hypotheses (1)	78	68–88
Discussion
	Alternate explanations for positive or negative results (0.33)	71	60–82
	Discussion of results in light of hypotheses (0.33)	91	84–98
	Discussion of results in terms of future study planning (0.33)	78	68–88

Where reporting variables were not applicable, that variable was removed from consideration and the remaining variables were reweighted to create the reporting domain score.

Where qualitative assessment of tissue staining was used (e.g. immunohistochemistry).

Reporting predictors

The use of a non-novel study drug showed positive but non-significant trend towards a higher RS (5.6 vs 5.2, P=0.219). Pharmacodynamics as a primary end point showed a significant positive relationship with RS (5.8 vs 5.1, P=0.04), as did the use of mandatory biopsy (5.9 vs 5.1, P=0.023). We found no relationship between RS and the year of publication, journal impact factor, funding source, or author assessment of the trial outcome.

Pharmacodynamic study outcome

The majority of articles (66%) reported some negative PD results and 10% of the articles reported all negative PD results. Fifty-six percent of studies reported at least one positive PD parameter. The majority of studies (61%) described their PD results as ‘positive' in discussions (for example, PD results provided evidence of the investigational agent having intended effects on molecular targets).

A large majority of articles (78%) contained a discussion of PD results in relation to the direction of future studies. Among these, 72% discussed possible amendments to the conduct or direction of future studies based on the PD findings of the current study.

Industry funding vs results positivity

Industry-funded trials were more likely to report all or some positive PD results than non-industry-funded studies. No industry-funded trial reported all negative results for PD parameters tested. Trials with industry funding trended towards greater positivity in discussion both in terms of support for the predicted method of action of the drug (75% vs 53%, P=0.11) and planning for future studies (80% vs 67%, P=0.359).

Discussion

Biopsies for PD in anticancer drug trials are often burdensome and entail non-trivial costs. Justification of procedures rests on a favourable gain of scientific knowledge (Weijer and Miller, 2004). Poor PD reporting does not adequately redeem burdens and can produce biased findings that lead to unsuccessful clinical development (Tan ). At present, there is little systematic evidence to inform the planning, implementation, and ethical evaluation of PD studies involving invasive tissue procurement.

Our study explored two relatively objective proxies of knowledge value in a convenience sample of PD studies using research biopsies. Encouragingly, a large fraction of studies reported tissue location, procurement method, and discussion of PD results. However, many important items were reported sporadically, including results of all planned tests, use of blinded histopathological assessment, biopsy dimensions, and description of patient flow through the PD portion of the trial. Previous studies of prognostic marker research reporting showed that over 90% of studies reported ‘positive' outcomes (Kyzas ; Kyzas ). Disproportionate reporting of positive results was also observed in genetic association studies (Ioannidis ). We entered this study expecting near-uniform positivity among PD reports. Instead, we found that two-thirds of articles contained negative outcomes, and a similar proportion described PD analysis as affirming hypotheses in discussion. This is evidence that PD is not characterised by overwhelming publication bias, and that results are not overdetermined at study inception. Nevertheless, that the fraction of studies reporting uniform positivity (34%) vs those reporting uniform negativity (10%) suggests, in our view, the presence of some bias. Whether this bias pertains to publication bias, or enhanced pre-test probability, we are unable to say. Analysis of positivity would be greatly aided if studies declared their primary hypothesis; the only instance where this occurred was in studies that reported only a single PD marker analysis. We further take the fact that a large fraction of PD studies were described as informing decisions for future studies as support for invasive PD evaluation. Future studies should investigate the fraction of PD findings that motivate actual new investigations.

Our study has several limitations. First, some might question the premises guiding our proxy indicators of knowledge value. Poorly reported studies can still hold value, and uniformly positive results can convert modest degrees of belief in drug effects into higher degrees of belief. Still, uniform confirmation would seem a modest gain of information for considerable burden. Second, some items in the RS, similar to blinded outcome assessment, straddle ‘good reporting' and ‘good methodological practice,' and high quality reporting can mask poor methodological practice (Huwiler-Muntener ; Toulmonde ). Third, in line with the exploratory orientation, our study did not capture a comprehensive sample of studies involving research biopsies. A larger sample might have produced different findings and our sample may have been underpowered to detect relationships between study characteristics and reporting quality. Fourth, although our article points to ways that reporting of PD might improve, nothing in our premises, data, or analysis provides a clear basis for deciding whether current research biopsy and PD study practices meet an adequate threshold of knowledge value. Last, our RS scale should be interpreted with caution. It was not the result of a consensus building process (unlike CONSORT and REMARK) (Harris, 2005; McShane ; Lai ; ‘How CONSORT began', 2008; Rios ; Toulmonde ). Furthermore, it gave uniform weighting for each criterion, which may not be appropriate, given that some items probably matter more than others with respect to valid study interpretation. Nevertheless, our scale was at least modelled on validated criteria and we believe its application is justified in the context of this exploratory exercise. Finally, although this study identifies deficiencies in current reporting practices and may aid in the development of consensus guidelines, it must be noted that delay to publication means that current study practices may not be accurately represented in our study.

Our study suggests several avenues investigators, funders, or IRBs might consider for improving the risk–benefit balance of PD studies. First, we recommend the research community develop formalized reporting guidelines similar to REMARK and CONSORT. Second, given our observation that separate PD-reporting trends towards higher quality, and that reporting quality for PD studies may be constrained by word counts at journals, we encourage investigators to consider separate PD publication, using standard methods described in a reference or reporting methods in supplementary materials (Toulmonde ). Journal editors may have a role in limiting ‘text limitation bias.' Third, given that PD components might not be registered in http//www.clinicaltrials.gov, IRBs might have a more active role in promoting reporting and publication by asking investigators to provide a detailed reporting plan for PD studies. A recent article recommended the creation of an online biomarker study registry similar to http//www.clinicaltrials.gov (Andre ). We support extending this initiative to PD.

Together with a previous study by our team, our results offer a complex picture of the quality of reporting for PD studies involving non-diagnostic biopsy. A preponderance of positive results, coupled with a finding that 63% of PD studies go unreported suggests biases. Low perceived quality of reports, and low reporting of basic factors like patient flow, suggests considerable room for improvement. On the other hand, some studies demonstrate careful reporting, many negative results are reported, and a large fraction of studies report that PD findings will help guide future investigations. In the end, we conclude that the evidence gathered above provides ammunition for proponents as well as opponents of research biopsies in cancer. In any event, our findings and analysis provide grounds for developing and disseminating PD-reporting standards.