Information on new drugs at market entry: retrospective analysis of health technology assessment reports versus regulatory reports, journal publications, and registry reports.

BACKGROUND: When a new drug becomes available, patients and doctors require information on its benefits and harms. In 2011, Germany introduced the early benefit assessment of new drugs through the act on the reform of the market for medicinal products (AMNOG). At market entry, the pharmaceutical company responsible must submit a standardised dossier containing all available evidence of the drug's added benefit over an appropriate comparator treatment. The added benefit is mainly determined using patient relevant outcomes. The "dossier assessment" is generally performed by the Institute for Quality and Efficiency in Health Care (IQWiG) and then published online. It contains all relevant study information, including data from unpublished clinical study reports contained in the dossiers. The dossier assessment refers to the patient population for which the new drug is approved according to the summary of product characteristics. This patient population may comprise either the total populations investigated in the studies submitted to regulatory authorities in the drug approval process, or the specific subpopulations defined in the summary of product characteristics ("approved subpopulations").
OBJECTIVE: To determine the information gain from AMNOG documents compared with non-AMNOG documents for methods and results of studies available at market entry of new drugs. AMNOG documents comprise dossier assessments done by IQWiG and publicly available modules of company dossiers; non-AMNOG documents comprise conventional, publicly available sources-that is, European public assessment reports, journal publications, and registry reports. The analysis focused on the approved patient populations.
DESIGN: Retrospective analysis. DATA SOURCES: All dossier assessments conducted by IQWiG between 1 January 2011 and 28 February 2013 in which the dossiers contained suitable studies allowing for a full early benefit assessment. We also considered all European public assessment reports, journal publications, and registry reports referring to these studies and included in the dossiers. DATA ANALYSIS: We assessed reporting quality for each study and each available document for eight methods and 11 results items (three baseline characteristics and eight patient relevant outcomes), and dichotomised them as "completely reported" or "incompletely reported (including items not reported at all)." For each document type we calculated the proportion of items with complete reporting for methods and results, for each item and overall, and compared the findings.Results 15 out of 27 dossiers were eligible for inclusion and contained 22 studies. The 15 dossier assessments contained 28 individual assessments of 15 total study populations and 13 approved subpopulations. European public assessment reports were available for all drugs. Journal publications were available for 14 out of 15 drugs and 21 out of 22 studies. A registry report in ClinicalTrials.gov was available for all drugs and studies; however, only 11 contained results. In the analysis of total study populations, the AMNOG documents reached the highest grade of completeness, with about 90% of methods and results items completely reported. In non-AMNOG documents, the rate was 75% for methods and 52% for results items; journal publications achieved the best rates, followed by European public assessment reports and registry reports. The analysis of approved subpopulations showed poorer complete reporting of results items, particularly in non-AMNOG documents (non-AMNOG versus AMNOG: 11% v 71% for overall results items and 5% v 70% for patient relevant outcomes). The main limitation of our analysis is the small sample size.
CONCLUSION: Conventional, publicly available sources provide insufficient information on new drugs, especially on patient relevant outcomes in approved subpopulations. This type of information is largely available in AMNOG documents, albeit only partly in English. The AMNOG approach could be used internationally to develop a comprehensive publication model for clinical studies and thus represents a key open access measure. © Köhler et al 2015.

Introduction

In 2013 alone, 46 new drugs entered the European market after a centralised procedure at the European Medicines Agency. In addition, 18 established drugs were approved for a new indication.1 Once new treatment options are available, patients and doctors require comprehensive information on their benefits and harms in the approved indications to enable decisions to be made about individual treatment. This information is best provided on the basis of a systematic review of all available evidence on the new drug. Such reviews are also required by medical societies to develop clinical practice guidelines and by health policy makers to inform reimbursement and pricing decisions.

It is well established that publicly available information from clinical studies is often incorrect as a result of reporting bias.2 3 The most comprehensive records of clinical studies—namely, the full clinical study reports submitted to regulatory authorities—are so far not publicly available.4 5 6 Publicly available reporting formats include journal publications, reports posted in study registries and results registries (“registry reports”), as well as regulatory documents. Regulatory documents include European public assessment reports published by EMA for every drug granted a central marketing authorisation.7 The European public assessment report is the key publicly available regulatory document on a new drug in Europe, reflecting the scientific conclusions of the relevant EMA committee; it thus represents a potentially important source of independent information.

Internationally agreed standards8 9 and national and international laws10 11 aim to improve the reporting quality of clinical studies. However, reporting is still insufficient in journal publications4 5 12 13 and registry reports.4 5 13 14 15 Criticism has also been raised about the quality of information in European public assessment reports16 and reports from other regulatory agencies.17 18

A second challenge to decision making after the approval of a drug is less well examined. Drug approval is based on clinical studies submitted by pharmaceutical companies to regulatory agencies. However, the patient population for which a drug is finally approved (that is, the approved indication according to the summary of product characteristics) might differ from the patient population investigated in the submitted clinical studies. This restriction may relate to the safety concerns of regulatory agencies or to the need to maximise the benefit-risk ratio in the approved indication. For example, the indication for rilpivirine in patients with HIV was restricted to those with a viral load of ≤100 000 HIV-1 RNA copies/mL. This was because post hoc subgroup analyses showed an increased risk of resistance and a lower virological response rate with rilpivirine in patients with a high baseline viral load (>100 000 copies/mL).19 Clinical studies in which a drug was used outside the scope of the approved patient population may overestimate or underestimate a drug’s benefit or harm. Information is therefore required on a drug’s effect in the approved population. At market entry of a new drug (new active substance or new indication of a known active substance), the following two requirements are thus essential: unbiased comprehensive information must be publicly available and this information should preferably cover the approved patient population.

In 2011, Germany introduced the early benefit assessment of new drugs through the act on the reform of the market for medicinal products (Arzneimittelmarktneuordnungsgesetz, AMNOG)20 21; see box 1. In short, at market entry the pharmaceutical company responsible must submit a standardised dossier (box 2) containing all available evidence of the drug’s added benefit over an appropriate comparator treatment.23 The Federal Joint Committee generally commissions the Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, IQWiG) to assess the evidence contained in the dossier. This “dossier assessment” and the main parts of the dossier are published online. The dossier assessment contains the relevant study information, including data from unpublished clinical study reports contained in the dossiers (box 2). The results of the dossier assessment serve as the basis for the Federal Joint Committee’s decision making process on the added benefit.

Box 1 Early assessment of benefit of new drugs in Germany according to AMNOG

Background

Until 31 December 2010, the price of a new drug introduced into the German market was not regulated or negotiated by a healthcare or governmental body but was solely set by the pharmaceutical industry. As a result Germany generally paid higher drug prices than other European countries.22 In addition, many (expensive) new drugs entering the market show no or little added benefit over existing treatment options. On 1 January 2011 Germany introduced the early benefit assessment of new drugs through the act on the reform of the market for medicinal products (AMNOG).20 21 The main goal of AMNOG is to inform pricing negotiations for new drugs.

Procedure

When a newly approved drug enters the German market, the pharmaceutical company responsible must submit a standardised dossier containing all available evidence of the drug’s added benefit over an appropriate comparator treatment.23 The added benefit is predominantly determined using patient relevant outcomes on benefits and harms—namely, mortality, morbidity (including adverse events), and health related quality of life.

The Federal Joint Committee (Gemeinsamer Bundesausschuss), the main decision making body within the German statutory health insurance system, is responsible for the procedure of early benefit assessment, and ultimately decides on the added benefit.24 The committee receives the dossiers from the pharmaceutical companies. It generally commissions the Institute for Quality and Efficiency in Health Care (IQWiG), Germany’s main health technology assessment agency, to assess the evidence contained in the dossier within three months after market entry (this IQWiG report is called a “dossier assessment”). The results of the dossier assessment serve as the basis for the committee’s decision making process on the added benefit. After publication of the dossier assessment, the committee conducts a commenting procedure and hearing, during which the pharmaceutical company responsible and other specified parties may submit comments. After evaluation of these comments, the committee issues a decision on the probability and extent of added benefit: this decision is used to inform pricing negotiations between the umbrella organisation of statutory health insurance and the pharmaceutical company. If the pharmaceutical company can prove that new scientific evidence is available, one year after the publication of the decision at the earliest, it can submit an application to the committee for reassessment of the drug. Alternatively, this reassessment can be initiated by the committee itself.24

Box 2 Details on the company dossier and dossier assessment

Dossier structure and requirements

The dossier submitted by the pharmaceutical company consists of five sections named “modules 1 to 5.” (In the present article, the terms “dossier” and “module” refer exclusively to documents submitted by companies to the Federal Joint Committee within the framework of AMNOG; they do not refer to documents submitted to regulatory authorities as part of the regulatory submission dossier, the common technical document.) The dossier is prepared in a mandatory (German language) template (included in “Anlage 2” (see supplementary appendix 2) on the website of the Federal Joint Committee25). This template contains detailed requirements on the information to be included in the dossier. In addition to methodological requirements and detailed table templates for data presentation, module 4 also includes a template for a tabular presentation of study methods with requirements derived from the CONSORT statement. The template also specifies the following main requirements for the systematic literature search conducted by the pharmaceutical companies: a search in specific bibliographic databases (Medline, Embase, and the Cochrane Central Register of Controlled Trials) and in specific study registries (for example, Clinical.Trials.gov), as well as inclusion of the relevant retrievals in the dossier. Furthermore, specific documents have to be provided (see fig 4). All documents required must be included in the dossier, together with the complete list of studies in the given indication. Compliance with the template is checked: deviation may lead to rejection of the dossier because of incompleteness and the conclusion of “no added benefit proven” for the drug under assessment.

Fig 3 Complete reporting of combined study methods and results items in study registries. *11 registry reports containing 14 individual assessments

Public availability of the dossier and dossier assessment

Modules 1 to 4 of the dossier are fully published on the website of the Federal Joint Committee (www.g-ba.de). Module 5 is classified as commercially confidential and thus not published. However, relevant information from clinical study reports and other documents included in module 5 and not presented in modules 1 to 4 is generally presented by IQWiG in its dossier assessment, and thus publicly available on www.g-ba.de and www.iqwig.de. A company may object to publication of data from module 5; however, in consequence its drug may be classified as having “no added benefit,” as any relevant data used in the assessment must be publicly available. The dossier assessment is fully published on the websites of IQWiG and the Federal Joint Committee. The dossier assessment and the published modules of the dossier are in German. The main sections of the dossier assessment are also available in English and are published on the IQWiG website.

Volume of the dossier and dossier assessment

In the pool of documents included in our analysis, modules 1 to 4 of the dossiers comprised a median of 446 pages (interquartile range 346-576 pages). In addition, module 5 comprised up to thousands of additional (unpublished) pages. The dossier assessments comprised a median of 83 pages (interquartile range 81-93 pages).

An important aspect of early benefit assessment in Germany is that it is based on the approved indication and patient population according to the summary of product characteristics.24 26 If this population is identical to the original patient populations investigated in the clinical studies submitted to the regulatory agencies during the drug approval process (hereinafter called the “total study populations”), the results reported in these studies can be used. However, as noted previously, the new drug may only have been approved for a specific subpopulation. In consequence, some dossier assessments refer to the total study populations, whereas others only refer to one or more approved subpopulations, and thus require additional data. In the dossiers submitted for early benefit assessment, analyses of these subpopulations can be provided by the pharmaceutical companies on the basis of individual patient data.

Besides informing pricing negotiations, the implementation of AMNOG has resulted in the provision of an additional publicly available information source for clinical studies of new drugs and could represent a key open access measure.

We determined the information gain from AMNOG documents compared with non-AMNOG documents for methods and results of studies available at the time of market entry of new drugs. AMNOG documents comprise IQWiG dossier assessments and publicly available modules of company dossiers; non-AMNOG documents comprise conventional, publicly available sources—that is, European public assessment reports, journal publications, and registry reports. We specifically considered the European context (inclusion of European public assessment reports), as new drugs assessed according to AMNOG are generally approved by EMA in a centralised procedure, and a European public assessment report is available for each drug. The analysis focused on the approved patient populations (total study populations or subpopulations).

Methods

Document selection

We originally considered all dossier assessments conducted by IQWiG between 1 January 2011 and 28 February 2013. From these we included all dossiers containing suitable studies allowing for a full early benefit assessment—that is, studies of a suitable design containing suitable data for the comparison of the benefits and harms of the new drug and the appropriate comparator treatment.

No dossier assessments of orphan drugs were included, as these are conducted by the Federal Joint Committee itself.

Each early benefit assessment can investigate one or more research questions—that is, one or more constellations of patients, interventions, and comparators. These are based on the approved indication (for example, telaprevir in chronic hepatitis C virus infection27), as well as the corresponding requirements for the patient population (for example, treatment naïve without cirrhosis) and the appropriate comparator treatment (for example, pegylated interferon alfa plus ribavirin); see supplementary table 1 for the two further research questions on telaprevir and examples of further drugs. Depending on the number of research questions investigated and clinical studies included, a dossier assessment may contain more than one individual assessment.

Our analysis was based on AMNOG documents. We compared the reporting of studies in these documents with the reporting of the same studies in non-AMNOG documents. The precondition for inclusion of non-AMNOG documents was their public availability at the time of market entry (that is, at the starting point of the AMNOG process). This meant that neither updates posted in study registries after the date of market entry were considered nor journal publications published after this date. In the AMNOG process it is mandatory for pharmaceutical companies to conduct a systematic literature search (see box 2 for details). This search is routinely checked by IQWiG for plausibility and completeness. Furthermore, if necessary, searches in study registries are repeated, and focused searches in PubMed conducted (the search techniques applied are described elsewhere28). No additional search was therefore conducted for the present analysis.

Data extraction, coding of information, and analysis

We used Microsoft Access software to develop a database for the information extracted from the selected documents. This database contained information on the drug assessed, the company responsible, study characteristics (study identification number, study sponsor), documents available for analysis, and the number and content of the research questions related to each assessment of benefit. To identify the patient relevant outcomes to be included in our analysis, we systematically screened the clinical study reports (including study protocols) and other parts of the dossiers to identify any planned or reported outcomes. (This deviated from the original methods in which we had planned to screen the dossier assessments to identify these outcomes.)

We classified the completeness of information (reporting quality) of each study and each available document by means of a standardised set of items (individual items or item categories) on study methods and results. Each study had eight methods items and 11 results items. The results items comprised three items on baseline characteristics and eight on patient relevant outcomes. Our definition of completeness of information was based on the requirements of authors of systematic reviews,8 29 and largely followed Wieseler and colleagues’ study.4 The total number of methods and results items expected to be reported for each document type for the individual assessments of the total study populations and subpopulations represented a completeness of information of 100%.

We classified information on methods items as (1) completely reported, (2) partly reported, or (3) not reported. On the basis of an intention to treat analysis, we classified information on results items as (1) completely reported including numerical data, (2) partly reported including numerical data, (3) verbally reported without numerical data, or (4) not reported. For results items, we aggregated the categories (2) and (3) to “partly reported” in the present analysis, as category (3) was rarely recorded (see supplementary tables 2 and 3 for full definitions of categories). We further aggregated the categories for both types of items and dichotomised them to “completely reported” (methods and results: category 1) or “incompletely reported” (methods: categories 2 and 3; results: categories 2 to 4).

Eight authors were involved in data extraction and coding, with one author covering all studies and data sources allocated to a specific benefit assessment. A second author checked extraction and coding. Discrepancies were resolved by consensus. Data analysis was performed by using SAS software, version 9.3.

Results

Document characteristics

Overall, 27 dossiers were submitted by 28 February 2013 (fig 1). Of these, 15 dossiers containing 22 studies were eligible for inclusion. The 15 dossier assessments presented 28 individual assessments of 15 total study populations and 13 approved subpopulations (see supplementary table 1 for details).

Fig 4 Structure and content of the company dossier and dossier assessment

Of the 15 drugs investigated, seven had been developed for the treatment of cancer, two each for hepatitis C and HIV infection, and one each for acute coronary syndrome, multiple sclerosis, prophylaxis of thrombosis after surgery, and prophylaxis of renal graft rejection (table 1).

Table 1

 Characteristics of included documents and indications

Characteristics	No
Dossier assessments	15
Clinical studies analysed in dossier assessments	22*
Research questions (patients/intervention/comparator) analysed in dossier assessments	22
Individual assessments	28*
Original (total) study population relevant for research question (%)	15 (54)
Subpopulations relevant for research question† (%)	13 (46)
Documents reporting clinical studies analysed in dossier assessments:
AMNOG documents‡	15
EPAR	15§
Journal publication¶	42
Registry report¶	22
With study results**	11††
Indications (tested drugs):
Acute coronary syndrome (ticagrelor)	1
Chronic hepatitis C virus infection genotype 1 (boceprevir, telaprevir)	2
Multiple sclerosis (fingolimod)	1
Thrombosis prophylaxis (apixaban)	1
Graft rejection prophylaxis after kidney transplantation (belatacept)	1
HIV infection (rilpivirin (emtricitabin, tenofovir))	2
Metastatic prostate cancer (abiraterone, cabazitaxel)	2
Metastatic breast cancer (eribulin)	1
Metastatic melanoma (vemurafenib, ipilimumab)	2
Kidney cancer (axitinib)	1
Lung cancer (crizotinib)	1

AMNOG=Arzneimittelmarktneuordnungsgesetz (act on reform of the market for medicinal products); EPAR= European public assessment report; IQWiG=Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (Institute for Quality and Efficiency in Health Care).

*Six studies referred to two research questions each, so a total of 28 individual assessments were conducted.

†Subpopulation of clinical study covering approved patient population (10 cases) or reflecting patient population treated with appropriate comparator treatment (three cases).

‡IQWiG dossier assessment and publicly available parts of company dossier.

§An EPAR was available for all 15 tested drugs. In addition, one dossier assessment (ticagrelor) considered an indirect comparison in the assessment of added benefit. Therefore, besides the EPAR on ticagrelor, the EPAR on the appropriate comparator treatment (prasugrel) was also considered.

¶Publications and registry reports referring to more than one research question were counted once only.

**Study results posted at time of dossier submission. Registry reports referring to more than one research question were counted once only.

††Refers to 14 out of 28 individual assessments.

As defined through the assessment procedure, there was one set of AMNOG documents (dossier assessment and publicly available parts of the dossier) for each drug investigated. All of the drugs were authorised for market access by the European Commission after assessment by EMA; therefore in all cases a European public assessment report was available. A total of 42 journal publications covering entire studies or specific aspects of these studies were available for 14 out of 15 drugs and 21 out of 22 studies. For each of these 21 studies, at least one publication contained study results. A registry report in ClinicalTrials.gov was available for all drugs and studies; however, only 11 out of 22 reports (referring to nine drugs) contained study results. Other registries only contained reports for six studies.

Reporting rates for combined methods and results items

Figure 2 shows the rates of completeness of reporting for the combined methods and results items by document type and relevant study population. In the analysis of total study populations, the AMNOG documents reached the highest grade of completeness, with about 90% of both types of items completely reported. For non-AMNOG documents, the overall rate was 75% for methods items (journal publications: 67%, European public assessment reports: 54%, registry reports: 46%); complete reporting of results items was noticeably poorer (overall rate: 52%, journal publications: 41%, European public assessment reports 29%, registry reports: 21%).

Fig 1 Flowchart for inclusion of documents in analysis. IQWiG=Institute for Quality and Efficiency in Health Care

In the analysis of approved subpopulations, in both AMNOG and non-AMNOG documents, the reporting rates for methods were similar to those in the total study populations. However, reporting of results was poorer, especially in non-AMNOG documents: whereas about 70% of results items were completely reported in AMNOG documents, the corresponding rate for non-AMNOG documents was only 11%, with rates of less than 10% for the three individual types of non-AMNOG documents. If one only considered patient relevant outcomes in the results items, the rate in non-AMNOG documents was even lower (non-AMNOG versus AMNOG: 5% v 70%).

Reporting rates for individual methods and results items

Table 2 shows the rates of completeness of reporting for individual methods items by document type and relevant study population. Table 3 shows the corresponding rates for results items.

Table 2

 Rates of complete information on study methods (by document type and relevant population for approval)

Study methods (items and item categories)	Complete study information by item and document type, No (%)
	Total study population relevant*						Subpopulation relevant†
	No of expected items‡	AMNOG§	EPAR	Publication	Registry report	Non-AMNOG total	No of expected items¶	AMNOG§	EPAR	Publication	Registry report	Non-AMNOG total
Randomisation	15	14 (93)	5 (33)	13 (87)	2 (13)	14 (93)	13	11 (85)	9(69)	13 (100)	0 (0)	13 (100)
Allocation concealment	15	14 (93)	5 (33)	12 (80)	0 (0)	13 (87)	13	11 (85)	6 (46)	11 (85)	0 (0)	11 (85)
Blinding	15	15 (100)	12 (80)	12 (80)	12 (80)	15 (100)	13	12 (92)	10 (77)	8 (62)	12 (92)	13 (100)
Definition of ITT dataset	15	12 (80)	11 (73)	11 (73)	3 (20)	12 (80)	13	11 (85)	10 (77)	10 (77)	4 (31)	13 (100)
No of patients in ITT dataset	15	15 (100)	9 (60)	10 (67)	3 (20)	10 (67)	13	13 (100)	9 (69)	8 (62)	5 (38)	9 (69)
Subtotal risk of bias items	75	70 (93)	42 (56)	58 (77)	20 (27)	64 (85)	65	58 (89)	44 (68)	50 (77)	21 (32)	59 (91)
Intervention**	67	67 (100)	61 (91)	60 (90)	59 (88)	66 (99)	62	61 (98)	60 (97)	60 (97)	48 (77)	62 (100)
Comparator**	64	64 (100)	43 (67)	54 (84)	51 (80)	60 (94)	62	55 (89)	53 (85)	53 (85)	40 (65)	56 (90)
Methods of outcome measurement††	163	133 (82)	54 (33)	74 (45)	38 (23)	87 (53)	144	105 (73)	45 (31)	65 (45)	37 (26)	87 (60)
Study methods total	369	334 (91)	200 (54)	246 (67)	168 (46)	277 (75)	333	279 (84)	202 (61)	228 (68)	146 (44)	264 (79)

ITT=intention to treat; AMNOG=Arzneimittelmarktneuordnungsgesetz (act on reform of the market for medicinal products); EPAR=European public assessment report.

*Original study population investigated in clinical studies (15 individual assessments).

†Subpopulation of clinical study covering approved patient population (10 individual assessments) or reflecting patient population treated with appropriate comparator treatment (three individual assessments).

‡Total number (100%) of methods items expected to be reported per document type for the 15 individual assessments of total study populations (items required according to CONSORT).

§IQWiG (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, Institute for Quality and Efficiency in Health Care) dossier assessment and publicly available parts of company dossier.

¶Total number (100%) of methods items expected to be reported per document type for the 13 individual assessments of subpopulations (items required according to CONSORT).

**Includes information on dosage, intervals for administering, duration of treatment, route for being administered and concomitant drugs.

††Determined by number of outcomes assessed according to clinical study report.

Table 3

 Rates of complete information on baseline characteristics and patient relevant outcomes (by document type and relevant population for approval)

Study results (items and item categories)	Complete study information by item and document type, No (%)
	Total study population relevant*						Subpopulation relevant†
	No of expected items‡	AMNOG§	EPAR	Publication	Registry report	Non-AMNOG total	No of expected items¶	AMNOG§	EPAR	Publication	Registry report	Non-AMNOG total
Baseline characteristics:
Age	15	15 (100)	10 (67)	12 (80)	6 (40)	14 (93)	13	7 (54)	2 (15)	5 (38)	1 (8)	5 (38)
Sex	13	13 (100)	9 (69)	12 (92)	7 (54)	12 (92)	13	13 (100)	2 (15)	5 (38)	1 (8)	5 (38)
Disease severity	9	9 (100)	6 (67)	7 (78)	1 (11)	7 (78)	10	7 (70)	2 (20)	2 (20)	0 (0)	2 (20)
Total baseline characteristics	37	37 (100)	25 (68)	31 (84)	14 (38)	33 (89)	36	27 (75)	6 (17)	12 (33)	2 (6)	12 (33)
Benefit outcomes:
Mortality	15	15 (100)	7 (47)	6 (40)	3 (20)	9 (60)	16	16 (100)	0 (0)	1 (6)	1 (6)	2 (13)
Clinical event**	23	23 (100)	5 (22)	8 (35)	3 (13)	9 (39)	20	19 (95)	4 (20)	1 (5)	0 (0)	5 (25)
Symptom††	7	4 (57)	0 (0)	0 (0)	0 (0)	0 (0)	8	1 (13)	0 (0)	0 (0)	0 (0)	0 (0)
Health related quality of life	19	7 (37)	0 (0)	0 (0)	1 (5)	1 (5)	9	5 (56)	0 (0)	0 (0)	0 (0)	0 (0)
Subtotal benefit outcomes	64	49 (77)	12 (19)	14 (22)	7 (11)	19 (30)	53	41 (77)	4 (8)	2 (4)	1 (2)	7 (13)
Harm outcomes:
Adverse event	15	14 (93)	3 (20)	6 (40)	8 (53)	11 (73)	13	11 (85)	0 (0)	0 (0)	0 (0)	0 (0)
Serious adverse event	15	14 (93)	2 (13)	5 (33)	6 (40)	8 (53)	13	10 (77)	0 (0)	0 (0)	0 (0)	0 (0)
Withdrawals due to adverse events	15	14 (93)	6 (40)	7 (47)	3 (20)	9 (60)	13	10 (77)	0 (0)	0 (0)	0 (0)	0 (0)
Specific adverse event‡‡	54	49 (91)	9 (17)	18 (33)	4 (7)	23 (43)	52	29 (56)	0 (0)	0 (0)	0 (0)	0 (0)
Subtotal harm outcomes	99	91 (92)	20 (20)	36 (36)	21 (21)	51 (52)	91	60 (66)	0 (0)	0 (0)	0 (0)	0 (0)
Total benefit and harm outcomes	163	140 (86)	32 (20)	50 (31)	28 (17)	70 (43)	144	101 (70)	4 (3)	2 (1)	1 (1)	7 (5)
Total study results	200	177 (89)	57 (29)	81 (41)	42 (21)	103 (52)	180	128 (71)	10 (6)	14 (8)	3 (2)	19 (11)

AMNOG=Arzneimittelmarktneuordnungsgesetz (act on reform of the market for medicinal products); EPAR=European public assessment report; HRQoL=health related quality of life; IQWiG=Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (Institute for Quality and Efficiency in Health Care).

*Original study population investigated in clinical studies (15 individual assessments).

†Subpopulation of clinical study covering approved patient population (10 individual assessments) or reflecting patient population treated with appropriate comparator treatment (three individual assessments).

‡Total number (100%) of results items expected to be reported per document type for the 15 individual assessments of total study populations (based on characteristics and patient relevant outcomes planned and reported in clinical study reports, including protocols).

§IQWiG dossier assessment and publicly available parts of company dossier.

¶Total number (100%) of results items expected to be reported per document type for the 13 individual assessments of subpopulations (based on baseline characteristics and patient relevant outcomes planned and reported in clinical study reports, including protocols).

**Clinical events (benefit outcome): any event (other than an adverse event) based on clinical diagnosis—for example, non-fatal stroke or non-fatal myocardial infarction, pulmonary embolism.

††Symptoms (benefit outcome): any sign of the disease based on description by patient—for example, pain, fatigue.

‡‡For example, adverse events of special interest in given indication.

In the analysis of total study populations, complete reporting rates above 90% were shown in AMNOG reports for 15 out of 19 methods and results items. Rates below 90% were only shown for the definition of the intention to treat population (80%), the reporting of outcome measurement methods (82%), and of (health related) quality of life and symptoms (37% and 57%). Rates of complete reporting (of an intention to treat analysis) were particularly low for quality of life and symptoms, as the respective analyses often included an insufficient number of patients. In comparison, in non-AMNOG documents, rates above 90% were shown for only six out of 19 items: quality of life was especially poorly reported (5%) and symptoms were not reported at all. Reporting of adverse events was particularly limited in our sample of European public assessment reports. However, even though only limited data on adverse event rates in individual studies were available, it should be noted that in contrast with the other formats, the European public assessment reports mainly reported adverse events rates that had been pooled over various studies.

In the analysis of subpopulations, reporting rates decreased for both AMNOG and non-AMNOG documents, but were noticeably lower in the non-AMNOG documents. In AMNOG documents, complete reporting rates above 90% were shown for six out of 19 items (three methods and three results items). Rates below 60% were shown for four results items (age, symptoms, quality of life, and specific adverse events). In non-AMNOG documents, rates above 90% were shown for five items (all referring to methods items). All rates for results items were below 40%; symptoms, quality of life, and all four adverse event outcomes were not reported at all.

Given the better overall reporting in AMNOG documents, we also analysed whether, for single methods or results items, non-AMNOG documents reported more information than AMNOG documents. Journal publications provided more information for 19 (1.8%) of the total number of 1082 methods and results items that were expected to be reported for each document type. The corresponding numbers for European public assessment reports and registry reports were 13 (1.2%) and 11 (1.0%) items, respectively.

Sensitivity analysis for registry reports

We hypothesised that the poor rate of complete reporting in registry reports was partly due to some reports not including study results at the time of market entry of the drugs. We therefore conducted a sensitivity analysis for the 11 registry reports that also contained results (fig 3 and supplementary tables 4 to 7).

Fig 2 Reporting of combined study methods and results items in AMNOG (act on reform of the market for medicinal products) documents versus conventional, publicly available sources (European public assessment reports, journal publications, registry reports (non-AMNOG total))

About 60% of methods items were completely reported (versus about 30% in registry reports without results), regardless of the population relevant for analysis. For results items (total study population), the rate was 45% (versus about 20% in the overall pool of study registries). However, only 3% of results items were completely reported in the approved subpopulation. Overall, the data show that even in registry reports containing results, completeness of reporting was limited, with registry reports not even covering 50% of the information required.

Discussion

Our findings show that, at the time of market entry of a new drug, a substantial amount of information needed for assessment of the corresponding clinical studies and for understanding of the drug’s benefits and harms is missing in publicly available European public assessment reports, journal publications, and registry reports (non-AMNOG documents). In contrast, AMNOG documents provide a considerably higher proportion of complete information on specific methods items and results items, including patient relevant outcomes. The apparent information gap is most striking when the drug is approved only in a certain subpopulation. In many of these cases, non-AMNOG documents, including European public assessment reports, reported no results data at all.

Information in European public assessment reports

Within the non-AMNOG documents, reporting rates in European public assessment reports were better than those in registry reports but worse than those in journal publications (for example, European public assessment reports versus journal publications: 29% v 41% for results items in the total study populations). Poor reporting of results has also been shown in an analysis of studies on psychiatric drugs reported in European public assessment reports: efficacy results were only available for five out of 15 indications.16

Suggestions to expand the European public assessment report to contribute to the relative effectiveness assessment of new drugs in health technology assessment (HTA) have been discussed between the European Medicines Agency and the European Network for Health Technology Assessment (EUnetHTA), which resulted in changes to the template for the European public assessment report.30 Only one of the European public assessment reports included in our sample (the one on ticagrelor) had been prepared before the revised template became effective. Therefore, in our sample even the new template did not result in full availability of the information required for our early benefit assessment. This may be partly because the new template was introduced during the EMA assessment process of the drugs in our sample: preliminary versions of EMA’s assessment reports may therefore not have been prepared according to the new standards, and the new template may not have been fully implemented.30 However, it is unclear whether future European public assessment reports will include the information required for HTA.

In addition, the new template for the European public assessment report does not require the same level of completeness of information as that required for the AMNOG documents. For example, the reporting of secondary outcomes is optional, so there is no requirement to report full numerical data on all (secondary) patient relevant outcomes. The instructions in the template rather suggest that “the level of detail should be adjusted to the data later needed for the discussion and conclusion on benefits, as well as the benefit-risk assessment.”31 While this seems reasonable from a regulatory point of view, it will inevitably exclude information on outcomes that are not primarily considered for approval, but might be useful for HTA. The issue of the total study population versus the approved subpopulation is not clearly dealt with in the template; notably, there is no requirement to report data on the approved subpopulation.

There has been a major development at EMA regarding data transparency. On 2 October 2014 the agency announced that its new “policy on publication of clinical data for medicinal products for human use” would be effective from 1 January 2015.32 According to this policy, EMA will make clinical study reports and parts of the regulatory submission dossier publicly available. However, it is unclear to what extent the documents will be redacted, and the terms of use have been criticised.33 Independently of the provision of comprehensive study information by EMA, improvement of the template for the European public assessment report should still be discussed so that an optimal summary format can be provided.

Information in journal publications

At the time of market entry of a new drug, journal publications reporting study results were available for the majority of studies analysed (21/22; 95%). This rate is higher than in other investigations of publication rates for studies submitted for regulatory approval.34 35 It is unclear whether this results from a generally improved publication rate or from the specific mix of indications and studies in our sample (mainly anticancer drugs with a few high profile studies for each drug). However, the high availability rate of journal publications in our sample did not correspond to a high reporting rate of study data, which is consistent with previous findings.5

In the present analysis, journal publications showed the highest reporting rate within non-AMNOG documents for methods, with more than two thirds of items completely reported. This is higher than the proportion of 40% shown in a previous analysis.4 Reasons for this might include the improved implementation of the CONSORT statement9 or the fact that AMNOG is limited to clinical studies of new drugs, which are often published in high impact journals with stringent demands on reporting quality. Journal publications also achieved better reporting rates (about 40%) than the two other non-AMNOG documents for the reporting of results in the total study population. However, if only patient relevant outcomes were considered, the rate dropped to 31%. In contrast with methods items, this rate was only slightly higher to that shown in a previous analysis investigating the same outcomes in a sample of older studies (31% v 23%).5 Although the reporting of certain results, such as a flow diagram for study participants36 or harms,37 has improved in journal publications over the past years (which is probably also due to the CONSORT statement) there is still much room for improvement.4 5 36 37

Information in registry reports

All of the studies included in our sample had been registered on ClinicalTrials.gov. However, only 11 out of the 22 registry reports (50%) contained study results, even though, according to the Food and Drug Administration Amendments Act, these should have been available in all cases. This rate is higher than in previous analyses.15 38 It is unclear whether this is from increasing compliance with the act or the characteristics of our study sample.

Of all sources analysed in our sample, the overall pool of registry reports provided the least complete information but achieved similar rates to journal publications and better rates than European public assessment reports, if only registry reports containing results were considered. However, given the specific requirements for the posting of results,39 the reporting rate of 45% still seems low. Better reporting rates in ClinicalTrials.gov than in journal publications have been shown in a previous analysis for a wide range of results (flow of participants, efficacy results, adverse events, and serious adverse events).13 None the less, overall the reporting of results in ClinicalTrials.gov is still inadequate.38 40

Information in AMNOG documents

The complete reporting rate of the publicly available AMNOG documents was about 90% for methods items and results items and thus comparable to rates in clinical study reports.4 This is due to the fact that, according to AMNOG, clinical study reports have to be submitted with the dossier. Although these reports are not published as part of the AMNOG process, the relevant study information is extracted and transferred to the publicly available AMNOG documents, with the aim of providing a complete and unbiased account of the studies. However, one cannot fully exclude that this additional filter may introduce bias, thus emphasising the need for the publication of full clinical study reports from the dossiers. This measure would probably also increase the information gain from the AMNOG process even further.

However, even after successful implementation of the new EMA policy stipulating the publication of full clinical study reports, AMNOG documents will still provide additional relevant information. Firstly, a systematic review of a new drug versus the appropriate comparator treatment, focusing on the approved subpopulation—AMNOG documents may thus contain subgroup analyses not available in the clinical study reports. Secondly, under the EMA policy, only reports submitted to EMA will be published, many of which refer to placebo controlled studies. Especially AMNOG documents on reassessed drugs (see box 1) may contain information from additional clinical study reports of active-controlled studies conducted after drug approval. AMNOG thus potentially enables access to information from a larger pool of clinical study reports than are available through EMA, with a higher proportion of reports of active-controlled studies.

Information on approved subpopulations

To the best of our knowledge, the problem of the lack of results data on drugs approved only in certain subpopulations has so far not been investigated. Our study shows that AMNOG documents are particularly helpful in these cases: complete reporting of results items in AMNOG documents compared with non-AMNOG documents was 71% versus 11%. These differences can largely be explained by the fact that the information reported in journal publications and registry reports mainly only summarises the information on the total study population as originally planned in the study protocol. It is, however, surprising that the reporting of approved subpopulations is so limited in European public assessment reports, which mainly summarise data on the total study population, irrespective of any restrictions imposed in the summary of product characteristics on the patient population.

For instance, the prescription of ticagrelor was restricted to a combination with an acetylsalicylic acid maintenance dose of a maximum of 150 mg, as the results of the Study of Platelet Inhibition and Patient Outcomes (PLATO) had indicated that increased doses of acetylsalicylic acid might lead to reduced efficacy.41 Since patients in the PLATO study were treated with up to 325 mg of acetylsalicylic acid, the analysis of the total study population alone might overestimate harms or underestimate benefits. Nevertheless, the European public assessment report contained no data on the approved subpopulation. The analysis of this subpopulation was provided by the company in the AMNOG dossier, and thus the results on the benefit and harms of ticagrelor in this population are now publicly available. Another example is rilpivirine, for which the European public assessment report presented results only on the approved subpopulation for the outcomes “virologic response” and “virologic failure.”19 Other outcomes (including adverse events) were not reported, even though they are important for the comprehensive assessment of the new drug, including the adequate weighing of benefits and harms.

These examples illustrate that evidence to assess new drugs in the indication finally approved can be generated from data available at the time of market access. Further discussion on implementation is required; for example, it should be explored whether the reanalysis of study data with the aim of generating information on the approved patient population could already be routinely provided by the regulatory process (for example, in European public assessment reports). Furthermore, the methods for reanalysis in specific situations might be explored to maximise the robustness of this analysis.

Information on specific outcomes

Current major public health decisions, such as those on the potential widespread prescription of statins,42 43 highlight the urgent need to provide complete information on harms. However, in our analysis, conventional publicly available sources reported only about 50% of the relevant data on harms for the total study populations and no data at all (0%) for the approved subpopulations. The corresponding rates in the AMNOG documents were 92% and 66%, indicating that there is still room for improvement in the reporting of approved subpopulations in the new assessment process.

Our analysis also highlights a further ongoing problem—namely, the inadequate reporting of symptoms and quality of life data, which seems to affect anticancer drugs in particular.44 Adequate information on these outcomes is especially important for decision making in oncology, where often only small gains in survival have to be placed in context with possible severe side effects.

Relevance of information from AMNOG for drugs approved outside Germany

In 2011 and 2012, 50 new drugs (excluding vaccines, combinations of known drugs, and extensions of indications) were centrally approved in Europe.45 By September 2014, 31 of these drugs underwent an AMNOG assessment, as described here. No suitable data for the assessment of added benefit were available for 14 of the 31 drugs. Ten of the 19 drugs not assessed according to the above procedure were orphan drugs, where a different procedure applies. Of the 68 new drugs approved by the FDA in 201146 and 2012,47 32 underwent an AMNOG assessment; no suitable data were available for nine drugs. The main reasons why FDA approved drugs were not assessed according to AMNOG were lack of approval in Europe (n=11), approval in Europe before AMNOG came into effect (n=11), or orphan drug status (n=8). Overall, AMNOG provides information on added benefit for a substantial proportion of new drugs approved in Europe and the United States.

Limitations of this study

Our analysis has several limitations. Firstly, it contains only a small sample of AMNOG documents and other publicly available documents used in the first early benefit assessments of new drugs in Germany. It therefore presents only preliminary descriptive findings on the potential information gain through AMNOG. A larger sample size would be needed to provide more robust conclusions. Secondly, we excluded those dossiers that provided no suitable studies for a full early benefit assessment. Therefore, our sample does not provide information on all new drugs entering the German market in the period investigated. Thirdly, we did not distinguish between possible grades of completeness in the category “partly reported.” Thus it is possible that, if combined across several document types, partly reported outcomes may have resulted in more complete information. Finally, the scope of our analysis is further limited because we did not consider FDA reports: as new drugs assessed according to AMNOG are generally approved by EMA, we only included European public assessment reports.

Implications for HTA

Our analysis shows that HTA can provide substantial information on new drugs as early as at the time of market entry. Although the primary goal of AMNOG was to inform price negotiations, the assessment process has made detailed information on a new drug’s effects available to the public, as a comprehensive reporting format and mandatory publication of this information is required by law. Although HTA processes in other countries also aim to include all relevant information, to the best of our knowledge no other HTA agency is supported by legislation to publish the extent of information included in AMNOG documents. Our results emphasise that HTA should not be based on conventional, publicly available sources or on any selected presentation of data by pharmaceutical companies but should include comprehensive study information, including clinical study reports, wherever possible.

Could the AMNOG reporting format serve as a model for open access study information?

The considerable information gaps seen in our sample of conventional, publicly available sources strongly support an expansion in the amount of publicly available information beyond that contained in summaries. This expansion would provide a level of detail potentially enabling the unbiased assessment of a study.

The AMNOG documents are only partly available in English, which is why they are of limited use to international researchers requiring comprehensive information on a new drug. However, the overall structure of the documents could be developed into a general model for comprehensive open access study information and would not be restricted to Germany. In this model a full public record of a clinical study would include the full clinical study report (including the study protocol and the analysis plan), anonymised individual patient data (for example, for analyses of subpopulations), the registry report, and all relevant journal publications. These documents could be used by different target groups for different purposes. For example, clinical study reports could be used by HTA agencies and other systematic reviewers to prepare HTA reports and systematic reviews so as to inform health policy and individual treatment decisions. Individual patient data could be used to further explore a drug’s effects in patient subpopulations or to address methodological questions such as surrogate validation. Any new analysis based on the study information could be linked to the study record, resulting in an increase in the publicly available information gained from the data.

While indepth study information would help to support the further use of the data, it is still important to publish summary formats; for instance, improved European public assessment reports and journal publications could be used by doctors to quickly gain an overview of the effects of a new drug. The availability of the full study record would also allow the scientific community to cross check summary information.

Conclusion

Conventional, publicly available sources (European public assessment reports, journal publications, and registry reports) provide insufficient information on new drugs, especially on patient relevant outcomes in approved subpopulations. This type of information is largely available in AMNOG documents, albeit only partly in English. Our results show that comprehensive information on new drugs is potentially available and should be provided to support decision making in health care, especially in approved subpopulations. The AMNOG approach could be used internationally to develop a comprehensive publication model for clinical studies and thus represent a key open access measure.

Selective publication of clinical studies and their results is a major problem in clinical research and leads to overestimation of benefits and underestimation of harms of treatments

Deficits in the reporting of clinical studies have been shown for publicly accessible documents such as regulatory reports—for example, European public assessment reports, journal publications, and registry reports

Clinical study reports submitted to regulatory authorities during the drug approval process provide additional relevant study information, but are so far not publicly available

Documents generated in accordance with the German act on the reform of the market for medicinal products (AMNOG) provide considerably more information on the methods and (patient relevant) outcomes of clinical studies on new drugs than other publicly available sources

The apparent information gap is most striking when the drug is approved only in a certain subpopulation

AMNOG documents could be used internationally to develop a comprehensive publication model for clinical studies