PhRMA – Adaptive designs software requirements subgroup
Adaptive Trials – User Requirements for Statistical Simulation and Analysis Software
Software requirements for adaptive designs using techniques which control the family wise error rate
Version 1.0; 21-Aug-08
TABLE OF CONTENTS
1 Introduction 3
2 Overview 4
3 Targeted audience / scope for software 5
4 Definitions 5
5 Design Requirements 7
5.1 Adaptive Trial Process 8
5.2 Design Requirement Specifics 10
6 Software Requirements 11
7 Statistical Methodology 18
8 Principles and Constraints 20
8.1 Data Principles & Constraints 20
1 Introduction
In the past few years, substantial effort within the pharmaceutical industry has been devoted to improving the existing drug development processes. Different approaches have been proposed for phase II and phase III clinical trials that give a significant opportunity to simultaneously improve decision making and saving both time and money; without compromising quality. Adaptive designs provide a promising approach to achieving this. A trial with an adaptive design incorporates the possibility of modifications in the trial design according to (pre-specified) rules, based on data collected from the on-going trial. A White Paper, consisting of several articles written by the members of the Adaptive Designs Working Group of PhRMA, discusses the areas in which adaptive designs can be beneficial and the various issues related to these designs. The White Paper appears in the journal, Drug Information Journal 2006; Vol 40, Issue 4.
Such adaptations, as dose selection, sub-group selection, sample size reassessments, stopping for futility or superiority, can be done to improve the characteristics of separate phase II dose finding trials or of phase III trials. In addition, it is often also possible to cover phase II (learning) and phase III objectives (confirming) in one seamless phase II/III trial. Such a trial combines what normally would be considered two separate trials into a single trial. If there is some form of data-driven adaptation and the data from both trials (or stages) of the seamless design are combined, then the trial is an Adaptive Seamless Design (ASD) Trial. The stages in a 2-stage adaptive design, e.g., are planned simultaneously allowing the 2nd stage to start immediately, adapted to the results of an interim analysis, after the 1st stage completes - otherwise the 2nd stage is aborted due to, e.g., lack of efficacy, or safety concerns.
Adaptive trial designs, in general, thus present many advantages over traditional approaches; improved decision making, more comprehensive exploration of dose-response relationships, better patient allocation to promising doses which are to be developed further (and reduced exposure to ineffective doses), reduced costs, adjustment of ongoing trials which may be heading towards failure, better use of the trial subject population, earlier stopping of futile trials, reduction of the time between trials or phases of development.
However, the adaptive and seamless design approach implies both increased complexity and a larger number of possible design options (like timing of interim analyses, decision rules for adaptations, population of the study arms, and early rejection or acceptance of hypotheses tested, analyses procedures etc.) that all need to be compared and validated to find optimal solutions with respect to appropriate success measures. Sometimes, this can be done analytically, but in many cases, it will require simulation of trials and their results under various design scenarios.
This paper provides initial and high-level proposals for the key business requirements of such adaptive design and analysis software. The scope of this document relates to those trials intended for submission purposes in which strict control of the false positive rate (a-level) is necessary, that is to say, adaptive trials within the latter part of drug development. Although the primary focus within this document is pivotal trials, many of the functional aspects within these user requirements are clearly applicable to explorative flexible designs, e.g., assessment of operating characteristics under varying assumptions, designs options, timing of analyses and so on.
2 Overview
The principal adaptations envisaged (and combinations thereof) in this version of the software requirements are:
§ Treatment arm selection (e.g. which dose(s) of the study drug to use in phase III out of the ranges used in phase II) – also possibly selection of comparison arm: placebo or active comparator
§ Sample size re-assessment (e.g. re-calculation of size of the phase III part of the trial in the light of the phase II results) - this could be blinded (looking at the combined-groups variance in the subject responses) or un-blinded (looking at the within-group variance or even taking into account the size of the response to the study drug)
§ Early termination – deciding to stop the study early for efficacy or futility because the actual results are markedly better or worse than expected
Less likely adaptations (to be considered for any future versions of the software requirements) may also allow analysis of:
§ Selection between alternative endpoints (e.g. which endpoint to use as the primary)
§ Selection of subject population (e.g. restrict phase III to a sub-population of those treated in phase II)
§ Dynamic randomization
§ Bayesian methodology
To protect against erroneously concluding a treatment effect (a-level control), different methods of multiplicity adjustment can be implemented and many of these methods can be combined with the closure principle. The closure principle may be used to adjust the rejection probabilities in cases of multiple tests of null hypotheses. Adjustments for analysis of data after each stage can be done by either a combination p-value or a conditional error function approach. It is also possible to include methods for multiple adjustments not based on the closure principle.
The potential benefits of adaptive designs as discussed in these requirements are envisaged as:
1. Reducing the number of subjects exposed to ineffective or unnecessarily high doses
2. Rapid termination of clinical trials that are highly likely to fail if otherwise completed
3. Increased power of the considered clinical trial
4. Thhe potential for continued monitoring of the relevant phase II subjects during phase III to provide earlier long term safety data.
5. Time savings, e.g., the elimination of the normal 6-9 month interval between the completion of a successful phase II and the start of the confirmatory phase III trial.
6. The ability to include relevant phase II subject responses in the phase III analysis – reducing the size (and hence time and cost) of the phase III part of the trial compared to a standalone trial.
Frequently, such an approach results in a more complex matrix of planning design options. Additionally, in order to successfully validate and optimise the trial design options, simulation techniques are required..
While this overview is focussed on confirmatory trials within the drug development process, it should be noted that the methodology for adaptive clinical trial is not limited to these types of trials. In fact, adaptive trials are intended to overcome some of the challenges within the phase I-IV paradigm (as alluded to within the Critical Path Initiative). The anticipated simulation functionality would also be applicable in other contexts, e.g. seamless phase I/II trials or combination of trials within a phase (IIa and IIb, for example). For many types of adaptive designs, leveraging statistical approaches that control the false positive rate in the strong sense (by construction) is not necessary. Such designs could utilize the approaches in this document however, many other analytic options / statistical approaches exist, e.g., Bayesian methodologies, and these may be preferred. The software user requirements for these types of adaptive designs are for future discussion.
3 Targeted audience / scope for software
The primary audience for this envisaged software are statisticians, or strongly quantitative persons who understand statistics but are not necessarily experts in adaptive designs. The software should allow for a graphical user interface (GUI) and/or writing a script (either via the GUI or external to the GUI) to run in batch mode.
4 Definitions
The following section contains definitions of the key concepts used in subsequent sections.:
DefinitionAdaptation / A predefined modification to a trial or between trials. In this scheme, this can comprise: early termination, treatment arm selection or sample size re-assessment.
Adaptive Seamless Design / The design of a program of trials where pre-defined modifications to the trial can be made between stages within and between the trial. All relevant data from the last break point on is used in the final analysis.
Alpha Spending Function / A function specifying the rate that the Type I error (a) is spent at the interim and final analyses
Beta Spending Function / A function specifying the rate that the Type II error (b) is spent at the interim and final analyses
Breakpoint / The period that separates independent trials or trial parts in a program, with no data carried over.
Clinical trial / Any investigation in human subjects intended to discover or verify the clinical, pharmacological, and/or other pharmacodynamic of one or more investigational medicinal product(s), and/or to study absorption, distribution, metabolism and excretion of one or more investigational medical product(s), with an objective of ascertaining its (their) safety and/or efficacy
Combination Analysis / Analysis of data collected across several stages since the last break point. As opposed to pooled data analysis, the data is combined across the interim points accounting for the interim decisions by some form of adjustment (e.g. combination p-values or a conditional error function approach). In these cases early rejection and un-blinded sample size adaptation is possible.
Combination fn analysis / Analysis of data collected across several stages since the last break point. As opposed to pooled data analysis, the data is combined across the interim points using a combination p-value rule. This is a special case of a combination analysis.
Combination p-value rule / Used to combine the p-values of the hypotheses in the different stages of a combination-fn analysis to produce an overall p-value.
Comparison Arm / A treatment arm that uses placebo or an active comparator (an approved alternative treatment); the effect of the treatment arms using the study drug are compared to this arm when analysing whether the study drug is effective or not.
Complete Simulation / A simulation, not using any observed data, that simulates a complete trial or trial part from start to finish including all interim points and all interim decisions. The simulation will involve two stages – simulating the results of individual subjects or directly simulating realizations of a summary measure like the test statistic, if this is possible, and statistical analysis of the combined simulations to determine the ‘simulated results’ from the whole study.
Conditional Simulation / A simulation that simulates the remaining stages of a trial conditional on the data observed up to a specified interim point and conditional on the decisions taken up to this point
Conventional Analysis / Standard test procedures applied to single-stage designs
Endpoint / The time and nature of the measure of subject’s responses, which (possibly relative to their score at baseline) indicates the final outcome of the trial for that subject.
Experimental Treatment Arm / A treatment arm where the subjects are given a particular dose or treatment regime that uses the study drug.
Final Analysis / The analysis of the study drugs effect compared to the comparison arm at the end of the last possible stage planned in the trial (real or simulated). All relevant data from the last break point on is used in the final analysis.
Group sequential design and analysis / A design with pre-planned interim analyses with the option to stop early for futility or efficacy, but no sample size reassessment and no treatment arm selection. Group sequential analysis is a special case of combination analysis.
Hypothesis (null or alternative) / The statement to be tested for rejection by a statistical test (null hypothesis) and the alternative to this statement (alternative hypothesis). Typically, the null hypothesis would be “no treatment effect”. The alternative hypothesis might then be “beneficial treatment effect”, for example.
Interim adaptation / A predefined modification to a trial or between trials, based upon the results of an interim analysis.
Interim Analysis / An un-blinded look at the study data at the end of a stage of a trial (real or simulated) in order to perform an adaptation. Decisions are made using the pooled data from the last break point up to that interim.
Interim Point / As opposed to a break point, the period that separates trial stages, with data carried over.
Intersection Hypothesis / An intersection of two or more hypotheses arising when applying the closed test procedure
Intersection p-value / P-value associated with an intersection hypothesis
Null Hypothesis / The generally accepted hypothesis for the study treatment having no effect (typically that there is no difference in response compared to the control group – however in a non-inferiority trial where the control group receives an alternative treatment the null hypothesis will be that the study treatment is to some specified degree inferior to the control group).
Pooled Analysis / Analysis of data collected across several stages since the last break point. The data is pooled over the interim points and analyzed as if there were no interim points.
Pooled data / Data collected across several stages since the last break point. The data is pooled over the interim points.
Scenario / A set of values for the unknown parameters of the trial, typically true magnitude of treatment effect or true variation of patient responses to treatment. Simulations are based on assumed values for these parameters.
Simulation / A probabilistic modeling of the trial for a given design and scenario. This is in two parts.- simulating the results for a set of sample subjects and analysing the combined set of subject simulations to provide a simulated result for the whole trial.
Simulation design (Design) / A set of values for all the controlled parameters of the trial.
Simulation results / The output from the simulation.
Single stage data / Data from a trial or trial part (bounded by break points) without an interim analysis.
Single-stage design. / A trial or trial part (bounded by break points) without an interim analysis
Stage / A section of a trial after which an analysis and possible adaptation is made. Stages are separated by interim analyses or break points.
Treatment arm / One of the (mutually exclusive) treatments that is being given to subjects in the trial in order to evaluate the study drug. A treatment arm is either a comparison arm or an experimental treatment arm.
Trial Part / Part of a trial starting and ending with a break point. A trial part may have one or several stages. A trial might consist of just one or several parts.
5 Design Requirements
5.1 Adaptive Trial Process
Figure 1 depicts an overview of the process associated with adaptive trial design and analysis.