DAMOCLES: Data Monitoring Committees; DMC: Data Monitoring Committee; RCT: Randomized Controlled Trial

Stopping a trial incurs significant cost and thus this is not a decision that should be taken lightly. Statistical boundary is a guideline and not a rule [1]. Therefore, the DAMOCLES protocol suggested Data Monitoring Committee (DMC) to understand all options and their implications [2]. We propose a real option framework to quantify the value of early stoppage to better understand the cost (both monetary and human) of early termination.

Stopping rule and its challenges

Rules of thumb-beyond reasonable doubt: The concept of beyond reasonable doubt is consistent with p<0.001 as in the simple Peto- Haybittle rule [1].

Statistical Power: Bayesian statistical reasoning has been proposed as a better way to make stopping decision. The decision to stop should be made by weighting the consequence of possible actions and then averaged over the distribution of future observations based on all available information [3]. Even though it is theoretically sound to adopt this approach, it might not be feasible because of the limitation to map out the future in Savage’s “large worlds” where relevant information must be estimated from small samples [4].

It is well known that treatment effects could be overestimated by chance due to “random high” [5]. It has also been suggested that stage wise ordering technique could be used to calculate the statistics [6].

Data-dependent stopping rules have also been developed to provide a framework to perform interim analyses [7]. These methods define p values for considering trial stoppage at an interim look while preserving the overall Type I error. One of the techniques is conditional power that accounted for the impact of power given a trend. These methods have been used effectively in monitoring randomized controlled trials.

Number and severity of adverse events

If the number of occurrence or the severity of adverse effect such as death or other permanent damage has reached an unacceptable level, we are ethically obligated to stop the trial even without high statistical power. It is called a symmetric stopping rule if benefit and harm are treated equally. Nevertheless, asymmetric stopping boundaries are often being set instead [7]. As a result, truncated RCTs tend to have greater effect sizes than other RCTs [8].

DMC (Data Monitoring Committee) potential bias

Cognitive dissonance or group think: It is common for new evidence on old treatment be ignored by practitioners and this could be explained by cognitive dissonance. The damage caused by an early stopping intensive insulin therapy could be a cautionary tale [9].

Cognitive bias: It is well documented that people are overconfident about their own relative abilities and unreasonably optimistic about their future [10]. Since it is likely that the DMC has some positive a priori information about the treatment effect, it has been shown that RCTs stopped early for benefit are becoming more common [11]. Besides, the decision of DMC could also be affected by decision framing as they might put more weighs on recent events [12].

Real option as at to evaluate a stopping decision

What is a real option?: Real option analysis is a conceptual framework to analyze a choice (an option) that the decision maker can act or not act on. This analytical framework allows us to quantify the value of this flexibility.

How can it help?: The original option pricing framework was created to price financial options traded in the commodity market. Later on, the option concept was extended to capital investment decision analysis. We use this tool to analyze a situation when we have an option to defer, to expand, to contract, to shut down, to abandon, or to switch [13]. Real option approach in health care capital has been used in investment analysis [14,15]. Even though the “net social benefit” concept and a simple hypothetical example have been used to explain the real option concept, the authors have not to link those concepts to real application [14]. Therefore, I am going to use the real option concept to estimate the value of being able to stop a real AIDS trial in Africa [16,17].

An illustrative example based on a RCT: A RCT was implemented to investigate the vitamin supplements in relation to transmission of HIV-1 through breastfeeding and early child mortality in Tanzania [16]. The vitamin A arm was dropped based on interim analyses of primary endpoints. When compared to the Pocock rule with a more conservative O’Brien-Fleming rule, a researcher found that O-F rule would not have allowed the DMC to stop early due to its extreme statistical boundaries [17]. My numerical examples below are based on his section on simulating early stopping with a loss function. It should be noted that the unit used below is for illustration purpose. The unit could be utility (based on economics theory) or benefit translated into monetary value (e.g. QALY-Quality of Adjusted Life Year) (Figure 1).

Figure 1: Decision tree of DMC.

Figure 1 is the decision tree based on a research on the aforementioned AIDS trial [17]. The author reconstructed the stopping rule and arrived at 41.6% efficacy. Therefore, the null hypothesis is: H0 is true if efficacy=0. The alternative hypothesis is: H1 is true if efficacy=41.6%. The author assumed that the cost of running the trial up to that point is-10000 unit and the cost of running the trial is three times its value (i.e.-30000) and

E (“stop early”, efficacy=0)=-10000

E (“stop early”, efficacy=41.6)=efficacy × number of patients+cost=41.6 × 180+(-10000)=-2512

E (“go until the end”, efficacy=0)=-30000

E (“go until the end”, efficacy=41.6)=41.6 × 1000-30000=11600

Let p be the probability that H1 is true, the trial should go ahead if and only if the expected payoff of “go until the end” is greater than that from “stop early”. Therefore, the trial should continue if p>0.59.

Here is an extension to this analysis. Suppose we cannot stop the trial early. So, the top arm is irrelevant and only the bottom arm remains. We will be indifferent to continue the trial or not if the expected payoff is 0. This implies

Expected payoff=-30000 (1-p)+11600 p=0 → p=30000/41600=0.72

Please note that the hurdle rate (the probability that H1 is true) is higher when stopping early is not an option. In another words, the difference between these two cases (0.72-0.59)=0.13 is the value of having an option to stop early. For example, if the payoff is 10000, one should pay only up to 0.13 × 10000=1300 to acquire the right to be able to stop the trial early if she wants.

It is generally not a good idea to stop a trial early as the results are sub-optimal. I proposed a real option framework to evaluate the value of the early stopping option. This novel approach offers a unified framework to assess the expected payoff in term of efficacy and cost. The researchers in trials can then adopt this framework to estimate the cost and the benefit of able to stop a trial early. This has strong implication of trial design because the trial designer may want to incorporate the flexibility of stopping a trial early given the cost of being able to do so. There are of course other reasons such as higher than expected mortality or severe complications for early stoppage. However, this paper offers a framework to look at this decision inside a consistent analytical framework.

The authors declare that they have no competing interests.

AL: conception and design, data collection and analysis, manuscript writing and final approval of the manuscript.

I work for the Blue Shield of California and this article reflects his personal opinion only. It does not necessarily reflect those of my employer or my any other affiliation. I do not receive any funding for this article. This article is based on one of my coursework for the RCT class held in Oxford on September 2012. Without those inspirational lectures, it is just impossible for me come up with this idea.