GET THE APP

“Get Real with Stopping Rule”-Evaluating the Value of being able to Stop Early using the Real Option Framework
Journal of Clinical Trials

Journal of Clinical Trials
Open Access

ISSN: 2167-0870

+44 20 3868 9735

Research Article - (2015) Volume 5, Issue 1

“Get Real with Stopping Rule”-Evaluating the Value of being able to Stop Early using the Real Option Framework

Aaron Lai*
Market Analytics, Blue Shield of California, CA 94104, USA
*Corresponding Author: Aaron Lai, Senior Manager, Market Analytics, Blue Shield of California, 50 Beale Street, San Francisco, CA 94104, USA

Abstract

Background: Stopping a trial incurs significant cost and thus this is not a decision that should be taken lightly. Data Monitoring Committee (DMC) is recommended to understand all options and their respective implications. Since this is a complicated decision, people often adopt some “rule of the thumb” guidelines rather than a comprehensive framework to assess the pros and cons. This article is a first step toward a unified approach to trial. It should be noted that the use of monetary amount is for illustration only-it is just a medium of a collective value and this is not unlike the use of Qualityadjusted Life Year (QALY). The framework is still valid if other measures such as utility are used instead. This article will review some major elements in making early stopping decision and then it will explain how a real option framework can be used to quantity the cost and the benefit of being able to make such decision. Method: This article proposes an analytical framework based on real option concept to estimate the cost and the benefit of being able to stop a trial early. Results: Flexibility usually is associated with some implicit and explicit additional cost. This method allows one to quantity the benefit of being able to stop early and thus one can make a more informed decision on whether to acquire that early stopping possibility intrial design. The information contained in an early-stopped AIDS trial is used to illustrate this. Conclusion: The challenges of stopping rules are discussed and then a real option approach to evaluate the value of being able to stop a trial early is proposed. This approach allows us to put the value of this choice into a boarder perspective and thus we are able to quantify the cost of early stoppage. This novel technique allows the trial designers to decide if they should incorporate the early stopping flexibility in their clinical trials.

Keywords: Stopping rule; Randomized controlled trial; Real option

Abbreviations

DAMOCLES: Data Monitoring Committees; DMC: Data Monitoring Committee; RCT: Randomized Controlled Trial

Background

Stopping a trial incurs significant cost and thus this is not a decision that should be taken lightly. Statistical boundary is a guideline and not a rule [1]. Therefore, the DAMOCLES protocol suggested Data Monitoring Committee (DMC) to understand all options and their implications [2]. We propose a real option framework to quantify the value of early stoppage to better understand the cost (both monetary and human) of early termination.

Method

Stopping rule and its challenges

Rules of thumb-beyond reasonable doubt: The concept of beyond reasonable doubt is consistent with p<0.001 as in the simple Peto- Haybittle rule [1].

Statistical Power: Bayesian statistical reasoning has been proposed as a better way to make stopping decision. The decision to stop should be made by weighting the consequence of possible actions and then averaged over the distribution of future observations based on all available information [3]. Even though it is theoretically sound to adopt this approach, it might not be feasible because of the limitation to map out the future in Savage’s “large worlds” where relevant information must be estimated from small samples [4].

It is well known that treatment effects could be overestimated by chance due to “random high” [5]. It has also been suggested that stage wise ordering technique could be used to calculate the statistics [6].

Data-dependent stopping rules have also been developed to provide a framework to perform interim analyses [7]. These methods define p values for considering trial stoppage at an interim look while preserving the overall Type I error. One of the techniques is conditional power that accounted for the impact of power given a trend. These methods have been used effectively in monitoring randomized controlled trials.

Number and severity of adverse events

If the number of occurrence or the severity of adverse effect such as death or other permanent damage has reached an unacceptable level, we are ethically obligated to stop the trial even without high statistical power. It is called a symmetric stopping rule if benefit and harm are treated equally. Nevertheless, asymmetric stopping boundaries are often being set instead [7]. As a result, truncated RCTs tend to have greater effect sizes than other RCTs [8].

DMC (Data Monitoring Committee) potential bias

Cognitive dissonance or group think: It is common for new evidence on old treatment be ignored by practitioners and this could be explained by cognitive dissonance. The damage caused by an early stopping intensive insulin therapy could be a cautionary tale [9].

Cognitive bias: It is well documented that people are overconfident about their own relative abilities and unreasonably optimistic about their future [10]. Since it is likely that the DMC has some positive a priori information about the treatment effect, it has been shown that RCTs stopped early for benefit are becoming more common [11]. Besides, the decision of DMC could also be affected by decision framing as they might put more weighs on recent events [12].

Results and Discussions

Real option as at to evaluate a stopping decision

What is a real option?: Real option analysis is a conceptual framework to analyze a choice (an option) that the decision maker can act or not act on. This analytical framework allows us to quantify the value of this flexibility.

How can it help?: The original option pricing framework was created to price financial options traded in the commodity market. Later on, the option concept was extended to capital investment decision analysis. We use this tool to analyze a situation when we have an option to defer, to expand, to contract, to shut down, to abandon, or to switch [13]. Real option approach in health care capital has been used in investment analysis [14,15]. Even though the “net social benefit” concept and a simple hypothetical example have been used to explain the real option concept, the authors have not to link those concepts to real application [14]. Therefore, I am going to use the real option concept to estimate the value of being able to stop a real AIDS trial in Africa [16,17].

An illustrative example based on a RCT: A RCT was implemented to investigate the vitamin supplements in relation to transmission of HIV-1 through breastfeeding and early child mortality in Tanzania [16]. The vitamin A arm was dropped based on interim analyses of primary endpoints. When compared to the Pocock rule with a more conservative O’Brien-Fleming rule, a researcher found that O-F rule would not have allowed the DMC to stop early due to its extreme statistical boundaries [17]. My numerical examples below are based on his section on simulating early stopping with a loss function. It should be noted that the unit used below is for illustration purpose. The unit could be utility (based on economics theory) or benefit translated into monetary value (e.g. QALY-Quality of Adjusted Life Year) (Figure 1).

clinical-trials-Decision-tree

Figure 1: Decision tree of DMC.

Figure 1 is the decision tree based on a research on the aforementioned AIDS trial [17]. The author reconstructed the stopping rule and arrived at 41.6% efficacy. Therefore, the null hypothesis is: H0 is true if efficacy=0. The alternative hypothesis is: H1 is true if efficacy=41.6%. The author assumed that the cost of running the trial up to that point is-10000 unit and the cost of running the trial is three times its value (i.e.-30000) and

E (“stop early”, efficacy=0)=-10000

E (“stop early”, efficacy=41.6)=efficacy × number of patients+cost=41.6 × 180+(-10000)=-2512

E (“go until the end”, efficacy=0)=-30000

E (“go until the end”, efficacy=41.6)=41.6 × 1000-30000=11600

Let p be the probability that H1 is true, the trial should go ahead if and only if the expected payoff of “go until the end” is greater than that from “stop early”. Therefore, the trial should continue if p>0.59.

Here is an extension to this analysis. Suppose we cannot stop the trial early. So, the top arm is irrelevant and only the bottom arm remains. We will be indifferent to continue the trial or not if the expected payoff is 0. This implies

Expected payoff=-30000 (1-p)+11600 p=0 → p=30000/41600=0.72

Please note that the hurdle rate (the probability that H1 is true) is higher when stopping early is not an option. In another words, the difference between these two cases (0.72-0.59)=0.13 is the value of having an option to stop early. For example, if the payoff is 10000, one should pay only up to 0.13 × 10000=1300 to acquire the right to be able to stop the trial early if she wants.

Conclusion

It is generally not a good idea to stop a trial early as the results are sub-optimal. I proposed a real option framework to evaluate the value of the early stopping option. This novel approach offers a unified framework to assess the expected payoff in term of efficacy and cost. The researchers in trials can then adopt this framework to estimate the cost and the benefit of able to stop a trial early. This has strong implication of trial design because the trial designer may want to incorporate the flexibility of stopping a trial early given the cost of being able to do so. There are of course other reasons such as higher than expected mortality or severe complications for early stoppage. However, this paper offers a framework to look at this decision inside a consistent analytical framework.

Competing Interest

The authors declare that they have no competing interests.

Author’s Contribution

AL: conception and design, data collection and analysis, manuscript writing and final approval of the manuscript.

Acknowledgements

I work for the Blue Shield of California and this article reflects his personal opinion only. It does not necessarily reflect those of my employer or my any other affiliation. I do not receive any funding for this article. This article is based on one of my coursework for the RCT class held in Oxford on September 2012. Without those inspirational lectures, it is just impossible for me come up with this idea.

References

  1. SJ Pocock (2006) Current controversies in data monitoring for clinical trials. Clin Trials 3: 513-521.
  2. DAMOCLES Study Group, NHS Health Technology Assessment Programme (2005) A proposed charter for clinical trial data monitoring committees: helping them to do their job well. Lancet 365: 711-722.
  3. Berry DA (1993) A case for bayesianism in clinical trials. Stat Med vol. 12: 15-16: 1377-1393.
  4. Volz KG,GigerenzerG (2012) Cognitive Processes in Decisions under Risk are not the Same as in Decisions Under Uncertainty. Front Neurosci 6: 105.
  5. Briel M1, Bassler D, Wang AT, Guyatt GH, Montori VM (2012) The dangers of stopping a trial too early. See comment in PubMed Commons below J Bone Joint Surg Am 94 Suppl 1: 56-60.
  6. JWittes(2012) Stopping a trial early-and then what? Clin. Trials9: 714-720.
  7. Schulz KF and Grimes DA (2005) Multiplicity in randomised trials II: subgroup and interim analyses. Lancet 365: 1657-1661.
  8. D Bassler, M Briel, VM Montori, M Lane, P Glasziou, et al.(2010) Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA 303: 1180-1187.
  9. GH Guyatt, M Briel, P Glasziou, D Bassler, VMMontori (2012) Problems of stopping trials early. BMJ344: e3863-e3863.
  10. CamererC,Lovallo D (1999) Overconfidence and Excess Entry: An Experimental Approach. Am Econ Rev 89: 306-318.
  11. Montori VM, Devereaux PJ, Adhikari NKJ, Burns KEA, CH Eggert, et al. (2005)Randomized trials stopped early for benefit: a systematic review. JAMA 294: 2203-2209.
  12. Tversky A, Kahneman D (1981) The framing of decisions and the psychology of choice. Science 211: 453-458.
  13. Trigeorgis L(1996) Real Options: Managerial Flexibility and Strategy in Resource Allocation. MIT Press, 446.
  14. Palmer S, Smith PC (2000) Incorporating option values into the economic evaluation of health care technologies. J Health Econ 19: 755-766.
  15. Williams DR, Hammes PH (2007) Real options reasoning in healthcare: an integrative approach and synopsis. J HealthcManag 52: 170-186.
  16. Fawzi WW, Msamanga GI, Hunter D, Renjifo B, Antelman G, et al. (2002) Randomized trial of vitamin supplements in relation to transmission of HIV-1 through breastfeeding and early child mortality. AIDS 16: 1935-1944.
  17. Stanev R (2012) Modelling and simulating early stopping of RCTs: a case study of early stop due to harm. J ExpTheorArtifIntell24: 513-526.
Citation: Lai A (2015) “Get Real with Stopping Rule”-Evaluating the Value of being able to Stop Early using the Real Option Framework. J Clin Trials 5:209.

Copyright: © 2015 Lai A. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.