Can multisite clinical trial results change clinical practice? Use of long-acting injectable risperidone nationally in the Veterans Health Administration

Short-term placebo-controlled randomized clinical trials (RCTs), conducted to obtain Food and Drug Administration (FDA) approval for commercial marketing of new drugs, do not provide adequate information for determining whether the longer-term effectiveness and safety of such medications merit widespread replacement of older, often less expensive, drugs already on the market [1]. Practical clinical trials or comparative effectiveness studies compare new drugs to widely used comparators at multiple sites, under longer-term, real-world clinical treatment conditions, with minimal exclusion criteria. These studies have become recognized as critical for the evaluation of health benefits of new medications and for fostering their widespread use [2, 3].

Since the 1940s, the Veterans Health Administration (VHA), the largest healthcare system in the USA, has committed itself to the conduct of such trials through its Cooperative Studies Program (CSP) which maintains a unique infrastructure of coordinating centers within the VHA to support comparative effectiveness research across more than 150 Veterans Affairs (VA) medical centers nationwide for the specific purpose of improving treatment in VHA [4]. The program has been responsible for landmark studies of antibiotics for tuberculosis, antihypertensive medications, antipsychotics for schizophrenia, and many other major studies to “produce innovative and effective solutions to Veteran and national healthcare problems” [4].

In spite of this clear objective, there has been limited evaluation of the impact of individual VA cooperative studies on clinical practice in the VA healthcare system. There has long been concern about the often prolonged gap between published scientific findings and changes in clinical practice [5, 6]. Limited attention has been paid to the naturalistic dissemination of trial findings, such as those from the VA CSP, into clinical practice, especially in the healthcare systems that sponsored the research and in which it was conducted.

Schizophrenia, the focus of this study, is perhaps the most disabling of psychiatric conditions affecting approximately 1% of all adults and almost 100,000 VA patients each year at a cost of thousands of dollars per patient [7, 8]. A major challenge in the treatment of schizophrenia is that many patients lack insight into their condition and are inconsistent in their use of prescribed medications or refuse them, leading to recidivism and costly rehospitalization [9, 10]. One tool to improve medication adherence in the treatment of schizophrenia is the use of long-acting injectable (LAI) antipsychotic medications—medications administered via intramuscular injection releasing active medication slowly into the bloodstream, yielding therapeutic blood levels for 2 weeks after each injection, and in recent formulations for 4 weeks or even 6 months [11]. LAI medications are thought to reduce medication non-adherence but meta-analyses have yielded mixed results on the effectiveness of these medications compared to oral drugs [12, 13] albeit with more positive results in a recent meta-analysis, which highlighted pre-post studies that claimed to better reflect the real-world practices of prescribing LAI antipsychotics [14].

Beginning in the 1990s, a new type of antipsychotic medication, second-generation antipsychotics (SGAs), emerged which had fewer neurological side effects than previously approved drugs, i.e., first-generation antipsychotics (FGAs). It was hoped they would thereby lead to greater acceptance and more consistent use of these medications, but, in fact also had mixed results [15, 16]. It was only in 2003 that the first LAI SGA, LAI risperidone (Risperdal Consta®) became available, combining the benefits of LAI administration with the reduced neurological side effects of SGAs. As only 11% of diagnostically appropriate VA patients [17] and 10–20% of non-VA patients [18] were prescribed LAI medications, it was of substantial importance to evaluate the potential benefits of this new treatment. In 2005, the VA CSP implemented a multisite study comparing LAI risperidone to providers’ choice of oral antipsychotic medication (CSP#555) to evaluate these potential benefits before the drug entered widespread use. Results reported 6 years later found no significant reductions in hospitalization rates (the primary outcome) after a year of LAI risperidone treatment and no differences in secondary outcomes including schizophrenia symptoms, major side effects, quality of life, or other health outcomes [19]. The study was thus designed to address a critical question that has not been well studied: did this costly treatment merit widespread use?

The current study sought to trace the impact of CSP#555, an important null study, on VA clinical practice by using national VHA administrative data to compare trends in the use of LAI risperidone and other types of antipsychotic medications by Veterans diagnosed with schizophrenia or schizoaffective disorder during the 4 years (2007–2010) before the publication of CSP#555 in 2011 and the 4 years after (2012–2015). As the paper was published in early March 2011, we focused on differences in the years before and after 2011 in the annual proportions of Veterans prescribed LAI risperidone for the first time (new starts) and in the proportions of Veterans receiving LAI risperidone at all (any use) using multivariate analysis to control for potentially confounding sociodemographic and clinical characteristics. To understand contextual factors, we also examined changes in the use of other antipsychotics during these years including both LAI and oral medications, as well as changes in VA expenditures for LAI risperidone and other antipsychotics in this population. The original protocol noted that a null result could justify systemwide discouraging of LAI risperidone use in VHA as it was many times more expensive than other approved antipsychotic medications [20]. Accordingly, we hypothesized that in the years following the publication of the trial use of LAI risperidone would decline and be associated with reduced costs for LAI risperidone and for antipsychotics generally in the VHA.

As hypothesized, new starts on LAI risperidone in VHA, as well as its total annual use and related expenditures declined following the publication of a multisite randomized clinical trial of the medicine with null results, thus failing to provide justification for use of this medication. The policy option of discouraging its use had, in fact, been identified in the trial protocol [20] as a justifiable response to null study results, although no such policy has been implemented.

Perhaps the most striking feature of these results is that they could not have been anticipated in 2005 when CSP#555 was designed. What was known was only that medication non-adherence was a determinant of relapse and rehospitalization in schizophrenia and that LAI medication improved compliance. There was far more limited evidence that LAI antipsychotics reduced relapse or improved symptom outcomes. Second-generation antipsychotics appeared to be better tolerated than conventional medications [14] and the combination of LAI delivery and SGA tolerability was promising and in need of evaluation.

Although CSP#555 offered no support for the value of LAI risperidone, or, by implication, for LAI SGAs more generally, it seems likely that the rapidly expanded use of LAI paliperidone reflected the greater face-value of a 4-week as contrasted to a 2-week LAI medication and vigorous marketing of this new but only modestly different treatment whose superiority to LAI risperidone or any other treatment had not been demonstrated.

The scientific questions posed by CSP#555, in fact, remained unresolved. A RCT quite similar to CSP#555, published around the same time, also found no difference in relapse or rehospitalization between LAI risperidone and oral antipsychotics [23] and a second RCT comparing LAI paliperidone and LAI haloperidol [24] found no clinical advantage for LAI paliperidone, while a secondary analysis of VHA data found that LAI haloperidol was the more cost-effective treatment [25]. In contrast, an observational study examined VHA administrative data and reported cost-savings with paliperidone as compared to oral medications, albeit without random assignment [26].

The most recent meta-analysis of 137 studies, published 10 years after CSP#555 [14], found significant but small effects in both randomized trials (RR = 0.88 for relapse in 29 studies of 7833, p = 0.03) and cohort studies (RR = 0.92 in 44 studies of 106,136 patients, p = 0.004). More substantial effects were found only in studies with weak pre-post designs (RR = 0.44 in 28 studies of 17,876 patients, p < 0.0001).

Perhaps the strongest evidence favoring LAIs was published in 2020 from a 489-patient cluster randomized trial showing less risk of hospitalization or relapse among patients with early episode schizophrenia at sites randomized to encourage the use of LAI aripiprazole as compared to sites that provided usual care. This more supportive evidence, published 9 years after CSP#555, is of unknown generalizability to adults with long-term illness such as those treated by VHA. Thus, VHA prescribers increasingly used LAI SGAs in the years after 2009, in spite of the null findings from CSP#555 or positive evidence from other studies.

The discrepancy between empirical findings and subsequent changes in clinical practice observed in this study highlights the role of unanticipated non-research commercial factors in shaping the impact of major effectiveness studies. A similarly paradoxical result was documented in response to the NIMH-funded CATIE study of 1432 patients diagnosed with schizophrenia that compared several oral SGAs to each other [15]. Following its publication, use of the drug that did best on the primary effectiveness outcome of the study (olanzapine) actually declined in clinical practice, most likely due to increasing concern about its metabolic side effects, and due to the approval of a new actively marketed drug with fewer metabolic side effects (aripiprazole), not evaluated in CATIE, that showed substantially increased use, also without evidence of its superiority [27]. VA cooperative study #504 evaluating oral risperidone for PTSD (CSP#504) also reported unexpected null results in 2011 [28]. A subsequent national study of VHA pharmacotherapy for PTSD from 2008 to 2018 showed an annual decline of about 8.3% in risperidone use before CSP#504 publication with a 7.4% annual decline (virtually unchanged) over the next 7 years. Here, again no impact of the trial was observed but rather a persistent decline in risperidone use which also appeared to have been replaced with the new actively marketed drug, aripiprazole, albeit without evidence of its effectiveness in PTSD [29].

In contrast to CSP#555, some trials quickly lead to major changes in practice. As soon as recent trials demonstrated that COVID-19 vaccines were highly effective, they received regulatory approval and major efforts were initiated to foster their widespread use [30]. Similarly, once a major trial clearly revealed the risk of cardiac mortality with rofecoxib [31], the drug was withdrawn from the market altogether. These examples point to a major factor that is likely to affect the impact of individual RCTs—the magnitude of effects and seriousness of the health benefits or harms they demonstrate. CSP#555 (as well as CSP#504) found virtually no positive or negative effect and this may account for its limited impact. Another influential factor is whether a study leads to explicit policy changes or directives by governmental agencies as in the case of the FDA and CDC promoting COVID-19 vaccines.

The declining use of LAI risperidone after the publication of null findings from CSP#555 likely reflected the expanded use of LAI paliperidone more than study results, although the declining use of LAI risperidone may have been reinforced by null study findings. Unanticipated changes in the therapeutic environment can complicate the implementation of clinical trial results and deserve further attention.