3 Committee discussion

The appraisal committee (section 6) considered evidence submitted by Bristol–Myers Squibb Pharmaceuticals Ltd. and a review of this submission by the evidence review group (ERG). See the committee papers for full details of the evidence.

Clinical need and current management

People with completely resected stage III and IV melanoma have a high unmet clinical need

3.1 Melanoma is becoming more common and often affects people at a younger age than some other cancers. It has a substantial effect on patients, their carers and the wider society. Five‑year survival estimates are about 50 to 55% for stage III disease, and 8 to 24% for stage IV disease. People with fully resected melanoma are still at high risk of disease recurrence; 5‑year relapse-free survival is 28 to 44% for stage III melanoma, and less for stage IV melanoma. The clinical and patient experts noted that significant developments in the treatment of melanoma in recent years, particularly the introduction of immunotherapies in the metastatic setting, have had a positive effect on the life expectancy and quality of life of people living with advanced disease. The patient expert emphasised the importance of access to additional treatment options, particularly in the adjuvant setting, for people living with melanoma. The committee concluded that people with fully resected stage III and IV melanoma have a high unmet clinical need and would value new treatment options.

Adjuvant treatment would change the treatment pathway for people with fully resected stage III and IV melanoma

3.2 The clinical experts noted that resection of the tumour and associated lymph nodes in people with evidence of regional node metastases is the standard first-line treatment for most people with stage III melanoma, and that some people with stage IV melanoma have resectable disease. They explained that surgical practice is changing for patients with stage IIIA disease because of publication of the MSLT2 trial; this showed that there is no overall survival benefit after complete resection of the remaining regional lymph nodes, compared with removal of the sentinel lymph nodes only. All patients in the key trial for adjuvant nivolumab (CheckMate 238) had had full resection. However, the clinical experts also explained that using nivolumab in the adjuvant setting was unlikely to influence surgical practice. Currently, the standard of care for people with completely resected stage III and IV melanoma is routine surveillance. This includes regular clinical review and imaging. Adjuvant radiotherapy and immunotherapy after tumour removal are not widely used in UK practice. The clinical experts explained that the aim of adjuvant treatment is to remove any residual microscopic disease after resection to reduce the risk of relapse and progression to metastatic disease, which is currently considered incurable. If the curative aims of adjuvant treatment are met, then this would represent a substantial benefit to patients. However, if nivolumab is used in the adjuvant setting, it might affect subsequent treatment for people who develop disseminated disease. The committee concluded that nivolumab as an adjuvant treatment will change the treatment pathway for stage III and IV fully resected melanoma.

Clinical evidence

Nivolumab has not been directly compared with routine surveillance in a clinical trial

3.3 There are no head-to-head trials comparing nivolumab with routine surveillance in the adjuvant setting. The key trial in the company submission was CheckMate 238, an ongoing multinational randomised double-blind study. It compared adjuvant nivolumab with adjuvant ipilimumab in 906 patients (aged 18 years or over) who have had complete resection of stage IIIB, IIIC or IV melanoma. Ipilimumab is not used for the adjuvant treatment of melanoma in England. At the latest data cut (19 December 2017), patients in CheckMate 238 had been followed for a minimum of 24 months. A statistically significant improvement in recurrence-free survival was seen with nivolumab compared with ipilimumab (hazard ratio [HR] 0.66, 95% confidence interval [CI] 0.54 to 0.81; p<0.0001). Investigator-assessed disease recurrence or death was reported in 171 (37.7%) and 221 (48.8%) patients who had nivolumab and ipilimumab respectively. The committee acknowledged that although median recurrence-free survival had been reached (at 30.8 months in the nivolumab arm compared with 24.1 months in the ipilimumab arm), the data were still immature, with heavy censoring in the Kaplan−Meier curve. It also noted that although median follow‑up had not been reached for the secondary outcome of distant metastasis-free survival at the most recent data cut (minimum of 24‑months follow‑up), a statistically significant benefit for nivolumab compared with ipilimumab had been shown (HR 0.76, 95% CI 0.59 to 0.98). The committee also accepted that nivolumab appears to be better tolerated than ipilimumab. It concluded that nivolumab is a more effective treatment than ipilimumab in terms of recurrence-free survival. However, it emphasised that CheckMate 238 has not provided any evidence on the relative efficacy of adjuvant nivolumab compared with routine surveillance, which is the comparison of interest for the appraisal.

The company's updated indirect treatment comparison for recurrence-free survival for the period between 12 weeks and 10 years is suitable to inform decision making

3.4 Given the lack of a direct comparison with routine surveillance, the company did an indirect treatment comparison for recurrence-free survival using data from CheckMate 238 and another multinational randomised double-blind trial (CA184‑029). CA184‑029 compared ipilimumab with placebo in 951 patients (aged 18 years or over) with high-risk stage III cutaneous melanoma who had had complete regional lymph node dissection. Because the company had access to the individual patient data for both trials, it chose to do an individual patient data meta-regression analysis. It used this approach to generate parametric survival curves for adjuvant nivolumab and routine surveillance to determine the treatment effect for recurrence-free survival between 12 weeks and 10 years (other sources were used up to 12 weeks and after 10 years). Log-logistic curves were selected based on goodness of fit to the observed data from CheckMate 238 (assessed on visual inspection and by statistical measures) and clinical plausibility according to expert opinion. The committee recognised that the additional evidence submitted by the company during consultation used the approach preferred by the committee and the ERG of excluding patients who had had ipilimumab for more than 1 year in CA184‑029 from the indirect treatment comparison. The committee still thought that differences in the inclusion criteria for CheckMate 238 and CA184‑029 about what stage disease people had were potentially not fully accounted for in the indirect treatment comparison adjustments. However, having considered comments from consultees and clinical experts, it concluded this was not of enough concern to undermine the reliability of the company's indirect treatment comparison for recurrence-free survival. The company's updated indirect treatment comparison for recurrence-free survival was therefore considered acceptable for decision making.

Adjuvant nivolumab may improve overall survival, but more data from CheckMate 238 are needed

3.5 Mature overall survival data from the ongoing CheckMate 238 trial were not available at the time of the most recent data cut (19 December 2017). The company provided overall survival data from an unplanned analysis of the patients in CheckMate 238 who had been followed up for at least 24 months. The committee noted that although the overall survival data were very immature, there was general agreement between consultees and invited clinical experts that, based on their wider experience with immunotherapy treatments in other cancers, it was reasonable to expect that the recurrence-free survival benefit seen in CheckMate 238 would be translated to some extent into an overall survival benefit. The committee therefore concluded that adjuvant nivolumab may improve overall survival compared with routine surveillance. However, until overall survival data are reported from CheckMate 238 and analysed in comparison with routine surveillance in a robust indirect treatment comparison, the survival benefit with adjuvant nivolumab cannot be confirmed.

Other considerations

Clinical experts believe that adjuvant nivolumab could provide a meaningful clinical benefit

3.6 The clinical expert recognised the immaturity of the CheckMate 238 data and the uncertainty resulting from the need for an indirect treatment comparison. They acknowledged that there is currently no evidence about whether adjuvant treatment will affect the effectiveness of subsequent treatments for metastatic disease, and that this is a key concern when deciding the potential benefits of making changes to the melanoma treatment pathway. They noted that the curative potential of adjuvant treatment, combined with the high risk of recurrence in patients with stage III melanoma, means that access to adjuvant nivolumab should be considered despite these uncertainties. They explained that the increased efficacy of immunotherapies in patients with low (compared with high) volume metastases provides a biologically plausible rationale for why it might be beneficial to use nivolumab earlier in the treatment pathway. They recognised that the uncertainty in the clinical evidence reinforces the need for discussion of the potential benefits and harms of preventative treatment when offering adjuvant nivolumab to patients. They also acknowledged that the benefit-to-risk ratio may differ for patients with stage IIIA disease because of their lower absolute risk of recurrence. However, they emphasised that it is important to consider adjuvant nivolumab for all patients with stage III and IV melanoma because there is no evidence of a difference in recurrence-free survival by stage of disease in CheckMate 238. The committee recognised that many of the comments made by the clinical expert were echoed by comments made by consultees who supported use of adjuvant nivolumab in the Cancer Drugs Fund while further evidence on efficacy is collected. The committee concluded that some clinicians are confident that, despite the limited evidence, adjuvant nivolumab could provide a meaningful clinical benefit for patients with stage III and IV resected disease.

Adverse events

Although nivolumab is well tolerated, toxicity risks are very important for a preventative treatment

3.7 CheckMate 238 showed that nivolumab was generally well tolerated. The clinical and patient experts explained that this was also the case in clinical practice, particularly compared with ipilimumab and chemotherapy. The committee noted that the common side effects that happen during treatment are generally manageable. However, immunotherapy (such as nivolumab) works by altering the immune system, and the clinical experts explained that about 10 to 20% of people develop irreversible endocrine disorders, in particular thyroiditis, with nivolumab. The committee was aware that some people who have fully resected stage III disease do not relapse (see section 3.1). Clinical experts explained that, for people considered to be at lower risk of relapse, careful assessment and discussion about the risks and potential benefits of adjuvant nivolumab would be needed. The committee concluded that although the risk of adjuvant nivolumab inducing serious adverse events is likely to be small, it could result in some people who would not have relapsed on routine surveillance having long-term irreversible adverse effects. It agreed with the experts that careful assessment of the likely benefits of treatment would be important.

The company's economic models

Both of the company's model structures are potentially acceptable for decision making

3.8 Following consultation, the company submitted 2 updated models comparing adjuvant nivolumab with routine surveillance: a partitioned survival model and a Markov model (which was an updated version of the Markov II model in its original submission). The incremental cost-effectiveness ratio (ICER) associated with each model was as follows:

  • Updated partitioned survival model ICER: £18,423 per quality-adjusted life year (QALY) gained compared with routine surveillance.

  • Updated Markov model ICER: £18,018 per QALY gained compared with routine surveillance.

    The company ICERs included a commercial access agreement for nivolumab and a patient access scheme for ipilimumab, but did not take account of commercial arrangements for other agents that may be used subsequently in the metastatic setting because the discounts are commercial in confidence. Exploratory analyses done by the ERG showed that when these discounts were included, the ICERs were slightly higher in both cases, but still within the range that may be considered cost effective. The committee recognised that, by providing alternative model options, the company showed that it had tried to investigate some of the uncertainties in its approach to estimating cost effectiveness. Both updated models had a cycle length of 28 days and a time horizon of 60 years, which was acceptable. The committee noted that both models used the same recurrence-free survival estimates based on the indirect treatment comparison, but used different data sources and assumptions about subsequent treatment to predict overall survival benefit. There were limitations relating to the overall survival and subsequent treatment inputs in both models (see sections 3.10 to 3.12). However, it also noted that, whereas the original versions of the models had produced very different ICERs, the results of the updated models were now more consistent. The committee noted that the company assumed the treatment benefit for recurrence-free survival continued after stopping treatment up to year 10. The committee noted the uncertainty in this assumption and considered that more mature data on both recurrence-free survival and overall survival would be crucial in determining the benefit of changing the current patient pathway of care to include adjuvant therapy. The committee concluded that, in principle, both model structures had the potential to be informative for decision making.

The company's calculation of recurrence-free survival before 12 weeks and after 10 years is complex

3.9 The results of the indirect treatment comparison only informed part of the company's analysis for recurrence-free survival for adjuvant nivolumab compared with routine surveillance (12 weeks to 10 years). Between 0 weeks and 12 weeks, the treatment effect for adjuvant nivolumab was based on the Kaplan−Meier data from CheckMate 238. For routine surveillance, a hazard ratio was applied to the Kaplan−Meier data from the placebo group in CA184‑029 (derived by fitting a Cox proportional hazards model to the ipilimumab groups of CheckMate 238 and CA184‑029). After 10 years, recurrence-free survival in each arm was calculated by applying a hazard ratio to the American Joint Committee on Cancer version 8 registry data for overall survival, based on a trial of interferon induction therapy in the adjuvant setting for people with completely resected melanoma (Agarwala et al. 2017). The committee noted that it was unclear whether the registry data reflected current overall survival in melanoma because of the recent advances in melanoma treatment. It also noted that interferon is not currently used in routine practice and has a different mechanism of action to nivolumab. Overall, the committee considered that the methodologies used to estimate recurrence-free survival for the comparison of interest were extremely complex and relied to some extent on data sources that were potentially inappropriate. This increased the uncertainty of the predictions. In summary, having taken account of the adjusted indirect treatment comparison results in the company's updated evidence submission and the ERG's critique, the committee concluded that, although the recurrence-free survival extrapolation seemed reasonable, the size of the benefit with nivolumab over the longer term is still uncertain.

Outstanding issues with the company's updated partitioned survival model

More mature overall survival data are needed to determine whether the company's updated ICER is robust

3.10 The company used overall survival data from CA184‑029 and CheckMate 238 to inform the overall survival extrapolations used in the updated partitioned survival model. Parametric survival curves fitted to the overall survival data for ipilimumab compared with placebo from the CA184‑029 trial were adjusted to account for the exclusion of patients with stage IV melanoma from CA184‑029. The company then added 0.5 to the mu parameter of the underlying survival function so that the ipilimumab curve was better aligned with the nivolumab arm of the Kaplan–Meier curve for overall survival from CheckMate 238. The effectiveness of the routine surveillance arm was also increased proportionally. The resulting model assumed that ipilimumab is as effective as nivolumab for overall survival (that is, they are equally effective for overall survival compared with routine surveillance). The ERG remained concerned about the company's approach because the relative treatment effect for adjuvant nivolumab compared with routine surveillance was entirely based on the ipilimumab compared with placebo effect in CA184‑029. This was partly driven by the difference in subsequent treatments that patients had in each arm of the trial. This may overestimate the relative effectiveness of adjuvant nivolumab compared with routine surveillance because more effective treatments are now available in the metastatic setting than at the time of the trial. Therefore ipilimumab may not offer such a large overall survival benefit compared with routine surveillance in current clinical practice. Furthermore, the company had not revised any of their original assumptions about subsequent treatments, which also affects the validity of the overall survival projections (see section 3.11). An ERG exploratory worst-case scenario (where adjuvant nivolumab was assumed to have no overall survival benefit over routine surveillance) produced an ICER of £80,401 per QALY gained (excluding confidential discounts for subsequent treatments other than nivolumab and ipilimumab). This was considerably higher than the company's equivalent base-case ICER, which was £18,423 per QALY gained (see section 3.8). The committee considered that this showed how sensitive the ICER is to the projected overall survival benefit. The committee noted that there were very limited overall survival data currently available from CheckMate 238 and concluded that it is not possible to resolve the uncertainty in the model predictions until more mature data are available.

More data are needed to validate the company's assumptions about subsequent treatments

3.11 Subsequent treatments that patients had in the adjuvant nivolumab and routine surveillance arms in the updated partitioned survival model were the same as those in the nivolumab and ipilimumab arms of the CheckMate 238 trial respectively. The updated model therefore continued to assume lower use of immunotherapy than might be expected in clinical practice. This meant that the subsequent treatment costs for the nivolumab arm were potentially too low. Also, because the overall survival extrapolations were linked directly to the overall survival data from CheckMate 238, it is not possible to explore different assumptions about the effectiveness of subsequent treatments in this model. The committee concluded that more real-world data on subsequent treatment used after adjuvant nivolumab in clinical practice would help to validate the model assumptions and reduce the uncertainty in the results.

Outstanding issues with the company's updated Markov model

The modelling of subsequent treatments and their effectiveness is still uncertain

3.12 The committee noted that the results of the Markov model were highly dependent on the subsequent treatments that patients had. The Cancer Drugs Fund clinical lead and clinical experts explained that people who relapse early after adjuvant nivolumab are likely to have ipilimumab alone, but re‑challenge with PD‑1 inhibitors (such as nivolumab and pembrolizumab) would be more likely for people who relapse after 2 years. In the updated model, the company adjusted the proportions of patients having different subsequent treatments in the adjuvant nivolumab arm. Early relapses (before 2 years) had ipilimumab; late relapses had the same range and proportions of subsequent therapies as patients in the routine surveillance arm of the model (matched to those that patients had in the ipilimumb arm of CheckMate 238). The ERG was concerned that there is no clinical evidence to support the assumption that PD‑1‑inhibitor re‑challenge at 2 years is as effective as shown in metastatic trials because the patients in the metastatic trials had not had previous treatment with a PD‑1 inhibitor in the adjuvant setting. It also noted that the company's evidence for the treatments used for metastatic disease in clinical practice suggested greater overall use of immunotherapies for metastatic disease compared with the proportions seen in the ipilimumab arm of CheckMate 238 (mostly because of more ipilimumab use), a greater use of pembrolizumab (based on prescribing data) and slightly lower usage of nivolumab (based on prescribing data and survey data from prescribing clinicians). The committee recognised the company's efforts in trying to address these issues. It also accepted the remaining uncertainty about which treatments people would have if their disease recurred after adjuvant nivolumab, together with the lack of evidence to support the assumption that giving PD‑1 inhibitors for a second time, 2 years after adjuvant nivolumab, would be equally effective as when these treatments are used for the first time in the metastatic setting (as assumed in the model). The committee concluded that neither the company's nor the ERG's analyses fully captured the true complexity of the post-recurrence treatment pathway.


Adjuvant nivolumab is not recommended for routine use in the NHS

3.13 The committee noted that when all commercial arrangements are taken into account, the company's base-case ICERs are within the range usually considered a cost-effective use of NHS resources (see section 3.8). However, the committee recognised that the clinical effectiveness of adjuvant nivolumab is very uncertain, because of the lack of mature overall survival data and the need for an indirect comparison. As a consequence, the cost-effectiveness estimates are also very uncertain. Also, the benefit of changing the strategy for managing completely resected stage III and IV melanoma from routine surveillance to adjuvant nivolumab is not known. Because of these uncertainties, the committee concluded that nivolumab for the adjuvant treatment of completely resected melanoma with lymph node involvement or metastatic disease cannot be recommended for routine commissioning in the NHS.

Cancer drugs fund

Adjuvant nivolumab is recommended for use in the Cancer Drugs Fund

3.14 Having concluded that nivolumab could not be recommended for routine use in the adjuvant setting, the committee considered whether it could be recommended within the Cancer Drugs Fund. It discussed the arrangements for the Cancer Drugs Fund agreed by NICE and NHS England in 2016, noting the addendum to the NICE process and methods guides. The committee accepted that the early trial results are promising. It noted that despite the current uncertainty about the clinical and cost effectiveness of adjuvant nivolumab, there is the plausible potential for nivolumab to be cost effective if further data confirm the current efficacy predictions for recurrence-free and overall survival. The committee also took the view that further data collection could help reduce the current uncertainty:

  • CheckMate 238 is ongoing and further data will become available for recurrence-free survival and overall survival, which would help to assess the validity of the overall survival extrapolations in the partitioned survival model and life-year gains predicted by the Markov model (see sections 3.10 to 3.12).

  • Observational data could be collected to provide information on the distribution of subsequent treatment used in clinical practice (see sections 3.11 and 3.12).

    Another ongoing trial, Keynote 054 (which is looking at the comparative efficacy of adjuvant pembrolizumab and placebo), may further contribute to the evidence of the clinical effectiveness of PD‑1 inhibitors as adjuvant treatments.

    The committee also noted that although both models were potentially acceptable, further survival data would be easier to incorporate into the partitioned survival model than the Markov model. The partitioned survival model is also less sensitive to the efficacy of follow‑on treatments, which is likely to still be uncertain at the point of review. The committee concluded that adjuvant nivolumab met the criteria for inclusion in the Cancer Drugs Fund and recommended nivolumab for use within the Cancer Drugs Fund as an option for people with melanoma with lymph node involvement or metastatic disease who have had complete resection if the conditions in the managed access agreement are followed.


The benefits of adjuvant nivolumab are captured in the measurement of QALYs

3.15 The company considered nivolumab to be an innovative treatment. The patient and clinical experts explained that there is an unmet need for adjuvant treatment. They explained that nivolumab has the potential to prevent people developing metastatic disease. The committee concluded that nivolumab would be beneficial for patients, but that it had not been shown evidence of any additional benefits that were not captured in the measurement of QALYs.

  • National Institute for Health and Care Excellence (NICE)