Early value assessment interim statement

This section of the early value assessment interim statement describes the processes and methods that are being used for the early value assessment pilots. Different approaches are being tested to identify where efficiencies can be made and where alternative approaches could be used, see table 3 in the appendix for different approaches being tested for each topic. The intention of this statement is to provide an overview of the methods and processes but not be prescriptive in detail, to allow flexibility to adapt and change approaches during the pilots. It should be read alongside NICE health technology evaluations: the manual (relevant sections of the manual have been referenced throughout this document).

The objectives of the early value assessment process and methods are to:

• identify the evidence that is available on the technologies

• explore if technologies have the potential to address the identified unmet need

• identify important evidence gaps to help direct evidence generation

• determine if technologies should be used while further evidence is generated.

The overall timeframe of an early value assessment is approximately 6 months. This includes 8 weeks for scoping, stakeholder identification and specialist committee member recruitment; 9 weeks for external assessment; and 7 weeks for guidance production and public consultation.

The following section outlines the stages of an early value assessment and the key areas for pilots (for a summary of the types of approaches being trialled per pilot see tables 3 and 4 in the appendix).

Topics for the early value assessment pilots include technologies that address an unmet need or priority area, different types of technology, different purposes or use cases and differing availability of infrastructure for real world data collection. The aim of selecting a wide variety of topics encompassing these aspects is to test that early value assessment is a useful approach for all different types of health technology and use cases, and to show proof of principle of what could be achieved through early value assessment when strong partnerships are in place.

Technologies selected for early value assessment will be scoped in line with section 2 of NICE's health technology evaluations: the manual, except for the following process changes:

• An advert for specialist committee members will be posted on the NICE website for 4 weeks. The panel for shortlisting and interviewing applicants will consist of the committee chair or vice chair and an associate director or their appointed deputy.

• Consultation of the scope may take place. When a consultation period is deemed necessary, it can take place before or after the scoping workshop.

• Where a scoping workshop is needed, all registered stakeholders will be invited to attend. After this, special committee members will be invited to an assessment subgroup workshop where the draft scope and key consultation comments will be discussed before final scope amendment and release. The scoping process will be expedited for the early value assessment pilot process.

The standard approach to assessing the evidence for a NICE evaluation is outlined in section 3 of NICE health technology evaluations: the manual. The way that NICE assesses evidence has been amended for the purpose of the early value assessment pilots.

The evidence assessment processes being piloted include:

• an external assessment group producing an assessment report for the early value assessment pilot projects in 8 to 10 weeks

• the assessment report may be released to stakeholders for comment before the committee discussion

• stakeholders being informed of the dates of any comment period at least 1 week before documents are released

• a streamlined approach to internal approval of final documents within NICE and the need for a resolution period.

The remainder of this section outlines the methodology for reviewing the evidence for an early value assessment.

It is expected that there will not be a comprehensive evidence base available for technologies included in early value assessment. The evidence considered by the committee should be relevant to the evaluation in terms of patient groups, comparators, perspective, outcomes and resource use as defined in the scope wherever possible. The aim of the evidence review is to identify the most relevant evidence relating to the decision question defined in the scope. If no evidence directly relevant to the evaluation is available, inclusion criteria should be expanded to look at a broader evidence base.

In addition to reviewing the evidence on the technologies, additional reviews may be needed to look for studies that report on relevant information, but do not include the intervention technologies. For example, evidence on treatment effect for a condition in which people have a biomarker identified by the new technology or evidence on a relationship between surrogate and final clinical outcomes. Identifying whether there is such evidence will help the committee identify whether there are data gaps across the care pathway that would lead to uncertainties in a future model.

Broad evidence mapping searches may need to be done to identify evidence on the technologies because articles may be published in less well-known journals, studies may not be well indexed or may only be presented as conference abstracts. Unpublished data provided by companies and other stakeholders could be considered.

The evidence reviews may be done using pragmatic approaches in order for the assessment to be done in the time available. For example, using rapid review methods such as single screening and data extraction. For evidence reviews of outcomes where a large amount of data could be available, a restricted number of databases could be searched or a review of review done.

Searches for ongoing studies should be done. Any ongoing studies that were identified should be checked to see if they will help to fill key evidence gaps.

A full critical appraisal of studies using a validated tool is not needed, but there should be discussion on the potential biases in key studies and how the risk of bias could affect key outcomes. The report should explicitly detail the potential sources of bias such as the main confounding factors. Comments on the generalisability of the results to clinical practice in the NHS should also be made.

Evidence synthesis of the key findings should be provided in a simple narrative and descriptive format, a quantitative meta-analysis is not expected. Where the direction of the effect (and associated confidence intervals) is reported in the literature it should be included. A cautious interpretation of the findings should be provided. The report should include a discussion of the evidence gaps and uncertainties in the identified evidence and suggestions of outcomes to focus on in future evidence generation.

Economic analyses for early value assessment may differ from the standard approach for guidance products. The standard approach is outlined in section 4 of NICE health technology evaluation: the manual. For early value assessment, the economic evaluation work that is likely to be most beneficial for committee decision making is likely to vary by topic, needing flexibility from assessment groups doing the work. Proposed work should be discussed between the NICE team and the external assessment group from an early stage to inform development of the protocol for the assessment.

The objectives of the economic evaluation are to:

• identify likely impacts of using technologies (while further data is collected) on:

  • people with the condition and, when relevant, carers

  • the NHS and personal social services (including costs)

• identify additional uncertainties that would not be apparent from technology related studies; for example, related to the structure of, or parameters used in, models that are anticipated to be needed for future guidance.

• identify uncertainties that are likely to be key drivers of model results and decision-uncertainty to inform decision making about further evidence generation.

The cost of reversing a decision to adopt the technologies while further evidence is generated may also be considered. This is to assess the potential implications if, after an initial decision by NICE to recommend use of the technologies while further evidence is generated, a later assessment (after this evidence is collected) shows the technology not to be cost effective or cost saving. This could include any costs that could not be recouped related to implementation such as fixed or up-front costs related to the purchase of equipment, training costs or changes to organisation of care pathways.

Efforts should be made to identify relevant economic evaluations related to the technologies or disease area that could inform the work. For example, through targeted literature searches or work highlighted by experts or stakeholders as potentially relevant. Any models highlighted or submitted by companies could be considered.

Economic evaluation should as a minimum consider model structure(s) that would be needed for a future analysis to support NICE committee decision making. Relevant experts (for example appointed specialist committee members) should be consulted to validate this. Comparisons should also be made with any existing model structures identified where they are deemed relevant or robust (see section 3.19).

Development of a preliminary or early coded model (that is, a model implemented in a software platform) should be considered to help meet the objectives in section 3.17, particularly to explore uncertainty with the aim of identifying key drivers of the model results to inform decision making about further evidence generation. Existing models could be used here, if considered suitable and an agreement is in place to allow models developed by third parties to be made available to stakeholders.

The reference case will be the same as described in section 4.2 of NICE health technology evaluations: the manual.

Guidance for presenting data and results of models is described in section 4.10 of NICE health technology evaluations: the manual. Outputs of models should be presented to easily allow clinicians to validate model results and the extent of work done to validate model outcomes should be described.

If no coded model is produced (see section 3.21), targeted searches should still be done to attempt to identify data (and highlight where there may be a paucity of data) for model parameters considered likely to be key drivers of future cost-utility analysis (see section 3.9 and section 3.10).

New approaches to decision making are being piloted for early value assessment. NICE's usual approach to decision making and committee recommendations is outlined in section 6 of NICE health technology evaluations: the manual. The key points are outlined below.

Decision making will be the responsibility of one of the following:

• the diagnostic advisory committee

• medical technologies advisory committee

• decision panel.
  
  Specialist committee members will also be invited to be decision makers for early value assessment.

When making decisions the committee should consider:

• the extent of the evidence that supports the likelihood of the technology addressing unmet need in the system

• the views and experiences of people using the technology

• any barriers to implementation and how these can be addressed as part of further evidence generation

• the key evidence gaps where further data is needed to resolve uncertainty for future decision making

• if further evidence generation, either from ongoing or new research or from real-world data, will sufficiently resolve the key evidence gaps

• if any identified risks or uncertainties could be mitigated if the technology is used while further data is generated (for example, by specifying how the device should be used or further research [with defined outcomes] that would need to be done before wider use, or by reductions in technology cost or ways in which the technology is charged for)

• the likelihood and size of impact of adopting the technologies for people using the technologies (and their carers) and the NHS personal social services while further data is collected, in terms of both potential benefits and risks

• whether provision should be made for special safety monitoring measures.

When multiple technologies are considered in a topic, each should be assessed independently. For example, considering any difference between technologies in terms of whether they can solve the specified unmet need and any differences in further evidence needs.

The committee or panel can make the following types of recommendations (specific wording subject to change):

• Conditionally recommended for use while further evidence is generated.

  • Made when the committee or panel consider it is plausible that the technology will address the unmet need and it is acceptable for the technology to be used in practice (potentially with measures in place to mitigate risks) while further evidence is generated. How evidence is to be collected (for example, collected while the technology is used in practice or from a research study) will be determined in the evidence generation plan developed (see section 4).

• Recommended in research

  • Made when the committee or panel are uncertain if the technology has the potential to solve the unmet need, or if the committee do not think it is acceptable for the technology to be widely used in practice while further evidence is generated. For example, if it is too uncertain about the extent to which the potential benefits of use outweigh the potential risks.

• Not recommended for use

  • Made where the committee or panel do not believe a technology has the potential to meet the unmet need, or where there are concerns about the potential harms associated with using the technology even in a research context.
    
    Figure B in the appendix illustrates the main considerations that will impact the recommendations.

For the conditional recommendation, the outcomes that the committee need further data on to support future decision making should be listed and include a focus on outcomes which are fundamental to decision making.