NICE process and methods
3 Evidence
3.1 Introduction
3.1.1 Consideration of a comprehensive evidence base is fundamental to the appraisal process. Evidence of various types and from multiple sources may inform the appraisal. To ensure that the guidance issued by the Institute is appropriate and robust, it is essential that the evidence and analysis, and their interpretation, are of the highest standard and are transparent.
3.1.2 The evaluation of effectiveness requires quantification of the effect of the technology under appraisal and of the relevant comparator technologies on survival, disease progression and health-related quality of life so that this can be used to estimate QALYs.
3.1.3 For costs, evidence requirements include quantifying the effect of the technology on resource use in terms of physical units (for example, days in hospital or visits to a GP) and valuing those effects in monetary terms using appropriate prices and unit costs.
3.1.4 In addition to evidence on treatment effects and costs, the appraisal of health technologies requires consideration of a range of other issues, for example:
-
the impact of having a condition or disease, the experience of undergoing specific treatments for that condition, and experience of the healthcare system for that condition
-
organisational issues that affect patients, carers or healthcare providers
-
NICE's legal obligations on equality and human rights
-
the requirement to treat people fairly.
3.2 Guiding principles for evidence
3.2.1 The evidence submitted to the Appraisal Committee should be:
-
relevant to the issue under consideration in terms of patient groups, comparators, perspective, outcomes and resource use as defined in the scope
-
inclusive of information on study design, such as the type of study, the circumstances of its undertaking and the rationale for the selection of outcomes, resource utilisation and costs
-
assembled systematically and synthesised in a transparent way that allows the analysis to be reproduced
-
analysed in a way that is methodologically sound and, in particular, minimises any bias.
To ensure that the evidence base for appraisals is consistent with these principles, NICE has defined a 'reference case' that specifies the methods it considers to be most appropriate for estimating clinical and cost effectiveness in technology appraisals (see section 5).
3.2.2 There are always likely to be deficiencies in the evidence base available for health technology assessment. For example, small sample sizes may result in some parameters being estimated with a low degree of precision, or evidence on effectiveness might come from outside the UK healthcare system or relate to groups of patients other than those of principal interest to the appraisal. Despite such weaknesses in the evidence base, decisions still have to be made about the use of technologies. Therefore, analyses should be explicit about the limitations of the evidence, and attempts to overcome these, and quantify as fully as possible how the limitations of the data are reflected in the uncertainty in the results of the analysis.
3.3 Types of evidence
3.3.1 Whatever the sources of evidence available on a particular technology and patient group, they should be integrated into a systematic review. A systematic review attempts to assemble all the available relevant evidence using explicit, valid and replicable methods in a way that minimises the risk of biased selection of studies. The data from the included studies can be synthesised (known as meta-analysis). All evidence should be critically appraised, and potential biases must be identified (see section 5.2).
Randomised controlled trials
3.3.2 Randomised controlled trials (RCTs) minimise potential external influences to identify an effect of 1 or more interventions on outcomes. Randomisation aims to prevent systematic differences between characteristics of participants assigned to different interventions at the start of the trial in terms of both known and unknown (or unmeasured) confounders. The trial should, in principle, provide a minimally biased estimate of the magnitude of any benefits or risks associated with the technology relative to those associated with the control group (participants receiving something other than the technology, for example no treatment, the standard treatment or placebo). RCTs are therefore considered to be most appropriate for measures of relative treatment effect.
3.3.3 The relevance of RCT evidence to the appraisal depends on both the external and internal validity of each trial. Internal validity is assessed according to the design and conduct of a trial and includes blinding (when appropriate), the method of randomisation and concealment of allocation, and the completeness of follow-up. Other important considerations are the size and power of the trial, the selection and measurement of outcomes and analysis by intention to treat. External validity is assessed according to the generalisability of the trial evidence; that is, whether the results apply to wider patient groups (and over a longer follow-up) and to routine clinical practice.
Non-randomised and non-controlled evidence
3.3.4 The problems of confounding, lack of blinding, incomplete follow-up and lack of a clear denominator and end point occur more commonly in non-randomised studies and non-controlled trials than in RCTs.
3.3.5 Observational (or epidemiological) studies do not apply an intervention, but instead compare outcomes for people who use the technology under appraisal with outcomes for people who do not use the technology. These studies may be biased in that the people who use the technology may fundamentally differ in their risk of the outcome than the people who do not use the technology. Some observational studies lack a control group, and include only people who receive the technology.
3.3.6 Inferences will necessarily be more circumspect about relative treatment effects drawn from studies without randomisation or control than those from RCTs. The potential biases of observational studies should be identified, and ideally quantified and adjusted for. When possible, more than 1 independent source of such evidence should be examined to gain some insight into the validity of any conclusions.
3.3.7 Evidence from sources other than RCTs is also often used for parameters such as the valuation of health effects over time into QALYs, and for costs. Study quality can vary, and so systematic review methods, critical appraisal and sensitivity analyses are as important for review of these data as they are for reviews of data on relative treatment effects from RCTs.
Qualitative research
3.3.8 In the context of technology appraisals the main purpose of qualitative research is to explore areas such as patients' experiences of having a disease or condition, their experiences of having treatment and their views on the acceptability of different types of treatment.
Economic evaluations
3.3.9 Evidence on cost effectiveness may be obtained from new analyses performed according to the NICE reference case; however, a systematic review of published, relevant evidence on the cost effectiveness of the technology should also be conducted.
3.3.10 Economic evaluations should quantify how the technologies under comparison affect disease progression and patients' health-related quality of life, and value those effects to reflect the preferences of the general population.
3.3.11 For all parameters (including effectiveness, valuation of health-related quality of life and costs) economic evaluation should systematically consider possible data sources, and avoid selection bias in the choice of sources.
Unpublished and part-published evidence
3.3.12 To ensure that the appraisal does not miss important relevant evidence, it is important that attempts are made to identify evidence that is not in the public domain. Such evidence includes unpublished clinical trial data and clinical trial data that are in abstract form only or are incomplete. Such information must be critically appraised and, when appropriate, sensitivity analysis conducted to examine the effects of its incorporation or exclusion.