The guidelines manual

1.1 Is the study population appropriate for the guideline?

The study population should be defined as precisely as possible and should be in line with that specified in the guideline scope and any related review protocols.

This includes consideration of appropriate subgroups that require special attention. For many interventions, the capacity to benefit will differ for participants with differing characteristics. This should be explored separately for each relevant subgroup as part of the base-case analysis by the provision of estimates of clinical and cost effectiveness. The characteristics of participants in each subgroup should be clearly defined and, ideally, should be identified on the basis of an a priori expectation of differential clinical or cost effectiveness as a result of biologically plausible known mechanisms, social characteristics or other clearly justified factors.

Answer 'yes' if the study population is fully in line with that in the guideline question(s) and if the study differentiates appropriately between important subgroups. Answer 'partly' if the study population is similar to that in the guideline question(s) but: (i) it differs in some important respects; or (ii) the study fails to differentiate between important subgroups. Answer 'no' if the study population is substantively different from that in the guideline question(s).

1.2 Are the interventions and services appropriate for the guideline?

All relevant alternatives should be included, as specified in the guideline scope and any related review protocols. These should include routine and best practice in the NHS, existing NICE guidance and other feasible options.

Answer 'yes' if the analysis includes all options considered relevant for the guideline, even if it also includes other options that are not relevant. Answer 'partly' if the analysis omits one or more relevant options but still contains comparisons likely to be useful for the guideline. Answer 'no' if the analysis does not contain any relevant comparisons.

1.3 Is the healthcare system in which the study was conducted sufficiently similar to the current UK NHS context?

This relates to the overall structure of the healthcare system within which the interventions were delivered. For example, an intervention might be delivered on an inpatient basis in one country whereas in the UK it would be provided in the community. This might significantly influence the use of healthcare resources and costs, thus limiting the applicability of the results to a UK setting. In addition, old UK studies may be severely limited in terms of their relevance to current NHS practice.

Answer 'yes' if the study was conducted within the UK and is sufficiently recent to reflect current NHS practice. For non-UK or older UK studies, answer 'partly' if differences in the healthcare setting are unlikely to substantively change the cost-effectiveness estimates. Answer 'no' if the healthcare setting is so different that the results are unlikely to be applicable in the current NHS.

1.4 Are costs measured from the NHS and personal social services (PSS) perspective?

The decision-making perspective of an economic evaluation determines the range of costs that should be included in the analysis. NICE works in a specific context; in particular, it does not set the budget for the NHS. The objective of NICE is to offer guidance that represents an efficient use of available NHS and PSS resources. For these reasons, the perspective on costs used in the NICE reference case is that of the NHS and PSS. Productivity costs and costs borne by patients and carers that are not reimbursed by the NHS or PSS are not included in the reference case. The reference case also excludes costs to other government bodies, although these may sometimes be presented in additional analyses alongside the reference case.

Answer 'yes' if the study only includes costs for resource items that would be paid for by the NHS and PSS. Also answer 'yes' if other costs have been included in the study, but the results are presented in such a way that the cost effectiveness can be calculated from an NHS and PSS perspective. Answer 'partly' if the study has taken a wider perspective but the other non-NHS/PSS costs are small in relation to the total expected costs and are unlikely to change the cost-effectiveness results. Answer 'no' if non-NHS/PSS costs are significant and are likely to change the cost-effectiveness results.

Some interventions may have a substantial impact on non-health outcomes or costs to other government bodies (for example, treatments to reduce illicit drug misuse may have the effect of reducing drug-related crime). In such situations, if the economic study includes non-health costs in such a way that they cannot be separated out from NHS/PSS costs, answer 'no' but consider retaining the study for critical appraisal. If studies containing non-reference-case costs are retained, use the comments column to note why.

1.5 Are non-direct health effects on individuals excluded?

In the NICE reference case, the perspective on outcomes should be all direct health effects, whether for patients or, when relevant, other people (principally carers). This is consistent with an objective of maximising health gain from available healthcare resources. Some features of healthcare delivery that are often referred to as 'process characteristics' may ultimately have health consequences; for example, the mode of treatment delivery may have health consequences through its impact on concordance with treatment. Any significant characteristics of healthcare technologies that have a value to people that is independent of any direct effect on health should be noted. These characteristics include the convenience with which healthcare is provided and the level of information available for patients.

This question should be viewed in terms of what is excluded in relation to the NICE reference case; that is, non-health effects.

Answer 'yes' if the measure of health outcome used in the analysis excludes non-health effects (or if such effects can be excluded from the results). Answer 'partly' if the analysis includes some non-health effects but these are small and unlikely to change the cost-effectiveness results. Answer 'no' if the analysis includes significant non-health effects that are likely to change the cost-effectiveness results.

1.6 Are both costs and health effects discounted at an annual rate of 3.5%?

The need to discount to a present value is widely accepted in economic evaluation, although the specific rate varies across jurisdictions and over time. NICE considers it appropriate to discount costs and health effects at the same rate. The annual rate of 3.5%, based on the recommendations of the UK Treasury for the discounting of costs, applies to both costs and health effects.

Answer 'yes' if both costs and health effects (for example, quality-adjusted life years [QALYs]) are discounted at 3.5% per year. Answer 'partly' if costs and health effects are discounted at a rate similar to 3.5% (for example, costs and effects are both discounted at 3% per year). Answer 'no' if costs and/or health effects are not discounted, or if they are discounted at a rate (or rates) different from 3.5% (for example, 5% for both costs and effects, or 6% for costs and 1.5% for effects). Note in the comments column what discount rates have been used. If all costs and health effects accrue within a short time (roughly a year), answer 'NA'.

1.7 Is the value of health effects expressed in terms of quality-adjusted life years (QALYs)?

The QALY is a measure of a person's length of life weighted by a valuation of their health-related quality of life (HRQoL) over that period.

Given its widespread use, the QALY is considered by NICE to be the most appropriate generic measure of health benefit that reflects both mortality and effects on HRQoL. It is recognised that alternative measures exist (such as the healthy-year equivalent), but few economic evaluations have used these methods and their strengths and weaknesses are not fully established.

NICE's position is that an additional QALY should be given the same weight regardless of the other characteristics of the patients receiving the health benefit.

Answer 'yes' if the effectiveness of the intervention is measured using QALYs; answer 'no' if not. There may be circumstances when a QALY cannot be obtained or where the assumptions underlying QALYs are considered inappropriate. In such situations answer 'no', but consider retaining the study for appraisal. Similarly, answer 'no' but retain the study for appraisal if it does not include QALYs but it is still thought to be useful for GDG decision-making: for example, if the clinical evidence indicates that an intervention might be dominant, and estimates of the relative costs of the interventions from a cost-minimisation study are likely to be useful. When economic evaluations not using QALYs are retained for full critical appraisal, use the comments column to note why.

1.8 Are changes in health-related quality of life (HRQoL) reported directly from patients and/or carers?

In the NICE reference case, information on changes in HRQoL as a result of treatment should be reported directly by patients (and directly by carers when the impact of treatment on the carer's health is also important). When it is not possible to obtain information on changes in patients' HRQoL directly from them, data should be obtained from carers (not from healthcare professionals).

For consistency, the EQ-5D is NICE's preferred measure of HRQoL in adults. However, when EQ-5D data are not available or are inappropriate for the condition or the effects of treatment, other multi-attribute utility questionnaires (for example, SF6D, QWB or HUI) or mapping methods from disease-specific questionnaires may be used to estimate QALYs. For studies not reporting QALYs, a variety of generic or disease-specific methods may be used to measure HRQoL.

Answer 'yes' if changes in patients' HRQoL are estimated by the patients themselves. Answer 'partly' if estimates of patients' HRQoL are provided by carers. Answer 'no' if estimates come from healthcare professionals or researchers. Note in the comments column how HRQoL was measured (EQ-5D, QWB, HUI and so on). Answer 'NA' if the cost-effectiveness study does not include estimates of HRQoL (for example, studies reporting 'cost per life year gained' or cost-minimisation studies).

1.9 Is the valuation of changes in HRQoL (utilities) obtained from a representative sample of the general public?

The NICE reference case specifies that the valuation of changes in HRQoL (utilities) reported by patients should be based on public preferences elicited using a choice-based method (such as the time trade-off or standard gamble) in a representative sample of the UK population.

Answer 'yes' if HRQoL valuations were obtained using the EQ-5D UK tariff. Answer 'partly' if the valuation methods were comparable to those used for the EQ-5D. Answer 'no' if other valuation methods were used. Answer 'NA' if the study does not apply valuations to HRQoL (for studies not reporting QALYs). In the comments column note the valuation method used (such as time trade-off or standard gamble) and the source of the preferences (such as patients or healthcare professionals).

1.10 Overall judgement

Classify the applicability of the economic evaluation to the clinical guideline, the current NHS situation and the context for NICE guidance as one of the following:

2.1 Does the model structure adequately reflect the nature of the health condition under evaluation?

This relates to the choice of model and its structural elements (including cycle length in discrete time models, if appropriate). Model type and its structural aspects should be consistent with a coherent theory of the health condition under evaluation. The selection of treatment pathways, whether health states or branches in a decision tree, should be based on the underlying biological processes of the health issue under study and the potential impact (benefits and adverse consequences) of the intervention(s) of interest.

Answer 'yes' if the model design and assumptions appropriately reflect the health condition and intervention(s) of interest. Answer 'partly' if there are aspects of the model design or assumptions that do not fully reflect the health condition or intervention(s) but these are unlikely to change the cost-effectiveness results. Answer 'no' if the model omits some important aspect of the health condition or intervention(s) and this is likely to change the cost-effectiveness results. Answer 'NA' for economic evaluations based on data from a clinical study which do not extrapolate treatment outcomes or costs beyond the study context or follow-up period.

2.2 Is the time horizon sufficiently long to reflect all important differences in costs and outcomes?

The time horizon is the period of analysis of the study: the length of follow-up for participants in a trial-based evaluation, or the period of time over which the costs and outcomes for a cohort are tracked in a modelling study. This time horizon should always be the same for costs and outcomes, and should be long enough to include all relevant costs and outcomes relating to the intervention. A time horizon shorter than lifetime could be justified if there is no differential mortality effect between options, and the differences in costs and HRQoL relate to a relatively short period (for example, in the case of an acute infection).

Answer 'yes' if the time horizon is sufficient to include all relevant costs and outcomes. Answer 'partly' if the time horizon may omit some relevant costs and outcomes but these are unlikely to change the cost-effectiveness results. Answer 'no' if the time horizon omits important costs and outcomes and this is likely to change the cost-effectiveness results.

2.3 Are all important and relevant health outcomes included?

All relevant health outcomes should include direct health effects relating to harms from the intervention (adverse effects) as well as any potential benefits.

Answer 'yes' if the analysis includes all relevant and important harms and benefits. Answer 'partly' if the analysis omits some harms or benefits but these would be unlikely to change the cost-effectiveness results. Answer 'no' if the analysis omits important harms and/or benefits that would be likely to change the cost-effectiveness results.

2.4 Are the estimates of baseline health outcomes from the best available source?

The estimate of the overall net treatment effect of an intervention is determined by the baseline risk of a particular condition or event and/or the relative effects of the intervention compared with the relevant comparator treatment. The overall net treatment effect may also be determined by other features of the people comprising the population of interest.

The process of assembling evidence for economic evaluations should be systematic – evidence must be identified, quality assessed and, when appropriate, pooled, using explicit criteria and justifiable and reproducible methods. These principles apply to all categories of evidence that are used to estimate clinical and cost effectiveness, evidence for which will typically be drawn from a number of different sources.

The sources and methods for eliciting baseline probabilities should be described clearly. These data can be based on 'natural history' (patient outcomes in the absence of treatment or with routine care), sourced from cohort studies. Baseline probabilities may also be derived from the control arms of experimental studies. Sometimes it may be necessary to rely on expert opinion for particular parameters.

Answer 'yes' if the estimates of baseline health outcomes reflect the best available evidence as identified from a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates are not derived from a systematic review but are likely to reflect outcomes for the relevant group of patients in routine NHS practice (for example, if they are derived from a large UK-relevant cohort study). Answer 'no' if the estimates are unlikely to reflect outcomes for the relevant group in routine NHS practice.

2.5 Are the estimates of relative treatment effects from the best available source?

The objective of the analysis of clinical effectiveness is to produce an unbiased estimate of the mean clinical effectiveness of the interventions being compared.

The NICE reference case indicates that evidence on outcomes should be obtained from a systematic review, defined as the systematic location, inclusion, appraisal and synthesis of evidence to obtain a reliable and valid overview of the data relating to a clearly formulated question.

Synthesis of outcome data through meta-analysis is appropriate provided that there are sufficient relevant and valid data obtained using comparable measures of outcome.

Head-to-head randomised controlled trials (RCTs) provide the most valid evidence of relative treatment effect. However, such evidence may not always be available. Therefore, data from non-randomised studies may be required to supplement RCT data. Any potential bias arising from the design of the studies used in the assessment should be explored and documented.

Data from head-to-head RCTs should be presented in the base-case analysis, if available. When head-to-head RCTs exist, evidence from indirect or mixed treatment comparison analyses may be presented if it is considered to add information that is not available from the head-to-head comparison. This indirect or mixed treatment comparison must be fully described and presented as additional to the base-case analysis. (A 'mixed treatment comparison' estimates effect sizes using both head-to-head and indirect comparisons.)

If data from head-to-head RCTs are not available, indirect treatment comparison methods should be used. (An 'indirect treatment comparison' is a synthesis of data from a network of trials that compare the interventions of interest with other comparators.)

When multiple interventions are being assessed that have not been compared within a single RCT, data from a series of pairwise head-to-head RCTs should be presented. Consideration should also be given to presenting a combined analysis using a mixed treatment comparison framework if it is considered to add information that is not available from the head-to-head comparison.

Only indirect or mixed treatment comparison methods that preserve randomisation should be used. The principles of good practice for standard meta-analyses should also be followed in mixed and indirect treatment comparisons.

The methods and assumptions that are used to extrapolate short-term results to final outcomes should be clearly presented and there should be documentation of the reasoning underpinning the choice of survival function.

Evidence for the evaluation of diagnostic technologies should normally incorporate evidence on diagnostic accuracy. It is also important to incorporate the predicted changes in health outcomes and costs resulting from treatment decisions based on the test result. The general principles guiding the assessment of the clinical and cost effectiveness of diagnostic interventions should be the same as for other technologies. However, particular consideration of the methods of analysis may be required, particularly in relation to evidence synthesis. Evidence for the effectiveness of diagnostic technologies should include the costs and outcomes for people whose test results lead to an incorrect diagnosis, as well as for those who are diagnosed correctly.

As for other technologies, RCTs have the potential to capture the pathway of care involving diagnostic technologies, but their feasibility and availability may be limited. Other study designs should be assessed on the basis of their fitness for purpose, taking into consideration the aim of the study (for example, to evaluate outcomes, or to evaluate sensitivity and specificity) and the purpose of the diagnostic technology.

Answer 'yes' if the estimates of treatment effect appropriately reflect all relevant studies of the best available quality, as identified through a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates of treatment effect are not derived from a systematic review but are similar in magnitude to the best available estimates (for example, if the economic evaluation is based on a single large study with treatment effects similar to pooled estimates from all relevant studies). Answer 'no' if the estimates of treatment effect are likely to differ substantively from the best available estimates.

2.6 Are all important and relevant costs included?

Costs related to the condition of interest and incurred in additional years of life gained as a result of treatment should be included in the base-case analysis. This should include the costs of handling non-adherence to treatment and treating side effects. Costs that are considered to be unrelated to the condition or intervention of interest should be excluded. If introduction of the intervention requires additional infrastructure to be put in place, consideration should be given to including such costs in the analysis.

Answer 'yes' if all important and relevant resource use and costs are included given the perspective and the research question in the economic study under consideration. Answer 'partly' if some relevant resource items are omitted but these are unlikely to affect the cost-effectiveness results. Answer 'no' if important resource items are omitted and these are likely to affect the cost-effectiveness results.

2.7 Are the estimates of resource use from the best available source?

It is important to quantify the effect of the interventions on resource use in terms of physical units (for example, days in hospital or visits to a GP) and valuing those effects in monetary terms using appropriate prices and unit costs. Evidence on resource use should be identified systematically. When expert opinion is used as a source of information, any formal methods used to elicit these data should be clearly reported.

Answer 'yes' if the estimates of resource use appropriately reflect all relevant evidence sources of the best available quality, as identified through a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates of resource use are not derived from a systematic review but are similar in magnitude to the best available estimates. Answer 'no' if the estimates of resource use are likely to differ substantively from the best available estimates.

2.8 Are the unit costs of resources from the best available source?

Resources should be valued using the prices relevant to the NHS and PSS. Given the perspective of the NICE reference case, it is appropriate for the financial costs relevant to the NHS/PSS to be used as the basis of costing, although these may not always reflect the full social opportunity cost of a given resource. A first point of reference in identifying costs and prices should be any current official listing published by the Department of Health and/or the Welsh Government.

When the acquisition price paid for a resource differs from the public list price (for example, pharmaceuticals and medical devices sold at reduced prices to NHS institutions), the public list price should be used in the base-case analysis. Sensitivity analysis should assess the implications of variations from this price. Analyses based on price reductions for the NHS will only be considered when the reduced prices are transparent and can be consistently available across the NHS, and if the period for which the specified price is available is guaranteed.

National data based on healthcare resource groups (HRGs) such as the Payment by Results tariff can be used when they are appropriate and available. However, data based on HRGs may not be appropriate in all circumstances (for example, when the definition of the HRG is broad, or the mean cost probably does not reflect resource use in relation to the intervention(s) under consideration). In such cases, other sources of evidence, such as micro-costing studies, may be more appropriate. When cost data are taken from the literature, the methods used to identify the sources should be defined. When several alternative sources are available, a justification for the costs chosen should be provided and discrepancies between the sources explained. When appropriate, sensitivity analysis should have been undertaken to assess the implications for results of using alternative data sources.

Answer 'yes' if resources are valued using up-to-date prices relevant to the NHS and PSS. Answer 'partly' if the valuations of some resource items differ from current NHS/PSS unit costs but this is unlikely to change the cost-effectiveness results. Answer 'no' if the valuations of some resource items differ substantively from current NHS/PSS unit costs and this is likely to change the cost-effectiveness results.

2.9 Is an appropriate incremental analysis presented or can it be calculated from the data?

An appropriate incremental analysis is one that compares the expected costs and health outcomes of one intervention with the expected costs and health outcomes of the next-best non-dominated alternative.

Standard decision rules should be followed when combining costs and effects, and should reflect any situation where there is dominance or extended dominance. When there is a trade-off between costs and effects, the results should be presented as an incremental cost-effectiveness ratio (ICER): the ratio of the difference in mean costs to the difference in mean outcomes of a technology compared with the next best alternative. In addition to ICERs, expected net monetary or health benefits can be presented using values placed on a QALY gained of £20,000 and £30,000.

For cost-consequence analyses, appropriate incremental analysis can only be done by selecting one of the consequences as the primary measure of effectiveness.

Answer 'yes' if appropriate incremental results are presented, or if data are presented that allow the reader to calculate the incremental results. Answer 'no' if: (i) simple ratios of costs to effects are presented for each alternative compared with a standard intervention; or (ii) if options subject to simple or extended dominance are not excluded from the incremental analyses.

2.10 Are all important parameters whose values are uncertain subjected to appropriate sensitivity analysis?

There are a number of potential selection biases and uncertainties in any evaluation (trial- or model-based) and these should be identified and quantified where possible. There are three types of bias or uncertainty to consider:

Answer 'yes' if an extensive sensitivity analysis was undertaken that explored all key uncertainties in the economic evaluation. Answer 'partly' if the sensitivity analysis failed to explore some important uncertainties in the economic evaluation. Answer 'no' if the sensitivity analysis was very limited and omitted consideration of a number of important uncertainties, or if the range of values or distributions around parameters considered in the sensitivity analysis were not reported.

2.11 Is there no potential conflict of interest?

The British Medical Journal (BMJ) defines competing interests for its authors as follows: "A competing interest exists when professional judgment concerning a primary interest (such as patients' welfare or the validity of research) may be influenced by a secondary interest (such as financial gain or personal rivalry). It may arise for the authors of a BMJ article when they have a financial interest that may influence, probably without their knowing, their interpretation of their results or those of others."

Whenever a potential financial conflict of interest is possible, this should be declared.

Answer 'yes' if the authors declare that they have no financial conflicts of interest. Answer 'no' if clear financial conflicts of interest are declared or apparent (for example, from the stated affiliation of the authors). Answer 'unclear' if the article does not indicate whether or not there are financial conflicts of interest.

2.12 Overall assessment

The overall methodological study quality of the economic evaluation should be classified as one of the following: