Methods for the development of NICE public health guidance (third edition)

1.1 Is the study population appropriate for the guideline?

The study population should be defined as precisely as possible and should be in line with that specified in the topic scope and any related review protocols.

This includes consideration of appropriate subgroups that require special attention. For many interventions, the capacity to benefit will differ for participants or communities with differing characteristics. This should be explored separately for each relevant subgroup as part of the base-case analysis by the provision of estimates of effectiveness and cost effectiveness. The characteristics of participants or communities in each subgroup should be clearly defined and, ideally, should be identified on the basis of an assumed expectation of differential effectiveness or cost effectiveness as a result of biologically, economic or sociologically plausible known mechanisms, social characteristics or other clearly justified factors.

Answer 'yes' if the study population is fully in line with that in the topic question(s) and if the study differentiates appropriately between important subgroups. Answer 'partly' if the study population is similar to that in the topic question(s) but: (i) it differs in some important respects; or (ii) the study fails to differentiate between important subgroups. Answer 'no' if the study population is substantively different from that in the topic question(s).

1.2 Are the interventions appropriate for the guideline?

All relevant alternatives should be included, as specified in the topic scope and any related review protocols. Where the guidance is primarily for the NHS, these should include routine and best practice in the NHS, existing NICE guidance and other feasible options. Where the guidance is primarily for local government, these should include routine and best practice in the local government, existing NICE guidance and other feasible options.

Answer 'yes' if the analysis includes all options considered relevant for the topic, even if it also includes other options that are not relevant. Answer 'partly' if the analysis omits 1 or more relevant options but still contains comparisons likely to be useful for the guidance. Answer 'no' if the analysis does not contain any relevant comparisons.

1.3 Is the system in which the study was conducted sufficiently similar to the current UK (NHS, local government or other) context?

This relates to the overall institutional structure in which the interventions were delivered. For example, an intervention might be delivered on an inpatient basis in 1 country whereas in the UK it would be provided in the community. This might significantly influence the use of healthcare and other resources and costs, thus limiting the applicability of the results to a UK setting. In addition, old UK studies may be severely limited in terms of their relevance to current NHS or local government practice. Other institutional structures include the differing legislative and legal systems, different working and sick-leave provisions, different social arrangements.

Answer 'yes' if the study was conducted in the UK and is sufficiently recent to reflect current NHS, local government and other relevant public, voluntary and private sector practice. For non-UK or older UK studies, answer 'partly' if differences in the setting are unlikely to substantively change the cost-effectiveness estimates. Answer 'no' if the setting is so different that the results are unlikely to be applicable in the current NHS or in other public, voluntary or private sectors.

1.4 Was/were the perspective(s) clearly stated?

The decision-making perspective of an economic evaluation determines the range of costs that should be included in the analysis. For public health guidance, one perspective that will usually be used is that of local government. Sometimes in public health, costs will be borne and benefits will accrue outside government departments (local government). When they are borne or accrue predominantly by other public sectors, it will also be appropriate to use a public sector perspective. In topics where interventions have a material effect on employment, the perspective may also need to reflect that. Where the cost effectiveness using a narrower perspective is clearly established, however, the requirement to embrace a wider perspective is much reduced. Answer 'yes' if the study clearly and correctly states the perspective used, and whether that perspective is appropriate. Answer 'partly' if the perspective stated is not the perspective used. Answer 'no' if the study does not state the perspective or that the perspective is not appropriate.

1.5 Are all direct health effects on individuals included, and are all other effects included where they are material?

For an NHS and personal social services (PSS) perspective, outcomes should include all direct health effects, whether for individuals directly affected or, when relevant, other people (often other family members). This is consistent with an objective of maximising health gain from available healthcare resources. Any significant characteristics of healthcare technologies that have a value to people independent of any direct effect on health should be noted. These characteristics include the convenience with which healthcare is provided and the level of information available for affected individuals.

For a local government perspective, outcomes should include both health and non-health benefits, and where relevant, should include not just the individuals directly targeted but also their families, friends and the community in general.

If an NHS and PPS perspective is used, answer 'yes' if the measure of health outcome excludes non-health effects (or if such effects can be excluded from the results) and if non-health effects are included in a wider perspective. Answer 'partly' if the analysis from an NHS and PSS perspective includes some non-health effects but these are small and unlikely to change the cost-effectiveness results. Answer 'no' if the analysis incorrectly includes or excludes significant non-health effects that are likely to change the cost-effectiveness results for a particular perspective.

If a local government perspective is used, answer 'yes' if the measure of health outcome includes non-health effects. Answer 'partly' if the analysis from a local government perspective includes some non-health effects but these are small and unlikely to change the cost-effectiveness results. Answer 'no' if the analysis incorrectly includes or excludes significant non-health effects that are likely to change the cost-effectiveness results for a particular perspective.

If a public sector perspective is used, answer 'yes' if the measure of health outcome includes non-health effects. Answer 'partly' if the analysis includes some non-health effects but these are small and unlikely to change the cost-effectiveness results. Answer 'no' if the analysis incorrectly includes or excludes significant non-health effects that are likely to change the cost-effectiveness results for a particular perspective.

If a societal perspective is used, answer 'yes' if the measure of health outcome includes non-health effects. Answer 'partly' if the analysis includes some non-health effects but these are small and unlikely to change the cost-effectiveness results. Answer 'no' if the analysis incorrectly includes or excludes significant non-health effects that are likely to change the cost-effectiveness results for a particular perspective.

1.6 Are all future costs and effects discounted appropriately?

The need to discount to a present value is widely accepted in economic evaluation, although the specific rate varies across jurisdictions and over time. There is currently some debate about the most appropriate discount rate for health benefits. The annual rate of 1.5% should be applied to both costs and effects, but an annual rate of 1.5% for health benefits and 3.5% for costs should routinely be used in sensitivity analysis, as should 3.5% for both costs and benefits.

Answer 'yes' if both costs and health effects (for example, quality-adjusted life years [QALYs]) are discounted at 1.5% per year, or if the health benefits are discounted at an annual rate of 1.5%. Answer 'no' if costs or health effects are not discounted, or if they are discounted at a rate (or rates) different from 1.5% (for example, 3.5% for both costs and effects). Note in the comments column what discount rates have been used. If all costs and health effects accrue in a short time (roughly a year), answer 'not applicable'.

1.7 Is the value of health effects expressed in terms of QALYs?

The QALY is a measure of a person's length of life weighted by a valuation of their health-related quality of life (HRQL) over that period.

Given its widespread use, the QALY is considered by NICE to be the most appropriate generic measure of health benefit that reflects both mortality and effects on HRQL. It is recognised that alternative measures exist (such as the healthy-year equivalent), but few economic evaluations have used these methods and their strengths and weaknesses are not fully established.

NICE's position is that an additional QALY should be given the same weight regardless of the other characteristics of the people receiving the health benefit.

Answer 'yes' if the health benefits of the intervention are measured using QALYs; answer 'no' if not. There may be circumstances when a QALY cannot be obtained or where the assumptions underlying QALYs are considered inappropriate. In such situations answer 'no', but consider retaining the study for appraisal. Similarly, answer 'no' but retain the study for appraisal if it does not include QALYs but it is still thought to be useful for public health advisory committee decision-making: for example, if the evidence indicates that an intervention might be dominant, and estimates of the relative costs of the interventions from a cost-minimisation study are likely to be useful. When economic evaluations not using QALYs are retained for full critical appraisal, use the comments column to note why. Retention of a study for appraisal will also depend on the extent and reliability of other evidence. In the absence of good alternative sources of evidence, it might be useful to retain a study.

1.8 Are costs and outcomes from other sectors fully and appropriately measured and valued?

Studies in public health often include costs accruing to other sectors of the economy or benefits gained by these sectors. Not all of these benefits can be translated into QALYs (for example, a reduction in school class size, the reduction in unwanted pregnancies or the ability to return to work earlier). Answer 'yes' if all the costs and all the benefits have been included, if they are appropriately measured and if they are appropriately valued. Answer 'partly' if omissions are not material and answer 'no' if some major cost or benefit is omitted, is improperly measured or improperly valued.

1.9 Overall judgement

Classify the applicability of the economic evaluation to the public health guidance, the current public sector situation and the context for NICE guidance as 1 of the following:

2.1 Does the model structure adequately reflect the nature of the topic under evaluation?

This relates to the choice of model and its structural elements (including cycle length in discrete time models, if appropriate). Model type and its structural aspects should be consistent with a coherent theory of the health condition under evaluation. The selection of treatment pathways, whether health states or branches in a decision tree, should be based on the underlying biological, economic or sociological processes of the topic under study and the potential impact (benefits and adverse consequences) of the intervention(s) of interest.

Answer 'yes' if the model design and assumptions appropriately reflect the topic and intervention(s) of interest. Answer 'partly' if there are aspects of the model design or assumptions that do not fully reflect the topic or intervention(s) but that are unlikely to change the cost-effectiveness results. Answer 'no' if the model omits some important aspect of the topic or intervention(s) and this is likely to change the cost-effectiveness results. Answer 'NA' for economic evaluations based on data from a study which do not extrapolate outcomes or costs beyond the study context or follow-up period.

2.2 Is the time horizon sufficiently long to reflect all important differences in costs and outcomes?

The time horizon is the period of analysis of the study: the length of follow-up for participants in a trial-based evaluation, or the period of time over which the costs and outcomes for a cohort are tracked in a modelling study. This time horizon should always be the same for costs and outcomes, and should be long enough to include all relevant costs and outcomes relating to the intervention. A time horizon shorter than lifetime could be justified if there is no differential mortality effect between options, and the differences in costs, HRQL and other relevant outcomes relate to a relatively short period (for example, in the case of an acute infection). Infectious disease models may need a time horizon beyond a lifetime.

Answer 'yes' if the time horizon is sufficient to include all relevant costs and outcomes. Answer 'partly' if the time horizon may omit some relevant costs and outcomes but these are unlikely to change the cost-effectiveness results. Answer 'no' if the time horizon omits important costs and outcomes and this is likely to change the cost-effectiveness results.

2.3 Are all important and relevant outcomes included?

All relevant outcomes should include direct health or other effects relating to harms from the intervention (adverse effects) as well as any potential benefits.

Answer 'yes' if the analysis includes all relevant and important harms and benefits. Answer 'partly' if the analysis omits some harms or benefits but these would be unlikely to change the cost-effectiveness results. Answer 'no' if the analysis omits important harms and/or benefits that would be likely to change the cost-effectiveness results.

2.4 Are the estimates of baseline outcomes from the best available source?

The sources and methods for eliciting baseline probabilities should be described clearly. These data might be based on patient outcomes in the absence of treatment or with routine care, sourced from cohort studies. Baseline probabilities may also be derived from the control arms of experimental studies. Sometimes it may be necessary to rely on expert opinion for particular parameters.

Answer 'yes' if the estimates of baseline outcomes reflect the best available evidence as identified from a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates are not derived from a systematic review but are likely to reflect outcomes for the relevant group of people in England (for example, if they are derived from a large UK-relevant cohort study). Answer 'no' if the estimates are unlikely to reflect outcomes for the relevant group in England.

2.5 Are the estimates of relative intervention effects from the best available source?

Evidence on outcomes should be obtained from a systematic review, defined as the systematic location, inclusion, appraisal and synthesis of evidence to obtain a reliable and valid overview of the data relating to a clearly formulated question.

Synthesis of outcome data through meta-analysis is appropriate if there are sufficient relevant and valid data obtained using comparable measures of outcome.

Head-to-head randomised controlled trials (RCTs) provide the most valid evidence of the effects of clinical interventions, but might not be as valid as other well-established forms of evidence for other types of intervention (such as the observational evidence for the need for a parachute when jumping from a plane, or the observation that raising the price will reduce a commodity's consumption). Therefore, data from non-randomised studies may supplement RCT data. Any potential bias arising from the design of the studies used in the assessment should be explored and documented.

When assessing multiple interventions that have not been compared in a single RCT, data from a series of pairwise head-to-head RCTs should be presented. Consideration should also be given to presenting a combined analysis using a mixed treatment comparison framework if it is considered to add information that is not available from the head-to-head comparison.

The principles of good practice for standard meta-analyses should be followed in mixed and indirect treatment comparisons.

The methods and assumptions that are used to extrapolate short-term results to final outcomes should be clearly presented.

Answer 'yes' if the estimates of treatment effect appropriately reflect all relevant studies of the best available quality, as identified through a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates of treatment effect are not derived from a systematic review but are similar in magnitude to the best available estimates (for example, if the economic evaluation is based on a single large study with treatment effects similar to pooled estimates from all relevant studies). Answer 'no' if the estimates of treatment effect are likely to differ substantively from the best available estimates.

2.6 Are all important and relevant costs included?

Costs related to the condition of interest and incurred in additional years of life gained as a result of treatment should be included in the base-case analysis. This should include the costs of handling non-adherence to treatment and treating side effects. Costs that are considered to be unrelated to the topic or intervention of interest should be excluded. If introduction of the intervention requires additional infrastructure to be put in place, consideration should be given to including such costs in the analysis.

Answer 'yes' if all important and relevant resource use and costs are included given the perspective and the research question under consideration. Answer 'partly' if some relevant resource items are omitted but these are unlikely to affect the cost-effectiveness results. Answer 'no' if important resource items are omitted and these are likely to affect the cost-effectiveness results.

2.7 Are the estimates of resource use from the best available source?

It is important to quantify the effect of the interventions on resource use in terms of physical units (for example, days in hospital, visits to a GP or hours spent in class strengthening behavioural resolve) and valuing those effects in monetary terms using appropriate prices and unit costs. Evidence on resource use should be identified systematically. When expert opinion is used as a source of information, any formal methods used to elicit these data should be clearly reported.

Answer 'yes' if the estimates of resource use appropriately reflect all relevant evidence sources of the best available quality, as identified through a recent well-conducted systematic review of the literature. Answer 'partly' if the estimates of resource use are not derived from a systematic review but are similar in magnitude to the best available estimates. Answer 'no' if the estimates of resource use are likely to differ substantially from the best available estimates.

2.8 Are the unit costs of resources from the best available source?

Resources should be valued using the prices relevant to the NHS and PSS or for local government (depending on who delivers the intervention) for health costs and in prices relevant to the respective sectors responsible for other costs. For the NHS and PSS perspective, it is appropriate for the financial costs relevant to the NHS and PSS to be used as the basis of costing, although these may not always reflect the full social opportunity cost of a given resource. A first point of reference in identifying costs and prices should be any current official listing published by the Department of Health or the Welsh Assembly Government.

When the acquisition price paid for a resource differs from the public list price (for example, pharmaceuticals and medical devices sold at reduced prices to NHS institutions), the public list price should be used in the base-case analysis. Sensitivity analysis should assess the implications of variations from this price.

National data based on healthcare resource groups (HRGs) such as the Payment by Results tariff can be used when they are appropriate and available. However, data based on HRGs may not be appropriate in all circumstances (for example, when the definition of the HRG is broad, or the mean cost probably does not reflect resource use in relation to the intervention(s) under consideration). In such cases, other sources of evidence, such as micro-costing studies, may be more appropriate. When cost data are taken from the literature, the methods used to identify the sources should be defined. When several alternative sources are available, a justification for the costs chosen should be provided and discrepancies between the sources explained. When appropriate, sensitivity analysis should have been undertaken to assess the implications for results of using alternative data sources.

Similar rules apply to the costs of other public sectors.

Answer 'yes' if resources are valued using up-to-date prices relevant to the NHS and PSS, and to other sectors where the perspective allows this. Answer 'partly' if the valuations of some resource items differ from current NHS and PSS unit costs or those of other sectors where the perspective allows this, but this is unlikely to change the cost-effectiveness results. Answer 'no' if the valuations of some resource items differ substantively from current NHS and PSS unit costs or those of other sectors where the perspective allows this and this is likely to change the cost-effectiveness results.

2.9 Is an appropriate incremental analysis presented or can it be calculated from the data?

An appropriate incremental analysis is one that compares the expected costs and health outcomes of one intervention with the expected costs and health outcomes of the next-best non-dominated alternative.

Standard decision rules should be followed when combining costs and effects, and should reflect any situation where there is dominance or extended dominance. When there is a trade-off between costs and effects, the results should be presented as an ICER: the ratio of the difference in mean costs to the difference in mean outcomes of a technology compared with the next best alternative. In addition to ICERs, expected net monetary or health benefits can be presented using values placed on a QALY gained of £20,000 and £30,000.

For CCA, appropriate incremental analysis can only be done by selecting 1 of the consequences as the primary measure of effectiveness, providing the consequences are independent of one another.

Answer 'yes' if appropriate incremental results are presented, or if data are presented that allow the reader to calculate the incremental results. Answer 'no' if: (i) simple ratios of costs to effects are presented for each alternative compared with a standard intervention; or (ii) if options subject to simple or extended dominance are not excluded from the incremental analyses.

2.10 Are all important parameters whose values are uncertain subjected to appropriate sensitivity analysis?

There are a number of potential selection biases and uncertainties in any evaluation (trial- or model-based) and these should be identified and quantified where possible. There are 3 types of bias or uncertainty to consider:

Answer 'yes' if an extensive sensitivity analysis was undertaken that explored all key uncertainties in the economic evaluation. Answer 'partly' if the sensitivity analysis failed to explore some important uncertainties in the economic evaluation. Answer 'no' if the sensitivity analysis was very limited and omitted consideration of a number of important uncertainties, or if the range of values or distributions around parameters considered in the sensitivity analysis were not reported.

2.11 Is there any potential conflict of interest?

The BMJ defines competing interests for its authors as follows: 'A competing interest exists when professional judgment concerning a primary interest (such as patients' welfare or the validity of research) may be influenced by a secondary interest (such as financial gain or personal rivalry). It may arise for the authors of a BMJ article when they have a financial interest that may influence, probably without their knowing, their interpretation of their results or those of others.'

Whenever a potential financial conflict of interest is possible, this should be declared.

Answer 'yes' if the authors declare that they have no financial conflicts of interest. Answer 'no' if clear financial conflicts of interest are declared or apparent (for example, from the stated affiliation of the authors). Answer 'unclear' if the article does not indicate whether or not there are financial conflicts of interest.

2.12 Overall assessment

The overall methodological study quality of the economic evaluation should be classified as 1 of the following: