Appendix D: summary of the methods used to develop this guidance

Introduction

The reports of the reviews and economic analysis include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E and are available from the NICE website.

The guidance development process

The stages of the guidance production process are outlined in the box below:

1. Draft scope

2. Stakeholder meeting

3. Stakeholder comments

4. Final scope and responses published on website

5. Reviews and cost-effectiveness modelling

6. Synopsis report of each review (executive summaries and evidence tables) circulated to stakeholders for comment

7. Comments and additional material submitted by stakeholders

8. Review of additional material submitted by stakeholders[a]

9. Synopsis, full reviews, supplementary reviews and economic modelling submitted to PHIAC

10. PHIAC produces draft recommendations

11. Draft recommendations published for comment

12. Responses to comments published

13. PHIAC amends recommendations

14. Final guidance published on website

[a] Submitted material is screened against inclusion criteria used in the reviews.

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and facilitated the development of recommendations by PHIAC.

1. Which methods of brief intervention are effective? Which methods are most effective? Which methods are least effective?

2. What are the costs and cost effectiveness associated with brief interventions?

3. What factors affect the effectiveness of brief interventions? Do they differ for different interventions?

4. Increasing the intensity, duration and/or frequency of a brief intervention can increase effectiveness. Is this increase additive or multiplicative?

5. To what extent is the effectiveness of a brief intervention influenced by previously received brief interventions?

6. To what extent is the effectiveness of a brief intervention influenced by previous quit attempts?

7a. Are some interventions more effective than others within population groups e.g. age within gender?

7b. Are some interventions more effective than others between population groups for example, with pregnant women and manual workers?

8. Are interventions tailored to sub-sets of the smoking population (for example, pregnant women, older smokers) more effective with them than generic interventions?

9. Does the setting/site of delivery of the intervention influence effectiveness?

10. Does the profession of the practitioner providing the intervention influence effectiveness? What are the significant features?

11. How applicable is what we know about the most effective brief interventions to the most disadvantaged smokers and pregnant smokers?

12. What are the barriers to delivering smoking cessation interventions?

13. What strategies are effective in encouraging primary care professionals and others to undertake smoking cessation interventions?

14. What, if any, negative consequences arise from brief interventions?

15. What is the impact on inequalities and health? For example, if a brief intervention is targeted at the whole population, will it lead to greater inequalities?

Key questions on referral:

1. What factors – training, incentives – influence the number of referrals?

2 What impact, if any, does the PCT have on referrals from primary care to the services?

3. What factors – mechanisms (e.g. card, fax, telephone, script), role of referrer (e.g. pharmacist, midwife), type and/or location of service – influence the likelihood of a patient following up the referral?

4. Does the method of promoting the specialist service (e.g. national advertising, referral from GPs and other health professionals, word of mouth) influence the number of referrals?

Reviewing the evidence of effectiveness

One review of effectiveness was conducted.

Identifying the evidence

The following databases were searched for recent systematic reviews (2000–August 2005) and for trials (1985–August 2005): MEDLINE, Cochrane Database of Systematic Reviews, Cochrane Controlled Trials Register (CENTRAL), Cochrane Tobacco Addiction Group Specialised Register, Reference Manager, DARE, ASSIA, British Nursing Index, Embase, Cinahl, PsycINFO Sociological Abstracts.

Additional searches using the same databases were conducted for information on barriers to implementation and for the question on referrals to NHS smoking services. In addition, a call for information on referrals was put out on Globalink UK, an international network of over 1000 tobacco control activists, smoking cessation workers and researchers. (This includes most smoking cessation coordinators in England.) Regional stop smoking service managers were also asked for referral data. Full details of the databases and search strategies can be found in the full effectiveness review.

Details of the search terms and strategies are included in the rapid review report.

Selection criteria

Reviews were excluded by two reviewers if:

  • the title or abstract did not primarily address smoking cessation

  • the study was clearly not conducted systematically

  • the study did not address any of the scope questions (in such cases lower level evidence was sought).

A similar process was used to exclude trials and other types of research studies. For these studies one reviewer judged the potential relevance of the evidence. Consistency of coding was assessed in a subset of 80 papers and a kappa of >0.6 obtained for inclusion versus exclusion. Papers where there was uncertainty about the intervention's classification were retained.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Guideline development methods[23]'. Each study was described by study type (categorised as types 1-4) and graded (++, +, -) to reflect the risk of potential bias arising from its design and execution:

Study type

1. Meta-analyses, systematic reviews of RCTs or RCTs (including cluster RCTs).

2. Systematic reviews of, or individual, non-randomised controlled trials, case-control studies, cohort studies, controlled before-and-after (CBA) studies, interrupted time series (ITS) studies, correlation studies.

3. Non-analytical studies (for example, case reports, case series).

4. Expert opinion, formal consensus.

Study quality

++ All or most of the criteria have been fulfilled. Where they have not been fulfilled the conclusions are thought very unlikely to alter.

+ Some criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

- Few or no criteria fulfilled. The conclusions of the study are thought likely or very likely to alter.

Study type and quality were described together, for example, as (1++) or (2-). The studies were also assessed for their applicability to the UK.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full reviews and the synopsis). Outcomes of interest included non-validated and validated smoking status (such as, self-reported smoking abstinence and bio-chemically validated smoking abstinence such as saliva cotinine).

Trials that included follow-ups of six months or more were the primary focus. Trials of shorter duration were included where necessary.

The findings from the review were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements reflect the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic appraisal

The economic appraisal consisted of a review of economic evaluations and a cost-effectiveness analysis.

Review of economic evaluations

A systematic search was carried out on 9 databases from January 1985 to August 2005: CINAHL, Cochrane Library, DARE, EMBASE, NHS HEED, NEED, HMIC, MEDLINE and PSYCINFO.

Where available, abstracts were used to identify papers that might be relevant to the review as appraised by two reviewers. Papers which included, or potentially included, cost-effectiveness results were identified and full copies obtained.

Studies were excluded if they:

  • did not contain any original evidence

  • did not report the costs (or cost effectiveness) of interventions

  • reported on interventions comprising more than one session (other than brief follow-up contacts).

Included studies were assessed for quality using a checklist based on the criteria developed by Drummond et al. (1997)[24]. Inclusion of QALYs as an outcome measure was essential at this stage. As with the review of effectiveness, studies were then given a score (++, +, -) to reflect the risk of potential bias arising from its design and execution. The evidence tables for the cost-effectiveness review are included in the review (see appendix E).

Cost-effectiveness analysis

A cost-effectiveness analysis was carried out for brief interventions in primary care.

An economic model was constructed to incorporate data from the reviews of effectiveness and cost effectiveness. The aim was to estimate the average QALYS gained over the simulation time period. The estimates were based on the estimated 12 month quit rates taken from the review of effectiveness. The estimates were calculated for different ages and gender of the population cohort. Because of the limited nature of the evidence a wide range of sensitivity analyses were performed.

A number of assumptions were made which could underestimate or overestimate the cost per QALY (see modelling report for further details).

How PHIAC formulated the recommendations

At its meeting in January 2006, PHIAC considered the evidence of effectiveness and cost effectiveness and comments from stakeholders to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective or ineffective, or whether it is equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on population health and/or reducing inequalities in health.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and the anticipated extent of change in practice that would be required.

PHIAC noted that the effectiveness of some interventions could vary according to the context in which they were delivered. For example, the social acceptability of smoking in a particular community might affect the way an intervention was received.

PHIAC also considered whether research should be a condition for a recommendation where evidence was lacking.

Where possible, recommendations were linked to an evidence statement(s) – see appendix A for details. Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in January/February 2006. PHIAC met in February 2006 to consider stakeholder comments and to revise the recommendations accordingly. The guidance was signed off by the NICE Guidance Executive in March 2006.



[23] National Institute for Clinical Excellence (2005) Guideline development methods. London: National Institute for Clinical Excellence.

[24] Drummond MF, et al. (1997) Critical assessment of economic evaluation. In Methods for the Economic Evaluation of Health Care Programmes. 2nd edition. Oxford: Oxford Medical Publications.