Appendix B: Summary of the methods used to develop this guidance

Introduction

The reports of the review, qualitative research and economic analysis include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E and are available from the NICE website.

The guidance development process

The stages of the guidance development process are outlined in the box below.

1. Draft scope

2. Stakeholder meeting

3. Stakeholder comments

4. Final scope and responses published on website

5. Reviews and cost-effectiveness modelling

6. Synopsis report of the evidence (executive summaries and evidence tables) circulated to stakeholders for comment

7. Comments and additional material submitted by stakeholders

8. Review of additional material submitted by stakeholders (screened against inclusion criteria used in reviews)

9. Synopsis, full reviews, supplementary reviews and economic modelling submitted to PHIAC

10. PHIAC produces draft recommendations

11. Draft recommendations published on website for comment by stakeholders and for field testing

12. PHIAC amends recommendations

13. Responses to comments published on website

14. Final guidance published on website

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and facilitated the development of recommendations by PHIAC. The five overarching questions were:

1. Which mass-media interventions are effective and cost effective in preventing children and young people from smoking?

2. Which interventions are effective and cost effective in reducing the illegal sale of tobacco to children and young people?

3. When appropriate interventions can be compared, which are most effective in preventing the uptake of smoking and the sale of tobacco to children and young people?

4. Are the interventions delaying rather than preventing the onset of smoking?

5. How would differences between the comparators used in published studies and the prevailing situation in England impact on the analysis of effectiveness and cost effectiveness?

Reviewing the evidence of effectiveness

A review of effectiveness was conducted.

Identifying the evidence

The following databases were searched for relevant systematic reviews, experimental studies and qualitative studies (from 1996–2006):

  • ASSIA (Applied Social Science Index and Abstracts)

  • British Nursing Index

  • CDSR

  • CENTRAL

  • CINAHL

  • Current Contents

  • DARE

  • EMBASE

  • HMIC

  • HSTAT

  • MEDLINE

  • National Research Register

  • PAIS

  • PsycINFO

  • Social Policy & Practice

  • Sociological Abstracts

  • TRIP

The following websites were also searched for relevant reports:

Further details of the databases, search terms and strategies are included in the review 'Interventions to prevent the uptake of smoking in children and young people'.

Selection criteria

Studies were included in the effectiveness review if they:

  • focused on children and young people aged under 18

  • used mass-media communications, including new media (such as podcasting, text messaging or social networking websites) to prevent the uptake of smoking

  • aimed to prevent sales of tobacco to children and young people

  • were published in English from 1990 onwards.

Studies were excluded if they:

  • focused on those aged 18 and over

  • focused on family, education or social interventions

  • were school-based

  • included counselling or self-help and did not involve the use of mass media

  • focused on price measures

  • were conducted in a developing country or not published in English.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for development of NICE public health guidance' (see appendix E). Each study was described by study type and graded (++, +, −) to reflect the risk of potential bias arising from its design and execution.

Study type
  • Meta-analyses, systematic reviews of randomised controlled trials (RCTs) or RCTs (including cluster RCTs).

  • Systematic reviews of, or individual, non-randomised controlled trials, case-control studies, cohort studies, controlled before-and-after (CBA) studies, interrupted time series (ITS) studies, correlation studies.

  • Non-analytical studies (for example, case reports, case series).

  • Expert opinion, formal consensus.

Study quality

++ All or most criteria fulfilled. Where they have not been fulfilled the conclusions are thought very unlikely to alter.

+ Some criteria fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

− Few or no criteria fulfilled. The conclusions of the study are thought likely or very likely to alter.

The interventions were also assessed for their applicability to the UK and a statement was included in the evidence statements.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full review).

The findings from the review were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements reflect the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic analysis

The economic analysis consisted of a review of economic evaluations and a cost-effectiveness analysis.

Review of economic evaluations

The following databases were systematically searched to identify studies published since 1990:

  • CINAHL (1990–July 2007)

  • EconLit (1990–June 2007)

  • EMBASE (1990–July 2007)

  • HEED (1990–July 2007)

  • HMIC (1990–July 2007)

  • MEDLINE (1990–July 2007)

  • NHS EED (1990–June 2007)

Studies were eligible for inclusion if:

  • they included children and young people aged up to 18

  • the population covered did not smoke at the start of the study

  • they reported on the cost and effectiveness of the prevention strategy.

The Drummond checklist (Drummond MF, Jefferson TO [1996] 'Guidelines for authors and peer reviewers of economic submissions to the BMJ'. British Medical Journal 313: 2075–283) was used for quality assurance.

Cost-effectiveness analysis

An economic model was constructed to incorporate data from the reviews of effectiveness and cost effectiveness. The results are reported in: 'Cost-effectiveness of a mass media campaign and a point of sale intervention to prevent the uptake of smoking in children and young people' by Dr Maria Raikou and Professor Alistair McGuire, LSE Health. It is available on the NICE website.

Qualitative research: focus groups

A range of both qualitative and quantitative research methods were used. The sample comprised: young people in school and sixth form colleges, two additional groups of young people who were at risk of, or who had been excluded from mainstream education and young people in contact with smoking cessation services. Full details of the methodology, data analysis and ethical approval for the project can be found in the report.

Fieldwork

Fieldwork was carried out to evaluate the relevance and usefulness of NICE guidance for practitioners and the feasibility of implementation. It was conducted in a variety of locations with practitioners and commissioners who are involved in tobacco control services. They included those working in the NHS, local authorities and the wider public sector, the criminal justice sector, marketing and retail.

The main issues arising from the fieldwork are set out in appendix C under fieldwork findings. The full fieldwork report 'NICE CPHE public health intervention guidance recommendations on measures to prevent the uptake of smoking by children and young people' is available on the NICE website.

How PHIAC formulated the recommendations

At its meeting in January 2008 PHIAC considered the evidence of effectiveness and cost effectiveness to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective or ineffective, or whether it is equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on population health and/or reducing inequalities in health.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and the anticipated extent of change in practice that would be required.

Where possible, recommendations were linked to an evidence statement(s) (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in March/April 2008. At its meeting in May 2008, the PDG considered comments from stakeholders and the results from fieldwork and amended the guidance. The guidance was signed off by the NICE Guidance Executive in June 2008.