Appendix B: Summary of the methods used to develop this guidance

Introduction

The reports of the reviews and economic analysis include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E and are available online.

Guidance development

The stages involved in developing public health intervention guidance are outlined in the box below.

1. Draft scope released for consultation

2. Stakeholder meeting about the draft scope

3. Stakeholder comments used to revise the scope

4. Final scope and responses to comments published on website

5. Evidence review(s) and economic analysis undertaken

6. Evidence and economic analysis released for consultation

7. Comments and additional material submitted by stakeholders

8. Review of additional material submitted by stakeholders (screened against inclusion criteria used in review/s)

9. Evidence and economic analysis submitted to PHIAC

10. PHIAC produces draft recommendations

11. Draft guidance released for consultation and for field testing

12. PHIAC amends recommendations

13. Final guidance published on website

14. Responses to comments published on website

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and were used by PHIAC to help develop the recommendations. The overarching question was:

  • Which school-based interventions, or combination of school-based interventions, are effective and cost effective in preventing children and young people from taking up smoking?

The subsidiary questions were:

  • What factors aid the delivery of effective school-based interventions to prevent the uptake of smoking?

  • What are the barriers to successful delivery?

These questions were made more specific for the reviews (see reviews for further details).

Reviewing the evidence of effectiveness

Both the effectiveness review and the qualitative evidence review made use of the same base literature search. This is detailed below, followed by more information on the specific criteria used by both reviews in selecting evidence.

Identifying the evidence

The following databases were searched for relevant studies (searches were conducted for studies published from January 1990 to November 2008):

  • ASSIA (Applied Social Science Index and Abstracts)

  • Cochrane Library (Wiley):

    • Cochrane Database of Systematic Reviews (CDSR)

    • Cochrane Central Register of Controlled Trials (CENTRAL)

  • EMBASE

  • ERIC

  • MEDLINE

  • Health Management Information Consortium (HMIC)

  • PsycINFO

  • York Centre for Reviews and Dissemination database (Database of Abstracts of Reviews of Effects [DARE] and Health Technology Assessment [HTA] database)

The database searches were supplemented by searches of the following websites:

Selection criteria

Inclusion and exclusion criteria for each review varied and details can be found online.

Review 1 (the effectiveness review) included:

  • randomised controlled trials (RCTs) with a follow-up of 6 months or more and with a sample size of 500 or greater. They had to include a comparator and report a change in smoking prevalence as an outcome.

Review 2 (the qualitative review) included:

  • studies that involved qualitative reporting of outcomes.

In general, studies were included in both reviews if they:

  • addressed the prevention of smoking among children and young people aged under 19 who attended an educational institution

  • were school-based or included a school-based component as part of a combined intervention

  • were conducted in Organization for Economic Cooperation and Development (OECD)-listed countries

  • were published in English from 1990 onwards

  • were reported in English.

Studies were excluded from review 1 if they were not an RCT, had a follow-up of less than 6 months and the sample size was less than 500.

Studies were excluded from both reviews if they:

  • focused on:

    • children under age 5 who do not attend an educational institution

    • children and young people who are educated at home

    • children and young people who are excluded from school

    • young people aged over 16 who are not in education

    • young people aged 19 and older

  • had no school component

  • were conducted in non-OECD countries

  • were published before 1990

  • were not published in English.

Quality appraisal

Included papers in both reviews were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for the development of NICE public health guidance' (see appendix E). Each study was graded (++, +, –) to reflect the risk of potential bias arising from its design and execution.

Study quality

++ All or most of the methodology checklist criteria have been fulfilled. Where they have not been fulfilled, the conclusions are thought very unlikely to alter.

+ Some of the methodology checklist criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

– Few or no methodology checklist criteria have been fulfilled. The conclusions of the study are thought likely or very likely to alter.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full reviews).

The findings from the reviews were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements were prepared by the external contractors (see appendix A). The statements reflect their judgement of the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic analysis

The economic analysis consisted of a review of economic evaluations and a cost-effectiveness analysis.

Review of economic evaluations

A systematic review of economic literature was undertaken by searching major bibliographic databases from their inception up to October 2008. These included:

  • ASSIA

  • Cochrane Library

  • EMBASE

  • ERIC

  • HMIC

  • MEDLINE

  • PsycINFO

  • York Centre for Reviews and Dissemination database.

This database search was supplemented by searches of selected websites.

Economic evaluations were selected if they:

  • were conducted in OECD countries

  • were published in English from 1990 onwards

  • assessed the cost-effectiveness of school-based interventions to prevent the uptake of smoking among children and young people.

The methodological quality of each study was rated using the Drummond checklist, and its applicability to the relevant population in the UK assessed. Applicability for economic evaluations was assessed on the basis of two dimensions:

  • Whether or not the population being studied was comparable to the current UK population.

  • Whether or not the methodology of the study was likely to yield results similar to a study based on the NICE reference case.

Cost-effectiveness analysis

A number of assumptions were made which could underestimate or overestimate the cost effectiveness of the interventions (see review modelling report for further details).

An economic model was constructed to incorporate data from the reviews of effectiveness and cost effectiveness. The results are reported in: 'School-based interventions to prevent the uptake of smoking among children and young people: cost-effectiveness model'.

Fieldwork

Fieldwork was carried out to evaluate how relevant and useful NICE's recommendations are for practitioners and how feasible it would be to put them into practice. It was conducted with practitioners and commissioners involved in tobacco control, health promotion and education services. This included those working in local authorities, educational establishments, the NHS and the voluntary sector.

The fieldwork comprised:

  • 18 focus groups carried out by GHK Consulting Ltd with commissioners, tobacco control specialists, Healthy Schools coordinators, teachers, youth workers and voluntary organisation staff in all nine English regions.

  • Four workshops carried out by the National Youth Agency with schoolchildren aged 11–17 years based in urban and rural locations.

The main issues arising from the resulting two studies are set out in appendix C under fieldwork findings. The full fieldwork reports, 'School-based interventions to prevent the uptake of smoking among children and young people' and 'NICE guidance on school-based prevention of smoking in children: consultation with young people', are available.

How PHIAC formulated the recommendations

At its meeting in May 2009 PHIAC considered the evidence reviews and review of cost effectiveness to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective, ineffective or equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on the target population's health.

  • Impact on inequalities in health between different groups of the population.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and any anticipated changes in practice.

Where possible, recommendations were linked to an evidence statement(s) (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in September 2009. At its meeting in November 2009, PHIAC amended the guidance in light of comments from stakeholders, experts and the fieldwork. The guidance was signed off by the NICE Guidance Executive in January 2010.

  • National Institute for Health and Care Excellence (NICE)