Appendix B: summary of the methods used to develop this guidance

Introduction

The report of the review and economic appraisal includes full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E and are available from the NICE website.

The guidance development process

The stages of the guidance development process are outlined in the box below.

1. Draft scope

2. Stakeholder meeting

3. Stakeholder comments

4. Final scope and responses published on website

5. Reviews and cost-effectiveness modelling

6. Synopsis report of the evidence (executive summaries and evidence tables) circulated to stakeholders for comment

7. Comments and additional material submitted by stakeholders

8. Review of additional material submitted by stakeholders (screened against inclusion criteria used in reviews)

9. Synopsis, full reviews, supplementary reviews and economic modelling submitted to PHIAC

10. PHIAC produces draft recommendations

11. Draft recommendations published on website for comment by stakeholders and for field testing

12. PHIAC amends recommendations

13. Responses to comments published on website

14. Final guidance published on website

Key questions

The key questions were established as part of the scope. They formed the starting point for the review of evidence and facilitated the development of recommendations by PHIAC. The overarching question was:

What are the most effective and cost-effective school-based interventions to prevent or reduce alcohol use among pupils?

The following subsidiary questions were considered:

  • What type of content works best (for example, should it focus on the harmful effects to health, legal issues or the social consequences of alcohol use)?

  • Is it better for the intervention to be delivered by a generalist, a specialist or someone else (for example, the police, a peer or a drug worker)?

  • What are the most cost-effective and appropriate interventions for different groups of young people (for example, males and females, different age groups, different social classes and different ethnic groups)?

  • Does the intervention lead to any adverse or unintended effects (for example, an increase in alcohol consumption)?

  • What factors might inhibit or facilitate implementation (for example, parents' views)?

Reviewing the evidence of effectiveness

One review of effectiveness was conducted.

Identifying the evidence

The following databases were searched for systematic reviews, randomised controlled trials (RCTs), non-RCTs, and controlled before and after studies published since 1990:

  • ASSIA (Applied Social Science Index and Abstracts)

  • CINAHL

  • Cochrane Library (CDSR, DARE, HTA and CCTR)

  • EMBASE

  • EPPI-Centre databases

  • ERIC

  • ETOH

  • Health Management Information Consortium

  • MEDLINE

  • National Guidelines Clearing House

  • National Research Register

  • Project Cork

  • PsycINFO

  • SIGLE

  • SOMED

  • SPECTR (Campbell Collaboration Trials Registry)

  • Web of Science (Science and Social Sciences citation indexes).

The following websites were searched:

In addition, information on current practice in English schools at a local and regional level was sought via Healthy Schools and DAAT coordinators. Further details of the search terms and strategies are included in the review report.

Selection criteria

Studies were included if they:

  • involved children and young people under 18 years old

  • were undertaken in primary and secondary state-sector maintained schools, city technology colleges, academies, grammar, non-maintained special and independent schools or pupil referral, secure training and local authority secure units, or further education settings

  • examined interventions in schools which aimed to prevent or reduce alcohol use, including:

    • lessons delivered by teachers or other professionals as part of a classroom-based curriculum

    • peer-led education by other pupils

    • external contributions (for example, from the police, theatre in education (TIE) organisations and life education centres)

    • implementation of school policies

    • activities carried out as part of the informal curriculum (for example, learning experiences in assembly/collective worship and parent evenings)

  • compared the intervention with a control or with another approach

  • reported changes in alcohol-related behaviour, including:

    • percentage who reported drinking alcohol (lifetime, monthly or weekly use)

    • amount of drinking and its frequency

    • age at which children/young people first drank alcohol

    • unsupervised alcohol use.

Studies were excluded if they examined interventions:

  • aimed at children and young people who did not attend any of the types of schools listed above, for example, those in secure institutions or receiving home education

  • without a school-based component, including:

    • 'server' and 'responsible beverage service' (RBS) training, media campaigns and diversionary activities delivered in the wider community

    • regulatory schemes such as taxation, restrictions on alcohol sales and advertising, proof of age schemes and warning labels

    • drink-driving schemes and driver training

    • treatment of alcohol misuse or alcohol dependence, including psychosocial interventions.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for development of NICE public health guidance' (see appendix E). Each study was described by study type and graded (++, +, -) to reflect the risk of potential bias arising from its design and execution.

Study type

  • Meta-analyses, systematic reviews of randomised controlled trials (RCTs) or RCTs (including cluster RCTs).

  • Systematic reviews of, or individual controlled non-randomised trials (CNRT), case-control studies, cohort studies, controlled before-and-after (CBA) studies, interrupted time series (ITS) studies, correlation studies.

  • Non-analytical studies (for example, case reports and case series).

  • Expert opinion, formal consensus.

Study quality

++ All or most criteria have been fulfilled. Where they have not been fulfilled the conclusions are thought very unlikely to alter.

+ Some criteria fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

- Few or no criteria fulfilled. The conclusions of the study are thought likely or very likely to alter.

The main reasons for studies being assessed as (-) were:

  • limited reporting of methodological details such as methods of random assignment

  • high level of participant attrition

  • lack of detail about baseline equivalence of intervention and control groups.

The interventions were also assessed for their applicability to the UK and the evidence statements were graded as follows:

A. harm-reduction approach and likely to be applicable across a broad range of settings and populations

B. harm-reduction approach and likely to be applicable across a broad range of settings and populations, assuming they are appropriately adapted

C. harm-reduction approach but applicable only to settings or populations included in the studies – broader applicability is uncertain, or approach unclear

D. clear abstinence approach or applicable only to settings or populations included in the studies.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full review).

The findings from the studies were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements reflect the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic appraisal

The economic appraisal consisted of a review of economic evaluations and a cost-effectiveness analysis.

Review of economic evaluations

The following databases were searched:

  • EconLit

  • Health Economic Evaluation Database (HEED)

  • NHS Economic Evaluation Database (NHS EED).

The inclusion and exclusion criteria were the same as those used for the effectiveness review. 'Cost per case averted' was chosen as the primary measure of cost and effect.

Cost-effectiveness analysis

The primary outcome produced by the economic analysis was the cost per case of averting hazardous/harmful drinking. An additional analysis was undertaken to estimate the quality of life years (QALY) gained before reaching a £20,000 or £30,000 per QALY threshold. A cost-consequence analysis was also carried out on non-health related outcomes.

An economic model was constructed to incorporate data from the reviews of effectiveness and cost effectiveness. The results are available on the NICE website.

Fieldwork

Fieldwork was carried out to evaluate the relevance and usefulness of NICE guidance for practitioners and the feasibility of implementation. It was conducted with practitioners and commissioners who are involved in providing alcohol education and advice to children and young people in schools. They included those working in the NHS, education, local authorities, the criminal justice sector and the wider public, voluntary and community sectors.

The fieldwork comprised:

  • Three meetings carried out in Liverpool, Manchester and Bristol with practitioners and commissioners working in education, health and the criminal justice sectors.

  • Twenty two semi-structured telephone interviews with professionals working in education, the NHS, children, young people and families' services, criminal justice and the voluntary and community sectors.

The main issues arising from the fieldwork are set out in appendix C under fieldwork findings. The full fieldwork report is available on the NICE website.

How PHIAC formulated the recommendations

At its meeting in May 2007 PHIAC considered the evidence of effectiveness and cost effectiveness to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective or ineffective, or whether it is equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of the evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on population health and/or reducing inequalities in health.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and the anticipated extent of change in practice that would be required.

Where possible, recommendations were linked to an evidence statement(s) (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in July 2007. At its meeting in September 2007, the PDG considered comments from stakeholders and the results from fieldwork, and amended the guidance. The guidance was signed off by the NICE Guidance Executive in October 2007.