Appendix B: Summary of the methods used to develop this guidance

Introduction

The review and economic analysis include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E and are available online.

Guidance development

The stages involved in developing public health intervention guidance are outlined in the box below.

1. Draft scope released for consultation

2. Stakeholder meeting about the draft scope

3. Stakeholder comments used to revise the scope

4. Final scope and responses to comments published on website

5. Evidence review(s) and economic analysis undertaken

6. Evidence and economic analysis released for consultation

7. Comments and additional material submitted by stakeholders

8. Review of additional material submitted by stakeholders (screened against inclusion criteria used in review/s)

9. Evidence and economic analysis submitted to PHIAC

10. PHIAC produces draft recommendations

11. Draft guidance released for consultation and for field testing

12. PHIAC amends recommendations

13. Final guidance published on website

14. Responses to comments published on website

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and were used by PHIAC to help develop the recommendations. The overarching question was:

Which 'whole-school' approaches and activities are effective and cost effective in promoting young people's emotional, social and psychological wellbeing and preventing bullying and violent behaviour?

The subsidiary questions were:

  1. What are the key features of an effective and cost-effective 'whole-school' approach?

  2. What types of lessons (scope, content, frequency, length, method) are most effective and cost effective?

  3. What are the most effective and cost-effective ways of protecting young people who are vulnerable to poor social and emotional health during key transition stages?

  4. What is the best (and most cost effective) way to ensure 'whole-school' approaches are sensitive to specific cultural, religious and ethnic needs?

  5. What is the role of teachers, other school-based practitioners and specialists (such as educational psychologists, counsellors, therapists or school nurses) and other professionals (such as youth workers) in ensuring young people's social and emotional wellbeing?

  6. What role should young people play in the design, delivery and assessment of 'whole-school' approaches? What are the most effective ways of involving them in decision-making?

  7. What is the role of governors and parents?

  8. How can schools effectively engage with parents living in disadvantaged circumstances?

  9. What mechanisms ensure effective links with external agencies (including youth services)?

  10. What is the role of voluntary and community agencies?

  11. What is the best way of evaluating the impact of different approaches?

  12. What are the barriers to – and facilitators of – effective implementation?

  13. Does the approach lead to any adverse or unintended effects?

Reviewing the evidence of effectiveness

A review of the effectiveness of universal interventions for the promotion of emotional and social wellbeing in secondary schools was conducted.

Identifying the evidence

The following databases were searched for primary studies and reviews published from 1990:

  • ASSIA(Applied Social Science Index and Abstracts)

  • CINAHL (Cumulative Index of Nursing and Allied Health Literature)

  • Cochrance Clinical Trials

  • Cochrane Databases of Systematic Reviews

  • DARE (Database of Abstracts of Reviews of Effectiveness)

  • EconLit

  • EMBASE (Excerpta Medica)

  • ERIC (Education Resources Information Centre)

  • Medline

  • Medline in Process

  • NHS EED (Economic Evaluation Database)

  • PsycINFO

  • Science Citation Index

  • Social Sciences Citation Index.

A search of the following website was also conducted:

The aim was to identify publications by key authors and those relating to specific interventions and programmes identified during the initial search. The programmes included Social and Emotional Aspects of Learning (SEAL), Social Emotional and Behavioural Skills (SEBS), School-wide positive behavioural support and Belly Busters (bully prevention).

Selection criteria

Studies were included in the effectiveness review if they:

  • featured young people aged 11–19 (including those with disabilities and other special needs)

  • were conducted in education settings (including state, independent, special school settings and pupil referral units)

  • described generic or whole-school interventions (including policies, systems and structures and the school's physical environment)

  • examined the school's links with parents and the community

  • examined the development and support given to teachers and other staff to ensure they can promote student – and their own - emotional and social wellbeing

  • were published from 1990 onwards.

Studies were excluded if they:

  • focused on young people who were not in full-time education

  • covered targeted interventions focused on specific groups/types of risk, conditions or behaviours (for example, young people already showing signs of depression or disruptive behaviour)

  • looked at wider community-based activities

  • described strategies which primarily aim to prevent self-harm and suicide

  • included young people under 11 or over 19

  • were not published in English

  • were undertaken in a developing country.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for the development of NICE public health guidance' (see appendix E). Each study was graded (++, +, –) to reflect the risk of potential bias arising from its design and execution using the NICE methodology (revised to reflect the nature of research in a school setting).

Study quality

++ At least seven of the methodology checklist criteria are well covered (if appropriate for the study design) and there is an attrition rate of <30%.

+ At least five of the methodology checklist criteria have been fulfilled (if appropriate for the study design) and/or there is an attrition rate of <50%.

– Less than five of the methodology checklist criteria have been fulfilled and/or there is an attrition rate of >50%.

The interventions were also assessed for their applicability to the UK and the evidence statements were graded as follows:

A. Intervention has been delivered in UK settings.

B. Intervention has been delivered to similar populations but might need adaptation.

C. Intervention has been delivered to specific cultural groups in the UK but might need adaptation.

D. Intervention has been delivered to a population that is entirely different from the UK.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full reviews).

The findings from the review were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements were prepared by the public health collaborating centres (see appendix A). The statements reflect their judgement of the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic analysis

The economic analysis consisted of a review of economic evaluations and a cost-effectiveness analysis.

Review of economic evaluations

As part of the systematic review of effectiveness, the economic databases EconLit and NHS Economic Evaluation Database (EED) were searched. Papers with economic content were retained for possible inclusion in the economic review.

Searches were run to retrieve papers on interventions designed to improve social behaviours and/or prevent bullying and disruptive behaviours. All studies of direct relevance to secondary schools (or that might contribute to linked themes in an economic model) were selected during the abstract sifting phase. None of the 53 studies identified presented an economic analysis of a school intervention. Of 12 initially selected because of their potential relevance to broader issues, only two included any economic analysis, neither of which proved pertinent.

Cost-effectiveness analysis

No studies of relevance were identified during the economic review. A protocol was therefore developed to search for literature that could link intervention outcomes to the modelling of cost effectiveness (for example, by establishing a link between a child's negative behaviour and various outcomes in later life, such as their health and employment status). The protocol considered the results of both primary data analysis and literature searching.

A number of assumptions were made which could underestimate or overestimate the cost effectiveness of the interventions (see review modelling report for further details).

An economic model was constructed to incorporate data identified using the protocol described above. The results are reported in 'Cost-effectiveness of universal interventions which aim to promote emotional and social wellbeing in secondary schools'.

Fieldwork

Fieldwork was carried out to evaluate how relevant and useful NICE's recommendations are for practitioners and how feasible it would be to put them into practice.

The fieldwork comprised two components:

  • A study to ascertain the views of practitioners working with young people in secondary education, undertaken by GHK Consulting Ltd. A sample of practitioners, working in 10 local authority areas (weighted with respect to deprivation) participated. They included teachers (including those in senior positions), support staff, governors, counsellors, those working in local authority children's services, PCTs and the voluntary sector).

  • A consultation with young people aged 11–20, undertaken by the National Youth Agency. Participants included students who had just started secondary school and those who were about to leave, as well as a number of students who had been excluded. The full reports are available online.

How PHIAC formulated the recommendations

At its meeting in December 2008 PHIAC considered the evidence of effectiveness and cost effectiveness to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective, ineffective or equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on the target population's health.

  • Impact on inequalities in health between different groups of the population.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and any anticipated changes in practice.

Where possible, recommendations were linked to an evidence statement(s) (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in March 2009. At its meeting in April 2009, PHIAC amended the guidance in light of comments from stakeholders, experts and the fieldwork. The guidance was signed off by the NICE Guidance Executive in September 2009.

  • National Institute for Health and Care Excellence (NICE) accreditation logo