Appendix B: Summary of the methods used to develop this guidance

Introduction

The reports of the reviews and economic appraisal include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E and are available from the NICE website.

The guidance development process

The stages of the guidance development process are outlined in the box below.

1. Draft scope

2. Stakeholder meeting

3. Stakeholder comments

4. Final scope and responses published on website

5. Reviews and cost-effectiveness modelling

6. Synopsis report of the evidence (executive summaries and evidence tables) circulated to stakeholders for comment

7. Comments and additional material submitted by stakeholders

8. Review of additional material submitted by stakeholders (screened against inclusion criteria used in reviews)

9. Synopsis, full reviews, supplementary reviews and economic modelling submitted to PHIAC

10. PHIAC produces draft recommendations

11. Draft recommendations published on website for comment by stakeholders and for field testing

12. PHIAC amends recommendations

13. Responses to comments published on website

14. Final guidance published on website

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and facilitated the development of recommendations by PHIAC. The overarching question was:

Which universal, 'whole school', indicated and targeted interventions effectively promote the mental wellbeing of children aged 4–11 in primary education?

The subsidiary questions included the following.

1. What elements of 'whole school' approaches are effective (and cost effective) in promoting the mental wellbeing of children aged 4–11 years?

2. What elements of targeted approaches are effective (and cost effective) in promoting the mental wellbeing of children aged 4-11 years?

3. What type of activities are most effective?

4. What is the frequency, length and duration of an effective intervention?

5. Is it better if teachers, school support staff or a specialist (such as a psychologist or school nurse) delivers the intervention?

6. What is the role of governors?

7. What is the role of parents?

8. What are the barriers to – and facilitators of – effective implementation?

9. Does the intervention lead to any adverse or unintended effects?

Reviewing the evidence of effectiveness

Three reviews of effectiveness were conducted.

Identifying the evidence

The following databases were searched for whole school, universal and targeted interventions (from January 1990 to June 2007):

  • ASSIA (Applied Social Science Index and Abstracts)

  • CENTRAL (BioMed Central)

  • CINAHL (Cumulative Index of Nursing and Allied Health Literature)

  • Cochrane Database of Systematic Reviews

  • DARE (Database of Abstracts of Reviews of Effectiveness)

  • EMBASE (Excerpta Medica)

  • ERIC (Education Resources Information Centre)

  • Medline

  • PsycINFO (Psychological Information)

  • SIGLE (System for Index of Grey Literature in Europe)

  • Sociological Abstracts.

Searches were also conducted of the following websites:

In addition, bibliographies of reviews and studies known to the research teams were searched to identify further studies that might be suitable for inclusion. Further details, including details of the databases, search terms and strategies used, are included in the review reports.

Selection criteria

Studies were included if they:

  • promoted the mental wellbeing of children aged 4–11 in primary education (maintained, independent and special schools)

  • (whole schools review) spanned primary and secondary schools but the mean age was below 12

  • (whole school review) adopted a whole school or universal approach

  • (targeted/indicated review) adopted a targeted/indicated approach

  • (targeted/indicated review) described interventions lasting more than 1 month.

Studies were excluded if they:

  • included children aged above 12 years

  • included children who did not attend school

  • (targeted/indicated review) were aimed at secondary school pupils

  • (targeted/indicated review) had no connection with school other than being delivered to school-aged children

  • (targeted/indicated review) were not based in school

  • (whole school/universal review) did not include a control group

  • (whole school/universal review) were not published in English

  • (whole school/universal review) were carried out in developing countries (according to World Bank/IMF classifications)

  • (whole school/universal review) were published before 1990.

For further details of the inclusion and exclusion criteria for each effectiveness review, see the NICE website.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for development of NICE public health guidance' (see appendix E).

Each study was described by study type and graded (++, +, -) to reflect the risk of potential bias arising from its design and execution.

Study type
  • Meta-analyses, systematic reviews of randomised controlled trials (RCTs) or RCTs (including cluster RCTs).

  • Systematic reviews of, or individual, non-randomised controlled trials, case-control studies, cohort studies, controlled before-and-after (CBA) studies, interrupted time series (ITS) studies, correlation studies.

  • Non-analytical studies (for example, case reports, case series).

  • Expert opinion, formal consensus.

Study quality

++ All or most criteria have been fulfilled. Where they have not been fulfilled the conclusions are thought very unlikely to alter.

+ Some criteria fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

- Few or no criteria fulfilled. The conclusions of the study are thought likely or very likely to alter.

The interventions were also assessed for their applicability to the UK.

Summarising the evidence and making evidence statements

The review data was summarised in evidence tables (see full reviews and the synopsis of the evidence).

The findings from the reviews were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements reflect the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic appraisal

The economic appraisal consisted of an economic review (covering universal approaches and targeted initiatives) and two cost-effectiveness analyses.

Review of economic evaluations

In addition to scanning the effectiveness evidence the following databases were searched:

  • Econlit

  • Health Economics Evaluation Database (HEED)

  • NHS EED (NHS Economics Evaluation Database).

The search strategies for these reviews were developed by NICE in collaboration with the Centre for Reviews and Dissemination at the University of York. Further detail can be found in the full reviews.

Studies were reviewed if they provided economic evidence directly linked to whole school, universal, targeted and indicated approaches. Published studies that met the inclusion criteria were rated to determine the strength of the evidence using the Drummond checklist. ('Guidelines for authors and peer reviewers of economic submissions to the BMJ' Drummond MF, Jefferson TO [1996] British Medical Journal 313: 2075–283.)

Cost-effectiveness analysis

An economic model was constructed to incorporate data from the whole school and targeted effectiveness reviews (reviews 1 and 2).

The 'Health utilities index mark 2' (HUI2) was used to estimate the cost effectiveness of a combined parent/classroom-based intervention in the short term. Modelling was used to predict how targeted interventions could lead to longer term cost savings for the health, social, voluntary and legal sectors (by improving children and young people's mental health and consequently, their behaviour).

The results are reported in 'Estimating the short-term cost effectiveness of a mental health promotion intervention in primary schools', and 'Cost effectiveness of mental health promotion in schools – focused interventions supplementary analysis'. They are available on the NICE website.

Fieldwork

Fieldwork was carried out in three stages. The aim was to evaluate the relevance and usefulness of NICE guidance for practitioners and the feasibility of implementation.

The first fieldwork stage comprised:

  • Qualitative interviews carried out by Dr Foster Intelligence with 91 professionals and parents, either in small groups or individually across 35 sites in London, Newcastle and Liverpool. Participants included: primary school head teachers; primary school teachers with responsibility for pastoral care, PSHE, SEAL, or Healthy Schools; PCT and LEA staff (including local authority directors of children's services and policy staff from local authorities, PCTs and CAMHS); school governors; and parents of children with emotional and behavioural issues.

The second fieldwork stage comprised:

  • An online discussion group, similar methodologically to a face-to-face group interview, on the revised draft recommendations. This was run by Dr Foster Intelligence. It involved four primary school teachers and a child counsellor from England and a primary school teacher from Scotland. All had responsibility for pastoral care in primary schools.

The third fieldwork stage comprised:

  • Qualitative research conducted by YoungMinds with 60 primary school-aged children and young people from central London. Participants were invited to a conference to express their views on what kind of emotional support they felt they need to improve their emotional wellbeing.

The main issues arising from the fieldwork are set out in appendix C under 'Fieldwork findings' and 'Qualitative report findings'. The full fieldwork report is available on the NICE website.

How PHIAC formulated the recommendations

At its meeting in July 2007 PHIAC considered the evidence of effectiveness and cost effectiveness to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective or ineffective, or whether it is equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on population health and/or reducing inequalities in health.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and the anticipated extent of change in practice that would be required.

Where possible, recommendations were linked to an evidence statement(s) (see appendix C for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in November 2007. At its meeting in January 2008 PHIAC considered comments from stakeholders and the results from fieldwork and amended the guidance. The guidance was signed off by the NICE Guidance Executive in January 2008.