Appendix D: summary of the methods used to develop this guidance

Introduction

The reports of the reviews and economic appraisal include full details of the methods used to select the evidence (including search strategies), assess its quality and summarise it.

The minutes of the PHIAC meetings provide further detail about the Committee's interpretation of the evidence and development of the recommendations.

All supporting documents are listed in appendix E and are available from the NICE website.

The guidance development process

The stages of the guidance development process are outlined in the box below:

1. Draft scope

2. Stakeholder meeting

3. Stakeholder comments

4. Final scope and responses published on website

5. Reviews and cost-effectiveness modelling

6. Synopsis report of the evidence (executive summaries and evidence tables) circulated to stakeholders for comment

7. Comments and additional material submitted by stakeholders

8. Review of additional material submitted by stakeholders (screened against inclusion criteria used in reviews)

9. Synopsis, full reviews, supplementary reviews and economic modelling submitted to PHIAC

10. PHIAC produces draft recommendations

11. Draft recommendations published for comment by stakeholders and for field testing

12. PHIAC amends recommendations

13. Responses to comments published

14. Final guidance published on website

Key questions

The key questions were established as part of the scope. They formed the starting point for the reviews of evidence and facilitated the development of recommendations by PHIAC. The overarching question was: How can one to one interventions contribute to the reduction of STIs (including HIV) and the reduction of the rate of under 18 conceptions? The subsidiary questions were:

  1. What is the aim/objective of the intervention? What is it trying to change?

  2. What outcome measures are used to assess effectiveness? How valid and appropriate are they?

  3. What is the content of the intervention? Does it influence effectiveness?

  4. Does the way it is carried out (the type/mode of communication, for example) influence effectiveness?

  5. Does effectiveness depend on the job title/position or other factors such as age, gender, sexuality or ethnicity of the deliverer (leader)? What are the significant features of an effective deliverer (leader)?

  6. Does the site/setting of delivery influence effectiveness?

  7. Does the intensity, length or frequency influence effectiveness/duration of effect?

  8. Does effectiveness vary according to age, gender, sexuality, socio-economic status or ethnicity of target audience?

  9. What evidence is there on cost effectiveness?

  10. What are the barriers to implementing effective interventions?

These questions were refined further in relation to the topic of each review (see reviews for further details).

Reviewing the evidence of effectiveness

Three reviews of effectiveness were conducted.

Review 1 'Contraceptive advice and provision for the prevention of under 18 conceptions and STIs: a rapid review'.

Review 2 'Review of evidence for the effectiveness of screening for genital chlamydial infection in sexually active young women and men'.

Review 3 'Review of evidence for the effectiveness of partner notification for sexually transmitted infections including HIV'.

Identifying the evidence

The following core databases were searched for randomised controlled trials, controlled before/after studies and qualitative studies (process only): Medline, Embase, Psychinfo, DARE and Sigle from 1990–2005. Reference lists from included studies were hand searched.

Further details of databases, search terms and strategies are included in the review reports.

Selection criteria

Inclusion and exclusion criteria for each review varied and details can be found online.

However, in general:

  • review 1 included one to one interventions which offered information, advice, condoms, counselling, cognitive behavioural therapy and/or activities that increase self-confidence, self-esteem and skill development

  • review 2 considered any activity described as screening or where testing for chlamydia was offered to asymptomatic sexually active adults

  • review 3 considered any intervention described as partner notification or contact tracing, or where partners were located and informed that they have been exposed to an infection

  • studies in both NHS and non-health settings were considered. Details of the studies that were excluded can be found in the reviews.

Quality appraisal

Included papers were assessed for methodological rigour and quality using the NICE methodology checklist, as set out in the NICE technical manual 'Methods for development of NICE public health guidance' (see appendix E). Each study was described by study type and graded (++, +, -) to reflect the risk of potential bias arising from its design and execution.

Study type
  • Meta-analyses, systematic reviews of RCTs or RCTs (including cluster RCTs).

  • Systematic reviews of, or individual, non-randomised controlled trials, case-control studies, cohort studies, controlled before-and-after (CBA) studies, interrupted time series (ITS) studies, correlation studies.

  • Non-analytical studies (for example, case reports, case series).

  • Expert opinion, formal consensus.

Study quality

++ All or most of the criteria have been fulfilled. Where they have not been fulfilled the conclusions are thought very unlikely to alter.

+ Some criteria have been fulfilled. Those criteria that have not been fulfilled or not adequately described are thought unlikely to alter the conclusions.

- Few or no criteria fulfilled. The conclusions of the study are thought likely or very likely to alter.

The studies were also assessed for their applicability to the UK.

Summarising the evidence and making evidence statements

Data from the reviews was summarised in evidence tables (see full reviews and the synopsis). Outcomes of interest included:

  • review 1: reductions in under 18 teenage conceptions and STIs including HIV (primary outcomes), and increased condom use, improved sexual health knowledge, and a reduction in the number of sexual partners and general sexual risk taking (intermediate outcomes)

  • review 2: reduction in the prevalence and incidence of chlamydia and female reproductive tract morbidity

  • review 3: reduction in the incidence and prevalence of STI (patient and index patient), increase in number of partners contacted, tested and treated.

The findings from the reviews were synthesised and used as the basis for a number of evidence statements relating to each key question. The evidence statements reflect the strength (quantity, type and quality) of evidence and its applicability to the populations and settings in the scope.

Economic appraisal

The economic appraisal consisted of a review of economic evaluations and a cost-effectiveness analysis.

Review of economic evaluations

A systematic search was carried out on four databases from January 1990 to December 2005: Econlit, NHS HEED, NEED, DARE. The results of these searches were supplemented by results from the parallel effectiveness reviews and additional papers identified by NICE. The main inclusion criteria were:

  • studies focused on one to one interventions

  • studies set in countries in Europe, US, Canada and Australia

  • studies set in prison, army, primary care and secondary care settings.

Included studies were assessed for quality using a checklist based on the criteria developed by Drummond et al. (1997). Studies were then given a score (++, +, -) to reflect the risk of potential bias arising from its design and execution. The evidence tables for the cost-effectiveness review are included in the review (see appendix E).

Cost-effectiveness analysis

Economic models were constructed to incorporate data from the reviews of effectiveness and cost effectiveness. The results are reported in 'PHIAC 6.7 economic modelling report' (University) and 'PHIAC 6.10 economic modelling report' (NERA consultancy). They are available on the NICE website.

Fieldwork

Fieldwork was carried out to evaluate the relevance and usefulness of NICE guidance for practitioners and the feasibility of implementation. It was conducted with practitioners and commissioners who are involved in sexual health services. They included those working across youth, community and clinical settings in the NHS, local authorities and voluntary sector.

The fieldwork comprised:

  • a qualitative study using structured focus groups in Birmingham, Leeds, Bristol and Plymouth, carried out by SHM Ltd.

  • an online Delphi study carried out in the North West and London regions by John.

The two studies were commissioned to ensure there was ample geographical coverage. The main issues arising from these two studies are set out in appendix A under 'Fieldwork findings'. The full fieldwork report is available on the NICE website.

How PHIAC formulated the recommendations

At its meetings in May 2006 and September 2006 PHIAC considered the evidence of effectiveness and cost effectiveness. In addition, at its meeting in December 2006, it considered comments from stakeholders and the results from fieldwork to determine:

  • whether there was sufficient evidence (in terms of quantity, quality and applicability) to form a judgement

  • whether, on balance, the evidence demonstrates that the intervention is effective or ineffective, or whether it is equivocal

  • where there is an effect, the typical size of effect.

PHIAC developed draft recommendations through informal consensus, based on the following criteria.

  • Strength (quality and quantity) of evidence of effectiveness and its applicability to the populations/settings referred to in the scope.

  • Effect size and potential impact on population health and/or reducing inequalities in health.

  • Cost effectiveness (for the NHS and other public sector organisations).

  • Balance of risks and benefits.

  • Ease of implementation and the anticipated extent of change in practice that would be required.

Where possible, recommendations were linked to an evidence statement(s) (see appendix A for details). Where a recommendation was inferred from the evidence, this was indicated by the reference 'IDE' (inference derived from the evidence).

The draft guidance, including the recommendations, was released for consultation in October and November 2006. The guidance was signed off by the NICE Guidance Executive in February 2007.