Process and methods

9 Reviewing the evidence

Good practice guidance (GPG) provides recommendations for good practice for people who are involved in governing, prescribing and commissioning medicines, and those involved in decision-making about medicines according to the best available evidence.

The nature of the GPG topics means that the best available evidence on which to produce the guidance may include evidence other than randomised controlled trials.

Studies identified during literature searches need to be reviewed to identify the most appropriate data to help address the review questions, and to ensure the GPG recommendations are based on the best available evidence.

The same rigour is applied to assessing fully and partially published studies, as well unpublished data supplied by stakeholders, for example from a call for evidence (see section 8.9.1 and section 8.9.2) or evidence provided by experts (see section 8.9.3).

The NICE project team appraises the evidence using a technique appropriate for the type of evidence, which includes applying inclusion and exclusion criteria. This allows the GDG to consider the strength of the evidence available when making recommendations (see section 9.1.5).

9.1 Methods of review

The method of review is based on the type of evidence retrieved. Given the nature of the topics, the best available evidence on which to produce the GPG may include evidence other than randomised controlled trials.

The review process for the evidence needs to be clearly documented, giving details of the inclusion and exclusion criteria that were applied.

9.1.1 Clinical studies

Before acquiring papers for assessment, the NICE project team sifts the evidence identified in the search to discard irrelevant material. First, the titles of the retrieved citations are scanned. Those falling outside the GPG topic are excluded. A review of the remaining paper identifies those that are clearly not relevant to the review questions and are therefore excluded.

Once sifting is complete, full versions of the selected studies are requested for assessment (see section 8.6). Full versions of the studies failing to meet the inclusion criteria are excluded; those meeting the criteria are assessed. Because there is a potential for error and bias in selecting the evidence, double sifting (sifting by 2 people) of a random selection of abstracts is performed periodically (Edwards et al. 2002). Any doubts about inclusion are resolved by discussion with the Guidance Development Group (GDG) before the results of the study are considered. The quality of individual research studies is assessed using an appropriate NICE methodological checklist and process for the type of study under consideration. See links provided below:

The checklists have been linked to the appendices of the guidelines manual. The term 'guideline' can be replaced with 'guidance' for purpose of this methods guide.

9.1.2 Legislation and policy

Relevant legislation or NHS policies may be identified in the literature search and used as evidence to develop the GPG. This does not need assessment, given the nature of the source.

Assessing the robustness of NHS policies for inclusion as evidence is discussed at GDG meeting 1. The NICE project team use a standard template for this assessment.

9.1.3 Published guidance

Relevant published guidance from other organisations may be identified in the search for evidence.

Any guidance produced by a NICE-accredited guidance producer is not subject to further quality appraisal having already undergone formal process appraisal by NICE.

Guidance not produced by a NICE-accredited guidance producer is assessed for quality using the AGREE II (appraisal of guidelines research and evaluation II) instrument (Brouwers et al. 2010) to ensure there is sufficient documentation to be considered. There is no cut-off point for accepting or rejecting guidance and each GDG needs to set its own parameters. These are documented in the methods section of the GPG, along with a summary of the assessment. The results are presented as an appendix to the guidance.

Reviews of evidence from other guidance that considers questions formulated by the GDG and the NICE project team may be considered as evidence if:

  • they are assessed using the appropriate methodology checklist from this methods guide and are judged to be of high quality

  • they are accompanied by an evidence statement and evidence table(s)

  • the evidence is updated according to the methodology for exceptional updates of NICE GPG (see section 16).

The GDG creates its own evidence summaries or statements. Evidence tables from other guidance are referenced with a direct link to the source website or a full reference of the published document. The GDG formulates its own recommendations, taking into consideration the whole body of evidence.

9.1.4 Oral testimony and written evidence

During development of the GPG, if there is limited primary research available as evidence on which to produce recommendations on, oral or written evidence may be needed. The written submission of evidence identifies the evidence most relevant to current NHS practice and the GPG topic, therefore informing GDG decision-making. Inclusion criteria for the evidence specifies relevant population and interventions for the review question, to support filtering and selection of oral and written submissions to be included for GDG review.

9.1.5 Other evidence

Given the nature of the GPG and the topic being covered, other sources of relevant evidence such as editorials, reports, audits, qualitative studies (which may include patient experience), surveys and standard operational procedures to guide recommendations may be included. Such evidence is assessed for the reasonableness and rigour of the process used to develop the source of evidence in question and their relevance to the topic under consideration.

9.2 Documenting evidence found outside of the literature search

All evidence reviewed is documented using a standard template for inclusion in an evidence table (see appendix C). Evidence tables help to identify the similarities and differences between studies, including the key characteristics of the study population and interventions or outcome measures. This provides a basis for comparison. Qualitative studies can be recorded using a similar table as for clinical studies (see appendix C).

Meta-analysis may be needed to pool treatment estimates from different studies. Recognised approaches to meta-analysis are used, as described in the manual from NHS Centre for Reviews and Dissemination (2009) and in the Cochrane handbook.

The NICE project team may produce evidence profiles if a body of evidence answers a review question. The profiles contain a 'quality assessment' section summarising the quality of the evidence and a 'summary of findings' section presenting the outcome data for each important outcome.

Information received in response to a call for evidence is entered into a reference management database, and the details cross-checked against evidence identified through database searching. It is assessed in the same way as published studies identified through the searches (see section 8).

9.3 Further reading

Brouwers M, Kho ME, Browman GP et al. for the AGREE Next Steps Consortium (2010) AGREE II: advancing guideline development, reporting and evaluation in healthcare. Canadian Medical Association Journal 182: E839–42

Centre for Reviews and Dissemination (2009) Systematic reviews: CRD's guidance for undertaking reviews in health care. University of York: Centre for Reviews and Dissemination

Edwards P, Clarke M, DiGuiseppi C et al. (2002) Identification of randomized trials in systematic reviews: accuracy and reliability of screening records. Statistics in Medicine 21: 1635–40

Higgins JPT, Green S, editors (2011) Cochrane handbook for systematic reviews of interventions. Version 5.1.0 (updated March 2011) [online]