Appendix B: Methodology checklist: systematic reviews and meta-analyses

Appendix B: Methodology checklist: systematic reviews and meta-analyses

Checklist

Study identification

Include author, title, reference, year of publication

Guideline topic:

Review question no:

Checklist completed by:

SCREENING QUESTIONS

In a well-conducted, relevant systematic review:

Circle or highlight one option for each question

The review addresses an appropriate and clearly focused question that is relevant to the guideline review question

Yes

No

Unclear

The review collects the type of studies you consider relevant to the guideline review question

Yes

No

Unclear

The literature search is sufficiently rigorous to identify all the relevant studies

Yes

No

Unclear

Study quality is assessed and reported

Yes

No

Unclear

An adequate description of the methodology used is included, and the methods used are appropriate to the question

Yes

No

Unclear

If the review does not meet some or all of these criteria, it may still be useful as a source of references, but should not be relied upon on its own to address a review question.

If you have insufficient information on the design or quality of individual studies, you should use the checklists for studies on interventions (see appendices C, D and E) to appraise each study. Each study should appear as a separate entry in the evidence table (see appendix J); the review should not appear in the evidence table.

If you plan to use the review as a whole, you will need to complete a row in an evidence table for the systematic review and input the results into an evidence profile as appropriate.

Notes on use of Methodology checklist: systematic reviews and meta-analyses

A systematic review uses explicit and systematic methods to identify, appraise and summarise the literature according to predetermined criteria. If the methods and criteria used to do this are not described or are not sufficiently detailed, it is not possible to make a thorough evaluation of the quality of the review.

The terms 'systematic review' and 'meta-analysis' are often used interchangeably. The term 'meta-analysis' is often used incorrectly to describe a systematic review that has used quantitative methods to summarise the results. However, it should be noted that meta-analysis refers only to the statistical techniques used to combine studies; thus not all meta-analyses are systematic reviews.

This checklist is intended for use with systematic reviews of questions about interventions and questions about diagnosis. It can potentially be used for any other types of question, although it has been designed primarily for the first two.

The aim of this checklist is to consider the suitability of the systematic review to answer a guideline review question. This assessment has two aspects: firstly, whether the question addressed by the review (in terms of the populations, interventions, comparisons and outcomes considered) is appropriate to answer the review question addressed by the guideline, and secondly, whether the methodology used for the review is sufficiently robust to permit a valid conclusion.

For each question in this section, you should indicate whether or not it has been addressed in the review. Choose 'unclear' if this aspect of the review process was ignored, or is not described in the report.

The review addresses an appropriate and clearly focused question that is relevant to the guideline review question

If the question addressed by the systematic review is not clearly stated, it will be difficult to determine whether the review is adequate to answer the guideline review question. If the question is not clear, the systematic review is unlikely to be a good one because it difficult to be systematic in addressing an unclear question. The review report should give a clear description of the population considered, the interventions, exposures or tests evaluated, comparators, and outcomes evaluated. Inclusion and exclusion criteria should be clearly described. Outcomes considered should be clearly described within the methodology, including a precise definition and acceptable methods of measuring. The appropriateness of the question addressed in the systematic review for answering the guideline review question can be determined by comparing these components. If the review does not consider all of the outcomes that are judged to be important to your guideline review question, you may still be able to use the outcome data but may need to review the individual studies to obtain other outcome data.

The review collects the type of studies you consider relevant to the guideline review question

You should be clear about the characteristics of studies that you consider will adequately address your guideline review question. These may relate to minimum design or quality characteristics (for example, randomised trials only). Systematic reviews should report the types of studies they sought, including any inclusion/exclusion criteria used. You can use this information to quickly assess the review's suitability for your purpose.

The literature search is sufficiently rigorous to identify all the relevant studies

Systematic and rigorous searches can help to minimise publication biases and identify as many relevant data as possible. Exact search terms depend on the review question, but there are core databases that should have been searched for every question. As a minimum, a well-conducted review should look at Embase and MEDLINE. For questions about interventions in particular, the Cochrane Database of Systematic Reviews (CDSR), the Database of Abstracts of Reviews of Effects (DARE)and the Cochrane Central Register of Controlled Trials (CENTRAL) should also be searched. The dates on which the searches were carried out should be given in the review. Good-quality reviews will also attempt to identify relevant studies by handsearching of key journals and examining reference lists of retrieved studies for further references.

If the methods used to locate studies are not clearly reported, it will be difficult to determine whether the review is likely to have missed important relevant studies. Ideally, the search strategy used should be reported in sufficient detail that the process could be replicated.

Any restrictions applied to the search (such as language or year of publication) should also be reported. You should consider how these might have influenced the findings of the review.

Advice from the information specialist (and/or other members of the Guideline Development Group) working on the guideline may be useful to determine whether any important search terms have been omitted.

If the search described in the review is judged to be inadequate to identify all relevant studies, it may be possible to expand the search by including additional databases or extra search terms within the search strategy, or by updating the search to identify more recently published studies. Any additional studies identified by this expanded search should be appraised for quality using the appropriate NICE checklist (see appendices C–I). They should appear individually in separate rows in an evidence table.

Study quality is assessed and reported

The inclusion of poor-quality studies within a review can result in biased estimates of effect. A well-conducted systematic review should have used clear criteria to assess whether individual studies had been appropriately designed and conducted, before deciding whether to include or exclude them. These criteria should be clearly described and should be reported for each study included. The quality appraisal checklists in appendices C–I, as appropriate for the type of question and study design, can be used as a guide to the types of quality criteria that should be considered.

If there is no indication of such a quality assessment, the review is unlikely to be reliable enough to be used in formulating guideline recommendations. It may be necessary to obtain and quality appraise the individual studies as part of your review.

An adequate description of the methodology used is included, and the methods used are appropriate to the question

In common with primary research, the approach used to analyse the data should be described and justified where appropriate. This may include the choice of statistical test used to analyse the outcome data, meta-analytical techniques and approaches to dealing with heterogeneity, including the specification of any subgroup analyses and sensitivity analyses.


This page was last updated: 30 November 2012