Appendix B Methodology checklist: systematic reviews and meta-analyses

Appendix B Methodology checklist: systematic reviews and meta-analyses

Appendices B–G include checklists for those study designs that are expected to be used in the evidence reviews for NICE social care guidance. Other checklists can found in the NICE clinical guidelines manual and Methods for the development of NICE public health guidance.

Checklist

Study identification

Include author, title, reference, year of publication

Guidance topic:

Review question no:

Checklist completed by:

SCREENING QUESTIONS

In a well-conducted, relevant systematic review:

Circle or highlight one option for each question

1 The review addresses an appropriate and clearly focused question that is relevant to the review question

Yes

No

Unclear

2 The review collects the type of studies you consider relevant to the guidance review question

Yes

No

Unclear

3 The literature search is sufficiently rigorous to identify all the relevant studies

Yes

No

Unclear

4 Study quality is assessed and reported

Yes

No

Unclear

5 An adequate description of the methodology used is included, and the methods used are appropriate to the question

Yes

No

Unclear

6 Overall assessment of internal validity. Are the results internally valid?

Rate the review for internal validity below (for further information see notes on using the methodology checklist)

++

+

-

Comments:

Overall assessment of external validity – Are the results externally valid (i.e. generalisable to the whole source population)? Consider participants, interventions, settings, comparisons and outcomes.

7 Rate the review for external validity below (for further information see notes on use of the methodology checklist)

++

+

-

If the review does not meet some or all of these criteria, it may still be useful as a source of references, but should not be relied upon on its own to address a review question.

If you have insufficient information on the design or quality of individual studies, you should use the checklists for studies on interventions (see appendices C, D and E) to appraise each study. Each study should appear as a separate entry in the evidence table (see appendix H); the review should not appear in the evidence table.

If you plan to use the review as a whole, you will need to complete a row in an evidence table for the systematic review and input the results into an evidence profile as appropriate.

Notes on use of Methodology checklist: systematic reviews and meta-analyses

A systematic review uses explicit and systematic methods to identify, appraise and summarise the literature according to predetermined criteria. If the methods and criteria used to do this are not described or are not sufficiently detailed, it is not possible to make a thorough evaluation of the quality of the review.

The terms 'systematic review' and 'meta-analysis' are often used interchangeably. The term 'meta-analysis' is often used incorrectly to describe a systematic review that has used quantitative methods to summarise the results. However, it should be noted that meta-analysis refers only to the statistical techniques used to combine studies; thus not all meta-analyses are systematic reviews.

This checklist is intended for use with systematic reviews of questions about interventions. It can potentially be used for any other types of question, although it has been designed primarily for this purpose.

The aim of this checklist is to consider the suitability of the systematic review to answer a guidance review question. This assessment has 2 aspects: firstly, whether the question addressed by the review (in terms of the populations, interventions, comparisons and outcomes [PICO] or setting, perspective, intervention, comparison, evaluation [SPICE]) is appropriate to answer the review question addressed by the guidance, and secondly, whether the methodology used for the review is sufficiently robust to permit a valid conclusion.

For each question in this section, you should indicate whether or not it has been addressed in the review. Choose 'unclear' if this aspect of the review process was ignored, or is not described in the report.

1 The review addresses an appropriate and clearly focused question that is relevant to the guidance review question

If the question addressed by the systematic review is not clearly stated, it will be difficult to determine whether the review is adequate to answer the guidance review question. If the question is not clear, the systematic review is unlikely to be a good one because it is difficult to be systematic in addressing an unclear question. The review report should give a clear description of the population considered, the interventions, settings, comparators, and outcomes evaluated. Inclusion and exclusion criteria should be clearly described. Outcomes considered should be clearly described within the methodology, including a precise definition and acceptable methods of measuring. The appropriateness of the question addressed in the systematic review for answering the guidance review question can be determined by comparing these components. If the review does not consider all of the outcomes that are judged to be important to your guidance review question, you may still be able to use the outcome data but may need to review the individual studies to obtain other outcome data.

2 The review collects the type of studies you consider relevant to the guidance review question

You should be clear about the characteristics of studies that you consider will adequately address your guidance review question. These may relate to minimum design or quality characteristics. Systematic reviews should report the types of studies they sought, including any inclusion/exclusion criteria used. You can use this information to quickly assess the review's suitability for your purpose.

3 The literature search is sufficiently rigorous to identify all the relevant studies

Systematic and rigorous searches can help to minimise publication biases and identify as many relevant data as possible. Exact search terms depend on the review question, and you will need to make a judgment on the appropriateness and comprehensiveness of the sources searched for relevant data, which will depend on the aim of the systematic review. The dates on which the searches were carried out should be given in the review. Good-quality reviews will also attempt to identify relevant studies by hand searching of key journals and examining reference lists of retrieved studies for further references.

If the methods used to locate studies are not clearly reported, it will be difficult to determine whether the review is likely to have missed important relevant studies. Ideally, the search strategy used should be reported in sufficient detail that the process could be replicated.

Any restrictions applied to the search (such as language or year of publication) should also be reported. You should consider how these might have influenced the findings of the review.

Advice from the information specialist (and/or other members of the Guidance Development Group) working on the guidance may be useful to determine whether any important search terms have been omitted.

If the search described in the review is judged to be inadequate to identify all relevant studies, it may be possible to expand the search by including additional databases or extra search terms within the search strategy, or by updating the search to identify more recently published studies. Any additional studies identified by this expanded search should be appraised for quality using the appropriate NICE checklist (see appendices C–G). They should appear individually in separate rows in an evidence table.

4 Study quality is assessed and reported

The inclusion of poor-quality studies within a review can result in biased estimates of effect. A well-conducted systematic review should have used clear criteria to assess whether individual studies had been appropriately designed and conducted, before deciding whether to include or exclude them. These criteria should be clearly described and should be reported for each study included. The quality appraisal checklists in appendices C–G, as appropriate for the type of question and study design, can be used as a guide to the types of quality criteria that should be considered.

If there is no indication of such a quality assessment, the review is unlikely to be reliable enough to be used in formulating guidance recommendations. It may be necessary to obtain and quality appraise the individual studies as part of your review.

5 An adequate description of the methodology used is included, and the methods used are appropriate to the question

In common with primary research, the approach used to analyse the data should be described and justified where appropriate. This may include the choice of statistical test used to analyse the outcome data, approaches to dealing with heterogeneity, and methods of analysis and synthesis.

6 Overall assessment of internal validity

Rate the review for internal validity according to the list below:

++ All or most of the checklist criteria have been fulfilled, where they have not been fulfilled the conclusions are very unlikely to alter

+ Some of the checklist criteria have been fulfilled, where they have not been fulfilled, or not adequately described, the conclusions are unlikely to alter

− Few or no checklist criteria have been fulfilled and the conclusions are likely or very likely to alter

7 Overall assessment of external validity

Rate the external validity of the review (also using ++, +, −). This is the extent to which the findings for the study participants apply to the whole source population (the population they were chosen from). If the review is of an 'intervention', then it should be assessed to see whether it would be feasible in settings other than that initially investigated, if restricted by setting, for example.