The social care guidance manual

1 The review addresses an appropriate and clearly focused question that is relevant to the guidance review question

If the question addressed by the systematic review is not clearly stated, it will be difficult to determine whether the review is adequate to answer the guidance review question. If the question is not clear, the systematic review is unlikely to be a good one because it is difficult to be systematic in addressing an unclear question. The review report should give a clear description of the population considered, the interventions, settings, comparators, and outcomes evaluated. Inclusion and exclusion criteria should be clearly described. Outcomes considered should be clearly described within the methodology, including a precise definition and acceptable methods of measuring. The appropriateness of the question addressed in the systematic review for answering the guidance review question can be determined by comparing these components. If the review does not consider all of the outcomes that are judged to be important to your guidance review question, you may still be able to use the outcome data but may need to review the individual studies to obtain other outcome data.

2 The review collects the type of studies you consider relevant to the guidance review question

You should be clear about the characteristics of studies that you consider will adequately address your guidance review question. These may relate to minimum design or quality characteristics. Systematic reviews should report the types of studies they sought, including any inclusion/exclusion criteria used. You can use this information to quickly assess the review's suitability for your purpose.

3 The literature search is sufficiently rigorous to identify all the relevant studies

Systematic and rigorous searches can help to minimise publication biases and identify as many relevant data as possible. Exact search terms depend on the review question, and you will need to make a judgment on the appropriateness and comprehensiveness of the sources searched for relevant data, which will depend on the aim of the systematic review. The dates on which the searches were carried out should be given in the review. Good-quality reviews will also attempt to identify relevant studies by hand searching of key journals and examining reference lists of retrieved studies for further references.

If the methods used to locate studies are not clearly reported, it will be difficult to determine whether the review is likely to have missed important relevant studies. Ideally, the search strategy used should be reported in sufficient detail that the process could be replicated.

Any restrictions applied to the search (such as language or year of publication) should also be reported. You should consider how these might have influenced the findings of the review.

Advice from the information specialist (and/or other members of the Guidance Development Group) working on the guidance may be useful to determine whether any important search terms have been omitted.

If the search described in the review is judged to be inadequate to identify all relevant studies, it may be possible to expand the search by including additional databases or extra search terms within the search strategy, or by updating the search to identify more recently published studies. Any additional studies identified by this expanded search should be appraised for quality using the appropriate NICE checklist (see appendices C–G). They should appear individually in separate rows in an evidence table.

4 Study quality is assessed and reported

The inclusion of poor-quality studies within a review can result in biased estimates of effect. A well-conducted systematic review should have used clear criteria to assess whether individual studies had been appropriately designed and conducted, before deciding whether to include or exclude them. These criteria should be clearly described and should be reported for each study included. The quality appraisal checklists in appendices C–G, as appropriate for the type of question and study design, can be used as a guide to the types of quality criteria that should be considered.

If there is no indication of such a quality assessment, the review is unlikely to be reliable enough to be used in formulating guidance recommendations. It may be necessary to obtain and quality appraise the individual studies as part of your review.

5 An adequate description of the methodology used is included, and the methods used are appropriate to the question

In common with primary research, the approach used to analyse the data should be described and justified where appropriate. This may include the choice of statistical test used to analyse the outcome data, approaches to dealing with heterogeneity, and methods of analysis and synthesis.

6 Overall assessment of internal validity

Rate the review for internal validity according to the list below:

++ All or most of the checklist criteria have been fulfilled, where they have not been fulfilled the conclusions are very unlikely to alter

+ Some of the checklist criteria have been fulfilled, where they have not been fulfilled, or not adequately described, the conclusions are unlikely to alter

− Few or no checklist criteria have been fulfilled and the conclusions are likely or very likely to alter

7 Overall assessment of external validity

Rate the external validity of the review (also using ++, +, −). This is the extent to which the findings for the study participants apply to the whole source population (the population they were chosen from). If the review is of an 'intervention', then it should be assessed to see whether it would be feasible in settings other than that initially investigated, if restricted by setting, for example.