Process and methods

5 Identifying the evidence: literature searching and evidence submission

5 Identifying the evidence: literature searching and evidence submission

5.1 Introduction

The systematic identification of evidence is an essential step in developing NICE guidelines. Literature searches should be systematic, transparent and reproducible to minimise 'dissemination biases'. These may affect the results of reviews and include publication bias and database bias.

This chapter provides advice on the sources to search and on how to develop strategies for systematic literature searches to identify the best available evidence. It also provides advice on other areas of information management that form an important part of guideline development. These include quality assurance, re‑running searches, documenting the search process, and the use of reference management or systematic review software. The methods for undertaking scoping searches are described in the chapter on the scope. For information on searching for economic evidence, see the chapter on incorporating economic evaluation. For information on searching for surveillance evidence, see the chapter on ensuring that published guidelines are current and accurate.

NICE encourages the use of search methods that balance recall and precision. The aim is to identify the best available evidence to address a particular question without producing an unmanageable volume of results. If review questions are so broad that the information specialist has to restrict the searches to complete the review in the time available, this should be acknowledged as a limitation in the final review document.

A flexible approach to the search for evidence should be adopted, guided by the subject of the question and type of evidence sought. When the guideline is an update, the approach can also be informed by searches for the existing guideline and subsequent surveillance review. Searching includes:

  • tailoring the search approach to the eligibility or inclusion criteria of the review question, as specified in the review protocol

  • selecting appropriate sources according to the eligibility or inclusion criteria of the review question, as specified in the review protocol

  • using additional search techniques, such as citation searching, as appropriate

  • continuous review of how best to find evidence and where.

A flexible approach will allow evidence to be identified both systematically and in the most efficient manner. For example, for a review question on the effectiveness of a pharmacological intervention it may be possible to search a relatively small number of sources (see the section on sources) and to develop a systematic search strategy using the PICO framework (see the section on developing search strategies). For questions about complex interventions, the evidence may be more widely scattered across sources and less consistently described. In these cases it may be necessary to search a wider range of sources and consider other question frameworks and search approaches, including the use of additional searching techniques.

The chapter on searching for and selecting studies in the Cochrane Handbook for Systematic Reviews of Interventions and the Campbell Collaboration's searching for studies methods guide offer good overviews of literature searching (Lefebvre et al. 2021; Kugley et al. 2017). The SuRe Info resource also provides research-based advice on information retrieval for systematic reviews and is updated twice a year.

5.2 Search protocols

Search protocols should be developed by the information specialist and agreed with the other members of the developer's team before undertaking a systematic search. Search protocols are part of the review protocol (see the appendix on the review protocol template), which is signed off by the committee. When developing search protocols, the information specialist may ask the committee for expert advice (for example, when a condition is described in many different ways in the literature).

Search protocols pre‑define how the evidence is identified and provide a basis for developing the search strategies. Search protocols should include the following elements:

5.3 Sources

Searches should include a mix of bibliographic databases, websites and other sources depending on the subject of the review question and the type of evidence sought.

For most searches there will be key sources that should be prioritised, and other potentially relevant sources that could be considered. It is important to ensure adequate coverage of the relevant literature and to search a range of sources, but there should be clear reasons, with only those likely to yield relevant results included. (See also the section on reviewing economic evaluations in the chapter on incorporating economic evaluation for information on searching for economic literature.)

The selection of sources will vary according to the requirements of the review question. For reviews of the effectiveness of pharmacological interventions the following should be prioritised for searching:

  • the Cochrane Central Register of Controlled Trials (CENTRAL)

  • the Cochrane Database of Systematic Reviews (CDSR)

  • Embase

  • MEDLINE

  • the Medicines and Healthcare products Regulatory Agency (MHRA) – for drug safety information.

For other questions, it might be as or more important to search other sources. Examples of other sources include, but are not limited to:

  • PsycINFO (psychology and psychiatry)

  • ASSIA (Applied Social Sciences Index and Abstracts)

  • Social Policy and Practice, Social Care Online

  • Sociological Abstracts

  • HMIC (Health Management Information Consortium)

  • HealthTalk.

For service delivery questions, some of the evidence may be in the form of modelling studies published in journals related to operational research, statistical and mathematical methods rather than in health journals. These are less likely to be indexed in healthcare databases such as MEDLINE, and consideration should be given to sources likely to retrieve this evidence.

The following sources may be useful for service delivery questions:

  • Science Citation Index

  • HMIC

  • Social Care Online or Social Policy Online

  • Kings Fund.

Evidence may also be primary data needed to inform parameters identified in the design-oriented conceptual model. For information on searching for model inputs, see the chapter on incorporating economic evaluation.

For some review questions, for example, where evidence is likely to be published in non-journal sources, it may be appropriate to search for grey literature. Useful sources of grey literature include:

  • HMIC

  • CORE.

Reports from organisations such as the European Medicines Agency and the US Food and Drug Administration may also be of value. The Canadian Agency for Drugs and Technology in Health (CADTH) Grey Matters may also be useful for identifying sources for grey literature, as may web search tools, such as NICE Evidence Search and TRIP. Topic experts on the committee may also be able to suggest appropriate sources for grey literature.

A list is provided in the appendix on sources for evidence reviews as a starting point for identifying potential relevant sources.

5.4 Developing search strategies

Devising a search strategy

The approach to devising and structuring search strategies should be informed by the review question. For example, the PICO (population, intervention, comparator and outcome) or the SPICE (setting, perspective, intervention, comparison, evaluation; Booth 2006) framework can be used to structure a search strategy for an intervention question. For other types of question, other frameworks may be more suitable. It may also be appropriate for the search framework to differ from that of the review question, and the choice should reflect the question type and search approach. Davies (2011) undertook a review of possible frameworks and SuRe Info has a useful summary of alternatives to PICO which have been evaluated. When using a framework, it is important to consider which concepts to include in the strategy because some concepts may not be mentioned in the titles, abstracts or subject headings of a database record. This is a particular challenge when the literature is less well defined and/or indexed. It is important to ensure that relevant studies are not missed as a result of an overly complex search structure.

Some topics are complex, for example, where search requirements evolve as evidence is identified and it is important to balance recall and precision. One approach, when the relevant literature for a question is less well defined or indexed, is to use a multi-stranded method. This involves developing several shorter search strategies (strands) with an emphasis on precision. Each strand should reflect 1 way in which the relevant literature may be described. The strands are then combined.

Another approach is to use an iterative and/or 'stepped' method. Searching is done in several stages, with each search taking into account the evidence that has already been retrieved. Searching in stages allows the reviewers to review the most relevant, high-quality information first. It also gives the opportunity to stop searching, omitting some steps if this is felt appropriate for the topic. Additional steps are added if the developer decides that the quantity or quality of the evidence already gathered is not sufficient. It may also be appropriate to add an additional search step to identify literature published between the final search date of a study and its inclusion in a systematic review. If the developer then finds there are topics of interest not covered by existing reviews, it would be appropriate to carry out additional searching on specific topics. A decision to use a stepped or iterative approach should be agreed by the developer and NICE staff with responsibility for quality assurance because it can affect timelines.

Some topics may have multiple information needs, for example, sometimes indirect evidence is needed for network meta-analyses. This may involve developing iterative searches for a set of relevant comparators as opposed to a search for pre-determined comparators (Hawkins et al. 2009). This type of searching can be time consuming and the developer should agree a decision to do this with NICE staff with responsibility for quality assurance. Searching for observational data for service delivery questions can also be very time consuming. For example, registry data can be a source of estimates of treatment effects, but if the committee cannot estimate the extent or direction of any biases, the value of the data is greatly reduced and the usefulness of searching is low.

For some types of review question, for example, questions for which qualitative research is more appropriate, it may not be necessary to identify all the literature on a topic. The objective may be to reach theoretical saturation, where any additional studies identified merely support the existing line of argument, rather than identify all relevant studies. In this context, it may be possible to undertake searches which are more precise. The search approaches for this type of evidence have been reviewed and summarised by Booth (2016) and can be used to guide practice.

Review questions that overlap and can be grouped together should be identified for searching purposes. For example, questions with the same population may involve comparing several interventions. This should make it possible to carry out a single search that covers all the interventions, although this approach may retrieve a large number of studies.

Identifying search terms

Search strategies should usually consist of a combination of subject headings and 'free‑text' terms from the titles and abstracts of relevant studies. When identifying subject headings, variations in thesaurus and indexing terms for each database should be taken into account: for example, MeSH (Medical Subject Headings) in MEDLINE, Emtree in Embase. Not all databases will have indexing terms and some databases will contain records which have not yet been indexed. Also, not all search concepts will have a subject heading, so free-text terms should also be used.

Free‑text terms may include synonyms, acronyms and abbreviations, spelling variants, old and new terminology, brand and generic medicine names, and lay and medical terminology. For a guideline that is being updated, previous search strategies, including surveillance searches, should be reviewed and used to inform search strategy design. New or changed terms should be identified, as well as any changes to indexing terms. This also applies when an existing review, for example, a Cochrane review, is being updated to answer a review question. Known key studies can be a useful source to identify search terms, but additional sources such as reports, guidelines, and topic-specific websites or topic experts can also be helpful.

Comprehensively identifying search terms may present challenges. For example, for public health, social care or implementation reviews, many databases do not use a controlled vocabulary for indexing records. Sometimes controlled vocabularies are used but do not include terms that adequately cover the search concept(s), which often cross a number of disciplines. In addition, the use of natural language varies between studies, and concepts may not be described in a consistent way. In light of these challenges, the development of a search strategy should always be an iterative process between the information specialists), the developer and, when necessary, the committee and NICE staff with responsibility for quality assurance (Alpi 2005, Papaioannou et al. 2010).

A further challenge may arise from websites or certain databases having limited search functionality. It may be necessary to simplify the search strategy, using fewer search terms or undertaking multiple searches of the same resource with different search term combinations.

It may be helpful to use frequency analysis or text mining to develop the search strategy (Stansfield et al. 2017, Hausner et al. 2012). Tools such as PubReMiner and Medline Ranker can help, either by highlighting search terms that might not otherwise be apparent, or by flagging terms of high value when exhaustive synonym searching is unfeasible or inadvisable. This is a rapidly developing area, but in principle NICE supports the use of such technologies to inform search development.

Limits and filters

Searches should be limited to studies reported in English. When there are likely to be animal studies in the evidence base, these can be excluded from the search results in some databases.

Limiting searches by date will depend on the topic and the nature of the evidence base (for example, when most of the research was published). When the guideline is an update, searches may be limited by date, but only if appropriate (for example, the inclusion and exclusion criteria for the update are the same as for the existing guideline). If there are relevant good-quality published systematic reviews (see the chapter on developing review questions and planning the evidence review), it may be possible to limit additional searching to the time since the searches for the published reviews were conducted. For some databases, restricting the search by date is difficult. When this is the case, searches can be run without date limits, using reference management or systematic review software to separate new results. The date range for the search, and the use of existing reviews, should be agreed before searching and documented in the search protocol (see the section on search protocols).

Depending on the review question, it may be appropriate to limit searches to particular study designs. For example, for review questions on the effectiveness of interventions, it may be more efficient to search for systematic reviews, followed by controlled trials followed by observational studies. This prevents unnecessary searching and review work. The best way to limit searches by study design is to use an appropriate search filter (strings of search terms), rather than using database publication type field limits, to ensure the search strategy is transparent and reproducible. Additional classifiers, such as the Cochrane RCT classifier, could be used to further attempt to limit the search to particular study types.

Other search filters relating to age, setting, geography, and health inequalities can also be applied as relevant. The most comprehensive list of available search filters is the search filter resource of the InterTASC Information Specialists' Sub-Group. This resource also includes critical appraisal tools, which can be used for filter selection. Additionally, McKibbon (2009) reviews the performance of 38 randomised controlled trial filters. Both sources may be useful when choosing a filter. Search filters should, however, be used with caution because concepts such as study design, age, setting and geography may not be adequately described in the title or abstract of a database record, and may not be captured by the indexing.

NICE is not prescriptive about which search filters should be used because there is often limited evidence on the performance of individual filters. Alternative methods for refining a search to achieve an adequate balance of recall and precision should be used when filters are not appropriate.

Additional or alternative search techniques

Additional or alternative search techniques should be used in addition to database searching when it is known, or reasonably likely, that relevant evidence is either not indexed in bibliographic databases and/or that it is difficult to retrieve from databases in a way that adequately balances recall and precision. These techniques might include forward and backward citation searching, journal hand-searches or contacting experts and stakeholders.

Reviews (for example, systematic reviews, literature reviews and meta-analyses) may provide an additional source of primary studies, with reference lists being used to identify these.

All search techniques should follow the same principles of transparency, rigour and reproducibility as other search methods.

Additional or alternative search techniques should be considered at the outset and documented in the search protocol (see the section on search protocols), if possible. They should also be documented in the evidence review.

5.5 Calls for evidence from stakeholders

Stakeholders' role in providing evidence

In some topic areas or for some review questions, NICE staff with responsibility for quality assurance, the developer or committee may believe that there is relevant evidence in addition to that identified by the searches. In these situations, the developer may invite stakeholders, and possibly also other relevant organisations or individuals with a significant role or interest (see the section on other attendees at committee meetings in the chapter on decision-making committees on expert witnesses), to submit evidence. A call for evidence is issued directly to registered stakeholders and via the NICE website. Examples and details of process are included in the appendix on call for evidence and expert witnesses. Confidential information should be kept to an absolute minimum.

5.6 Health inequalities and equality and diversity

All searches should be inclusive, capturing evidence related to health inequalities or impacts on equality relevant to the guideline topic. For example, if the population group is 'older people' a search for 'older people' should pick up subpopulations such as 'disabled older people' or 'black and minority ethnic older people'. Similarly, if the setting is 'communities and religious places', the search terms should cover all relevant faith settings (such as 'church', 'temple' and 'mosque').

5.7 Quality assurance

Quality assuring the literature search is an important step in guideline development. Studies have shown that errors do occur (Sampson 2006). Although developed specifically for Cochrane reviews, the Methodological Expectations of Cochrane Intervention Reviews (MECIR) manual may be useful to guide practice. They set out the components which can ensure that the identification of the evidence base is comprehensive, transparent and reproducible (Higgins et al. 2021).

For each search (including economic searches), the principal database search strategy should be quality assured by a second information specialist to maintain a consistently high standard for identifying the evidence. A checklist should be used to ensure clarity and consistency when quality assuring search strategies. An example is the PRESS 2015 Guideline Evidence-Based Checklist (McGowan et al. 2016).

Each time the principal database strategy is adapted for use in another database, it is good practice for it to be peer reviewed by a second information specialist to ensure quality and consistency is maintained.

As part of quality assurance, there is also an opportunity to undertake an audit of search results to analyse how the evidence for the guideline was identified. Using the final list of included studies, it is possible to determine the contribution of individual sources and search techniques. This can provide valuable data for informing the search approach for future surveillance and guideline updates.

5.8 Reference management

Electronic records of the references retrieved by searches should be stored using EPPI-Reviewer.

5.10 Re-running searches

Searches undertaken to identify evidence for each review question (including economics searches) may be re‑run to identify any further evidence that has been published since the search was last run. For example, searches should be re‑run if the evidence base changes quickly, or if there is reason to believe that substantial new evidence exists, or if the development time is longer than usual. Searching PubMed or OVID's MEDLINE Epub Ahead of Print at this stage, in addition to MEDLINE, is useful to identify 'ahead-of-print' citations.

A decision to re‑run searches will be taken by the developer and members of NICE staff with a quality assurance role.

If undertaken, searches should be re‑run at least 6 to 8 weeks before the final committee meeting before consultation.

If evidence is identified after the last cut‑off date for searching but before publication, a judgement on its impact should be made by the developer and NICE staff with a quality assurance role. In exceptional circumstances, this evidence can be considered if its impact is judged as substantial.

5.11 References and further reading

Alpi K (2005) Expert searching in public health. Journal of the Medical Library Association 93: 97–103

Beynon R, Leeflang MM, McDonald S et al. (2013) Search strategies to identify diagnostic accuracy studies in MEDLINE and Embase. Cochrane Database of Systematic Reviews 2013, Issue 9

Booth A (2006) Clear and present questions: formulating questions for evidence based practice. Library Hi Tech 24: 355–68

Booth A (2016) Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews 5:74

Booth A, Harris J, Croot E et al. (2013) Towards a methodology for cluster searching to provide conceptual and contextual 'richness' for systematic reviews of complex interventions: case study (CLUSTER). BMC Medical Research Methodology 13: 118

Brunton J, Stansfield C, Thomas J (2017) Finding relevant studies. In: Gough D, Oliver S, Thomas J, editors. Introduction to systematic reviews. London: Sage

Canadian Agency for Drugs and Technologies in Health (2019) Grey Matters: a practical tool for searching health-related grey literature [online; accessed 21 January 2021]

Centre for Reviews and Dissemination (2009) Systematic reviews: CRD's guidance for undertaking reviews in health care. University of York: Centre for Reviews and Dissemination

Davies K (2011) Formulating the evidence based practice question: a review of the frameworks. Evidence Based Library and Information Practice 6: 75–80

Glanville J, Lefebvre C, Wright K (editors) (2008, updated 2017) The InterTASC Information Specialists' Subgroup Search Filters Resource [online; accessed 31 August 2018]

Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and methods. Systematic Reviews 1: 28.

Hausner E, Waffenschmidt S, Kaiser T et al. (2012) Routine development of objectively derived search strategies. Systematic Reviews 1: 19

Hawkins N, Scott DA, Woods B. (2009) How far do you go? Efficient searching for indirect evidence. Medical Decision Making 29: 273–81

Higgins JPT, Lasserson T, Chandler J et al. (2021) Methodological expectations of Cochrane intervention reviews (MECIR). London: Cochrane.

Jenkins M (2004) Evaluation of methodological search filters – a review. Health Information and Libraries Journal 21: 148–63

Kugley S, Wade A, Thomas J et al. (2017) Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: The Campbell Collaboration

Lefebvre C, Glanville J, Briscoe S et al. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Cumpston M et al. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021

McGowan J, Sampson M, Salzwedel DM et al. (2016) PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement. Journal of Clinical Epidemiology 75: 40–6

McKibbon KA, Wilczynski NL, Haynes RB. (2009) Retrieving randomized controlled trials from MEDLINE: a comparison of 38 published search filters. Health Information and Libraries Journal 26: 187-202

Morrison A, Polisena J, Husereau D et al. (2012) The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. International Journal of Technology Assessment in Health Care 28: 138–44

Paisley S, Levay P, Ananiadou S et al. Public health, policy and the evidence-based medicine paradigm: challenges and innovations in methods of searching. Research Synthesis Methods (in press)

Papaioannou D, Sutton A, Carroll C et al. (2010) Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Information Libraries Journal 27: 114–22

Rethlefsen M, Kirtley S, Waffenschmidt S et al. (2021) PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Systematic Reviews 10: 39

Sampson M, McGowan J (2006) Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology 59: 1057–63

Stansfield C, O'Mara-Eves A, Thomas J (2017) Text mining for search term development in systematic reviewing: a discussion of some methods and challenges. Research Synthesis Methods 8: 355-65

Summarized research for Information Retrieval in HTA (SuRe Info). [online; accessed 31 August 2018]