NICE process and methods

8 Identifying the evidence: literature searching and evidence submission

The systematic identification of evidence is an essential step in developing the good practice guidance (GPG). Systematic literature searches are undertaken by NICE's guidance information services[3] to identify the evidence. Sources used to search for evidence depend on the review questions identified from the scoping workshop (see section 4). Search terms are provided to the information service for the literature search (see appendix A).

8.1 Methods of obtaining evidence

This section contains the main methods used to gather evidence for developing the GPG. The NICE project team will use a standard template to request literature search from NICE guidance information services[3].

8.1.1 Devising a search strategy

The evidence underpinning the GPG is derived from review questions arising from the scope and scoping workshop with stakeholders. The NICE project team develops the review questions into search terms for request using a PICO (population, intervention, comparator and outcome) framework (see section 4.4.1).

The search strategy is constructed by NICE guidance information services[3] for terms relating to the population; this can be combined with terms relating to the interventions and comparators (if any) to be evaluated. Not all components of a search term or review question are always mentioned in the abstracts or subject headings of database records – outcomes are often not mentioned. Therefore, these components may not be included when developing a strategy. For GPGs being updated, previous strategies can be used to inform the search strategy design.

8.1.2 Identifying search terms

Search strategies usually consist of a combination of subject headings and 'free-text' terms from the titles and abstracts of relevant studies. Search terms used are derived from the review questions identified during the scoping process. Subject headings are used to identify the main theme of an article. When identifying subject headings, variations in thesaurus and indexing terms for each database are included; for example, MeSH in MEDLINE and the Cochrane Library, and Emtree in Embase. Free-text terms may include synonyms, acronyms and abbreviations, differences in terminology across national boundaries, different spellings, old and new terminology, brand and generic drug names, and lay and medical terminology. Misspellings or typographical errors may also affect a search, particularly with records being indexed, for which there may be only a title and no abstract or subject headings.

8.1.3 Sensitivity and precision

The key attributes of a search strategy are:

  • sensitivity: the number of relevant records retrieved by a search strategy as a proportion of the total number of relevant records (Jenkins 2004)

  • precision: the number of relevant records retrieved by a search strategy as a proportion of the total number of records retrieved (Jenkins 2004).

Both attributes are influenced by the time period covered and by the search terms used. Although it is important that searches for systematic reviews attempt to identify all the relevant literature, there needs to be a trade-off between conducting an exhaustive search requiring additional resources compared with undertaking a more modest search that may miss some studies. Identifying key studies for a review question can assist in checking search sensitivity; such studies can also act as a guide to search terms.

8.1.4 Grouping review questions

Review questions that overlap and can be grouped together are identified for searching purposes (for example, questions about factors contributing to medication errors in care homes).

Questions that have the 'population' and 'intervention' in common but a different comparator can be grouped together by identifying and combining search terms for the population and intervention only.

8.1.5 Limiting searches

Using certain parameters to limit searches can improve precision without affecting sensitivity:

  • Date parameters depend on the GPG topic and when most of the research was published. The date range for the search is agreed by the NICE project team, in consultation with experts in the area if appropriate. If relevant good-quality published systematic reviews exist, additional searching may be limited to updating the reviews, covering the time period since the searches for the published systematic reviews were conducted. Existing reviews may not address all of the relevant outcomes, in which case new searches may be needed. Authors of published reviews may be contacted for updates, particularly for reviews found in the Cochrane Database of Systematic Reviews.

  • Depending on the review question, it may be appropriate to limit searches to particular study designs. The best way to do this is to use an appropriate search filter rather than limiting searches by the publication type field (see appendix A).

  • If a decision has been taken to limit a review to studies reported in English, the appropriate database limit function can be used to improve precision.

8.2 Databases and sources to search

Databases and other sources that are searched to identify evidence depend on the review question.

8.2.1 Core and subject-specific databases

The core databases listed in table 8.1 are searched for every review question. Additional subject-specific databases and other resources may also need to be searched, depending on the subject area of the review question and the type of evidence sought. Links are provided in table 8.1 and table 8.2 for sources that can be easily accessed.

Table 8.1 Databases that can be searched

Question type


Review questions about interventions, systems and processes, patient experience and service delivery

Core databases:

Subject-specific database/websites (this list is not exhaustive):

An awareness of the strengths and weaknesses of each database is important when undertaking a systematic literature search.

The various databases use different:

  • index journals

  • subject headings

  • time periods

  • amounts of bibliographic information.

For example, Embase is considered stronger than MEDLINE in covering pharmacology, toxicology, drug research and psychiatric literature, but contains selected dental and nursing literature. MEDLINE contains a collection of scope notes for its subject heading (MeSH) terms, which can assist in developing the search strategy. Records retrieved from the different databases for a particular review question may overlap; the extent of this overlap for MEDLINE and Embase is reported as being between 10% and 87% depending on the topic (Lefebvre et al. 2008). Cross-database searching, although time-consuming, is needed to comprehensively identify evidence for GPG development.

8.2.2 Other sources of information

The sources listed in table 8.2, including databases and websites, can provide useful information about ongoing research, patient experience, audits and statistics to help guide the Guidance Development Group's (GDG's) decision-making. Because the GPG is aimed at providing advice and guide good practice for people involved in handling, prescribing, commissioning and decision-making about medicines, for most of the topics it is necessary to use regulatory authorities and professional bodies as a source of information. This list is not exhaustive; the 'Searching for studies' chapter in the Cochrane handbook provides an overview and further examples of sources to search (Lefebvre et al. 2011). Other sources used depend on the GPG topic.

Table 8.2 Other sources of information



NICE-accredited organisations (this list is not exhaustive)

All Wales Medicines Strategy Group (AWMSG)

Healthcare Improvement Scotland

Medicines and Healthcare products Regulatory Agency


Social Care Institute for Excellence (SCIE)

Scottish Intercollegiate Guidelines Network (SIGN)


Department of Health

International Standard Randomised Controlled Trial Number Register

National organisations

NHS England

Public Health England

National Audit Office

Care Quality Commission

Regulatory Authorities (this list is not exhaustive)

General Dental Council

General Medical Council

General Optical Council

General Pharmaceutical Council

Health Professions Council

Health and Safety Executive

Council for Healthcare Regulatory Excellence

Nursing and Midwifery Council

Professional bodies


Royal College of Nursing

British Medical Association

Royal Pharmaceutical Society

British Dental Association

Web of Knowledge

Conference Papers Index

The King's Fund

NHS Education for Scotland

Health Education England

Hospital Episode Statistics

Patient Episode Database for Wales

National or regional audits

Search by topic or geographical area for appropriate audit data.

Information about patient experiences

Surveys of patients' experiences

Search for relevant patient organisation websites; condition-specific or topic-specific as appropriate.




Drugs and Therapeutic Bulletin (DTB)


Regional Drug & Therapeutics Centre

Office of Health Economics

8.3 Quality assurance of search strategies

The quality and accuracy of search strategies during the development of the GPG is checked. Although it is not usually possible to check all strategies for every search, the following approaches may be used to ensure that the key studies are retrieved:

  • Ask GDG members to identify, for example, key studies, legislation and policies (if relevant to the GPG topic) that are already published, but not included in the search results.

  • Check search strategies used in existing published systematic reviews.

  • Check the bibliographies of included studies to ensure all relevant papers have been retrieved by the search strategy used.

  • If relevant papers have not been retrieved by the search strategy, investigate and amend the strategy if appropriate.

8.4 Reference management software

Electronic records of the references retrieved by searches are stored using the reference management software 'Reference Manager'. Records are exported from bibliographic databases such as MEDLINE and imported automatically into the software using import filters by NICE guidance information services[3] who then forward the prepared reference management files to the NICE project team. Details of any additional references identified in guidance development or any oral submissions needed by the GDG are added to the files manually.

In addition to storing records of references, the NICE project team use the reference management software for the following:

  • coding the references with additional information, such as:

    • the source of the reference

    • the review question it was identified to answer

    • the study design

    • selection decisions

  • providing links to the full text of articles, where possible

  • logging the ordering and/or receipt of articles

  • keeping track of the printed copies of papers.

The coding scheme is determined and agreed by the NICE project team before working with a reference management database to ensure consistency of use (see appendix B).

8.5 Sifting

8.5.1 First sift

A first sift reviews each article by title and abstract against search criteria, removing any irrelevant results. This may be done by either the NICE project team or NICE guidance information services[3] (see appendix A).

8.5.2 Second sift

Two members of the NICE project team independently sift each article by title and abstract in the reference management database. An opinion is recorded on whether each article meets the criteria for inclusion or exclusion taken from the PICO framework (see section 4.4.1), drawn up to answer the review questions for the GPG.

Once complete, a reconciliation meeting assesses and resolves any difference of opinion on articles over which there is disagreement. If there is no resolution, a third opinion that is binding is sought from a third NICE project team member. A final list of articles for inclusion is signed off by the NICE project lead after agreement by the GDG at GDG meeting 1. Any articles excluded at this stage have a rationale for exclusion recorded in the reference management database.

8.6 Acquiring full text of references

The NICE project team accesses the full text of references from several sources and asks the NICE guidance information services[3] for those references that cannot be sourced using these methods.

The full text of references can be obtained from several sources:

8.6.1 Free online journal articles

Many journals provide free access to some or all of their content. Several journals apply this to all material more than 1 or 2 years old; others provide access to particular types of articles only (for example, the British Medical Journal provides free access to all research articles). Individual articles can be purchased from the websites of most journals that do not allow free access, but this can be expensive.

Some websites provide links to medical journal web pages with freely available articles. Two that are used are Free Medical Journals and Genamics JournalSeek.

NICE Evidence Services and its Welsh equivalent, NHS Wales e\u2011\Library for health, provide free access to some journals for all NHS staff and staff in organisations such as NICE using an Athens log-in. This log-in can be obtained by applying to the NICE guidance information services[3].

8.6.2 Free online reports

Many institutions make their reports and guidelines freely available online, so the relevant websites can be checked.

8.6.3 Libraries

Many libraries that stock a wide range of journals, books and reports have an inter-library loan or document delivery service. All supply articles within copyright law and some loan documents. There is usually a charge for this service, and for loans the cost of postage is usually extra. Some libraries provide articles at a reduced cost if an annual subscription is taken out. Three major libraries offering this level of service are the British Library, the British Medical Association (BMA) Library and the Royal Society of Medicine Library. A British Library account also allows users to pay for articles from other libraries that accept payment in this way.

When these references are needed, the NICE project team uses NICE guidance information services[3] to source the reference.

8.7 Documenting search strategies

An audit trail is kept of the searches for all types of evidence carried out during the GPG development process to ensure the process for identifying the evidence is transparent and reproducible.

8.7.1 Internal documentation

The following information is recorded for each search carried out during the GPG development process:

  • details of the question for which the search was conducted

  • the names of the databases and database host systems used

  • the database coverage dates; for example, Ovid MEDLINE 1950 to February week 3 2012

  • the date on which the search was conducted

  • the search strategy (this should be stored in an easily accessible form, such as Microsoft Word)

  • any limits applied to the search or to study designs searched for

  • the number of records retrieved from each database

  • a text file and/or database (Reference Manager) of results.

Enough detail is provided to allow searches to be repeated if needed.

8.7.2 Documentation for the good practice guidance

A full description of the searching process is included in the methods section of the GPG, which should include:

  • details of the scoping search (see section 4.4.1)

  • details of the development of the search strategies

  • dates on which the searches were carried out

  • any limits placed on the type of evidence searched for and details of methodological search filters, if used

  • names of the databases, database host systems and any other sources searched

  • date or language limits applied to searches

  • a consort diagram.

8.8 Gap analysis

From the evidence identified in the main literature search, the NICE project team determines if there is sufficient published evidence to address the review questions identified from the scoping workshop. The gap analysis documents the sources, type (nature of the published information) and, where appropriate, the quality of the retrieved published literature. Information describing current practice, the level of activity and any significant regional variation is summarised alongside the published evidence. The gap analysis is provided to the GDG for discussion at GDG meeting 1. This helps the GDG to:

  • identify the gaps between current practice, service provision and patient experience

  • shape the GPG and formulate recommendations likely to have the greatest impact on practice, including clinical outcomes.

8.9 Additional evidence

8.9.1 Call for written evidence

For some review questions, the GDG and NICE project team may agree that information exists that has not been found using standard searches. Therefore, the GDG and the NICE project team may 'call for evidence'. This call goes to all stakeholders and/or relevant organisations or individuals with a significant role or interest in the GPG topic being developed. It specifies the question(s) being addressed and detail the type of evidence being sought, for example, in terms of PICO framework (see section 4.4.1) and study design for questions of effectiveness. The need for a call for evidence is agreed at GDG meeting 1. The opportunity to provide written evidence is usually open for 2 weeks. All written evidence submissions are considered at GDG meeting 2. Confidential Information

In addition to published studies, organisations or individuals may submit relevant unpublished information in response to a call for evidence. When the NICE project team and the GDG send out a call for evidence, respondents are asked to complete a checklist that lists and identifies the location of all confidential information contained in their submission. The NICE project team keeps the checklists for their records to ensure that the draft and final versions of the GPG do not contain confidential information.

Box 8.1 summarises what may and may not be considered confidential by NICE.

Box 8.1 Information on what may and may not be considered confidential

Data that may be included as confidential include those that may influence share price values ('commercial in confidence') or are intellectual property ('academic in confidence'; that is, awaiting publication).

Confidential information should be kept to an absolute minimum; for example, just the relevant part of a sentence, a particular result from a table or a section of code.

NICE does not allow a whole study to be designated confidential. As a minimum, a structured abstract of the study or economic model needs to be made available for public disclosure during consultation on the GPG.

Results derived from calculations using confidential data are not considered confidential unless releasing those results would enable back-calculation to the original confidential data.

In addition to completing the checklist, respondents indicate the part of their submission that contains the confidential information, for example, by using a highlighter pen on a hard copy, or the highlighter function in an electronic version. These markings are then maintained on those sections so that the GDG knows which parts are confidential. When the draft and final version of the GPG is prepared for publication, the NICE project team ensures that these sections are replaced by a note stating that confidential information has been removed, so that readers know exactly where confidential data have been used.

Following the principles in box 8.1, the amount of confidential information is kept to a minimum. As a minimum, a summary is publicly available by the time of the consultation on the GPG. NICE needs to be able to justify the recommendations in the GPG on the basis of the evidence considered by the GDG. The NICE project team therefore works with the data owners to agree a balance between confidentiality and transparency. Information not eligible for submission

Organisations or individuals are asked not to submit the types of evidence listed in box 8.2, as these will not be considered.

Box 8.2 Material not eligible for consideration by the GDG

Studies with weak designs if better designed studies are available

Promotional literature

Papers, commentaries and editorials that interpret the results of a published paper

8.9.2 Call for oral evidence

After review of the written evidence submissions, the GDG select up to 10 submissions to be called to give oral evidence at GDG meeting 3. The NICE project team records a written summary of the oral evidence presented.

8.9.3 Experts

If the GDG and the NICE project team identify gaps in the evidence that can be addressed by experts in that topic, the identified experts can be contacted directly to provide evidence. This can either be as written evidence or can be presented directly at a GDG meeting (either in person or virtually). An example could be to aid interpretation of legislation or regulations.

8.9.4 Using formal consensus methods outside the Guidance Development Group

Exceptionally, if the literature search has found no evidence that addresses the review question, the GDG may identify best practice by using formal consensus methods (for example, the Delphi technique or the nominal-group technique). Any formal consensus method used is documented in the GPG.

[3] Where NICE guidance information services are referred to, an externally commissioned information service may be used to provide support in this area in some cases.