Process and methods

10 The validation process for draft guidelines, and dealing with stakeholder comments

Consultation with stakeholders is an integral part of the guideline development process. Comments received from registered stakeholders are a vital part of the quality-assurance and peer-review processes, and it is important that they are addressed appropriately. Registered stakeholders are notified of the consultation dates in advance via the guideline page on the NICE website, and are reminded by email. (See NICE's webpage on registering as a stakeholder for more information.)

This chapter describes the validation process for draft guidelines. It includes information on what happens during the consultation, the principles of responding to stakeholder comments after the consultation and when a second consultation may be needed.

Before the draft guideline is signed off for consultation, an equality impact assessment is completed by the developer and the committee chair to show which equality issues have been identified and considered during guideline development. The equality impact assessment is signed off by a member of NICE staff with responsibly for quality assurance, and published on the NICE website with the draft guideline. The assessment is updated by the developer and the committee chair after the consultation.

10.1 What happens during consultation

Commenting on the draft guideline

The draft version of the guideline (recommendations, rationales, committee discussions, evidence reviews and methods) is posted on the NICE website for consultation with registered stakeholders. Stakeholders can register at any point during guideline development. NICE informs registered stakeholders that the draft is available, via email and through its promotional channels, and invites them to comment by the deadline. Questions for stakeholders are posted with the draft guideline. The purpose of these questions is to seek stakeholder views on factors such as the potential equality impact. NICE also asks stakeholders to comment on recommendations identified as likely to substantially increase costs, and their justification, and to consider whether any other draft recommendations are expected to add substantial costs. Questions related to implementation may also be included to identify practitioners who are already implementing the draft recommendations, or resources that could be fed into the NICE endorsement scheme.

Consultation usually lasts for 6 weeks. A 4‑week consultation may be used for partial updates of guidelines or small guidelines (for example, guidelines on systems and processes that relate to the use of medicines in different care settings and within provider and commissioning organisations).

NICE is unable to accept:

  • more than 1 set of comments from each registered stakeholder organisation

  • comments that are not presented correctly on the form provided

  • comments with attachments such as research articles, letters or leaflets.

In these cases, NICE will invite a registered stakeholder to resubmit a single set of comments with no attachments before the consultation deadline. NICE is unable to accept any comments received after the deadline.

Comments should be constructed as reasoned argument and be submitted for the purpose of improving the draft guideline. NICE reserves the right not to respond to comments that are hostile or inappropriate.

Stakeholders should make sure that any confidential information or information that the owner would not wish to be made public is clearly underlined and highlighted. Confidential information should be kept to a minimum. Stakeholders should explain why the information is confidential and if and when it will become publicly available.

Where views on the guideline are shared by more than 1 stakeholder organisation, NICE encourages these organisations to work together to produce a joint response. This should be submitted by 1 registered stakeholder; other stakeholders supporting the joint response should respond to the consultation noting their endorsement.

When registering, and when commenting on the draft scope and draft guideline, stakeholders are asked to disclose whether their organisation has any direct or indirect links to, or receives or has ever received funding from, the tobacco industry. Disclosures will be included with the published consultation responses. Tobacco companies and those who speak for them or are funded by them (collectively referred to as 'tobacco organisations') cannot register as stakeholders. This is in line with NICE's obligation under Article 5.3 of the WHO Framework Convention on Tobacco Control (FCTC) to protect public health policies from the commercial and other vested interests of the tobacco industry. Tobacco organisations are simply referred to as 'respondents' and any comments received during consultation are reviewed for factual inaccuracy claims and are made public along with any responses.

Approaches to additional consultation

When a draft guideline on novel, complex or sensitive areas is issued for consultation the developer may, in exceptional cases, decide that the feasibility of the draft recommendations should also be tested. This may be done in fieldwork with people providing services, or a targeted consultation with people using services.

For example, it may be important to explore how easy it will be for policy makers, commissioners, practitioners and professionals to implement the draft recommendations and how the recommendations might work in practice.

It may also be important to test draft recommendations directly with people using services, and where appropriate their families or carers.

Sometimes additional consultation will be considered because there is a lack of evidence on the views and experiences of people affected by the guideline, or NICE's standard processes need to be adapted or supplemented to incorporate user perspectives. Examples of where this could occur include guidelines covering children and young people, or people with a learning disability or cognitive impairment (see the section on forming the committee in the chapter on decision-making committees and the appendix on approaches to additional consultation for more information).

The main criteria for considering additional consultation are:

  • the topic covers novel or sensitive areas or

  • the evidence, such as evidence on users' views is lacking or

  • people affected by the guideline are not participants in the guideline's decision-making and consultation processes (for example, children).

The developer should document the reasons for the additional consultation, with a proposal including consideration of the methods to be used, and the anticipated costs. The proposal should be discussed with members of NICE staff with a quality assurance role, and approved by the centre director. If the work is approved, the reasons and methods should be documented in the guideline.

Additional consultation usually happens at the same time as draft guideline consultation, but can be undertaken earlier (for example, to seek views and experiences of people affected by the guideline when there is a lack of evidence, or to validate selected draft recommendations with people using services, policy makers, commissioners, practitioners and professionals before guideline consultation). Occasionally additional consultation may be needed at more than one stage of guideline development.

The results of all consultations are considered by the committee. The committee uses this information to refine and prioritise the recommendations after consultation. This includes making them more specific or targeting them at different groups of practitioners, where appropriate.

Further details on approaches to additional consultation are given in the appendix on approaches to additional consultation.

External expert review

Although NICE does not routinely commission peer review from external experts, members of NICE staff with a quality assurance role, or the developer, may occasionally consider arranging additional external expert review of part or all of a guideline, or an evidence review, executable model or economic analysis. For example, review by external experts may be valuable if novel methods have been used in developing an evidence review.

External expert reviewers may include practitioners, those commissioning care, academics (for example, with expertise in economic or meta-analysis), or people with a lay perspective. Experts are selected on the basis of their experience in the particular issue under review.

External expert review may take place during guideline development or during consultation on the draft guideline. If it occurs during development the comments are not published, but the reviewer(s) should be named in the guideline. Comments from external expert reviewers during the development of the guideline should be discussed by the committee. If the reviewers also comment during consultation, their comments are responded to in the same way as comments from registered stakeholders and are published in the guideline consultation table on the NICE website under 'external expert reviewers'. All external expert reviewers are required to complete a declaration of interests form (see the section on code of conduct and declaration of interests in the chapter on decision-making committees).

10.2 Principles of responding to stakeholder comments

After consultation the committee discusses the comments received during consultation, proposes any changes needed to the guideline, and agrees the final wording of the recommendations.

This section describes how developers should respond to consultation comments. The same principles apply when responding to comments on the draft scope (see the chapter on the scope).

Developers must take the following key points into account when responding to comments from registered stakeholders:

  • Each comment must be acknowledged and answered as directly, fully and with as much information as possible.

  • For a draft guideline, the committee must consider whether changes to the guideline are needed as a result of consultation comments; any changes to the guideline must be agreed by the committee before publication.

  • If changes are made to a guideline as a result of a consultation comment, this must be made clear in the response to the comment. If no changes have been made, it should be clear from the response why not.

  • Developers should maintain an audit trail of any changes made to the guideline.

Registered stakeholders who have commented on the draft guideline are sent the final guideline, and comments and responses, in confidence 2 weeks before publication (see the chapter on finalising and publishing the guideline). Comments and responses are made available on the NICE website when the final guideline is published.

NICE reserves the right to summarise and edit comments received during consultations, or not to publish them at all, if we consider the comments are too long, or publication would be unlawful or otherwise inappropriate.

Comments received from non-registered stakeholders and individuals are reviewed by the committee. A formal response is not given and these comments are not made available on the NICE website. Comments received from 'respondents' are reviewed for factual inaccuracy claims and are made public along with any responses.

Comments received after the deadline are not considered and are not responded to; in such cases the sender will be informed.

When evidence is highlighted by stakeholders during consultation, this should be considered for inclusion in the guideline. The developer will take the evidence into account:

  • if it meets all of the inclusion criteria for the relevant review (as set out in the review protocol), and should have been identified in the guideline searches/screening

  • if it falls within the timeframe for the guideline search parameters.

Any effects on the guideline of including new evidence will be considered, and any further action agreed between the developer and NICE staff with a quality assurance role.

If the new evidence falls outside of the timeframe for the guideline searches, the impact on the guideline will still need to be considered, and any further action agreed between the developer and NICE staff with a quality assurance role.

10.3 When a second consultation may be needed

In exceptional circumstances, NICE may consider the need for a further 4‑week stakeholder consultation after the first consultation. This additional consultation may be needed if either:

  • information or data that would significantly alter the guideline were omitted from the first draft or

  • evidence was misinterpreted in the first draft and the amended interpretation significantly alters the draft recommendations.

NICE staff with responsibility for guideline quality assurance make the final decision on whether to hold a second consultation.