NICE process and methods

10 The validation process for draft guidelines, and dealing with stakeholder comments

Consultation with stakeholders is an integral part of the guideline development process. Comments received from registered stakeholders are a vital part of the quality-assurance and peer-review processes, and it is important that they are addressed appropriately. Registered stakeholders are notified of the consultation dates in advance via the guideline page on the NICE website, and are reminded by email. (See NICE's webpage on registering as a stakeholder for more information.)

This chapter describes the validation process for draft guidelines. It includes information on what happens during the consultation, the principles of responding to stakeholder comments after the consultation and when a second consultation may be needed.

Before the draft guideline is signed off for consultation, an equality impact assessment is completed by the developer and the committee chair to show which equality issues have been identified and considered during guideline development. The equality impact assessment is signed off by a member of NICE staff with responsibly for quality assurance, and published on the NICE website with the draft guideline. The assessment is updated by the developer and the committee chair after the consultation.

10.1 What happens during consultation

Commenting on the draft guideline

The draft version of the guideline (recommendations, rationales, committee discussions, evidence reviews and methods) is posted on the NICE website for consultation with registered stakeholders. Stakeholders can register at any point during guideline development. NICE informs registered stakeholders that the draft is available, via email and through its promotional channels, and invites them to comment by the deadline. Questions for stakeholders are posted with the draft guideline. The purpose of these questions is to seek stakeholder views on factors such as the potential equality impact. NICE also asks stakeholders to comment on recommendations identified as likely to substantially increase costs, and their justification, and to consider whether any other draft recommendations are expected to add substantial costs. Questions related to implementation may also be included to identify practitioners who are already implementing the draft recommendations, or resources that could be fed into the NICE endorsement scheme.

The length of time for consultation depends on the size of the guideline and the number of review questions. Consultation on a new guideline or full update consisting of 15 to 20 review questions usually lasts for 6 weeks. A 4‑week consultation may be used for partial updates of guidelines with less than 15 review questions, while small updates with 1 or 2 review questions will normally have a 2-week consultation. NICE staff with responsibility for quality assurance will decide how long the consultation will last, and stakeholders will be told well in advance.

NICE is unable to accept:

  • more than 1 set of comments from each registered stakeholder organisation

  • comments that are not presented correctly on the form provided

  • comments with attachments such as research articles, letters or leaflets.

In these cases, NICE will invite a registered stakeholder to resubmit a single set of comments with no attachments before the consultation deadline. NICE is unable to accept any comments received after the deadline.

Comments should be constructed as reasoned argument and be submitted for the purpose of improving the draft guideline. NICE reserves the right not to respond to comments that are hostile or inappropriate.

Stakeholders should make sure that any confidential information or information that the owner would not wish to be made public is clearly underlined and highlighted. Confidential information should be kept to a minimum. Stakeholders should explain why the information is confidential and if and when it will become publicly available.

Where views on the guideline are shared by more than 1 stakeholder organisation, NICE encourages these organisations to work together to produce a joint response. This should be submitted by 1 registered stakeholder; other stakeholders supporting the joint response should respond to the consultation noting their endorsement.

When registering, and when commenting on the draft scope and draft guideline, stakeholders are asked to disclose whether their organisation has any direct or indirect links to, or receives or has ever received funding from, the tobacco industry. Disclosures will be included with the published consultation responses. Tobacco companies and those who speak for them or are funded by them (collectively referred to as 'tobacco organisations') cannot register as stakeholders. This is in line with NICE's obligation under Article 5.3 of the WHO Framework Convention on Tobacco Control (FCTC) to protect public health policies from the commercial and other vested interests of the tobacco industry. Tobacco organisations are simply referred to as 'respondents' and any comments received during consultation are reviewed for factual inaccuracy claims and are made public along with any responses.

Approaches to additional consultation and commissioned primary research

Additional consultation is a targeted engagement exercise to obtain a range of views, experiences and expertise, independent from the committee. An additional consultation for a guideline is considered only on an exceptional basis and is additional to the routine stakeholder consultations.

Commissioned primary research can occasionally be useful for addressing topic areas where there is limited or no evidence. Commissioned primary research should start during the development of the guideline, and the findings should be used by the committee as part of the evidence base for developing recommendations.

For more information, see the appendix on approaches to additional consultation and commissioned primary research.

External expert peer review

Although NICE does not routinely commission peer review from external experts, members of NICE staff with responsibility for quality assurance, or the developer, may occasionally consider arranging additional external expert peer review of part or all of a guideline, or an evidence review, executable model or economic analysis. For example, review by external experts may be valuable if novel methods have been used in developing an evidence review.

External expert peer reviewers may include practitioners, those commissioning care, academics (for example, with expertise in economic or meta-analysis), or people with a lay perspective. Experts are selected on the basis of their experience in the particular issue under review.

External expert peer review may take place during guideline development or during consultation on the draft guideline. If it occurs during development the comments are not published, but the reviewers should be named in the guideline. Comments from external expert peer reviewers during the development of the guideline should be discussed by the committee, and their decisions fully documented in the committee discussion section of the relevant evidence review. If the reviewers also comment during consultation, their comments are responded to in the same way as comments from registered stakeholders and are published in the guideline consultation table on the NICE website under 'external expert peer reviewers'. All external expert peer reviewers are required to complete a declaration of interests form (see the section on code of conduct and declaration of interests in the chapter on decision-making committees).

10.2 Principles of responding to stakeholder comments

After consultation the committee discusses the comments received during consultation, proposes any changes needed to the guideline, and agrees the final wording of the recommendations.

This section describes how developers should respond to consultation comments. The same principles apply when responding to comments on the draft scope (see the chapter on the scope).

Developers must take the following key points into account when responding to comments from registered stakeholders:

  • Each comment must be acknowledged and answered as directly, fully and with as much information as possible.

  • For a draft guideline, the committee must consider whether changes to the guideline are needed as a result of consultation comments; any changes to the guideline must be agreed by the committee before publication.

  • If changes are made to a guideline as a result of a consultation comment, this must be made clear in the response to the comment. If no changes have been made, it should be clear from the response why not.

  • Developers should maintain an audit trail of any changes made to the guideline.

Registered stakeholders who have commented on the draft guideline may be sent the final guideline, and comments and responses, in confidence before publication (see the chapter on finalising and publishing the guideline). Comments and responses are made available on the NICE website when the final guideline is published.

NICE reserves the right to summarise and edit comments received during consultations, or not to publish them at all, if we consider the comments are too long, or publication would be unlawful or otherwise inappropriate.

Comments received from non-registered stakeholders and individuals are reviewed to decide whether they have raised any key issues that the committee need to consider. A formal response is not given and these comments are not made available on the NICE website. Comments received from 'respondents' are reviewed for factual inaccuracy claims and are made public along with any responses.

Comments received after the deadline are not considered and are not responded to; in such cases the sender will be informed.

When evidence is highlighted by stakeholders during consultation, this should be considered for inclusion in the guideline. The developer will take the evidence into account:

  • if it meets all of the inclusion criteria for the relevant review (as set out in the review protocol), and should have been identified in the guideline searches/screening

  • if it falls within the timeframe for the guideline search parameters.

Any effects on the guideline of including new evidence will be considered, and any further action agreed between the developer and NICE staff with responsibility for quality assurance.

If the new evidence falls outside of the timeframe for the guideline searches, the impact on the guideline will still need to be considered, and any further action agreed between the developer and NICE staff with responsibility for quality assurance.

10.3 When a second consultation may be needed

In exceptional circumstances, NICE may consider the need for a further stakeholder consultation after the first consultation. This additional consultation may be needed if either:

  • information or data that would significantly alter the guideline were omitted from the first draft or

  • evidence was misinterpreted in the first draft and the amended interpretation significantly alters the draft recommendations.

NICE staff with responsibility for guideline quality assurance make the final decision on whether to hold a second consultation, how long it should be.