Tools and resources

Appendix B: Approaches to additional consultation

An additional consultation for a guideline is considered only on an exceptional basis and is additional to the routine stakeholder consultations (see section 10.1 of the manual). Additional consultation is a targeted engagement exercise to obtain a range of views and experiences, independent from the committee, either to inform the evidence and draft recommendations, or to test the feasibility of implementing the draft recommendations or their relevance and acceptability to those affected by the guideline. This appendix outlines approaches that could be used when additional consultation is needed involving specific groups of professionals or people using services and carers. Additional consultation may be conducted during guideline development or at the same time as the public consultation on the draft guideline.

Points to consider include:

  • deciding whether additional consultation is needed

  • aim of additional consultation

  • commissioning process

  • obtaining ethical approval

  • the proposal and time frame of the additional consultation

  • reporting findings.

This appendix also describes how findings from an additional consultation are used to finalise the recommendations.

Deciding whether additional consultation is needed

Reasons for additional consultation will vary depending on the topic, and may become apparent at different stages of guideline development. They might include a new area for NICE guidelines during update, a lack of evidence on the views and experiences of people using services, or concerns raised by key stakeholders.

Sometimes health and social care inequalities or impacts on equality are a particular concern, for example, people affected by the guideline find it difficult to engage with health and social care services.

Sometimes a particularly complex topic needs a whole system approach. Configuration of services may be central to the efficacy of a set of recommendations and input from a particular group of health and social care practitioners may be needed.

Occasionally a guideline includes an area of rapidly changing practice, with publication of evidence lagging behind change. It may be necessary to test the draft recommendations with frontline practitioners, or providers or commissioners of services.

In some exceptional cases, the developer may commission an additional consultation with people affected by the guideline to obtain:

  • their views on specific aspects of the guideline, review questions or issues raised by the committee

  • their views and experiences of relevant health and social care services.

The developer may also wish to commission an additional consultation with people affected by the guideline to test the relevance and acceptability of selected draft recommendations. This may be undertaken at the stakeholder consultation stage (see also section 10.1 of the manual), or earlier in the process to validate emerging draft recommendations.

Examples of how guidelines have used the methods described above include:

  • Due to limited evidence and in the absence of representative views from the committee, young people with life-limiting and life-threatening conditions were asked for their views and opinions on selected review questions, including their preferences for place of care, information and communication provision, personalised care planning, and psychological care (Report, appendix L, NICE guideline on end of life care for infants, children and young people with life-limiting conditions).

  • In the absence of evidence, the developer worked with Alder Hey Children's Hospital to survey children about their views and experiences of sedation for diagnostic and therapeutic procedures. Trust staff obtained real-time feedback via hand-held touch screen computers which young children can use (chapter 7, full guideline on sedation in children and young people).

  • In the absence of representative user experiences, people in prison were consulted on their experiences of prison health services to help refine draft recommendations. The developer commissioned User Voice to conduct focus groups with a range of serving prisoners, including people with disabilities, women, older people, long- and short-term prisoners, and those with a history of substance misuse (appendix V, NICE guideline on physical health of people in prison).

  • Children and young people on the autistic spectrum were consulted on emerging draft recommendations (developed from a qualitative literature review) for improving access to and experience of care. The purpose was to validate findings where appropriate and to allow feedback on areas in which the children and young people felt that the qualitative literature was either not representative of their views or that evidence was missing (chapter 4, section 5.2, full guideline on management of autism in children and young people).

  • In the absence of conclusive evidence, healthcare professionals working in neonatology and midwives across the country were consulted on the consensus bilirubin thresholds for managing babies 38 weeks or more gestational age with hyperbilirubinaemia (addendum to NICE guideline on jaundice in newborn babies under 28 days). The additional consultation was conducted during the development of the guideline before public consultation. The aim of the additional consultation was to seek validation from healthcare professionals and midwives on the consensus bilirubin thresholds for managing babies 38 weeks or more gestational age with hyperbilirubinaemia before wider public consultation.

  • Due to a lack of published evidence, additional consultation with adult and paediatric neurologists, general practitioners and other healthcare professionals was conducted during guideline development to run a 1-round modified Delphi to gain consensus on signs and symptoms associated with suspected neurological conditions presented in primary care (NGXX Suspected neurological conditions, publication date: to be confirmed).

Aim of additional consultation

The aim of an additional consultation must be clearly stated in the proposal for NICE as well as in the guideline methods. The aim could include, for example:

  • obtaining expert view or opinions, or testing the feasibility of recommendations with policy makers, commissioners, health and social care providers and practitioners

  • identifying barriers and facilitators to implementing recommendations with policy makers, commissioners, health and social care providers and practitioners

  • obtaining users' views and experience of health and social care services to fill evidence gaps

  • obtaining users', and their families' or carers', experience and views to fine-tune the recommendations.

These are just a few examples. Developers should consult NICE staff with responsibility for quality assurance for initial discussion as soon as the need for additional consultation is identified. If the work is likely to involve people using services or their carers, the developer should also discuss their plans with NICE public involvement staff, who can advise on options and methods for involving people affected by the guideline, including targeted consultation to obtain their views. They can also signpost to external resources and sources of more specialist advice.

Agreeing who should be commissioned to do the work

Once the aim of additional consultation is agreed, the developer should then discuss the commissioning process with NICE staff with responsibility for quality assurance. Additional consultation may be conducted by the developer or by an external contractor.

When the decision is made to commission an external contractor, the developer and NICE should consider an academic or research organisation, or an organisation that works with people affected by the guideline and has research expertise. This organisation should be separate from the team involved in compiling evidence reviews for the guideline and the committee, unless there are exceptional circumstances. For example, specific expertise in the topic or access to specialist networks is needed. However, the team may be asked to help the contractor, for example, by generating a list of participants.

The contractor should have a good record of qualitative or participatory research and, ideally, should have experience in the topic area, as well as expertise in working with people affected by the guideline.

The developer should document the reasons for the additional consultation, with a proposal including the methods to be used, and the anticipated time and costs. The proposal should be discussed with members of NICE staff with a quality assurance role, and approved by the centre director. If the work is approved, the reasons and methods should be documented in the guideline.

If an external contractor is commissioned, the commissioning process should follow NICE's Standing Financial Instructions. This involves developing a project specification, issuing invitations to tender and selecting a contractor based on clear and auditable criteria.

Obtaining ethical approval

In principle, additional consultation falls into the category of 'service evaluation' and so is outside the remit of NHS research ethics committees. However, NICE, the developer and external contractor (if commissioned) should consider the ethical issues each time an additional consultation is planned to ensure appropriate expertise, and that policies and procedures are in place for the safety and welfare of participants. If there is any doubt, the developer or external contractor should consult the national Research Ethics Service. The developer or external contractor (if commissioned) is responsible for seeking ethical approval, if required.

For topics covering children and young people, NICE's patient and public involvement policy includes a set of principles for involving them and has an appendix about safeguarding. The national Research Ethics Service should also be consulted for topics covering children and young people and other vulnerable groups such as adults with learning disabilities or frail older people.

The proposal

The proposal for the additional consultation should include information on the:

  • aim and objectives

  • recruiting participants

  • methods used

  • timeframe of the additional consultation

  • analysis of data

  • feedback mechanism.

The proposal and the final report of the additional consultation should be included as part of the guideline or guideline appendices.

The developer or the external contractor (if commissioned) should agree with NICE the approaches and methods to use, including a summary of the issues to be covered. Similarly, the methodology and any questions or support materials used must be developed and agreed with NICE. For example, NICE should:

  • be briefed by the developer or external contractor (if commissioned) in detail before work begins

  • agree final documents and comment on draft recruitment letters

  • help develop topic guides (for example, summaries of the recommendations and key questions for discussion)

  • agree sampling frames and samples, and other supporting materials

  • discuss how to get participants from key groups involved, including people who work with or are from seldom heard groups or those who share characteristics protected under equality legislation

  • have access to transcripts of all data

  • discuss and agree techniques for data analysis and themes for data presentation

  • comment on the additional consultation report before the final draft is submitted.

Aim and objectives

The aim of the additional consultation should be clearly stated in the proposal. The proposal should also state the expected outputs, for example, the final report may summarise themes from participants' views, which would be used to inform or fine-tune the final recommendations.

Recruiting participants

The developer and external contractor (if commissioned) should consider the recruitment strategy carefully, taking into account the purpose of the additional consultation, the topic, the groups, the range of views required, and other relevant issues.

If the purpose of the consultation is to test the feasibility of implementing recommendations, participants should be chosen to represent a broad range of stakeholder groups in the statutory, non-statutory and voluntary sectors, where applicable. This may include people who work with the target populations covered by the guideline and other users of the guideline, such as health and social care practitioners, commissioners, policy makers, people using services, and if appropriate their families or carers. Participants do not have to be from an organisation that is registered as a NICE stakeholder.

When planning an additional consultation with children and young people, school holidays and exam schedules should be taken into account.

Equality issues should be fully considered when choosing participants. This may mean getting a representative spread of practitioners or people using services, but it may also mean focusing on participants from seldom heard groups or people with recent experience of working with them. When testing the feasibility of implementing recommendations, the approach should be based on the content of the draft recommendations, whether or not they refer to the whole population or subgroups, and service delivery and policy issues.

Different sampling methods may be used to recruit participants. Sampling should be guided by the topic and will depend on the:

  • stakeholder groups identified as being responsible for taking action

  • the make-up of the population affected by the guideline

  • scope

  • research questions

  • inclusion criteria for the evidence reviews.

'Snowballing' (gathering participants via other participants or networks) and purposive or other non-random techniques may be used to ensure all relevant groups are represented.

Random sampling (randomly selecting participants from the relevant groups) or quota sampling (selecting a fixed number of participants, randomly or purposively from these groups) may be useful for large-scale surveys. Random and quota sampling may also be useful where there are a large number of potential participants, but there are not enough of them in each relevant geographical area.

The proposal should explicitly state the groups of participants to be recruited, the recruitment strategy, including sampling method, the number of participants to be recruited, considerations of consent, confidentiality and data protection. The developer or external contractor (if commissioned) should ensure the sampling frame and sample take account of equality issues. It should be agreed with NICE.

Methods used

Additional consultation is a targeted engagement exercise to obtain a range of views and experiences either to inform the evidence and draft recommendations, or to test the feasibility of implementing the draft recommendations or their relevance and acceptability to those affected by the guideline. Additional consultation can involve a number of approaches and methods. NICE, the relevant committee and the developer or external contractor should consider the choice of methods carefully, taking into account the topic, the groups involved and other issues. When involving people affected by the guideline, the methods and materials used should be tailored to the age, ability and culture of participants. Additional consultation may include the use of groups, 1-to-1 or paired in-depth interviews or surveys. In some cases – for example, if a range of groups are involved – a combination of approaches may be used.

Group-based methods

Group-based methods include focus groups, participative workshops and 'virtual' (electronic) groups. These may be appropriate when:

  • potential participants have clear 'professional identities' and the 'field' is well established

  • the developer (with support from NICE) can contact enough people in a geographical region to set up a focus group or workshop

  • the issues discussed are unlikely to be confidential or sensitive and anonymity will not be necessary.

The developer or external contractor (if commissioned) may also want to consider the following:

  • more than 1 participative workshop or focus group or 'virtual' (electronic) group could be convened; these should take place in more than 1 geographical region and will normally be a half day but may take up to a day; if it is not feasible to organise this many workshops or groups, the decision on how many should be convened must be agreed with NICE

  • if it suits the needs of the project, separate participative workshop or focus group or 'virtual' (electronic) group can be arranged for different practitioner or user groups; this will depend on the number of participants and should be agreed with NICE

  • for some topic areas, researchers may be included in the additional consultation; in such cases, a separate meeting should be convened for them, using the same processes; this should be agreed with NICE

  • topic guides, prompts or supporting materials (such as the draft recommendations and the key areas of concern) must be developed in collaboration with, and agreed by, NICE

  • if the purpose of the additional consultation is to test the feasibility of implementing guideline recommendations, a member of the NICE field team should attend at least 1 meeting.

1-to-1 or paired in-depth interviews

Interviews may be carried out face-to-face, by telephone or online. They may be appropriate when:

  • it is not possible to get groups together because the topic is a relatively new area, the number of possible participants is limited or there are geographical or time constraints

  • the issues discussed are likely to be confidential or sensitive and anonymity may be needed

  • in-depth responses are needed.

Interviews may be structured or semi-structured, depending on the topic and the groups involved. Semi-structured interviews allow complex or difficult issues to be explored and so are likely to be more useful than a fixed-format interview. They should focus on, for example, areas in which views and experiences are needed, or the draft recommendations.

Individual or paired interviews are usually more expensive to set up than group work, and the need for in-depth or individual contact should be weighed against the available resources at the planning stage.

Surveys

Group-based methods and 1-to-1 or paired interviews are the best way to find out opinions. But they are not suitable in all circumstances, for example, because of the sensitivity of the topic, confidentiality issues, or difficulties in recruiting participants. In such cases, surveys that use semi-structured and open-ended questions could be more appropriate. Surveys may be carried out by telephone, online, on paper or by using vote casting or polling.

Surveys gather opinions in a quick and less obtrusive manner than group-based approaches and interviews. The responses can also be quantified. But surveys do not allow the same depth of exploration and, generally, should only be used if other methods are unsuitable. Formal consensus methods such as Delphi survey and RAND appropriateness could be modified for the survey if appropriate.

Analysis of data

There are different ways of analysing data from additional consultation, depending on the methods used for data collection. Some descriptive summary statistics should be provided, for example, characteristics of participants and attendance or response rates.

Group-based methods and interviews are likely to generate qualitative data. Analysis may be performed using qualitative research software, or by hand, but the method should be fully reported in the proposal and the final report.

Qualitative data can be broken down into common and consistent themes for each of the questions asked, using, for example, a content analysis approach. Usually, 1 researcher should prepare an initial analysis, which should be verified by 'blind' coding and sorting of a sample of the transcript by a second researcher. For examples of this kind of analysis, see part 3 (chapters 7 to 13) of Silverman (2004) or Ritchie and Spencer (1993).

Once the analysis is complete, participants' quotes may be selected to illustrate each theme. These quotes should be coded to keep participants anonymous and to allow the quotes to be distinguished. Where transcripts are processed, ensure confidentiality and data protection are fully considered. As with data from clinical trials, transcripts should be kept for at least 5 years (see www.ct-toolkit.ac.uk).

Surveys are likely to involve a mixture of quantitative and qualitative data. Quantitative data may be analysed and presented using summary statistics. These could be generated using various statistical software or calculators. Where informal consensus methods such as Delphi survey and RAND appropriateness have been modified for the survey, specific analytical methods, for example, thresholds for agreement, should be stated in the proposal and the final report.

The developer or external contractor (if commissioned) should ensure the methods for analysing the data are discussed and agreed with NICE.

Feedback mechanism

The developer should ensure that all participants receive feedback on their contribution or the findings of the consultation and how this information has been used. For commissioned work, the external contractor should agree with the developer a process for giving feedback to all participants. Providing feedback to participants should be specified in contracts. This may include an evaluation exercise, a follow-up session or sharing interim findings via email.

Reporting and using the findings

The final report of the additional consultation should follow the same structure as the proposal. It should include sections on aim and objectives, recruiting participants, methods used, analysis of data and all the findings from the additional consultation.

These findings should be used to inform the guideline recommendations. The developer may present a summary of all the findings to the committee, and the committee should use this information to refine and prioritise the recommendations before or after the public consultation, depending on when the additional consultation is conducted. How the summary findings are used to inform committee's decision-making should be documented in the committee's discussion of the evidence.

Both the proposal and the final report of the additional consultation should be available as appendices on publication of the guideline.

Further information

References and further reading

Green J, Thorogood N (2004) Qualitative methods for health research: qualitative methods for health. London: Sage

Kelly MP, Chambers J, Huntley J et al. (2004) Method 1 for the production of effective action briefings and related materials. London: Health Development Agency

Lightfoot J, Sloper P (2002) Having a say in health: guidelines for involving young patients in health services development. University of York: Social Policy Research Unit

Pope C, Ziebland S, Mays N (2000) Qualitative research in health care: analysing qualitative data. BMJ 320: 114–6

Ritchie J, Spencer L (1993) Qualitative data analysis for applied policy research. In: Bryman A, Burgess R, editors. Analysing qualitative data. London: Routledge pp 173–94

Shaw C, Brady L-M, Davey C (2011) Guidelines for research with children and young people. London: National Children's Bureau

Silverman D, editor (2004) Doing qualitative research: a practical handbook. London: Sage

Tashakkori A, Teddlie C (2002) Handbook of mixed methods in social and behavioural research. London: Sage


This page was last updated: 31 October 2018