Shared learning database

NHS South of Tyne and Wear
Published date:
January 2012

Development and testing of review criteria for assessing compliance with NICE guidelines in primary care, using osteoporosis and depression as examples.

Guidance the shared learning relates to:
Does the example relate to a general implementation of all NICE guidance?
Does the example relate to a specific implementation of a specific piece of NICE guidance?


Aims and objectives

To develop and field test review criteria to measure adherence to selected NICE guidelines in primary care.
1. To use a stakeholder consensus process in the development of valid review criteria to measure adherence to key recommendations from two NICE guidelines.
2. To evaluate the feasibility and reliability of applying the review criteria in measuring clinical performance and examining variations in practice.
3. To assess the utility of the information assembled to stakeholders in directing improvement work and identify the skills and resources needed to replicate the data collection elsewhere.

Reasons for implementing your project

There is an obligation on NHS commissioners to ensure compliance with NICE guidelines, and all PCTs have an assurance system in place to discharge this responsibility. This obligation is become more significant, with the requirement to comply with NHS Quality Standards, many of which are based on NICE guidelines.

The assurance system in NHS South of Tyne and Wear (NHS STW) for primary care compliance with NICE guidelines is based on descriptions of practice rather than analysis of care of individual patients. Our working assumption was that the assurance processes for primary care in NHS STW were typical of commissioning organisations, but wanted to explore how greater assurance about standards of care could be obtained.

How did you implement the project

Phase 1: Development of review criteria
The review criteria were developed by the research team from national and local guidelines. Consensus panels of 12 people comprising local GPs, primary care health workers, trust representatives and patient representatives used a RAND consensus process to select a group of quality standard measures (33 measures for osteoporosis and 31 measures for depression) considered to be both clinically important and important to document in medical records.

Phase 2: Measurement and validation of review criteria in general practice records
Practice recruitment
- 20 practices were successfully recruited to the study: Sunderland (9); South Tyneside (6) and Gateshead (5).
Patient recruitment:
Using MIQUEST searches the practices identified lists of eligible patients and letters of invitation were sent.
160 patients were recruited for the osteoporosis and 150 patients for the depression study. Assessing utility of chosen indicators.

Each indicator was rated for clinical importance, Validity, Feasibility of data collection, relevance to primary care, the frequency with which data were recorded, and ease of collecting were all assessed, giving a total 'quality and feasibility' score out of a total maximum of 19.

Key findings

Data was collected for 29 osteoporosis and 28 depression review criteria. Key findings for osteoporosis were: 50% of patients had a DEXA scan, 30% did not have secondary causes excluded and recording of parental hip fractures was low. The majority of patients with a fragility fracture were treated first line with alendronate with Ca/Vit D. 35% of patients who did not meet the criteria for treatment were prescribed alendronate. Falls data was poorly recorded. Key findings for depression were: 24% of patients had a comprehensive assessment and over 70% had a suicide risk assessment. Over 70% of severe depression was treated with a combination of antidepressants and psychological interventions and 50% of chronic continuous depression has > 2 years antidepressant treatment. Mild depression also had a high rate of use of antidepressants and 27% of patients with new or episodic depression received less than 6 months antidepressants. The majority of patients had an SSRI first line and an alternative SSRI or new generation antidepressant second line. The average number of consultations was 3 in the first 3 months and the majority saw only 1 or 2 GPs in the first 6 months of care.

An important limitation in determining standards of depression care was in determining severity (which often had to be imputed) and classifying episodes of care. The guidance relies heavily on defining distinct episodes of depression, which we found difficult to map on to many cases we selected. Many of the indicators with high 'quality and feasibility' scores showed good clinical performance. However, significant omissions in data impaired assessment. For example, in the management of osteoporosis, poor recording of fragility fractures reduced performance against the standard on risk registers. In depression management, there was little documentation of formal suicide risk assessment or referral to crisis intervention teams, which again limited performance.

Key learning points

It is possible to improve the ways in which standards of primary care in relation to NICE guidelines are assessed.

Deficiencies in documentation of care can make assessment of adherence to quality standards difficult. In management of depression, the scenarios described in the guidance are difficult to map on to typical care in general practice.

Assessment of many quality criteria requires clinical input, greatly increasing the resource implications of undertaking these assessments.

Assessing feasibility and validity of quality assessment criteria is a helpful exercise for determining compliance, and will be useful for determining compliance with NHS Quality Standards.

Contact details

Dr Mark Lambert
Consultant in Public Health Medicine
NHS South of Tyne and Wear

Invalid Form Data
Is the example industry-sponsored in any way?