Appendix C The evidence

This appendix lists the evidence statements from four evidence reviews and commissioned research provided by external contractors (see appendix A and appendix E) and links them to the relevant recommendations. See appendix B for the meaning of the (++), (+) and (−) quality assessments referred to in the evidence statements.

The additional evidence section of this appendix also lists 19 expert papers and their links to the recommendations and sets out a brief summary of findings from the economic modelling.

The evidence statements are short summaries of evidence in a review, report or paper (provided by an expert in the topic area). Each statement has a short code indicating which document the evidence has come from. The letter(s) in the code refer to the type of document the statement is from, and the numbers refer to the document number, and the number of the evidence statement in the document.

Evidence statement number 1.1 indicates that the linked statement is numbered 1 in the review 'Identifying the key elements and interactions of a whole system approach to obesity prevention'. Evidence statement 2.1 indicates that the linked statement is numbered 1 in the review 'The effectiveness of whole system approaches to prevent obesity'. Evidence statement 3.1 indicates that the linked statement is numbered 1 in the review 'Barriers and facilitators to effective whole system approaches'. Evidence statement 4.1 indicates that the linked statement is numbered 1 in the review 'Whole system approaches to obesity prevention: review of cost-effectiveness evidence'. Evidence statement CR1 indicates that the linked statement in numbered 1 in the commissioned report 'Implementing community-wide action to prevent obesity: opinions and experiences of local public health teams and other relevant parties'.

See the full reviews, commissioned research, expert papers and economic modelling report. Where a recommendation is not directly taken from the evidence statements, but is inferred from the evidence, this is indicated by IDE (inference derived from the evidence).

Where the Programme Development Group (PDG) has considered other evidence, it is linked to the appropriate recommendation below. It is also listed in the additional evidence section of this appendix.

Recommendation 1: evidence statements 1.6, 2.5, 3.1, 3.2, 3.5, CR1; expert papers 2, 3, 5, 6, 7, 9, 12, 14

Recommendation 2: evidence statements 1.2, 1.6, 3.1, 3.2, 3.4, 3.7, 3.8, CR1, CR3; expert papers 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 18

Recommendation 3: evidence statements 1.6, 3.4, 3.5, 3.7, CR1, CR5; expert papers 2, 3, 5, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18

Recommendation 4: evidence statements 1.2, 1.6, 3.2, 3.3, 3.4, 3.5, 3.7, CR2, CR3, CR4; expert papers 1, 4, 5, 6, 7, 8, 9, 10, 12, 14, 16

Recommendation 5: evidence statements 1.6, 3.3, 3.4, CR1, CR2, CR3, CR4; expert papers 2, 3, 5, 6, 8, 9, 11, 12, 14, 15, 16

Recommendation 6: evidence statements 1.2, 1.6, 3.1, 3.2, 3.3, 3.4, 3.7, CR1, CR4, CR5; expert papers 2, 4, 5, 7, 8, 9, 10, 12, 15, 17

Recommendation 7: evidence statements 1.3, 1.4, 1.6, 3.3, 3.6, 3.8, CR3, CR4, CR5; expert papers 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 15, 16, 19

Recommendation 8: evidence statements 1.6, 3.2; expert papers 2, 5, 8, 11, 18; IDE

Recommendation 9: evidence statements 3.2, 3.5, CR3; expert papers 2, 5, 11, 18; IDE

Recommendation 10: evidence statements 1.4, 1.6, 3.6, 3.8, 4.3, CR4, CR5; expert papers 2, 3, 4, 5, 9, 11, 12, 13, 14, 16, 19

Recommendation 11: evidence statements 1.4, 1.6, 3.6, 3.8, 4.3, CR4, CR5; expert papers 2, 3, 4, 5, 9, 11, 12, 13, 14, 16, 19

Recommendation 12: evidence statements 4.3; CR6; expert papers 5, 9, economic modelling report

Recommendation 13: evidence statements 1.6, 3.2, 3.3, 3.6, CR3, CR4; expert papers 2, 5, 7

Recommendation 14: evidence statements 3.7, 3.8, CR1, CR4; IDE

Evidence statements

Please note that the wording of some evidence statements has been altered slightly from those in the evidence review(s) to make them more consistent with each other and NICE's standard house style. The superscript numbers refer to the studies cited beneath each statement. The full references for those studies can be found in the reviews.

Evidence statement 1.1: Whole systems theory

Authors may interpret what is meant by a whole system in different ways; there is a clear division in views between those advocating 'complexity theory' and those discussing a more mechanistic approach.

A whole-system approach to achieving change in organisations, communities or individuals shares conceptual underpinnings with complexity science and complex adaptive systems. Systems continually evolve, with complex outcomes arising from a few simple rules of interaction. Self-regulation occurs within systems, and efforts to contain them may be counterproductive. Systems include formal and informal relationships or networks; these relationships are of great importance. Systems can exist in single or multi-sector organisations.

Evidence statement 1.2: Implications of whole-system theory for ways of working

Whole system theory suggests that organisation or community goals may best be achieved by:

  • Creating more flexible organisational structures.

  • Recognising that relationships are crucial.

  • Understanding how positive and negative feedback loops within a system operate – giving insights into how to increase or sustain positive outcomes.

  • Genuine engagement and discussion about the issues to be addressed – developing shared meaning and purpose – before moving on to 'problem-solving'. This must include a diverse range of actors and community members at all organisational levels.

  • All actors understanding the system in which they operate (and their role within it).

  • Awareness of the divisions between traditional ways of working and whole-system working. The former may involve hierarchical leadership and complex targets and plans while the approach of the latter may be to increase opportunities for natural adaption.

Evidence statement 1.3: Implications of whole-system theory for those working within the system

Individuals participate in their own capacity rather than as a representative of an organisation, community or profession so that they only agree to do what is in their power.

Successful and productive communication within or across organisations may require innovative approaches to break down traditional restrictions stemming from hierarchies and differing expectations of organisations, professions and individuals.

The personal qualities of individuals working within the system may be important. Personal qualities such as optimism, empathy, humility and tenacity may increase the likelihood of success.

A willingness to take the 'long view' rather than go for the 'quick fix' is essential for a systems approach to be effective.

Evidence statement 1.4: Implications of a whole-system approach for evaluation

In a whole-system approach, it is the function rather than the form of activities that is standardised.

The change in behaviour of individuals working within the system, through developing relationships and creating robust networks, is central.

Evaluating a systems approach is complex. Different techniques for evaluation may be required to assess the added benefit of taking a systems approach. Process outcomes and the robustness of the systems are of particular interest (over and above short term outcomes).

Evaluation of a systems approach needs to consider the networks that have been established and the relationships and synergies between and within settings.

Evaluation of a systems approach may be time consuming.

Evidence statement 1.5: Potential challenges of whole-system working

Challenging long-standing assumptions can be uncomfortable. Traditional organisational structures are culturally embedded and change may appear chaotic.

Evidence statement 1.6: The features of a systems approach to tackle health problems

Identifying a system: explicit recognition of the public health system with the interacting, self-regulating and evolving elements of a complex adaptive system. Recognise that a wide range of bodies with no overt interest or objectives referring to public health may have a role in the system and therefore that the boundaries of the system may be broad.

Capacity building: an explicit goal to support communities and organisations within the system. For example, increasing understanding about obesity in the community and by potential partner organisations or training for those in posts directly or indirectly related to obesity.

Creativity and innovation: mechanisms to support and encourage local creativity and/or innovation to address obesity. For example, mechanisms that allow the local community to design locally relevant activities and solutions.

Relationships: methods of working and specific activities to develop and maintain effective relationships within and between organisations. For example, establishing and maintaining relationships with organisations without a health remit or an overt focus on obesity.

Engagement: clear methods to enhance the ability of people, organisations and sectors to engage community members in programme development and delivery. For example, sufficient time in projects allocated to ensuring that the community can be involved in planning and assessing services.

Communication: mechanisms to support communication between actors and organisations within the system. For example, ensuring sufficient face-to-face meeting time for partners, having planned mechanisms for feeding back information about local successes or changes.

Embedded action and policies: practices explicitly set out for obesity prevention within organisations within the system. For example, local strategic commitments to obesity, aligning with wider policies and drivers (such as planning or transport policy) and ensuring obesity is an explicit concern for organisations without a health remit.

Robust and sustainable: clear strategies to resource existing and new projects and staff. For example, contingency planning to manage risks.

Facilitative leadership: strong strategic support and appropriate resourcing developed at all levels. For example, specific methods to facilitate and encourage bottom-up solutions and activities.

Monitoring and evaluation: clear methods to provide ongoing feedback into the system, to drive change to enhance effectiveness and acceptability. For example, developing action-learning or continuous-improvement models for service delivery.

Evidence statement 2.1: paucity of evidence

There is a paucity of evidence on the effectiveness of community-wide programmes displaying features of a whole-system approach to prevent obesity. Of the eight community-wide obesity prevention programmes included in this review – two before-and-after (one [−] and one [+]) three non-randomised control trials (all [+]) one controlled before-and-after study (+); one longitudinal epidemiological study (+); and one repeated cross-sectional survey (+) – none were undertaken in the UK and all targeted children below 14 years. Although they stated an aim to influence the wider community through the programme, including parents, childcare centre workers, teachers and other members of the community. This evidence is judged to be partially applicable to communities of a similar size in the UK.

Evidence statement 2.2: Range of whole-system approach (WSA) features in obesity prevention programmes

None of the eight obesity prevention programmes included in the review demonstrated evidence of explicit recognition of the public health problem as a system. All programmes demonstrated inconsistent evidence of local creativity. Seven programmes demonstrated more robust evidence of capacity building, robustness and sustainability and community engagement, but this was still inconsistent across the groups and all these features did not appear across the same seven programmes. Five obesity prevention programmes demonstrated inconsistent evidence of a focus on the embeddedness of actions and policies, and of developing working relationships within and between partners. Four of the obesity prevention programmes demonstrated inconsistent evidence of a focus on enhancing communication between actors and organisations within the system, facilitative leadership and the use of well-articulated methods for monitoring and evaluation of activities.

Evidence statement 2.3: The effectiveness of obesity prevention programmes – anthropometric outcomes

Overall, there is evidence from a range of community-wide obesity programmes that they can have a beneficial effect on body mass index (BMI) scores, weight gain or the prevalence of overweight and obesity in children. However, these observed differences tended to be relatively small and were not always significant. There is no clear evidence of a relationship between features of system working and programme effectiveness. Studies reported lower BMI scores (one [+] controlled before-and-after; one non-randomised control trial; and one [+] repeated cross-sectional survey). Lower BMI z scores (and one [+] before-and-after and one [+] non-randomised control trial); weight gain (and one cross-sectional [+] survey in France); increase in waist circumference or the prevalence of overweight or obesity (and one [+] longitudinal study). Only one before-and-after (+) study in New Zealand reported a statistically non-significant increase in the prevalence of overweight or obesity among the intervention group.

Evidence statement 2.4: The effectiveness of obesity prevention programmes – diet and physical activity outcomes

There is some evidence that community-wide obesity programmes can have a beneficial effect on diet or physical activity outcomes in children. However, there is no clear evidence of a relationship between features of a system working and the programme's effectiveness. Studies reported a significant decrease in the number of daily servings of 'less healthy' foods and increased daily servings of vegetables and less TV viewing (one controlled before-and-after [+] study). A statistically significantly higher percentage of children passing a fitness test post intervention (one before-and-after [+] study) and a statistically significant increase in diet and activity 'best practice' at childcare centres (one before-and-after [−] study). One non-randomised control trial study also reported a decrease in the number of children unhappy with their body size post intervention.

Evidence statement 2.5: Relationship between system working and effectiveness of obesity prevention programmes

Due to the degree of variation across studies, the small number of the included studies, and the wide range of outcomes reported, the relationship between the presence of features of system working and the effectiveness of community-based programmes to prevent obesity remains ambiguous. It is therefore not possible to suggest a clear relationship.

Two community programmes based in Australia demonstrated the strongest evidence for system working. One controlled before-and-after (+) study describes nine out of the ten features of system working, and demonstrated favourable (though statistically non-significant) between-group differences in anthropometric outcomes. The programme also reported favourable outcomes relating to nutrition (that were statistically significant) and physical activity (that were statistically non-significant). The other study, a (+) non-randomised control trial, shows clear evidence of six out of ten features of a whole system approach, and makes implicit reference to an additional three features. This study reports statistically non-significant between-group decreases in BMI, weight gain and the prevalence of overweight and obesity.

Three community programmes in the US showed five to seven features of whole-system working. One (+) study clearly demonstrates the presence of four WSA features and implies another three features. This study reported a non-significant decrease in BMI z scores. Another (+) study describes three WSA features and makes reference to another three features. It reported a statistically significant change in the prevalence of obesity and improvements in fitness among children post-intervention. Another (−) study describes only two WSA features and makes reference to another three features. No anthropometric outcomes were reported, but the authors reported a statistically non-significant post-intervention increase in diet and activity 'best practices' at childcare centres.

The remaining three community programmes clearly displayed evidence of four or fewer features of whole-system working.

One longitudinal epidemiological (+) study based in France clearly demonstrated evidence of four features, and demonstrated unclear evidence of two additional features. Another, related, repeated cross-sectional (+) survey in France demonstrated unclear evidence of four features. Both studies showed significant pre-/post-reductions in obesity prevalence. One (+) non-randomised control trial from New Zealand provides unclear evidence of two features and reported a between-group statistically significant and favourable change in BMI z scores.

Evidence statement 3.1: System recognition

According to three UK studies (one [−] and two [+]) and one (−) USA study, it is important to recognise the system in which public health problems such as obesity exist. The importance of collaborative working practices (such as partnership working, using novel networks, or managing meetings in a constructive, non-hierarchical way) was also recognised.

Evidence statement 3.2: Ownership and involvement

According to three studies (one [+] and one [++] based in the UK and one [−] based in the USA), partner organisations need to feel that they are actively involved and have some 'ownership' of a strategy. This can help reduce the strain between partner organisations. It is important to develop shared awareness and perspectives (for example, through pre-engagement work or training), but this may take considerable time (that is, years rather than months). Consultations should be focused to prevent partners becoming disillusioned and community concerns recognised, even if these are at odds with those envisaged in the public health programme.

Evidence statement 3.3: Capacity building

According to three (−) studies – one from the USA, one from the UK and one from New Zealand, adequate time and resources need to be set aside for capacity building. Training and awareness-raising may be particularly important – for example to increase staff evaluation (or other technical) skills or bring health onto the agenda of bodies that do not have public health as a primary concern (for example, city planners), according to four (+) UK studies.

Evidence statement 3.4: Partnerships

According to eight studies (two [−] from the UK; three [+] from the UK; one [++] from the UK; one [+] from the USA; and one [−] from New Zealand) partnerships may encounter problems in establishing consensus on the design, delivery and priorities of a programme. Partnerships need time and space to develop and are likely to be stronger where:

  • there is active involvement from both the community and senior staff in key organisations (with communication downwards and upwards)

  • organisations have a positive historical relationship

  • actors form natural communities and share at least some interests or areas of work

  • pre-existing tensions are resolved

  • there is strategic leadership

  • a common language is developed (poor communication can lead to silo working and strained relationships).

Studies also found joint working is easier where programme workers have the skills to establish a relationship with the local community and key individuals can act as 'boundary spanners' across organisations, linking their concerns (two [−] UK; six [+] UK; one [++] UK; one [−] New Zealand and one [−] USA).

Such individuals can be vital to the success of a programme, but this has implications for sustainability (one [+] UK).

Evidence statement 3.5: Embeddedness

Whole-system working is more likely to become embedded where whole systems principles are integrated into strategy and policy documents (one [+] UK) and actions and policies are present at both strategic and operational levels (one [−] UK).

Evidence statement 3.6: Sustainability

The sustainability of whole-systems approaches may be hindered by traditional organisational structures (one [++] UK) or poor experience from previous projects (one [+] UK).

According to seven studies (two [−] UK; one [+] UK; one [++] UK; one [−] USA; one [+] USA; one [−] New Zealand) funding issues impact on the sustainability of a whole-system approach for a range of reasons including:

  • difficulties in making the case for funding for diffuse objectives

  • the lack of continuity and stability inherent in short-term funding for addressing long-term issues

  • inadequate staffing levels.

Evidence statement 3.7: Leadership

According to four UK studies (three [+] and one [++]) strategic leadership was considered important when implementing a whole-system approach – for example, ensuring focus in programme meetings, providing clarity on staff roles, managing tensions between programme staff, providing active leadership at local level and demonstrating personal commitment. However, implementing formal accountability arrangements in cross-organisation partnerships can be difficult. Leadership may face a range of problems including difficulties in achieving consensus between partners (one [+] UK); tensions between local and national priorities, ensuring the overall strategic direction does not stifle local leadership (one [+] UK) and difficulties ensuring inclusive working with minimal resources. Studies have noted implementation problems related to management decisions taken without staff consultation, autonomy of local staff and clarity of management structures, and local programme staff feeling isolated from a national programme (one [−] UK).

Evidence statement 3.8: Monitoring and evaluation

According to two UK studies (one [−] and one [+]) the usefulness of evaluation may be limited by a lack of clarity about objectives and a lack of specificity about outcomes to be measured. Six studies (one [+] USA, one [++] UK and two [−] UK) found intermediate or broader outcome measures may be more appropriate for assessing whole-system approaches, at least in the first instance, rather than specific short-term health outcomes. Broader indicators of success may have the added benefit of fostering partnership working.

It may be particularly difficult to evaluate non-health outcomes and 'reward' partners who do not have a traditional health role. Problems may arise with data collection where staff responsible for collecting the data are unclear about its usefulness or relevance, partners use different information systems or where organisations struggle to reach a consensus on appropriate outcome measures. Unresolved organisational issues or the promotion of a working culture where partners feel unable to openly discuss problems in implementation may act as a barrier to organisational learning (one [+] UK). There may be an unfounded assumption at national level that local agencies have the capacity to develop and deliver a whole system approach.

Evidence statement 3.9: National policy and priorities

According to two studies (both [+] one USA and one UK) the broader political climate may open a 'national policy window' that facilitates policy change, influencing the ability to take a systems approach. Three UK studies (all [+]) found this would enable partnerships that focus on addressing health inequalities. Supportive national policy can help foster partnerships and influence the local agenda. However, changes in national policy may create uncertainty (one [−] UK) and reduce the credibility of local programmes. Targets or funding attached to narrowly defined areas of health, and limited timeframes may limit the ability to take a systems approach (one [−] UK).

Evidence statement 4.1: Quantity and quality of published cost effectiveness and obesity modelling evidence

Only four published economic evaluations were found which related to community-wide multi-faceted obesity prevention or smoking prevention programmes. Two of the economic evaluations (a conference poster relating to the 'Be active eat well' programme in Australia, and a 3-page section of a larger evaluation report on the 'Breathing space' smoking prevention intervention in Edinburgh) were not presented in sufficient detail to warrant a full summary or critical appraisal. The other two cost-effectiveness analyses were not comparable because they were:

  • A small pilot-trial based cost-effectiveness analysis of a school-based community-wide child obesity prevention programme (in New Zealand, results presented in $NZ per kg of weight gain prevented after 2 years).

  • A modelling-based study of the cost-effectiveness of two US-based community-wide campaigns to promote physical activity (the 'Stanford five cities project' and 'Wheeling walks' programme for older people – results presented in cost per life-year and cost per quality-adjusted life-year).

Evidence statement 4.2: Cost-effectiveness findings

There is evidence from only one community-wide obesity prevention programme that estimated incremental cost-effectiveness ratios, and can be judged as having used appropriate methods (of the APPLE pilot project in four small towns in New Zealand). However, while having some community-based activities, the APPLE project was judged to only weakly exhibit two of the ten defined features of a whole-system approach. Only four published economic evaluations were identified that were potentially relevant to the scope of this guidance. Two of these studies were so under-reported that their findings cannot be relied upon. The other included cost-effectiveness study was of two community-wide physical activity promotion campaigns in the USA.

Evidence statement 4.3: Approaches to modelling of obesity and for obesity prevention

Simulation modelling of obesity or obesity policies is still at a relatively early stage of development. However, in some cases methods for modelling outcomes in the area of obesity and obesity prevention policies or programmes has already become so complex and advanced that the usefulness (or even feasibility) of attempting to develop credible new models without significant modelling capacity, access to national data, and significant modeller time and other resources is questionable. Instead, with limited resources, any realistic modelling of alternative local community-wide obesity prevention policies should aim to make best use of one of the well-established and tested existing population-level obesity models (such as the National Heart Forum's micro-simulation model, or the ACE Obesity model framework).

Evidence statement CR1: Establishing a community-wide approach to preventing obesity – key actors and players

A genuinely community-wide approach to preventing obesity includes a vast range of actors and agencies. For such a network to be effective, partners must share an overarching vision around obesity prevention, with each organisation 'buying in' and feeling a sense of ownership.

At the strategic level, the impetus for a community-wide approach begins with local elected members and senior managers (particularly from the NHS and the local authority). Public health is best placed to provide investment and leadership for the network of partners, aided by the health and wellbeing board that needs to exert its influence on the clinical commissioning group to ensure investment and 'buy in' across community health services.

In order to build the network of partners, local communities and services should be viewed from the perspective of individual citizens, to identify the most relevant services regularly used and trusted by key groups such as parents. Once signed up as partners, these services can be leveraged to make every contact count.

Information needs to be shared and relationships developed both 'horizontally' across partner organisations, and 'vertically' inside individual organisations. Failure to ensure middle managers and frontline workers share the vision and understand the community-wide approach is perhaps the most common factor limiting the effectiveness of such partnerships.

The main delivery organisations (for example, community projects with provider contracts) must have credibility in their local communities. Community engagement is the key activity in building and developing this credibility.

Evidence statement CR2: Facilitators of an effective community-wide approach

Having a central coordination and communications function is considered to be essential and must engage beyond senior management level in the partner organisations, striving to ensure middle managers share the vision, and are well informed about the wider network. Concise briefings on key issues are important for middle managers and frontline staff, to build confidence, capacity and consistency in messaging across the wide range of partners.

Partner organisations should be expected to make an explicit commitment of what they will contribute, and this should be publicised across the network. Those making investment decisions should build on proven success by 'backing winners', and concentrate investment where it is most likely to succeed.

Strategy should take an iterative approach, reviewing progress regularly.

Evidence statement CR3: Barriers to an effective community-wide approach

Starting conversations about obesity with individual clients and patients is difficult, and there are numerous reasons why staff may not have the confidence or the motivation to do so, even among primary care professionals. It is very important to build confidence and capability among customer-facing staff in both primary care and community settings, as the credibility of messages from the latter will be seriously undermined if inconsistent with messages from the former.

In terms of population-wide primary prevention, the term 'obesity' can be off-putting, and engagement with target audiences may be easier if the focus is framed as 'healthy lifestyles'. This more broad-based approach may also be more stable in terms of long-term funding.

Financial barriers are significant for many low-income groups, particularly in terms of the cost of transport and accessing services. Cultural minorities and disabled people face additional barriers in accessing information and services, and their specific needs should be considered carefully when assessing needs.

A significant contribution can be made by volunteers (health champions and peer mentors), but their effectiveness may be limited by the willingness of health professionals to make referrals to them.

The prevention of obesity is a long-term objective, but most project funding is short term. There are complex personal, family and socioeconomic causes applying to many obese and overweight people. Both commissioners and providers would like to be able to commit to longer-term contracts for obesity prevention work, in recognition of the considerable time and resources needed to successfully engage with clients with complex needs, for whom positive short-term outcomes are less likely.

Evidence statement CR4: Sustainability

It is inevitable that funding streams will change over time. By recognising that obesity is an essential concern for many health and social issues, it should be possible to be flexible and creative in justifying ongoing funds for obesity prevention work, despite such changes.

The strategy and the wider network of partners must be sustainable. The maintenance and development of the shared vision is fundamental for sustainability, and this requires effective communication to maintain the engagement, particularly with politicians and middle managers. Frontline staff and organisations may see themselves as peripheral to the issue of obesity. Having a strong local brand or identity is important, particularly for workers in the network of organisations, as it is important for them to feel part of a bigger picture.

A key message in this communication must be the commitment to evaluation and ongoing service improvement. If pump-priming funds (that is, short-term funds, aimed at stimulating future investment from mainstream sources) are made available to establish the network, plans to transfer responsibilities to mainstream budgets should be built in wherever possible. However, in the context of current public expenditure constraints, mainstream incorporation cannot be guaranteed.

The community-wide approach should seek to build on existing community assets. This will build capacity in people and institutions that will continue, even if obesity-specific funding diminishes. Commissioners should also consider that at some point in the future, they may be relying on influence and goodwill rather than contractual obligations.

A clear separation of strategic and operational management, using boards and forums with distinctive terms of reference, may be helpful.

Evidence statement CR5: Evaluation

Data collection and monitoring can contribute to project sustainability, project management, keeping all parties focused on goals and service improvement. Evaluation is primarily considered for individual programmes, projects and interventions; a complex, community-wide approach is seldom evaluated.

Further consideration needs to be given to the applicability and acceptability of different types of evidence, in the context of the very limited time and resources available at a local level. There is concern that while obesity prevention is a long-term challenge, with long timescales for return on investment, funding is very often short term, with unrealistic outcome expectations. Consideration should be given to the acceptance of intermediate outcomes in commissioning contracts. The example of 'job readiness' in employment-related community work was cited, with the suggestion that 'weight-loss readiness' was a similarly legitimate intermediate outcome. There is a tension between the use of narrow, quantitative outcome criteria (often the focus of commissioners), versus a broader range of outcome measures including qualitative data of community wellbeing (often the focus of providers).

Evaluation is often focused on contract performance management. There was little evidence of a systematic approach to building a local evidence base. Project timetables and budgets rarely allow for the establishment of robust baselines on which to base evaluations. Evaluation often ignores clients who had dropped out of the programme or intervention. This would seem to be a significant gap in the development of evidence.

Providers express concerns about the burden of data collection and monitoring, particularly those receiving funding from multiple sources. There is frustration at the inconsistency of data required by different funders. Evaluators should properly brief those collecting the data on the rationale and requirements.

Evidence statement CR6: Cost effectiveness

Very little true cost-effectiveness evaluation is undertaken at a local level due to the lack of specialist skills. To commission externally is expensive, and if the skills are available internally it is very time intensive. Thus, cost-effectiveness analysis may be considered not justified on grounds of cost effectiveness.

There seems to be relatively little scrutiny of cost effectiveness (as opposed to cost management). Budget holders at a higher level appear to have limited understanding of cost-effectiveness analysis, and as a result, there is little pressure to undertake such work.

Some participants expressed concern that public health investment might be disadvantaged by more exposure to cost-effectiveness analysis, due to public health delivering longer-term returns on investment, and the difficulty of attributing cause and effect (relative to clinical treatment). There was also a concern that truly like-for-like comparisons are difficult to achieve in cost-effectiveness analysis. In this view there was a risk of simplistic interpretation, in which differences between programmes and interventions may be caused by underlying socioeconomic factors that were not visible in the calculation.

Additional evidence

Expert paper 1: 'Whole systems – adapted and designed'

Expert paper 2: 'Lessons from tobacco control'

Expert paper 3: 'Systems and system failure'

Expert paper 4: 'Whole system approaches to obesity – progress and future plans'

Expert paper 5: 'Insight, experiences and evidence of the Childhood Obesity National Support Team'

Expert paper 6: 'Cycling cities/cycling demonstration towns initiative'

Expert paper 7: 'The contribution of health trainers, community health champions and the general public'

Expert paper 8: 'Well London'

Expert paper 9: 'Tower Hamlets healthy borough programme'

Expert paper 10: 'Healthy places, healthy lives – tackling childhood obesity in Luton case study'

Expert paper 11: 'Exeter cycling demonstration town 2005 to 2011'

Expert paper 12: 'Commissioning – learning from Sheffield and Rotherham'

Expert paper 13: 'Evaluation in Hull'

Expert paper 14: 'Working in partnership: An example from a rural area – South Gloucestershire'

Expert paper 15: 'Tackling obesity in a rural county'

Expert paper 16: 'West and Mid Essex local commissioning experience'

Expert paper 17: 'Effective partnership working and stakeholder engagement in the delivery of obesity prevention and treatment programmes in Kirklees'

Expert paper 18: 'Short paper on organisational issues'

Expert paper 19: 'Evaluating complex community-based interventions (CBIs) for obesity prevention'

Economic modelling report

Where two organisations decide to work in partnership to implement an intervention more effectively than they could while working alone and there is a low initial cost, the partnership can usually be considered cost effective. When the partnership is known to lead to cost savings (especially as a result of sharing resources), it will be cost effective provided that the health benefits are not diminished when the organisations work together. In more complex situations, it is unclear whether or not partnerships are cost effective, because conventional cost-effectiveness methods cannot be applied.

On funding for projects, a simple model suggests that obesity projects with long-term funding are likely to be more cost effective than equivalent projects with less secure funding.

Previous modelling suggests that any public health interventions costing £10 or less per head will be cost effective for all except the smallest weight losses (or weight gains prevented).

Engaging with local communities can, for a relatively low cost, ensure aspects of a large project that have not been acceptable to a community may be modified, and result in large community gains that would otherwise have been rejected. The decision to engage will depend on whether the original plans are likely to succeed without engagement, and the likelihood that engagement will succeed in producing a consensus in favour of a modified project.

  • National Institute for Health and Care Excellence (NICE)