3 Approach to evidence generation

3.1 Evidence gaps and ongoing studies

Table 1 summarises the evidence gaps and ongoing studies that might address them. Information about evidence status is derived from the external assessment group's report; evidence not meeting the scope and inclusion criteria is not included. The table shows the evidence available to the committee when the guidance was published.

**Table 1 Evidence gaps and ongoing studies**
Technology	Clinical effectiveness	Resource and service impact	Engagement and acceptability	Uptake in different subgroups
Activate Your Heart	Limited evidence	Limited evidence	Limited evidence	No evidence
D REACH-HF	Limited evidence	Limited evidence	No evidence	No evidence Ongoing study
Digital Heart Manual	No evidence	Limited evidence	Limited evidence	No evidence
Gro Health HeartBuddy	Limited evidence Ongoing study	Limited evidence	Limited evidence	No evidence
KiActiv	Limited evidence Ongoing study	Limited evidence	Limited evidence Ongoing study	No evidence
myHeart	Limited evidence Ongoing study	Limited evidence	Limited evidence	No evidence
Pumping Marvellous Cardiac Rehab Platform	No evidence	No evidence	No evidence	Limited evidence

3.2 Data sources

There are several data collections that have different strengths and weaknesses that could potentially support evidence generation. NICE's real-world evidence framework provides detailed guidance on assessing the suitability of a real-world data source to answer a specific research question.

The National Audit of Cardiac Rehabilitation (NACR) is the data source that is most likely to be able to collect the real-world data necessary to address the essential evidence gaps. The audit collects data to support the monitoring and improvement of cardiovascular prevention and rehabilitation services. Currently, data is collected at the start and end of a cardiac rehabilitation program. The audit currently indicates if cardiac rehabilitation has included Activate Your Heart, or the manual or digital version of D REACH-HF or Heart manual. There are future plans to link patient-level data to other datasets such as the Hospital Episode Statistics and Office for National Statistics for collection of longer-term outcomes. Additional data collection is planned around the mode of delivery of cardiac rehabilitation as part of the audit.

Other useful sources of data are the National Institute for Cardiovascular Outcomes Research and the National Cardiac Audit Programme.

The quality and coverage of real-world data collections are of key importance when used in generating evidence. Active monitoring and follow up through a central coordinating point is an effective and viable approach of ensuring good-quality data with broad coverage.

Ongoing studies

There are 2 highly relevant, ongoing studies that may address some of the clinical effectiveness, resource-impact and service-impact evidence gaps. Both are due to end in 2025.

3.3 Evidence collection plan

The suggested approach to addressing the evidence gaps for the technologies is a real-world historical control study with propensity score methods. The study would compare outcomes before and after implementation of the technologies. Quantitative data for the historical control arm is likely to exist in the NACR dataset. This dataset includes patient-level data such as components of cardiovascular risk profiles; exercise capacity; health-related quality of life; psychological wellbeing; and nutrition. The dataset also details service-user characteristics such as age, sex, ethnicity, geographical location and employment status, which will enable subgroup analyses. These baseline cohort differences may affect clinical outcomes and should be adjusted for in future analyses. In 2024, there were 40 services collecting 12‑month assessment period data.

Qualitative data could be generated through appropriate methods such as surveys, focus groups or interviews, as highlighted in NICE's real world evidence framework. This could include reported outcomes (acceptability, usability and preferences) from people using the technologies.

Despite consistent eligibility criteria, non-random assignment to interventions can lead to confounding bias, complicating interpretation of the intervention effect. To minimise bias and identify a suitable control group, appropriate statistical approaches that balance confounding factors across comparison groups should be used, for example, propensity score matching. The comparator group of primary interest is cardiac rehabilitation face-to-face sessions, or a hybrid programme of in-person group-based and home-based programmes (including paper manuals, live online classes, home visits or telehealth) without digital cardiac rehabilitation technologies. NICE's real-world evidence framework provides further detailed guidance on the planning, conduct and reporting of real-world evidence studies assessing comparative effects.

3.4 Data to be collected

Study criteria

At recruitment, eligibility criteria for the suitability of the digital technologies for the participant and inclusion in the real-world study should be reported. This should include the referral pathway for participants. There should be detailed descriptions of each technology, including its training requirements, digital-safety assurance and its specific version.

Service-user characteristics and clinical outcomes

These should include:

Information about individual characteristics at baseline, for example, sex, age, ethnicity, first language, medicines, diagnosis, comorbidities, socioeconomic status, and location, with other important covariates chosen with input from clinical specialists. Characteristics should include those needed for adjustment to address confounding, and for subgroup analysis.
Measures recorded at baseline and follow up (at least 12 months later, ideally up to 18 months) of:
- exercise capacity (for example the shuttle walk test)
- cardiovascular risk profile (including blood pressure, weight, height, and cholesterol)
- psychological wellbeing (Patient Health Questionnaire-9, Generalised Anxiety Disorder‑7 or Hospital Anxiety and Depression Scale, Cardiac Distress Inventory-Short Form)
- health-related quality of life (EQ-5D or Dartmouth COOP)
- nutrition status (Mediterranean Diet Score tool)
- medication adherence.
Adverse events.

Resource and system use

This should include:

time from post-discharge referral to start of core cardiac rehabilitation programme
number and cost of face-to-face cardiac rehabilitation sessions (and details about the health professional including the banding of staff leading or supporting the sessions)
referrals to other specialist services
number of appointments in primary, secondary and community care
costs of digital technologies for supporting cardiac rehabilitation, including:
- licence fees
- healthcare professional staff time, staff banding, and training costs to support the service
- integration with digital NHS systems
- implementation costs
other technology costs.

Engagement and acceptability

This should include:

usability and acceptability of the technologies
intervention adherence, uptake, completion and attrition rates (including reasons for not using the technology).

It is also important to report and specify if any optional features of the technologies are being used (for example additional artificial intelligence modules) during evidence generation.

Data collection should follow a predefined protocol, and quality assurance processes should be put in place to ensure the integrity and consistency of data collection. See NICE's real-world evidence framework, which provides guidance on the planning, conduct, and reporting of real-world evidence studies.

3.5 Evidence generation period

This will be 3 years to allow for setting up, implementing the test, data collection, analysis and reporting.

3.6 Following best practice in study methodology

It is important to follow best practice in conducting studies to ensure the reliability and validity of the research findings. Adherence to rigorous guidelines and established standards is crucial for generating credible evidence that can ultimately improve patient care. The NICE real-world evidence framework details some key considerations.

In the context of an early value assessment, a key factor to consider as part of the informed consent process is making sure that patients (and their carers, as appropriate) understand that data will be collected to address the evidence gaps identified in section 2. Where applicable this should take account of NICE's guidance about shared decision making.

Digital platforms to support cardiac rehabilitation: early value assessment

Evidence generation plan for digital platforms to support cardiac rehabilitation: early value assessment