3 Approach to evidence generation

3.1 Evidence gaps and ongoing studies

There are 2 ongoing studies evaluating Phio Engage that may contribute some clinical-effectiveness data. These are due to end in December 2025.

Table 1 summarises the evidence gaps and ongoing studies that might address them. Information about evidence status is derived from the external assessment group's report; evidence not meeting the scope and inclusion criteria is not included. The table shows the evidence available to the committee when the guidance was published.

**Table 1 Evidence gaps and ongoing studies**
Evidence gap	Clinical effectiveness	Resource use and service impact	User engagement and acceptability	Clinical effectiveness and resource use in different subgroups
getUBetter	Limited evidence	No evidence	No evidence	No evidence
Good Boost	Limited evidence	No evidence	No evidence	No evidence
Hinge Health	Limited evidence	No evidence	Limited evidence	No evidence
Joint Academy	Limited evidence	Limited evidence	Limited evidence	No evidence
Phio Engage	Limited evidence Ongoing studies	No evidence	No evidence	No evidence
re.flex	Limited evidence	No evidence	Limited evidence	No evidence
Sword Thrive	Limited evidence	No evidence	No evidence	No evidence
TrackActive Me	Limited evidence	No evidence	No evidence	No evidence

3.2 Data sources

NICE's real-world evidence framework provides detailed guidance on assessing the suitability of a real-world data source to answer a specific research question. The NHS England Secure Data Environment (SDE) service could potentially support this research. This platform provides access to high-standard NHS health and social care data that can be used for research and analysis. SDEs are data storage and access platforms that bring together many sources of data, such as from primary and secondary care, to enable research and analysis. They could be used to collect data to address the evidence gaps. Subnational SDEs are designed to be agile and can be modified to suit the needs of new projects. Within an SDE, the data can be linked to other useful data, such as that from primary care, and could provide information on important confounders (for example, comorbidities).

The Osteoarthritis Initiative (OAI) could potentially provide additional information for comparison. The quality and coverage of real-world data collections are of key importance when used in generating evidence. Active monitoring and follow up through a central coordinating point is an effective and viable approach of ensuring good-quality data with broad coverage.

3.3 Evidence collection plan

NICE suggests a mixed-methods approach to address the identified evidence gaps: a prospective real-world comparative cohort study combined with a qualitative survey. The qualitative component should explore user experience, engagement and barriers to access in more depth.

Data could be collected through a combination of:

primary data (for example, outcome measures and surveys)
data generated through the technology itself (for example, engagement metrics and session completion)
routinely collected real-world data sources (for example, Clinical Practice Research Datalink [CPRD] and Hospital Episode Statistics [HES]).

Prospective real-world comparative cohort study

In this type of study, data should be collected from healthcare services where the digital technology is offered and compared with services where it is not. People in both groups should be followed from the point at which they would typically be offered the technology. The comparison group should include people from similar services with comparable patient populations and standard care pathways but without access to the digital technology. Ideally, the study should be conducted across multiple centres to reflect the diversity of the NHS service provision. Non-random assignment to interventions introduces a risk of confounding bias. So, appropriate methods, such as matching or adjustment (for example, propensity score methods), should be used to minimise selection bias and balance confounding factors between groups. High-quality data on patient characteristics will be essential to support these methods. The identification of key confounders should be informed by expert input during protocol development.

Qualitative survey

Qualitative data could be generated through appropriate methods, such as surveys, focus groups or interviews. This should include reported outcomes (acceptability, usability and preferences) from people using the technologies. The robustness of the findings will depend on:

broad and inclusive distribution across eligible users
the sample of respondents being representative of the population of potential users.

3.4 Data to be collected

Data collection should follow a predefined protocol, and quality assurance processes should be put in place to ensure the integrity and consistency of data collection. NICE's real-world evidence framework provides guidance on the planning, conduct and reporting of real-world evidence studies.

Study criteria

At recruitment, eligibility criteria for the suitability of using the digital technologies and inclusion in the real-world study should be reported. Detailed descriptions of the technologies should include their training requirements, digital-safety assurance and the specific products and versions, as well any optional features of the products that are being used (including any artificial intelligence add-ons for program selection).

Service-user characteristics and clinical outcomes

Information about individual characteristics at baseline, for example, sex, age, ethnicity, first language, medicines, diagnosis and affected body area, comorbidities, socioeconomic status and location, with other important covariates chosen with input from clinical specialists. Characteristics should include those needed for adjustment to address confounding and for subgroup analysis.
Measures recorded at baseline and follow up (at least 12 months, ideally 18 months), of:
- health-related quality of life (EQ-5D)
- patient-reported outcomes, including pain and stiffness, psychological outcomes, physical function, activity impairment and self-efficacy (for example, Musculoskeletal Health Questionnaire, Numeric Pain Rating Scale (0 to 10) for the target joint, Patient Specific Functional Scale, Work Ability Index or Arthritis Self-Efficacy Scale).
Number of referrals for corticosteroid injections.
Adverse events (including reporting detail around whether or not they are intervention related).

Engagement and acceptability

Usability, satisfaction and acceptability of the technologies.
Intervention adherence, uptake, completion and attrition rates (including reasons for not using the technology).

Resource and system

Number and cost of face-to-face physiotherapy sessions (and details about profession and banding of staff leading or supporting the sessions).
Number and cost of appointments in primary, secondary and community care.
Medication use.
Any additional interactions with healthcare professionals outside of appointments (for example, time to review data provided by the technologies, where relevant).
Referrals to secondary care and other specialist services.
Costs of digital technologies for supporting management of osteoarthritis, including:
- licence and maintenance fees with subscription duration
- healthcare professional staff time and training costs to support the service
- integration with digital NHS systems
- implementation costs
- other technology costs (including additional hardware or software).

It is also important to report and specify if any optional features of the technologies are being used during evidence generation.

3.5 Evidence generation period

This will be 3 years to allow for setting up, implementing the test, data collection, analysis and reporting.

3.6 Following best practice in study methodology

Following best practice in conducting studies is paramount to ensuring the reliability and validity of the research findings. Adhering to rigorous guidelines and established standards is crucial for generating credible evidence that can ultimately improve patient care. NICE's real-world evidence framework details some key considerations.

For early value assessments, a key factor to consider as part of the informed consent process is to ensure that patients (and their carers, as appropriate) understand that data will be collected to address the evidence gaps identified in section 2. Where applicable, this should take account of NICE's guidance about shared decision making.

Digital technologies for managing mild to moderate symptoms of hip or knee osteoarthritis: early value assessment

Evidence generation plan for digital technologies for managing mild to moderate symptoms of hip or knee osteoarthritis: early value assessment