4 Approach to evidence generation

This section presents an approach to generating evidence and a description of real-world data that may be able to support evidence generation.

The technologies have varied positions with regards to their ongoing studies and the evidence gaps. All ongoing studies will complete within the evidence generation period. OSI has studies that may address most of the evidence gaps including its comparative effectiveness. Other technologies have studies that could address some of the evidence gaps, but not comparative effectiveness, and will need additional evidence generation.

4.1 Evidence generation plan

To address the evidence gaps a historical control study with propensity score methods is recommended

In a prospective study with propensity score matching, data is collected on a control group before the intervention is implemented among children and young people at the point at which they would have been eligible for the intervention. This is followed by data collection in the intervention group after implementation of the technology. In this approach, propensity score methods are used to balance observed characteristics to create comparable cohorts, reducing bias from confounding.

This approach allows direct comparison between an individual technology and the comparator (routine care) and can produce the evidence necessary for NICE decision making. There are limitations to this approach; head-to-head comparison between the technologies will not be possible and good information on important confounding factors is needed for each individual. It will also be important to control for seasonality. Ideally, the study should be run in multiple services or centres for each technology.

The experimental design will compare the technologies against routine care. As described in the scope, routine care is the current first-line treatment for children and young people with mild to moderate symptoms of anxiety or low mood. This may include education, advice, support and signposting.

Although the early value assessment for these technologies does not describe a particular setting in which they are to be used, services provided in schools may be a good option for evidence generation. These services are likely to identify children and young people for whom the technologies may be appropriate and follow up after treatment is likely to be more straightforward.

Data collection is best approached using multiple sources, for example:

  • some information generated through the technologies themselves (such as engagement and drop out)

  • some information generated through linked mental health-GP electronic health records (such as important confounding factors, for example, comorbidities or medications or safety outcomes such as suicide or self-harm)

  • other information collected by clinicians or research staff working with the children and young people (symptom measures such as the Revised Children's Anxiety and Depression Scale).

The technologies are well suited to collecting engagement information, but the outcome scales are clinical tools not designed for use in this way and are better administered by trained practitioners.

Data collection should follow a predefined protocol and quality assurance processes should be put in place to ensure the integrity and consistency of data collection. Analysis and reporting of the data should be completed by an independent party. See NICE's real-world evidence framework, which provides guidance on the planning, conduct, and reporting of real-world evidence studies.

4.2 Real-world data collections

The technologies will be able to collect some of the specific outcome measures such as use and level of engagement. But this data would need to be integrated with other data collected. There are several different data collections that could potentially support evidence generation with different strengths and weaknesses. NICE's real-world evidence framework provides detailed guidance on assessing the suitability of a real-world data source to answer a specific research question.

The Mental Health Services Data Set (MHSDS) is a mandated national data collection that could potentially collect the necessary data. But it may not routinely collect all the outcome measures identified in the early value assessment. Also, there are potential issues with data quality and whether data on all people who are eligible has been submitted. NHS England has suggested that modification of MHSDS may take up to 2 years, so it is unlikely that it could be modified in time to support data collection.

There may be local or regional data collections that do collect the specific outcome measures recommended in the early value assessment. For example, questionnaires administered by Child and Adolescent Mental Health Services or the South London and Maudsley NHS Foundation Trust Biomedical Research Centre Case Register Interactive Search tool. These datasets or others that are similar, may be available through sub-national secure data environments (SDEs). Within an SDE the data may be linked to other useful data such as that from primary care and could provide information on important confounders (for example, comorbidities).

The quality and coverage of real-world data collections are of key importance when used in generating evidence. Active monitoring and follow up through a central coordinating point is an effective and viable approach of ensuring good-quality data with high coverage.

4.3 Data to be collected

The following information has been prioritised for collection, for both control and treatment groups:

  • Revised Children's Anxiety and Depression Scale, recorded at baseline and after treatment at 3 months and ideally at 6 months

  • ideally, the Strengths and Difficulties Questionnaire should also be recorded at baseline and after treatment at 3 months and ideally at 6 months

  • engagement and drop-out information, including reasons for stopping

  • incidence of self-harm and suicide

  • information about potential confounding factors at baseline (for example, comorbidities and socioeconomic status), evidence developers should seek clinical input to ensure important confounding factors are captured to support robust propensity score methods

  • information about a subgroup of children and young people with low mood only

  • information about a subgroup of neurodivergent children and young people

  • CHU-9D, recorded at baseline and after treatment at 3 months and ideally at 6 months.

4.4 Evidence generation period

It is proposed that the evidence generation period is 3 years, this would leave enough time for ongoing studies to complete, and the time needed to implement evidence generation, collect the necessary information and analyse collected data.