4 Evidence and interpretation

The Appraisal Committee (Appendix A) considered evidence from a number of sources (Appendix B).

4.1 Clinical effectiveness

4.1.1 The Assessment Report reports effect sizes (ESs) for the trials. An effect size (ES) quantifies the size of the difference within or between groups by relating it to the corresponding standard deviation. A positive ES indicates a favourable effect in the direction of the intervention against the comparator within or between groups. The ESs were not weighted. The literature on ESs suggests that an ES can be thought of as 'small' when ES = 0.2, 'medium' when ES = 0.5 and 'large' when ES = 0.8. The importance of these ESs will also depend on the clinical effectiveness of the comparator therapy in each comparison.

4.1.2 Fourteen studies (six RCTs, two non-RCTs and six non-comparative studies) were identified for the five packages included in the review. Some of trials had considerably more female than male participants, particularly in the case of depression. In most studies, the mean age of patients was in the range 30–45 years. In the majority of trials, patients were included who were also taking medications for their particular disorder. Multiple outcomes using multiple measures were collected in the trials.

Depression

Beating the Blues

4.1.3 Two RCTs (one unpublished) and one non-comparator trial (unpublished) were used in the assessment of Beating the Blues. However, the unpublished RCT was marked 'academic in confidence'; hence the results are not reported in this review. The published RCT contained 274 patients at baseline assessment but pre-treatment values were recorded in only 241 patients, and only around two-thirds of patients were assessed at the end of the follow-up period of 6 months. Randomisation was stratified according to whether drug treatment was also being received. All of the trials were conducted in a primary care setting in the UK and all trials recruited patients through GP referral or screening with the General Health Questionnaire (GHQ).

4.1.4 The published RCT compared Beating the Blues (n = 146) with treatment as usual (TAU) (n = 128), which was defined as 'whatever treatment their GP prescribed'. Beating the Blues statistically significantly improved scores for depression (Beck Depression Inventory [BDI] – the primary outcome in the trial on which power calculations were based), negative and positive attributional style (Attributional Style Questionnaire [ASQCoNeg/Co Pos]), and work and social adjustment (WSA) compared with TAU. It was found, from an informal investigation of interaction, that treatment interacted with severity for anxiety (Beck Anxiety Inventory [BAI]) and positive attributional style. ESs, calculated between Beating the Blues and TAU for the BDI, BAI and WSA, were 0.65, 0.25 and 0.31, respectively.

4.1.5 With regard to patient satisfaction for Beating the Blues, the published RCT found that Beating the Blues patients were significantly more satisfied (as measured on a single item) with treatment than TAU patients, although values were not reported. In the submission from the manufacturer of Beating the Blues, an unpublished paper discussing the credibility and satisfaction of the programme was included. This open trial reported that nine out of ten patients stated that they would recommend the programme to others and over half stated that it was better than any other treatment that they had received.

COPE

4.1.6 Two non-comparator trials were used in the assessment of COPE (n = 39 and n = 41 patients), and the follow-up period for both was a maximum of 12 weeks. One trial was conducted in the UK and the other in both the USA and the UK, over the telephone. Recruitment for one of the studies was through self-referral and the other was through self- and healthcare professional-referral. Both trials reported information on patient history, such as duration of symptoms and previous therapy or medication. Only one of the trials gave reasons for loss to follow-up. In one of the trials, it was reported that patients felt comfortable with the system, found it easy to use and found the booklets helpful, while 75% of the 28 completers said that COPE had improved the quality of their lives.

Overcoming Depression: a Five Areas Approach

4.1.7 One non-comparator trial was used in the assessment of Overcoming Depression (n = 20), with a follow-up period of 3 months. The trial was conducted in the UK. Recruitment in this trial was through consecutive referrals to a clinical psychology service. The trial did not report information on patient history or reasons for loss to follow-up. Of those who gave an opinion, all (n = 15) stated they would recommend the programme to others. At 6 weeks, 60% rated treatment usefulness as 'a lot' and 40% as 'a little'. At the end of treatment, 80% said they would prefer a CD-ROM over book treatment.

Panic/phobia

FearFighter

4.1.8 Two RCTs and two non-RCTs were used in the assessment of FearFighter. The total numbers of patients in the trials ranged from 27 to 93, with follow-up periods ranging from 1 to 4 months. Two of the trials were conducted in a secondary care setting in the UK, and two were conducted in non-healthcare settings in the UK. Three of the trials used a mixture of self- and healthcare professional-referral, whereas one used self-referral only. All the trials reported information on patient history, such as duration of symptoms and previous therapy or medication, but only the two RCTs reported reasons for loss to follow-up.

4.1.9 When FearFighter was compared with therapist-led cognitive behaviour therapy (TCBT), the results of the non-RCT showed that both groups (FearFighter n = 54; TCBT n = 31) improved statistically significantly from pre-treatment scores, although the TCBT group scores were more severe at baseline. An ES, calculated between FearFighter and TCBT for Fear Questionnaire (FQ) total, was –0.12. For the RCT (FearFighter n = 37; TCBT n = 39), there was a large number of drop-outs, with 30 patients lost to follow-up at 1 month. Both the FearFighter and TCBT groups improved statistically significantly at 3 months compared with baseline. ESs, calculated between FearFighter and TCBT for Main Problems (MP), Goals, Global Phobia and WSA, were –0.22, 0.26, –0.89 and –0.04, respectively. A relaxation group (n = 17) was also included in this RCT. The relaxation group had no statistically significant improvement compared with the CCBT and TCBT groups. Almost twice as many patients dropped out of the FearFighter group than TCBT, although an intention-to-treat (ITT) analysis was carried out.

4.1.10 For the RCT that compared FearFighter (n = 45) with another computer programme (n = 23) (also delivered by the Internet) with cognitive components but no exposure (Managing Anxiety), both groups improved statistically significantly from baseline. An ES, calculated between FearFighter and Managing Anxiety for Total Phobia, was –0.19. At the 1-month follow-up, FearFighter was statistically significantly more effective than the other computer programme on some measures.

4.1.11 One non-RCT compared the two delivery methods of FearFighter. Delivery of FearFighter in a clinical setting comprised seven sessions (n = 17), whereas the Internet group had unlimited access at home over a 12-week period (n = 10). Both groups improved statistically significantly on all measures. An ES, calculated between groups for FQ total, was –0.11.

4.1.12 With regard to patient satisfaction, only one RCT reported ratings of treatment helpfulness and found no statistically significant differences between FearFighter, TCBT and relaxation. Satisfaction ratings in the other RCT did not differ statistically significantly between FearFighter and the Managing Anxiety programme. In one of the non-RCTs, Internet users were said to be generally satisfied, although no data were reported. Three of ten Internet users said they would have preferred face-to-face guided self-help to Internet-guided self-help.

Obsessive-compulsive disorder

BTSteps (now OCFighter)

4.1.13 Two RCTs and two non-comparator trials were used in the assessment of BTSteps. The total numbers of patients in the trials ranged from 21 to 218, with follow-up periods ranging from 14 to 22 weeks. Patient history and socio-economic information were reported in three of the studies, but none of the trials reported reasons for loss to follow-up. Two of the trials were conducted in the UK, one in the US and Canada, and one was conducted at two centres in the US and UK. Three of the four studies used DSM-III-R criteria for diagnosis, and one used ICD-10. Both of the RCTs had methodological problems; one did not use blind assessment or did not report reasons for loss to follow-up and the other did not report the method of randomisation, use blind assessment (except for a subgroup of 41% that was rated blind at the two main time periods) and did not report reasons for loss to follow-up.

4.1.14 One RCT compared BTSteps (n = 74) with TCBT (n = 69) and relaxation (n = 75). TCBT was found to be statistically significantly more effective than BTSteps at 14 weeks post treatment. ESs, calculated between BTSteps and TCBT for the Yale-Brown Obsessive Compulsive Scale (Y-BOCS), the Hamilton Rating Scale for Depression (HAM-D) and the Work and Social Adjustment Scale (WSA), were –0.45, –0.25 and –0.23, respectively. Relaxation was shown to be ineffective compared with BTSteps and TCBT.

4.1.15 The other RCT compared BTSteps plus scheduled support (n = 22) with BTSteps plus on-demand support (n = 22). Statistically significant greater improvement was reported in the scheduled support group at 17 weeks. An ES of 0.77 was calculated between the groups for the Y-BOCS.

4.1.16 With regard to patient satisfaction, one of the RCTs reported that patients who received TCBT or BTSteps were significantly more satisfied than patients who received relaxation, and patients treated with TCBT tended to be more satisfied than patients who used BTSteps. Another separate report also found that when patients who had received BTSteps went on to clinician-guided care (n = 9), patients were statistically significantly more satisfied with TCBT than BTSteps.

4.2 Cost effectiveness

4.2.1 The Assessment Group identified one published economic evaluation. No formal analyses of cost effectiveness were included in the manufacturer submissions. The Assessment Group developed its own economic models for the three disease areas: depression, panic/phobia and OCD.

Depression

4.2.2 One published economic study detailed a cost-effectiveness analysis of Beating the Blues versus TAU. Data on resource use were collected prospectively alongside the clinical effectiveness trial. The costs of Beating the Blues were supplied by the manufacturer. The study covered a wide range of NHS resource usage, and estimates were also made of the indirect costs of lost production (costs were reported separately with and without indirect costs). Resource use data were collected for the period 6 months prior to study entry and for the 8 months' duration of the study. Comparisons were made between the mean costs of Beating the Blues and TAU using a bootstrapping technique to generate 95% confidence intervals (CIs). The analyses were conducted on an ITT basis and revealed that the mean service cost for CCBT was £397 compared with £357 for TAU, resulting in a mean incremental service cost of £40 (90% CI, –£28 to £148). Total costs including lost employment were less for the Beating the Blues group at £533 compared with £900 for TAU. Data on reported clinical outcomes were combined with cost data to produce a cost per point reduction in the BDI and a cost per symptom-free day. A cost–utility analysis was undertaken by applying a utility value to days with and without symptoms. The mean number of depression-free days was 89.7 (standard deviation [SD] 74.2) for Beating the Blues compared with 61 (SD 67.1) for TAU. From a published review of utilities studies of patients, figures of 0.59 and 1.0 for depression days and depression-free days, respectively, were taken, resulting in an estimated quality-adjusted life year (QALY) gain of 0.032, or a cost per QALY gained (CQG) of £1250. Valuing a one-unit improvement in the BDI at £40 was associated with an 81% chance of Beating the Blues being cost effective.

4.2.3 The Assessment Group developed a decision analytic model, which was used to assess the cost effectiveness of the three products (Beating the Blues, COPE and Overcoming Depression). The model compared CCBT with TAU over an 18-month period. The main model results were based on the initial distribution over the three severity classes from the Beating the Blues trial. A subgroup analysis was performed to examine the variation in cost effectiveness by severity of depression. The Assessment Group presented two scenarios in the model: one assumed patients received one cycle of CCBT, and the second assumed 70% of patients relapsed and received a second cycle of CCBT; this relapse rate equals the relapse rate for traditional CBT. A sensitivity analysis was carried out for the one cycle of CCBT.

4.2.4 The Assessment Group reported the costs resulting from the following: licence fees, computer hardware, screening of patients for suitability, clinical support, capital overheads and training of staff. The basic principles of costing were very similar for all three products; however, the Assessment Group assumed a lower screening time for COPE. CCBT also had an impact on reducing the level of depression compared with TAU, which has consequences for use of other services.

4.2.5 The Assessment Group used utility values obtained from a study that provided data on 62 patients with BDI total scores and Euroqol EQ-5D. Mean scores for three depression categories were estimated: mild to moderate 0.78 (SD 0.20), moderate to severe 0.58 (SD 0.31) and severe 0.38 (SD 0.32). For the minimal category, it was assumed that the mean score was 0.88 (SD 0.22).

Beating the Blues

4.2.6 Mean package costs (cost per patient) for Beating the Blues were estimated at £219.30 for a single-copy licence and £104.62 for a 20-copy licence. These estimates come with large ranges, reflecting the uncertainties around the unit costs and the expected numbers of treated patients at practice level. The transition probabilities for Beating the Blues were estimated from changes on the BDI in the pivotal trial.

4.2.7 The CQG of Beating the Blues over TAU was reported as £1801 for the single-copy licence (assumed mean number of patients treated 37.5). If the licence was to be offered to PCTs for 20 copies (assumed mean number of patients treated 750) the CQG would fall to £415. Running the model for one cycle increased the CQG to £4961. For the subgroup analysis, the results showed that the mild to moderate group had the lowest mean CQG of £1802, but there was little difference among groups.

COPE

4.2.8 Two costings were undertaken for practice level licences, one assuming that the practice would provide computer access and the other assuming that patients could access the Internet from home or some other location (cost-free to the NHS). Both options included the cost for a telephone support line. The estimated mean cost per patient was £171.30 for no practice computer access and £195.86 with practice computer access. At PCT level, the cost fell to £110.53. These estimates come with large ranges, reflecting the uncertainties around the unit costs and the expected numbers of treated patients at practice level. For COPE, the transition probabilities were estimated from one of the non-comparative trials and no individual level data were available.

4.2.9 The CQG of COPE over TAU was reported as £7139 when modelled using a GP practice licence (assumed mean number of patients treated 37.5). If the licence were to be offered to PCTs (assumed mean number of patients treated 750), the CQG would fall to £3915. Limiting the model to one cycle increased the CQG to £16,469.

Overcoming Depression: a Five Areas Approach

4.2.10 Costs were estimated for a single licence with one and two copies and a PCT licence of 20 copies at £72.64 and £66.64 per treated patient, respectively. These estimates come with large ranges, reflecting the uncertainties around the unit costs and the expected numbers of treated patients at practice level. For Overcoming Depression, the transition probabilities were estimated from individual level data provided by the non-comparative trial.

4.2.11 The incremental CQG of Overcoming Depression over TAU was £5391 when modelled using a GP practice licence (assumed mean number of patients treated 37.5). If the licence was to be offered to PCTs (assumed mean number of patients treated 750), the CQG would fall to £4856. Limiting the model to one cycle increased the CQG to £26,087.

Panic/phobia

FearFighter

4.2.12 The Assessment Group developed a discrete-state Markov model in which patients were assumed to be either well or having panic/phobia. The model runs for four cycles, each lasting 3 months. The costs associated with the product in terms of licence fees, computer hardware, screening of patients for suitability, clinical support, capital overheads and the training of staff were the same as those for netCOPE. Utilities data were obtained from the European Study of the Epidemiology of Mental Disorders (ESEMeD) survey.

4.2.13 The estimated mean cost of FearFighter was £171.30 per patient if patients could access the Internet from home or some other location, and this increased to £195.86 if the practice had to provide computer access. At PCT level, the cost fell to £110.53 per patient. These estimates come with large ranges, reflecting the uncertainties around the unit costs and the expected numbers of treated patients at practice level. If the disorder resulted in a lower throughput than for depression, then the average costs would be higher than for netCOPE.

4.2.14 No clear dominance between interventions was reported. FearFighter achieved a CQG of £2380 over relaxation when modelled using a GP practice licence (assumed mean number of patients treated 37.5). TCBT, in the same GP practice licence scenario, resulted in a CQG over FearFighter of £17,608. A sensitivity analysis using costs at PCT level (assumed mean number of patients treated 750) resulted in the CQG of FearFighter over relaxation being reduced to £901 and the CQG of TCBT over FearFighter being increased to £25,432.

4.2.15 As part of the response to the Appraisal Consultation Document (ACD) the manufacturer of FearFighter submitted an economic analysis by the same author as that of the published analysis for Beating the Blues. Costs for FearFighter and TCBT used in this model were similar to those used by the Assessment Group. Outcomes analysed in the model were self-rating of (i) the patient's main problem and (ii) global phobia. Cost effectiveness was presented as cost per unit improvement in either one of these two scales. A 'net-benefit' approach was used to present cost-effectiveness acceptability curves based on a range of hypothetical values that society would place on a unit improvement on the two scales. Valuing a one-unit improvement in the main problem score at £60 was associated with a 50% chance of FearFighter being cost effective over relaxation. No incremental analysis of TCBT versus CCBT was presented, nor was a CQG estimate.

Obsessive-compulsive disorder

BTSteps (now OCFighter)

4.2.16 The Assessment Group developed a decision-tree model for OCD with two cycles. The model runs for 18 months. The costs associated with the product in terms of licence fees, computer hardware, screening of patients for suitability, clinical support, capital overheads and the training of staff were the same as for netCOPE. The only difference was that the number of patients with OCD was significantly lower. The lower throughput of BTSteps compared with COPE resulted in a lower level of helpline support required per copy of the programme used. Otherwise, the total costs were the same as for COPE. This resulted in costs per treated patient that were substantially higher than the other CCBT products. For BTSteps, the model drew heavily on one of the RCTs. The Assessment Group found little evidence on the health state utility values of people with OCD but used the outcomes of the ESEMeD survey in which changes in scores on the obsessive scale of the Y-BOCS were correlated with changes mapped on the EQ-5D. They found that a one-point reduction in the obsessive scale was equivalent to a 0.04 reduction in the EQ-5D preference scale. The Assessment Group subsequently used changes on the Y-BOCS from the pivotal trial to define a group of responders and non-responders and calculated relevant EQ-5D values: 0.92 (SD 0.07) for responders (that is, they have a Y-BOCS value equivalent to a post-treatment score of 16) and 0.80 (SD 0.15) for non-responders (that is, they have a Y-BOCS value equivalent to the mean treatment score assumed to be 25).

4.2.17 Two costings were carried out for a practice level licence, one assuming that practices would provide computer access and the other assuming that the patients could access the Internet from home or some other location. The estimated mean cost per patient was £837.23 and £714.49, respectively. At PCT level, assuming the recruitment of 249 patients, the cost fell to £248.83 per patient. These estimates come with large ranges, reflecting the uncertainties around the unit costs and the expected numbers of treated patients at practice level.

4.2.18 In the base-case analysis, when modelled using a GP practice licence (assumed mean number of patients treated 7.5), BTSteps was dominated by TCBT and had an incremental CQG of £52,000 versus relaxation. A sensitivity analysis using costs at PCT level (assumed mean number of patients treated 249) resulted in a CQG for TCBT over BTSteps of £22,484, and of BTSteps over relaxation of £15,581. An additional, intermediate, scenario was also reviewed that included a maximum licence fee of £200 per patient for the GP practice licence. This intermediate scenario resulted in BTSteps being dominated by a scenario in which a proportion of patients receive TCBT and a proportion of patients receive relaxation.

4.2.19 As part of the response to the ACD the manufacturer of BTSteps submitted an economic analysis by the same author as that of the published analysis for Beating the Blues. BTSteps was assumed to cost £200 per person (the price charged by the manufacturer for a minimum of 30 users). Ten 1-hour sessions of therapist time were provided to the TCBT group costed at £69 per hour. As outcome measure the model used incremental changes in scores on the Y-BOCS from baseline. Cost effectiveness was presented as cost per unit improvement on the Y-BOCS. A 'net-benefit' approach was used to present cost-effectiveness acceptability curves based on a range of hypothetical values that society would place on a unit improvement on the Y-BOCS. Valuing a one-unit improvement in the Y-BOCS score at £50 was associated with a 50% chance of BTSteps being cost effective over relaxation. No incremental analysis of TCBT versus CCBT was presented; nor was a CQG estimate.

4.3 Consideration of the evidence

4.3.1 The Committee reviewed the data available on the clinical and cost effectiveness of CCBT for the management of depression and anxiety, having considered evidence on the nature of the condition and the value placed on the benefits of CCBT by people with depression and anxiety, those who represent them, and clinical experts. It was also mindful of the need to take account of the effective use of NHS resources.

4.3.2 The Committee noted that NICE guidelines have been published for the management of depression in primary and secondary care and for the management of anxiety (panic disorder, with or without agoraphobia, and GAD) in adults in primary, secondary and community care. Both guidelines place CCBT as an option only within step 2 of a stepped-care approach. The Committee also heard testimony from the patient and clinical experts that there are several methods available for delivering CBT in the NHS and that the comparators used in the studies reviewed had excluded some important ones such as group CBT and bibliotherapy. The Committee appreciated that CCBT would not necessarily be the best delivery method for CBT for all patients. However, the Committee heard that enabling patient choice between various methods of delivery of CBT was likely to increase the motivation to comply with treatment. The Committee was also informed by the experts that, in general, the CCBT packages being appraised could be considered as being as 'safe' as CBT in that the information and guidance delivered by these packages were similar to that given in a standard CBT approach. The experts also stated that CCBT would not be appropriate for the management of severe depression.

4.3.3 The Committee carefully considered all the comparators that had been used in the trials. The Committee heard testimony from the clinical experts that TAU was an appropriate comparator for CCBT for patients with mild and moderate depression, but that relaxation was considered an imperfect comparator for the management of panic and phobia and OCD. The Committee also noted that TCBT could be considered as an appropriate comparator for all three disease areas but took on board the issue of the lack of availability of trained therapists and consequently the long waiting times for treatment.

Depression

4.3.4 The Committee noted that the evidence base for Beating the Blues had improved since the previous appraisal. In particular, more data relating to the single RCT had become available. The results from this RCT showed that Beating the Blues was more effective than TAU on a number of outcome measures, and the ES for the Beck Depression Inventory was particularly notable. The Committee noted some limitations in the evidence base which were only partially reflected in the uncertainty around the cost-effectiveness estimates. However, the Committee was persuaded that the evidence base was sufficient for the CQG estimate for Beating the Blues to be acceptable.

4.3.5 The Committee noted that there is no RCT evidence for COPE or Overcoming Depression for the management of depression. Therefore, the Committee could not establish with a reasonable degree of certainty that either of these packages is a clinically or cost-effective method of treating people with depression over and above other management options such as TAU. Furthermore, it was not able to conclude that the CCBT packages for depression could be considered to be equivalent as in a 'class', because of the differences between the packages' presentation, style and complexity.

4.3.6 Therefore, on the basis of currently available data, the Committee concluded that, of the three CCBT packages for depression, only Beating the Blues could currently be recommended as an option for delivering CBT in the management of mild and moderate depression as outlined in the current NICE clinical guideline for the stepped-care management of depression in primary and secondary care (see Section 8 – Related guidance).

Panic/phobia

4.3.7 The Committee considered the evidence base for FearFighter for the management of panic and phobia, and noted that there was one relevant RCT that established that FearFighter was more effective than TCBT on one outcome measure, but less effective on others.

4.3.8 The Committee noted that the use of FearFighter could be considered to be cost effective versus relaxation. It was mindful, however, of the fact that the comparison of TCBT versus CCBT resulted in a CQG of £18,000 (with a sensitivity analysis increasing this to £25,000 depending on the purchasing scenario). On this basis, TCBT would be the preferred option for the management of panic/phobia. However, the Committee considered that, not only were both TCBT and CCBT to some extent effective, but also that there was substantial uncertainty in the greater effectiveness of TCBT implied by the economic model.

4.3.9 The Committee concluded that it was acceptable to consider FearFighter as an option for delivering CBT in the management of panic/phobia as outlined in the current NICE clinical guideline for the stepped-care management of anxiety (panic disorder, with or without agoraphobia, and generalised anxiety disorder) in adults in primary, secondary and community care (see Section 8 – Related guidance).

Obsessive-compulsive disorder

4.3.10 The Committee considered the RCT evidence for BTSteps for the management of OCD in which BTSteps was compared with TCBT and relaxation. The Committee was concerned with the methodology reported for the RCT with regard to aspects of the study design. The Committee noted that in the randomised clinical trials BTSteps was never more effective than TCBT. It also noted that patients were more satisfied with TCBT than with BTSteps.

4.3.11 Regarding the cost effectiveness the Committee noted that there were a number of scenarios modelled for purchasing BTSteps in the NHS. In all these scenarios TCBT was more cost effective than BTSteps and relaxation. Only in the scenario in which an average PCT has to purchase a bulk licence is BTSteps more cost effective than relaxation. However, the Committee did not consider that an average PCT should reasonably seek to treat a mean number of 250 patients with BTSteps when TCBT is the most cost-effective option for that PCT to deliver CBT.

4.3.12 The Committee considering both the clinical and cost-effective evidence concluded that BTSteps should not be recommended as an option for delivering CBT in the management of OCD.