4 Outcomes

The Diagnostics Advisory Committee (section 8) considered evidence from a number of sources (section 9). Full details of all the evidence are in the Committee papers.

How outcomes were assessed

4.1 The assessment consisted of a systematic review of the evidence on test performance and clinical‑effectiveness data for the LightCycler SeptiFast Test MGRADE, SepsiTest and IRIDICA BAC BSI and comparator tests.

4.2 Studies were included if they evaluated 1 of the interventions, compared with either blood culture or blood culture with MALDI‑TOF mass spectrometry (MS), to analyse whole blood samples collected from people being treated for suspected sepsis. Studies that compared 1 of the interventions with another intervention were also included. In total, 66 studies met the inclusion criteria. Diagnostic‑accuracy data were reported in 62 of the 66 studies and were included in meta‑analyses, which were based on a bivariate normal model with Markov Chain Monte Carlo simulation. Inter‑study heterogeneity was explored using meta‑regression. Intermediate or clinical outcome measures were reported in 41 of the 66 studies and were included in a narrative analysis.

4.3 Sixty four of the 66 studies were single‑index test, single‑gate studies, that is, studies in which only patients with the target condition (suspected sepsis) were recruited. Three of these 64 studies were randomised controlled trials (RCTs). The remaining 2 studies were single‑gate studies that reported results for both the LightCycler SeptiFast Test MGRADE and SepsiTest.

4.4 Only 3 of the 66 studies included patients from the UK. Most of the studies were done in other European countries. Of the studies that included patients from the UK, 1 study (Dark et al. 2009) used the SeptiFast assay to test 50 patients and 1 other study (Vincent et al. 2015) used the IRIDICA assay to test 529 patients from 6 European countries. The third UK study (Warhurst et al. 2015) reported the use of SeptiFast in 795 patients with sepsis and was judged to be the highest quality and most applicable included study.

4.5 All studies were assessed using the QUADAS‑2 tool. The results of 65 of the 66 studies were considered to be at risk of bias and may not be applicable to the decision problem. The issues of greatest uncertainty included patient selection and blinding to the index test or reference standard. The External Assessment Group also reported concerns about 21 of the 66 studies, which did not report whether the blood samples for the index test and reference standard were drawn at the same time, and 6 of the 66 studies, which used a mixture of reference standards. In addition, only 28 of the 66 studies reported using blood sampling and test methods that were in accordance with the company's instructions for use. Studies also reported different units of analysis for diagnostic‑accuracy data, such as per patient, per sample, per episode of sepsis, and species or pathogen level.

Diagnostic accuracy

4.6 Of the 62 studies that reported diagnostic‑accuracy data, 55 reported data for the LightCycler SeptiFast Test MGRADE; 5 reported data for SepsiTest; and 4 reported data for the IRIDICA BAC BSI assay. Two of the 62 studies that reported data for both the Light Cycler SeptiFast Test MGRADE and SepsiTest were counted as individual studies for each test.

LightCycler SeptiFast Test MGRADE

4.7 There were 54 studies that compared the LightCycler SeptiFast Test MGRADE with blood culture and were combined in a meta‑analysis. The pooled estimate for sensitivity was 0.65 (95% credible interval [CrI] 0.60 to 0.71; 95% prediction interval 0.29 to 0.90) and for specificity was 0.86 (95% CrI 0.84 to 0.89; 95% prediction interval 0.62 to 0.96). The proportion of discordant results varied across studies from 6% to 46% (median 17%).

4.8 One study (Tafelski et al. 2015) compared the LightCycler SeptiFast Test MGRADE with blood culture plus MALDI‑TOF MS. It reported a sensitivity of 0.58 (95% confidence interval [CI] 0.30 to 0.86) and a specificity of 0.74 (95% CI 0.64 to 0.85).

4.9 Reasons for heterogeneity in sensitivity and specificity estimates between studies were explored using meta‑regression for clinically relevant variables. The following variables were explored:

  • age (neonates and children)

  • exposure to antibiotics before blood sample collection

  • suspected community- or healthcare‑acquired infection

  • febrile neutropenia

  • studies with inclusion or exclusion of contaminants.

    There was no evidence that sensitivity and specificity estimates were affected by these variables.

SepsiTest

4.10 Four studies compared SepsiTest with blood culture and were combined in a meta‑analysis. The pooled estimate for sensitivity was 0.48 (95% CrI 0.21 to 0.74; 95% prediction interval 0.07 to 0.90) and for specificity was 0.86 (95% CrI 0.78 to 0.92; 95% prediction interval 0.66 to 0.95). The proportion of discordant results varied between studies and ranged from 14% to 26% (median 22%).

4.11 One study (Loonen et al. 2014) compared SepsiTest with blood culture plus MALDI‑TOF MS. The study reported a sensitivity of 0.11 (95% CI 0.00 to 0.23) and specificity of 0.96 (95% CI 0.92 to 1.00). No subgroup analyses were possible for the SepsiTest.

IRIDICA BAC BSI

4.12 Four studies compared the IRIDICA BAC BSI assay with blood culture and were combined in a meta‑analysis. Two of these studies reported data using an earlier version of the IRIDICA PCR/ESI‑MS analyser known as the PLEX‑ID system, which has different desalter and mass spectrometry modules. The pooled estimate for sensitivity was 0.81 (95% CrI 0.69 to 0.90; 95% prediction interval 0.55 to 0.94) and for specificity was 0.84 (95% CrI 0.71 to 0.92; 95% prediction interval 0.50 to 0.96). The proportion of discordant results varied between studies and ranged from 7% to 30% (median 18%).

4.13 No studies compared the IRIDICA BAC BSI assay with blood culture plus MALDI‑TOF MS and no subgroup analyses were possible for this intervention.

Intermediate and clinical outcomes

4.14 There were 41 studies included that reported data relating to the time to pathogen identification for the index test, time to treatment, test‑failure rate, mortality, duration of intensive care unit or hospital stay, duration of antibiotic therapy or reported changes in antimicrobial treatment plan. None of the included studies reported data on re‑admission rates, adverse events associated with broad‑spectrum antimicrobial use, morbidity, changes in disease severity over time, rates of superinfection, rates of resistant infection, or health‑related quality of life.

Light Cycler SeptiFast Test MGRADE

4.15 There were 37 studies that reported data on intermediate and clinical outcomes for the LightCycler SeptiFast Test MGRADE. In addition, 1 study (Schreiber et al. 2013) reported data for both the LightCycler SeptiFast Test MGRADE and SepsiTest. No studies compared the LightCycler SeptiFast Test MGRADE with the IRIDICA BAC BSI assay.

Time to result (pathogen identification)

4.16 There were 21 studies using the LightCycler SeptiFast Test MGRADE that reported turnaround times of a minimum of 4 hours to a median of 26.25 hours for pathogen identification. Some of these studies also reported the time for pathogen identification using blood cultures, which ranged from a turnaround time of a minimum of 24 hours to a median of 80 hours.

Time-to-treatment change

4.17 Time-to-treatment change for the LightCycler SeptiFast Test MGRADE was reported in 3 RCTs:

  • Tafelski et al. (2015) reported a mean time of 18.8 hours (standard deviation [SD] 5.6) from taking the blood sample to changing treatment using the LightCycler SeptiFast Test MGRADE and a mean time of 38.3 hours (SD 14.5) using blood culture and MALDI‑TOF MS.

  • Rodrigues et al. (2013) reported a mean time of 9.7 hours from taking the blood sample to a change in treatment using the LightCycler SeptiFast Test MGRADE compared with a mean time of 50.1 hours using blood culture (p=0.004).

  • Idelevich et al. (2015) reported a mean time to changing treatment of 21.4 hours (range 16.2 to 46.3 hours) in the LightCycler SeptiFast Test MGRADE group compared with 47.5 hours (range 7.3 to 59.2 hours) in the blood culture group (p=0.018).

Test‑failure rates

4.18 There were 7 studies that reported test‑failure rates for the LightCycler SeptiFast Test MGRADE, which ranged from 1.5% to 24.2%. It is not clear why there is a large variation in failure rates between studies.

Duration of stay in intensive care unit, hospital or both

4.19 Duration of stay in an intensive care unit, or hospital, or both were reported in 13 studies that compared the LightCycler SeptiFast Test MGRADE with blood culture. In most of these studies, it was unclear if the duration of stay was recorded from before, during or after blood sampling. Also, most of the studies did not present comparative data. Of the 4 studies that did report between group differences, 1 study (Alvarez et al. 2012) reported a statistically significant difference (p<0.05) in intensive care unit and hospital duration of stay in favour of the LightCycler SeptiFast Test MGRADE. Three other studies (Idelevich et al. 2014; Mancini et al. 2014; Rodrigues et al. 2013) reported no significant difference in duration of stay.

Duration of broad- and narrow‑spectrum antibiotic therapy

4.20 One RCT (Tafelski et al. 2015) reported a duration of empirical antimicrobial therapy (antibiotics which are prescribed based on clinical presentation) of 18.8 hours (SD ±5.6) for patients in the LightCycler SeptiFast Test MGRADE group compared with 38.3 hours (SD ±14.5) for patients in the blood culture with MALDI‑TOF MS group.

Change in antimicrobial treatment

4.21 There were 14 studies that reported details of change in antimicrobial treatment, 10 of which did not report comparative data. Three studies compared the LightCycler SeptiFast Test MGRADE with blood culture. One RCT (Rodrigues et al. 2013) reported that therapy was adjusted for 35% of patients in the LightCycler SeptiFast Test MGRADE group compared with 24% of patients in the blood culture group. In contrast, a further RCT (Idelevich et al. 2015) reported that 9.5% of patients in the LightCycler SeptiFast Test MGRADE had an adjustment to therapy compared with 10.5% in the blood culture group. One study based on propensity score matching (Mancini et al. 2014) reported no differences in management.

4.22 One RCT (Tafelski et al. 2015) compared the LightCycler SeptiFast Test MGRADE with blood culture plus MALDI‑TOF MS. Testing with the LightCycler SeptiFast Test MGRADE resulted in a change of treatment for 9.8% of patients compared with 13.5% of patients in the blood culture plus MALDI‑TOF MS group.

Mortality

4.23 Mortality data were reported in 17 studies, 12 of which reported data on a cohort level only. The mortality rates reported ranged from 4% to 61%; but the length of follow‑up was highly variable across the studies. One study (Alvarez et al. 2012) reported no statistically significant differences between the LightCycler SeptiFast Test MGRADE and blood culture for both 28‑day and 6‑month mortality. One other study (Rodrigues et al. 2013) also reported no statistically significant difference in 28‑day mortality.

4.24 One propensity score matching study (Mancini et al. 2014) reported no statistically significant difference in mortality (p=0.39) between a prospective cohort (LightCycler SeptiFast Test MGRADE) and retrospective cohort (blood culture). Although, when more strict matching criteria were applied, the LightCycler SeptiFast Test MGRADE was associated with a statistically significant reduction in mortality (3.13% compared with 14.71%; p=0.04). A reduction in mortality associated with using the LightCycler SeptiFast Test MGRADE was reported in 2 further studies (Idelevich et al. 2015; Tafelski et al. 2015), but the reductions were not statistically significant.

SepsiTest

Mortality

4.25 One study (Loonen et al. 2014) reported a mortality rate of 3.2% for the study cohort but the duration of follow‑up was not reported. In addition, Schreiber et al. (2013) reported an intensive care unit mortality rate of 16% and a 28‑day mortality rate of 24% for the study cohort.

4.26 No other intermediate or clinical‑outcome data were reported for the SepsiTest.

IRIDICA BAC BSI

Test‑failure rates

4.27 One study, which was unpublished at the time of guidance development, reported data relevant to test‑failure rates for the IRIDICA BAC BSI assay. These data are considered to be academic in confidence and cannot be reported at this time.

Change in antimicrobial‑treatment plan

4.28 One study (Vincent et al. 2015) reported that an adjudication panel of 3 clinical experts retrospectively recommended a change in management based on the IRIDICA BAC BSI assay for 41% of all patients. This increased to 57% of patients when the IRIDICA BAC BSI assay was positive and blood culture was negative.

Mortality

4.29 One study (Vincent et al. 2015) reported a mortality rate of 29% for the study cohort, but did not report the duration of follow‑up.

Costs and cost effectiveness

4.30 The External Assessment Group conducted a search to identify studies investigating the cost effectiveness of the LightCycler SeptiFast Test MGRADE, SepsiTest or the IRIDICA BAC BSI assay. The External Assessment Group also constructed a conceptual economic model to determine the cost effectiveness of the technologies.

Systematic review of cost-effectiveness evidence

4.31 Four studies were included and were assessed according to their relevance to the decision problem: 3 studies included the LightCycler SeptiFast Test MGRADE, 2 of which were within‑study cost‑minimisation analyses (that is, a cost‑minimisation analysis conducted within a clinical study), and 1 was a cost‑effectiveness analysis. The remaining study included a cost‑minimisation analysis of the IRIDICA PLEX‑ID hybrid assay. The target population, condition and setting varied across the 4 studies.

4.32 The 2 studies that were within‑study cost‑minimisation analyses of using the LightCycler SeptiFast Test MGRADE when compared with blood culture reported cost savings of €178.75 per sample (Mancini et al. 2014) and €183.00 per patient (Alvarez et al. 2012). The third study, Lehmann et al. (2010), reported incremental cost‑effectiveness ratios (ICERs) of €11,477 per incremental survivor and €3,107 per quality‑adjusted life year (QALY) gained when using the LightCycler SeptiFast Test MGRADE compared with blood culture. When the use of an IRIDICA‑PLEX‑ID hybrid system was compared with blood culture, Bilkovski et al. (2014) reported cost savings of $1,123,372 per 422 tests. None of the studies considered the effect of a potential reduction in antibiotic resistance. The External Assessment Group concluded that the existing economic evaluations had limited relevance to either the UK or the decision problem because of differences in patient populations, costs of the interventions and standard care. In particular, Mancini et al. (2014) included haematology patients for whom relatively expensive empirical antifungals were prescribed that are unlikely to be representative of the UK treatment pathway.

Economic analysis

4.33 The External Assessment Group developed a conceptual economic model designed to explore the cost effectiveness of the LightCycler SeptiFast Test MGRADE, SepsiTest and the IRIDICA BAC BSI assay. The population included in the model was hospitalised patients with suspected bloodstream infection.

Model structure

4.34 The model comprised a decision tree with a lifetime time horizon and took the perspective of the NHS and personal social services. The key clinical outcomes included in the model were 30‑day mortality, duration of stay in intensive care unit, duration of hospital stay and antimicrobial treatment.

Model inputs

4.35 Data on the diagnostic accuracy of the interventions, intermediate outcomes and clinical outcomes were taken from the clinical‑effectiveness systematic review when possible. Expert opinion was also sought to populate key clinical outcomes and supplement the data available from the systematic review. Routine sources of costs and prevalence data were also used when appropriate. A discount rate of 3.5% per annum was applied to both costs and effects. The potential effect of the tests on antimicrobial stewardship was not included in the model, because there was insufficient evidence to show how the tests would affect antimicrobial use and the subsequent development of resistant organisms.

Costs

4.36 The incremental cost per test was calculated using the cost of the test, the net effect on duration of intensive care unit and hospital stay, and changes in the costs of antimicrobial treatment. The estimated cost per day for an intensive care unit bed was £1057 and for a general ward bed was £275. A course of empirical antimicrobial treatment was estimated to cost £350.

4.37 It was assumed that the cost per test was dependent upon both test throughput and whether laboratory equipment needed to be bought to use the tests. The range of technology costs included in the model were as follows:

  • LightCycler SeptiFast Test MGRADE £153.67 to £205.54

  • SepsiTest £108.30 to £149.53

  • IRIDICA BAC BSI £197.35 to £314.61

  • MALDI‑TOF MS £6.94 to £232.39.

Health-related quality of life and QALY decrements

4.38 Incremental QALYs were calculated by assuming 11.32 discounted QALYs per 30‑day mortality avoided, based on the estimated number of discounted life years for an adult patient with sepsis and the estimated quality of life after an episode of sepsis. The model assumed a mean age of 58 years and that 60% of the cohort were male. Patients were assumed to have a utility value of 0.68 at 5 years after an episode of severe sepsis (Cuthbertson et al. 2013) unless the utility value predicted for the age and sex profile of a patient in the general population was lower. In these instances, the lower utility value was applied.

Economic-analysis results

4.39 Five deterministic analyses were done:

  • Base case 1: interventions compared with blood culture, with clinical‑outcome data taken from the systematic review.

  • Base case 2: interventions compared with blood culture, with clinical‑outcome estimates taken from expert opinion.

  • Threshold analyses.

  • Interventions compared with MALDI‑TOF MS.

  • Data taken from studies comparing more than 1 intervention.

4.40 The following assumptions were common to all analyses:

  • The only parameter to affect QALY gain or loss was 30‑day mortality rate.

  • Negative rapid tests did not affect any of the 4 key outcomes.

  • Failed rapid tests did not affect any of the 4 key outcomes.

  • If 2.4 tests per day were run, laboratories ran tests Monday to Friday only, with 3 times the number of tests run on Monday to account for samples building up over a weekend.

  • If 17 or 68 tests per day are run, laboratories did 3 runs per day and worked 24 hours a day, 7 days a week.

  • The purchase cost of machines needed for the interventions and comparators was equally divided over 7 years of use.

  • It was assumed that no additional staff costs or laboratory estate costs were incurred when using the interventions.

  • The time scale of testing was 1 year although discounted QALYs accrued in subsequent years were included.

  • Incremental QALYs were accrued through the number of avoided 30‑day mortalities.

  • If accuracy data from Warhurst et al. (2015) were used, the LightCycler SeptiFast Test MGRADE had a failure rate of 6.9%. A failure rate of 1.4% was assumed when pooled accuracy data was used.

  • IRIDICA BAC BSI had a failure rate of 1.9%.

  • SepsiTest had a failure rate of 0%.

  • Patients were treated with either 18 g per day of piperacillin/tazobactam or 3 g per day of meropenem for 7 days.

  • 30‑day mortality rates were assumed to be either 13% or 29%.

  • MALDI‑TOF MS was only used on positive samples (8.7% of all blood cultures).

  • MALDI‑TOF MS had a sensitivity of 79.8% at species level compared with blood culture.

  • LightCycler SeptiFast test MGRADE diagnostic-accuracy data were derived from Warhurst et al. (2015) unless otherwise specified.

  • SepsiTest and IRIDICA BAC BSI diagnostic-accuracy data were derived from the External Assessment Group's meta‑analyses unless otherwise specified.

Base-case-1 results

4.41 In this analysis, clinical‑outcome data from the clinical‑effectiveness review were included. This resulted in the assumption that there were no clinical benefits associated with the interventions for 30‑day mortality, duration of stay in the intensive care unit or duration of stay in hospital. The costs of antimicrobials were also unchanged in this analysis. All interventions were compared with blood culture only.

4.42 The results of the analysis showed that all the interventions were dominated by blood culture (that is, blood culture was less expensive and more effective than all of the interventions). Regardless of the test throughput assumed in different scenarios, the interventions remained dominated (more expensive with no additional clinical benefit) because of the lack of QALYs gained.

4.43 In addition, a threshold analysis was done for base case 1 to assess the reduction in antimicrobial costs that would be needed for each intervention to be cost neutral. The results suggested that the reductions needed would be 44% to 59% for the LightCycler SeptiFast Test MGRADE, 31% to 43% for the SepsiTest and 56% to 90% for the IRIDICA BAC BSI, although the rate of positive tests associated with each intervention suggested that their costs could not be offset solely by a reduction in antimicrobial therapy use.

Base-case-2 results

4.44 In this analysis, the key clinical‑outcome parameters were populated using an average of estimated values provided by clinical experts. The External Assessment Group used these values in a range of scenarios that assumed a 30‑day mortality rate of either 13% or 29%, a throughput of 2.4, 17 or 68 tests per day and a maximum acceptable ICER of £20,000 or £30,000 per QALY gained. The comparator used in this analysis was blood culture.

4.45 For each scenario, the net monetary benefit of each intervention was estimated. A positive net monetary benefit suggests that the benefits associated with the intervention outweigh the costs, and the intervention with the largest net monetary benefit is estimated to be the most cost effective. MALDI‑TOF MS was also included in the analysis to estimate the relative cost effectiveness between the 2 comparators included in the assessment.

4.46 In all scenarios modelled, MALDI‑TOF MS produced a positive net benefit compared with blood culture. In 1 scenario (30‑day mortality rate 13%, 2.4 tests per day, maximum acceptable ICER of £20,000 per QALY gained), SepsiTest had the highest net monetary benefit when it was assumed that equipment to run the test had to be bought. In the same scenario, the IRIDICA BAC BSI assay had the highest net monetary benefit when only the test reagents and consumables were purchased. In all other modelled scenarios, the IRIDICA BAC BSI assay had the highest net monetary benefit.

4.47 ICERs were also calculated using the data from expert opinion. When it was assumed that no additional equipment had to be bought or the 30‑day mortality rate was 29%, the ICERs became more favourable because of either a decrease in incremental costs or an increase in incremental QALY gain.

4.48 The External Assessment Group also explored the effect of applying the pooled estimates of sensitivity and specificity from the meta‑analyses to the LightCycler SeptiFast Test MGRADE. This assumption produced more favourable ICERs for the LightCycler SeptiFast Test MGRADE through increasing the estimated sensitivity of the test (65% pooled estimate compared with 51% from Warhurst et al. 2015), while maintaining specificity at 86%.

Threshold analyses

4.49 The External Assessment Group used a range of threshold analyses to explore the effect of key clinical outcomes. In all analyses, it was assumed that the comparator equipment had already been bought but that the equipment for the interventions needed to be bought. The threshold levels resulting from the analyses, which assumed 2.4 tests run per day and a maximum acceptable ICER of £20,000 per QALY gained, suggested reductions in 30‑day mortalities ranging from 0.09 to 0.14 per 100 tests would be needed for the interventions to be considered cost effective compared with blood culture. Antimicrobial costs would need to reduce by £149.53 to £314.61 per 100 tests. The results were similar when the interventions were compared with MALDI‑TOF MS. The threshold analyses that assumed either 17 or 68 tests run per day produced lower threshold values. The values of the reductions needed were also lower when a maximum acceptable ICER of £30,000 per QALY gained was assumed.

Cost effectiveness of the LightCycler SeptiFast Test MGRADE and SepsiTest compared with MALDI‑TOF MS

4.50 The External Assessment Group also explored the cost effectiveness of both the LightCycler SeptiFast Test MGRADE and SepsiTest compared with MALDI‑TOF MS, based on data from 2 studies (Tafelski et al. 2015; Loonen et al. 2014) that used MALDI‑TOF MS in addition to blood culture. The effect estimates based on expert opinion were also included in the analysis. It was assumed that both interventions had a failure rate of 0% and that equipment to run the tests needed to be bought. The results of these analyses suggested that when compared with MALDI‑TOF MS (and blood culture), the LightCycler SeptiFast Test MGRADE dominated (less costly and more effective) MALDI‑TOF MS (and blood culture), and SepsiTest had ICERs ranging from £23,375 to £34,848 per QALY gained with a 30‑day mortality rate of 13% and from £10,479 to £15,621 per QALY gained with a 30‑day mortality rate of 29%.

Results from studies comparing the LightCycler SeptiFast Test MGRADE and SepsiTest simultaneously with blood culture

4.51 An analysis was run using data from 2 studies (Schreiber et al. 2013; Leitner et al. 2013), which evaluated both the LightCycler SeptiFast Test MGRADE and SepsiTest with blood culture. The analysis was done to compare the relative cost‑effectiveness estimates with those derived in base case 2 that were based on indirect comparisons of the relative effectiveness of the interventions from expert opinion. The analysis assumed a 0% test‑failure rate for both interventions and that equipment to run the tests needed to be bought. A range of scenarios were presented with 30‑day mortality rates of 13% or 29% and a throughput of 2.4, 17 or 68 tests per day. In all scenarios, the ICER for the LightCycler SeptiFast Test MGRADE was greater than £30,000 per QALY gained when compared with SepsiTest.

  • National Institute for Health and Care Excellence (NICE)