Appendix E Methodology checklist: case–control studies

Appendix E Methodology checklist: case–control studies

Appendices B–G include checklists for those study designs that are expected to be used in the evidence reviews for NICE social care guidance. Other checklists can found in the NICE clinical guidelines manual and Methods for the development of NICE public health guidance.

Checklist

Study identification

Include author, title, reference, year of publication

Guidance topic:

Review question no:

Checklist completed by:

Section 1: Internal validity

Circle or highlight 1 option for each question

1.1

The study addresses an appropriate and clearly focused question.

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

Selection of participants

1.2

The cases and controls are taken from comparable populations

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

1.3

The same exclusion criteria are used for both cases and controls

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

1.4

What was the participation rate for each group (cases and controls)?

Cases:

Controls:

1.5

Participants and non-participants are compared to establish their similarities or differences

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

1.6

Cases are clearly defined and differentiated from controls

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

1.7

It is clearly established that controls are not cases

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

Assessment

1.8

Measures were taken to prevent knowledge of primary exposure from influencing case ascertainment

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

1.9

Exposure status is measured in a standard, valid and reliable way

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

Confounding factors

1.10

The main potential confounders are identified and taken into account in the design and analysis

Well covered

Adequately addressed

Poorly addressed

Not addressed

Not reported

Not applicable

Statistical analysis

1.11

Have confidence intervals been provided?

1.12 Overall assessment of internal validity. Are the study results internally valid?

Rate the study for internal validity below (for further information see notes on using the methodology checklist)

++

+

Comments:

Section 2: Overall assessment of external validity. Are the study results externally valid (i.e. generalisable to the source population)?

Consider participants, interventions, settings, comparisons and outcomes. Rate the study for external validity below (for further information see notes on use of the methodology checklist)

++

+

Comments:

Section 3: Description of the study

(This information is required for evidence tables to facilitate cross-study comparisons. Please complete all sections for which information is available.)

Please print clearly

3.1

How many people participated in the study?

List the numbers of cases and controls separately.

.

.

.

.

3.2

What are the main characteristics of the study population?

Include all characteristics used to identify both cases and controls – for example, age, sex, social class, level of need.

.

.

.

.

3.3

What environmental or prognostic factor is being investigated?

.

.

.

.

3.4

What comparisons are made?

Normally only 1 factor will be compared, but in some cases the extent of exposure may be stratified. Note all comparisons here.

.

.

.

.

3.5

For how long are participants followed up?

This is the length of time over which participant histories are tracked in the study.

.

.

.

.

3.6

What outcome measure(s) is/are used?

List all outcomes that are used to assess the impact of the chosen environmental or prognostic factor.

.

.

.

.

.

3.7

What size of effect is identified?

Effect size should be expressed as an odds ratio. If any other measures are included, note them as well. Include p-values and any confidence intervals that are provided.

.

.

.

.

3.8

How was the study funded?

List all sources of funding quoted in the article, whether government, voluntary sector or industry.

.

.

.

.

3.9

Does this study help to answer your guidance review question?

Summarise the main conclusions of the study and indicate how it relates to the review question.

.

.

.

.

Notes on use of the Methodology checklist: case–control studies

Case–control studies are designed to answer questions of the type 'What are the factors that caused this event?'. They involve comparison of individuals who have an outcome with other individuals from the same population who do not have the outcome. These studies start after the outcome of an event, and can be used to assess multiple causes of a single event. They are generally used to assess the causes of a new problem but they may also be useful for the evaluation of population-based interventions. .

The questions in section 1 are aimed at establishing the internal validity of the study under review – that is, making sure that it has been carried out carefully, and that any link between events and outcomes is clearly established. Each question covers an aspect of methodology that has been shown to make a significant difference to the conclusions of a study. In question 1.12 you are, based on the results from the study checklist, asked to provide an overall assessment of the internal validity of the study.

In section 2 you are then asked to rate the external validity of the study (also using ++, +, −). External validity assesses the extent to which the findings for the study participants apply to the whole 'source population' (that is, the population they were chosen from).

Case–control studies need to be designed very carefully – the complexity of their design is often not appreciated by investigators, and so many poor-quality studies are conducted. The questions in this checklist are designed to identify the main features that should be present in a well-designed study. There are few criteria that should, alone and unsupported, lead to rejection of a study. However, if a study fails to address or report on more than 1 or 2 of the questions in the checklist you should consider whether to reject the study

For each question in this section you should choose 1 of the following categories to indicate how well it has been addressed in the study:

  • well covered

  • adequately addressed

  • poorly addressed

  • not addressed (not mentioned, or this aspect of study design was ignored)

  • not reported (mentioned, but with insufficient detail to allow assessment to be made)

  • not applicable.

Question

1.1 The study addresses an appropriate and clearly focused question

Unless a clear and well-defined question is specified, it will be difficult to assess how well the study has met its objectives or how relevant it is to the question you are trying to answer.

Selection of participants

1.2 The cases and controls are taken from comparable populations

Study participants may be selected from the target population (all individuals to which the results of the study could be applied), from the source population (a defined subset of the target population from which participants are selected) or from a pool of eligible people (a clearly defined and counted group selected from the source population). A study that does not include clear definitions of the source population may be rejected.

1.3 The same exclusion criteria are used for both cases and controls

All selection and exclusion criteria should be applied equally to cases and controls. Failure to do so may introduce a significant degree of bias into the results of the study.

1.4 What was the participation rate for each group (cases and controls)?

Differences between the eligible population and the study participants are important because they may influence the validity of the study. A participation rate can be calculated by dividing the number of study participants by the number of people who are eligible to participate. It is more useful if it is calculated separately for cases and controls. If the participation rate is low, or there is a large difference in rate between cases and controls, the study results may be invalid because of differences between participants and non-participants. In these circumstances the study should be downgraded, and rejected if the differences are very large.

1.5 Participants and non-participants are compared to establish their similarities or differences

Even if participation rates are comparable and acceptable, it is still possible that the participants selected to act as cases or controls may differ from other members of the source population in some significant way. A well-conducted case–control study will look at samples of those not participating among the source population to ensure that the participants are a truly representative sample.

1.6 Cases are clearly defined and differentiated from controls

The method of selection of cases is of critical importance to the validity of the study. Investigators have to be certain that cases are truly cases, but must balance this with the need to ensure that the cases admitted into the study are representative of the eligible population. The issues involved in case selection are complex, and should ideally be evaluated by someone with a good understanding of the design of case–control studies. If there is no information on how cases were selected it is probably safest to reject the study as a source of evidence.

1.7 It is clearly established that controls are not cases

Just as it is important to be sure that cases are true cases, it is important to be sure that controls do not have the outcome under investigation. Controls should be chosen so that information on exposure status can be obtained or assessed in a similar way to that used for the selection of cases. If the methods of control selection are not described, the study may be rejected. If different methods of selection are used for cases and controls, the study should be evaluated by someone with a good understanding of the design of case–control studies.

Assessment

1.8 Measures were taken to prevent knowledge of primary exposure from influencing case ascertainment

If there is a possibility that case ascertainment was influenced by knowledge of exposure status, assessment of any association is likely to be biased. A well-conducted study should take this into account in the design of the study.

1.9 Exposure status is measured in a standard, valid and reliable way

The inclusion of evidence from other sources or previous studies that demonstrate the validity and reliability of the assessment methods, or the fact that the measurement method is a recognised procedure, should increase confidence in study quality.

Confounding factors

1.10 The main potential confounders are identified and taken into account in the design and analysis

Confounding is the distortion of a link between exposure and outcome by another factor that is associated with both exposure and outcome. The possible presence of confounding factors is one of the principal reasons why observational studies are not more highly rated as a source of evidence. The report of the study should indicate which potential confounders have been considered, and how they have been assessed or accounted for in the analysis. Social care practitioner (or healthcare professional where appropriate) judgement should be used to consider whether all likely confounders have been taken into account. If the measures used to address the potential effects of confounders are considered inadequate, the study should be downgraded or rejected, depending on how serious the risk of confounding is considered to be. A study that does not address the possibility of confounding should be rejected.

Statistical analysis

1.11 Have confidence intervals been provided?

Confidence intervals are the preferred method for indicating the precision of statistical results, and can be used to differentiate between an inconclusive study and a study that shows no effect. Studies that report a single value with no assessment of precision should be treated with caution.

Internal validity

1.12 Overall assessment of internal validity

Rate the study for internal validity according to the list below:

++ All or most of the checklist criteria have been fulfilled, where they have not been fulfilled the conclusions are very unlikely to alter

+ Some of the checklist criteria have been fulfilled, where they have not been fulfilled, or not adequately described, the conclusions are unlikely to alter

− Few or no checklist criteria have been fulfilled and the conclusions are likely or very likely to alter

External validity

Section 2 of the checklist asks you to rate the external validity of the study (also using ++, +, −). External validity assesses the extent to which the findings for the study participants apply to the whole 'source population' (that is, the population they were chosen from). This may also involve an assessment of the extent to which, if the study were replicated in a different setting but with similar population parameters, the results would have been the same or similar. If the study includes an 'intervention', then it should be assessed to see whether it would be feasible in settings other than that initially investigated.

Section 3 of the checklist asks you to summarise key points about the study that will be added to an evidence table (see appendix H) in the next stage of the process.