Tools and resources

The guidelines manual: appendices B–I

Appendix I: Methodology checklist: prognostic studies

Appendix I: Methodology checklist: prognostic studies

The criteria used in this checklist are adapted from: Hayden JA, Cote P, Bombardier C (2006) Evaluation of the quality of prognosis studies in systematic reviews. Annals of Internal Medicine 144: 427–37.

Checklist

Study identification

Include author, title, reference, year of publication

Guideline topic:

Review question no:

Checklist completed by:

Circle or highlight one option for each question

1.1

The study sample represents the population of interest with regard to key characteristics, sufficient to limit potential bias to the results

Yes

No

Unclear

1.2

Loss to follow-up is unrelated to key characteristics (that is, the study data adequately represent the sample), sufficient to limit potential bias

Yes

No

Unclear

1.3

The prognostic factor of interest is adequately measured in study participants, sufficient to limit potential bias

Yes

No

Unclear

1.4

The outcome of interest is adequately measured in study participants, sufficient to limit potential bias

Yes

No

Unclear

1.5

Important potential confounders are appropriately accounted for, limiting potential bias with respect to the prognostic factor of interest

Yes

No

Unclear

1.6

The statistical analysis is appropriate for the design of the study, limiting potential for the presentation of invalid results

Yes

No

Unclear

Notes on use of Methodology checklist: prognostic studies

The studies covered by this checklist are designed to answer questions about prognosis. Such questions address the likelihood of an outcome for patients from a population at risk for that outcome, based on the presence of a proposed prognostic factor. Prognostic factors may be disease-specific (for example, presence or absence of particular disease feature), demographic (for example, age, sex), or relate to the likely response to treatment or the presence of comorbidities.

This checklist is based on a checklist for the quality appraisal of studies about prognosis developed by Hayden and co-workers (2006).

Checklist items are worded so that a 'yes' response always indicates that the study has been designed and conducted in such a way as to minimise the risk of bias for that item. An 'unclear' response to a question may arise when the answer to an item is not reported or is not reported clearly.

1.1 The study sample represents the population of interest with regard to key characteristics, sufficient to limit potential bias to the results

Measures of prognosis can vary substantially when obtained from populations with different clinical or demographic features. Estimates of prognosis are not useful without information about the population from which they were obtained.

To minimise bias, the study population should be clearly defined and described and should represent the source population of interest. Points to consider include the following:

  • Are the source population or the population of interest adequately described with respect to key characteristics?

  • Are the sampling frame and recruitment adequately described, possibly including methods to identify the sample (number and type used; for example, referral patterns in healthcare), period of recruitment and place of recruitment (setting and geographical location)?

  • Are inclusion and exclusion criteria adequately described (for example, including explicit diagnostic criteria or a description of participants at the start of the follow-up period)?

  • Is participation in the study by eligible individuals adequate?

  • Is the baseline study sample (that is, individuals entering the study) adequately described with respect to key characteristics?

1.2 Loss to follow-up is unrelated to key characteristics (that is, the study data adequately represent the sample), sufficient to limit potential bias

Attrition refers to the loss of participants during the course of a study. Consideration should be given to why participants dropped out, as well as how many dropped out. Attrition bias occurs when there are systematic differences between participants lost to the study and those who remain.

To minimise bias, completeness of follow-up should be described and adequate. Points to consider include the following:

  • Is the response rate (that is, proportion of study sample completing the study and providing outcome data) adequate?

  • Are attempts to collect information on participants who dropped out of the study described?

  • Are reasons for loss to follow-up provided?

  • Are the key characteristics of participants lost to follow-up adequately described?

  • Are there any important differences in key characteristics and outcomes between participants who completed the study and those who did not?

If your review addresses more than one outcome, you should score this item for each outcome individually.

1.3 The prognostic factor of interest is adequately measured in study participants, sufficient to limit potential bias

The prognostic factor under study should be well defined. It should be clear how the investigators determined whether participants were exposed or not to the factor. The same definition and measurement should be used for all participants in the study. Often there may be more than one way of determining the presence or absence of the factor (for example, physical or laboratory tests, questionnaire, reporting of symptoms). The method of measurement should be valid (that is, it measures what it is claimed to measure) and reliable (that is, it measures something consistently).

To minimise bias, prognostic factors should have been defined and measured appropriately. Points to consider include the following:

  • Is a clear definition or description of the prognostic factor(s) measured provided (including dose, level, duration of exposure, and clear specification of the method of measurement)?

  • Are continuous variables reported, or appropriate cut-off points (that is, not data-dependent) used?

  • Are the prognostic factors measured and the method of measurement valid and reliable enough to limit misclassification bias? (This may include relevant outside sources of information on measurement properties, as well as characteristics such as blind measurement and limited reliance on recall.)

  • Are complete data for prognostic factors available for an adequate proportion of the study sample?

  • Are the method and setting of measurement the same for all study participants?

  • Are appropriate methods employed if imputation is used for missing data on prognostic factors?

1.4 The outcome of interest is adequately measured in study participants, sufficient to limit potential bias

The outcome under study should be well defined. It should be clear how the investigators determined whether participants experienced, or did not experience, the outcome. The same methods for defining and measuring outcome should be used for all participants in the study. Often there may be more than one way of measuring an outcome (for example, physical or laboratory tests, questionnaire, reporting of symptoms). The method of measurement used should be valid and reliable.

To minimise bias, the outcome(s) of interest should be defined and measured appropriately. Points to consider include the following:

  • Is a clear definition of the outcome of interest provided, including duration of follow-up?

  • Are the outcomes that were measured and the method of measurement valid and reliable enough to limit misclassification bias? (This may include relevant outside sources of information on measurement properties, as well as characteristics such as 'blind' measurement and limited reliance on recall.)

  • Are the method and setting of measurement the same for all study participants?

If your review addresses more than one outcome, you should score this item for each outcome individually.

1.5 Important potential confounders are appropriately accounted for, limiting potential bias with respect to the prognostic factor of interest

Confounding can occur when there are differences between participants, apart from the presence or absence of the prognostic factor, that are related to both the outcome and the prognostic factor. An example of this is if the participants are recruited at different stages of disease progression. The design and analysis of prognostic studies are usually based on some conceptual model about how factors interact to lead to the outcome.

This question is not relevant where the study is being reviewed for the purposes of identifying the absolute risk of the outcome in the group with the prognostic factor.

To minimise bias, important confounders should be defined and measured, and confounding should be accounted for in the design or analysis. Points to consider include the following:

  • Are all important confounders, including treatments (key variables in the conceptual model), measured? Are clear definitions of the important confounders measured (including dose, level and duration of exposures) provided?

  • Is measurement of all important confounders valid and reliable? (This may include relevant outside sources of information on measurement properties, as well as characteristics such as 'blind' measurement and limited reliance on recall.)

  • Are the method and setting of measurement of confounders the same for all study participants?

  • Are appropriate methods employed if imputation is used for missing data on confounders?

  • Are important potential confounders accounted for in the study design (for example, matching for key variables, stratification or initial assembly of comparable groups)?

  • Are important potential confounders accounted for in the analysis (that is, appropriate adjustment)?

If your review addresses more than one outcome, you should score this item for each outcome individually.

1.6 The statistical analysis is appropriate for the design of the study, limiting potential for the presentation of invalid results

Analysis undertaken within the study that is incorrect or inappropriate for the study design may result in false conclusions being drawn from the data.

To minimise bias, the statistical analysis undertaken should be clearly described and appropriate for the design of the study. Points to consider include the following:

  • Is the presentation of data sufficient to assess the adequacy of the analysis?

  • Where several prognostic factors are investigated, is the strategy for model building (that is, the inclusion of variables) appropriate and based on a conceptual framework or model?

  • Is the selected model adequate for the design of the study?

  • Is there any selective reporting of results?

  • Are only pre-specified hypotheses investigated in the analyses?

In some circumstances it may be possible to reanalyse the data using the information supplied in the study report, in order to remove bias.


This page was last updated: