NICE process and methods
Appendix C Methodology checklist: randomised controlled trials
Appendix C Methodology checklist: randomised controlled trials
Appendices B–G include checklists for those study designs that are expected to be used in the evidence reviews for NICE social care guidance. Other checklists can found in the NICE clinical guidelines manual and Methods for the development of NICE public health guidance.
Checklist
|
Study identification Include author, title, reference, year of publication |
||||||||||||||
|
Guidance topic: |
Review question no: |
|||||||||||||
|
Checklist completed by: |
||||||||||||||
|
Circle or highlight 1 option for each question |
||||||||||||||
|
A. Selection bias (systematic differences between the comparison groups) |
||||||||||||||
|
An appropriate method of randomisation was used to allocate participants to intervention groups (which would have balanced any confounding factors equally across groups) |
Yes |
No |
Unclear |
N/A |
||||||||||
|
There was adequate concealment of allocation (such that investigators, social care practitioners, healthcare professionals and participants cannot influence enrolment or allocation to groups) |
Yes |
No |
Unclear |
N/A |
||||||||||
|
The groups were comparable at baseline, including all major confounding factors |
Yes |
No |
Unclear |
N/A |
||||||||||
|
Based on your answers to the above, in your opinion was selection bias present? If so, what is the likely direction of its effect? |
||||||||||||||
|
Low risk of bias |
Unclear/unknown risk |
High risk of bias |
||||||||||||
|
Likely direction of effect: . . . . . |
||||||||||||||
|
B. Performance bias (systematic differences between groups in the care provided, apart from the intervention under investigation) |
||||||||||||||
|
The comparison groups received the same care and support apart from the intervention(s) studied |
Yes |
No |
Unclear |
N/A |
||||||||||
|
Participants receiving care and support were kept 'blind' to intervention allocation |
Yes |
No |
Unclear |
N/A |
||||||||||
|
Individuals administering care and support were kept 'blind' to intervention allocation |
Yes |
No |
Unclear |
N/A |
||||||||||
|
Based on your answers to the above, in your opinion was performance bias present? If so, what is the likely direction of its effect? . . . . |
||||||||||||||
|
Low risk of bias |
Unclear/unknown risk |
High risk of bias |
||||||||||||
|
Likely direction of effect: . . |
||||||||||||||
|
C. Attrition bias (systematic differences between the comparison groups with respect to loss of participants) |
||||||||||||||
|
All groups were followed up for an equal length of time (or analysis was adjusted to allow for differences in length of follow-up) |
Yes |
No |
Unclear |
N/A |
||||||||||
|
a. How many participants did not complete the intervention in each group? |
||||||||||||||
|
b. The groups were comparable for intervention completion (that is, there were no important or systematic differences between groups in terms of those who did not complete the intervention) |
Yes |
No |
Unclear |
N/A |
||||||||||
|
a. For how many participants in each group were no outcome data available? . |
||||||||||||||
|
b. The groups were comparable with respect to the availability of outcome data (that is, there were no important or systematic differences between groups in terms of those for whom outcome data were not available). |
Yes |
No |
Unclear |
N/A |
||||||||||
|
Based on your answers to the above, in your opinion was attrition bias present? If so, what is the likely direction of its effect? . . . |
||||||||||||||
|
Low risk of bias |
Unclear/unknown risk |
High risk of bias |
||||||||||||
|
Likely direction of effect: . . . . |
||||||||||||||
|
D. Detection bias (bias in how outcomes are ascertained, diagnosed or verified) |
||||||||||||||
|
The study had an appropriate length of follow-up |
Yes |
No |
Unclear |
N/A |
||||||||||
|
The study used a precise definition of outcome |
Yes |
No |
Unclear |
N/A |
||||||||||
|
A valid and reliable method was used to determine the outcome |
Yes |
No |
Unclear |
N/A |
||||||||||
|
Investigators were kept 'blind' to participants' exposure to the intervention |
Yes |
No |
Unclear |
N/A |
||||||||||
|
Investigators were kept 'blind' to other important confounding factors |
Yes |
No |
Unclear |
N/A |
||||||||||
|
Based on your answers to the above, in your opinion was detection bias present? If so, what is the likely direction of its effect? . . |
||||||||||||||
|
Low risk of bias |
Unclear/unknown risk |
High risk of bias |
||||||||||||
|
Likely direction of effect: |
||||||||||||||
|
E. Overall assessment of internal validity. Are the study results internally valid? Rate the study for internal validity below (for further information see notes on using the methodology checklist) |
||||||||||||||
|
++ |
+ |
− |
||||||||||||
|
Comments: |
||||||||||||||
|
F. Overall assessment of external validity – Are the study results externally valid (i.e. generalisable to the whole source population)? Consider participants, interventions, settings, comparisons and outcomes. Rate the study for external validity below (for further information see notes on use of the methodology checklist) |
||||||||||||||
|
++ |
+ |
− |
||||||||||||
Notes on use of Methodology checklist: randomised controlled trials
The studies covered by this checklist are designed to answer questions about the relative effects of social care interventions, such as task-centred interventions, crisis interventions or psychosocial interventions. Please note some of the items on this checklist may need to be filled in individually for different outcomes reported by the study. It is therefore important that the systematic reviewer has a clear idea of what the important outcomes are before appraising a study. You are likely to need input from the Guidance Development Group in defining the important outcomes.
Checklist items are worded so that a 'yes' response always indicates that the study has been designed/conducted in such a way as to minimise the risk of bias for that item. An 'unclear' response to a question may arise when the item is not reported or not clearly reported. 'N/A' should be used when a randomised controlled trial cannot give an answer of 'yes' no matter how well it has been done.
This checklist is designed to assess the internal and external validity of the study. Internal validity implies that the differences observed between groups of participants allocated to different interventions may (apart from the possibility of random error) be attributed to the intervention under investigation. Biases are characteristics that are likely to make estimates of effect differ systematically from the truth. External validity assesses the extent to which the findings for the study participants apply to the whole 'source population' (that is, the population they were chosen from).
The checklist contains 5 sections (A–E) on internal validity. Sections A–D each address potential sources of bias. At the end of each section you are asked to give your opinion on whether bias is present and to estimate the likely direction of this bias – that is, whether you think it will have increased or decreased the effect size reported by the study. It will not always be possible to determine the direction of bias, but thinking this through can help greatly in interpreting results. In section E you are asked to give an overall assessment of the internal validity of the study (using ++, +, −). Section F then requires you to assess and rate the external validity of the study.
A: Selection bias
Selection bias may be introduced into a study when there are systematic differences between the participants in the different intervention groups. As a result, the differences in the outcome observed may be explained by pre-existing differences between the groups rather than because of the intervention itself. For example, if the people in one group are in poorer health or have higher levels of need, then they may be more likely to have a bad outcome than those in the other group, regardless of the effect of the intervention. The intervention groups should be similar at the start of the study – the only difference between the groups should be the intervention received.
Randomisation
There are 2 aspects to randomisation:
-
generation of the random allocation sequence that results in groups that differ only randomly
-
allocation concealment, so that both the participant and the investigator are unaware of which group the next participant will be allocated to when entering the study.
A1. An appropriate method of randomisation was used to allocate participants to intervention groups
If an appropriate method of randomisation has been used, each participant should have an equal chance of ending up in any of the intervention groups. Examples of random allocation sequences include random numbers generated by computer, tables of random numbers, and drawing of lots or envelopes. Any method of allocation used should be judged as to its potential for bias.
There are some more complicated ways of allocating people to intervention groups that minimise the differences between groups, such as block randomisation and minimisation. Although these are not truly random, they are usually considered to be adequate for the purpose. If a study does not report the method of randomisation used, this should be scored as 'unclear'.
A2. There was adequate concealment of allocation
If investigators are aware of the allocation group for the next participant being enrolled in the study, there is potential for participants to be enrolled in an order that results in imbalances in important characteristics. For example, a social care practitioner (or healthcare professional where appropriate) might feel that participants who are more unwell or who have a higher level of need are likely to do better on a new, experimental, intervention and be tempted to enrol such participants when they know they will be allocated to that group. This would result in the participants in the intervention group being, on average, more unwell. Concealment of intervention group may not always be feasible, but concealment of allocation up until the point of enrolment in the study should always be possible.
The information presented within the paper should provide some assurance that allocations were not known until at least the point of enrolment. Centralised allocation, computerised allocation systems and the use of coded identical containers are all regarded as adequate methods of concealment. Sealed envelopes can be considered as adequate concealment if the envelopes are serially numbered and opaque, and allocation is performed by a third party. Poor methods of allocation concealment include alternation, or the use of case record numbers, date of birth or day of the week.
If the method of allocation concealment used is regarded as poor, or relatively easy to subvert, the study must be given a lower quality rating. If a study does not report any concealment approach, this should be scored as 'unclear'.
A3. The groups were comparable at baseline, including all major confounding factors
Studies may report the distributions of potential confounding factors in the comparison groups, or important differences in the distribution of these factors may be noted.
Formal tests comparing the groups are problematic – failure to detect a difference does not mean that a difference does not exist, and multiple comparisons of factors may falsely detect some differences that are not real.
Social care practitioner (or healthcare professional where appropriate) input may be required to determine whether all likely confounders have been considered. Confounding factors may differ according to outcome, so you will need to consider potential confounding factors for all of the outcomes that are of interest to your review.
B: Performance bias
Performance bias refers to systematic differences between the comparison groups in the care and support provided to the participants, other than the intervention under investigation.
This may consist of additional care, support, or advice, or even simply a belief about the effects of an intervention. If performance bias is present, it can be difficult to attribute any observed effect to the experimental intervention rather than to the other factors.
B1. The comparison groups received the same care apart from the intervention(s) studied
There should be no differences between the intervention groups apart from the intervention(s) received. If some participants received additional care or support (known as 'co-intervention'), this intervention is a potential confounding factor that may compromise the results.
Blinding
Blinding (also known as masking) refers to the process of withholding information about intervention allocation from those involved in the study who could potentially be influenced by this information. This can include participants, investigators, those administering care and support and those involved in data collection and analysis. If people are aware of the intervention allocation ('unblinded'), this can bias the results of studies, either intentionally or unintentionally, through the use of other effective co-interventions, decisions about withdrawal, differential reporting of symptoms or influencing concordance with the intervention. Blinding of those assessing outcomes is covered in section D on detection bias.
Blinding of participants and carers is not always possible, particularly in studies of non-drug interventions used in social care, and so performance bias may be a particular issue in these studies. It is important to think about the likely size and direction of bias caused by failure to blind.
The terms 'single blind', 'double blind' and even 'triple blind' are sometimes used in studies. Unfortunately, they are not always used consistently. Commonly, when a study is described as 'single blind', only the participants are blind to their group allocation. When both participants and investigators are blind to group allocation, the study is often described as 'double blind'. It is preferable to record exactly who was blinded, if reported, to avoid misunderstanding.
B2. Participants receiving care and support were kept 'blind' to intervention allocation
The knowledge of assignment to a particular intervention group may affect outcomes, such as a study participant's reporting of symptoms, self-use of other known interventions or even dropping out of the study.
B3. Individuals administering care and support were kept 'blind' to intervention allocation
If individuals who are administering the intervention and/or other care and support to the participant are aware of intervention allocation, they may treat participants receiving one intervention differently from those receiving the comparison intervention; for example, by offering additional co-interventions.
C: Attrition bias
Attrition refers to the loss of participants during the course of a study. Attrition bias occurs when there are systematic differences between the comparison groups with respect to participants lost, or differences between participants lost to the study and those who remain. Attrition can occur at any point after participants have been allocated to their intervention groups. As such, it includes participants who are excluded after allocation (and may indicate a violation of eligibility criteria), those who do not complete intervention (whether or not they continue measurement) and those who do not complete outcome measurement (regardless of whether or not intervention was completed). Consideration should be given to why participants dropped out, as well as how many. Participants who dropped out of a study may differ in some significant way from those who remained as part of the study throughout. Drop-out rates and reasons for dropping out should be similar across all intervention groups. The proportion of participants excluded after allocation should be stated in the study report, and the possibility of attrition bias considered within the analysis; however, these are not always reported.
C1. All groups were followed up for an equal length of time (or analysis was adjusted to allow for differences in length of follow-up)
If the comparison groups are followed up for different lengths of time, then more events are likely to occur in the group followed up for longer, distorting the comparison. This may be overcome by adjusting the denominator to take the time into account.
C2a. How many participants did not complete intervention in each group?
A very high number of participants dropping out of a study should give concern. The drop-out rate may be expected to be higher in studies conducted over a longer period of time. The drop-out rate includes people who did not even start the intervention; that is, they were excluded from the study after allocation to intervention groups.
C2b. The groups were comparable for intervention completion (that is, there were no important or systematic differences between groups in terms of those who did not complete the intervention)
If there are systematic differences between groups in terms of those who did not complete the intervention, consider both why participants dropped out and whether any systematic differences in those who dropped out may be related to the outcome under study, such as potential confounders. Systematic differences between groups in terms of those who dropped out may also result in intervention groups that are no longer comparable with respect to potential confounding factors.
C3a. For how many participants in each group were no outcome data available?
A very high number of participants for whom no outcome data were available should give concern.
C3b. The groups were comparable with respect to the availability of outcome data (that is, there were no important or systematic differences between groups in terms of those for whom outcome data were not available)
If there are systematic differences between groups in terms of those for whom no outcome data were available, consider both why the outcome data were not available and whether there are any systematic differences between participants for whom outcome data were and were not available.
D: Detection bias (this section should be completed individually for each important relevant outcome)
The way outcomes are assessed needs to be standardised for the comparison groups; failure to 'blind' people who are assessing outcomes can also lead to bias, particularly with subjective outcomes. Most studies report results for more than 1 outcome, and it is possible that detection bias may be present in a study for some, but not all, outcomes. It is therefore recommended that this section is completed individually for each important outcome that is relevant to the guidance review question under study. To avoid biasing your review, you should identify the relevant outcomes before considering the results of the study. Social care practitioner (or healthcare professional where appropriate) input may be required to identify the most important outcomes for a review.
D1. The study had an appropriate length of follow-up
The follow-up of participants after intervention should be of an adequate length to identify the outcome of interest. This is particularly important when different outcomes of interest occur early and late after an intervention. A study that is too short will give an unbalanced assessment of the intervention. For events occurring later, a short study will give an imprecise estimate of the effect, which may or may not also be biased. For example, a late-occurring side effect will not be detected in the intervention arm if the study is too short.
D3. A valid and reliable method was used to determine the outcome
The outcome under study should be well defined. It should be clear how the investigators determined whether participants experienced, or did not experience, the outcome. The same methods for defining and measuring outcomes should be used for all participants in the study. Often there may be more than 1 way of measuring an outcome (for example, questionnaires, reporting of symptoms and functioning). The method of measurement should be valid (that is, it measures what it claims to measure) and reliable (that is, it measures something consistently).
D5. Investigators were kept 'blind' to other important confounding factors
In this context the 'investigators' are the individuals who are involved in making the decision about whether a participant has experienced the outcome under study. Investigators can introduce bias through differences in measurement and recording of outcomes, and making biased assessments of a participant's outcome based on the collected data. The degree to which lack of blinding can introduce bias will vary depending on the method of measuring an outcome, but will be greater for more subjective outcomes, such as reporting of pain or quality of life.
E. Overall assessment of internal validity
Rate the study for internal validity according to the list below:
++ All or most of the checklist criteria have been fulfilled, where they have not been fulfilled the conclusions are very unlikely to alter
+ Some of the checklist criteria have been fulfilled, where they have not been fulfilled, or not adequately described, the conclusions are unlikely to alter
− Few or no checklist criteria have been fulfilled and the conclusions are likely or very likely to alter
F. Overall assessment of external validity
Rate the external validity of the study (also using ++, +, −). This is the extent to which the findings for the study participants apply to the whole source population (the population they were chosen from). This may also involve an assessment of the extent to which, if the study were replicated in a different setting but with similar population parameters, the results would have been the same or similar. If the study includes an 'intervention', then it should be assessed to see whether it would be feasible in settings other than that initially investigated.