NICE process and methods

Appendix G Methodology checklist: qualitative studies

Appendix G Methodology checklist: qualitative studies

Appendices B–G include checklists for those study designs that are expected to be used in the evidence reviews for NICE social care guidance. Other checklists can found in the NICE clinical guidelines manual and Methods for the development of NICE public health guidance..

This checklist is based on checklists from:

Checklist

Study identification

Include author, title, reference, year of publication

Guidance topic:

Key research question/aim:

.

Checklist completed by:

Circle or highlight 1 option for each question

Section 1: theoretical approach

1.1 Is a qualitative approach appropriate?

For example:

Does the research question seek to understand processes or structures, or illuminate subjective experiences or meanings (in social care this would apply to how care and support is organised and service user or carer experience)? Or could a quantitative approach better have addressed the research question?

Appropriate

Inappropriate

Not sure

Comments:

1.2 Is the study clear in what it seeks to do?

For example:

Is the purpose of the study discussed – aims/objectives/research question(s)?

Are the values/assumptions/theory underpinning the purpose of the study discussed?

Clear

Unclear

Mixed

Comments:

Section 2: study design

2.1 How defensible/rigorous is the research design/methodology?

For example:

Are there clear accounts of the rationale/justification for the sampling, data collection and data analysis techniques used?

Defensible

Not defensible

Not sure

Comments:

Section 3: data collection

3.1 How well was the data collection carried out?

For example:

Are the data collection methods clearly described?

Were the data collected appropriate to address the research question?

Appropriate

Inappropriate

Not sure/ inadequately reported

Comments:

Section 4: validity

4.1 Is the context clearly described?

For example:

Are the characteristics of the participants and settings clearly defined?

Were observations made in a variety of circumstances and from a range of respondents?

Was context bias considered (that is, did the authors consider the influence of the setting where the study took place)?

Clear

Unclear

Not sure

Comments:

4.2 Were the methods reliable?

For example:

Were data collected by more than 1 method?

Were other studies considered with discussion about similar/different results?

Reliable

Unreliable

Not sure

Comments:

Section 5: analysis

5.1 Are the data 'rich'?

For example:

How well are the contexts of the data described?

Has the diversity of perspective and content been explored?

Has the detail of the data that were collected been demonstrated?

Are responses compared and contrasted across groups/sites?

Rich

Poor

Not sure/not reported

Comments:

5.2 Is the analysis reliable?

For example:

Did more than 1 researcher theme and code transcripts/data?

If so, how were differences resolved?

Were negative/discrepant results addressed or ignored?

Is it clear how the themes and concepts were derived from the data?

Reliable

Unreliable

Not sure/not reported

Comments:

5.3 Are the findings convincing?

For example:

Are the findings clearly presented?

Are the findings internally coherent (that is, are the results credible in relation to the study question)?

Are extracts from the original data included (for example, direct quotes from participants)?

Are the data appropriately referenced so that the sources of the extracts can be identified?

Is the reporting clear and coherent?

Convincing

Not convincing

Not sure

Comments:

5.4 Are the conclusions adequate?

For example:

How clear are the links between data, interpretation and conclusions?

Are the conclusions plausible and coherent?

Have alternative explanations been explored and discounted?

Are the implications of the research clearly defined?

Is there adequate discussion of any limitations encountered?

Adequate

Inadequate

Not sure

Comments:

Section 6: ethics

6.1 Was the study approved by an ethics committee?

Yes

No

Not sure/not reported/not applicable

Comments:

6.2 Is the role of the researcher clearly described?

For example:

Has the relationship between the researcher and the participants been adequately described?

Is how the research was explained and presented to the participants described?

Clear

Not clear

Not sure/not reported

Comments:

Section 7: Overall assessment

As far as can be ascertained from the paper, how well was the study conducted (see guidance notes)

++

+

Comments

After completion of quality appraisal using the checklist, the included studies can be presented in a 'Quality of the included studies' table, which summarises the quality of each study under the main criteria of population, methods and analysis, and also the relevance of the study to the population being considered in the guidance.

Notes on use of Methodology checklist: qualitative studies

The studies covered by this checklist are those that collect and analyse qualitative data – usually (but not exclusively) textual (written), spoken or observational data. Qualitative data are occasionally collected using structured questionnaires (for example, as thematically organised free-text comments) but such research needs to be scrutinised carefully because it may not meet acceptable quality criteria for consideration as a qualitative study.

There is considerable debate over which quality criteria should be used to assess qualitative studies. Quality in qualitative research can be assessed using the same broad concepts of validity (or trustworthiness) used for quantitative research, but these need to be put in a different contextual framework to take into account the aims of qualitative research. This checklist is based on the broadly accepted principles that characterise qualitative research and that may affect its validity; it is concerned with adequate reporting of key factors that affect the quality of qualitative research studies. The questions in the checklist are framed to encompass the variety of ways in which qualitative research is conducted. Care must be taken to apply the checklist in a way that matches the research methodology.

The following notes provide suggestions for completing the checklist. A list of publications on qualitative research is provided at the end of these notes for further reading on this topic.

Note that the sub-questions given as examples under each question in the checklist are intended to highlight some of the key issues to be considered for that question – they are not intended to be exhaustive. Please add any additional considerations in the comments box.

Section 1: theoretical approach

This section deals with the underlying theory and principles applied to the research.

1.1 Is a qualitative approach appropriate?

A qualitative approach can be judged to be appropriate when the research sets out to investigate phenomena that are not easy to quantify or measure accurately, or where such measurement would be arbitrary and inexact.

Qualitative research in social care settings often measure:

  • personal experiences (for example, of a service, intervention or package of care)

  • processes (for example, action research, practitioner or service user views on assessments of social care and support needs)

  • personal values and beliefs (for example, about need, disability and dignity)

  • interactions and relationships (for example between service users and social workers or between service users and personal carers)

  • service evaluations (for example, what was good or bad about service user or carer experiences of a re-ablement package).

If clear numerical measures could reasonably have been put in place, then consider whether a quantitative approach may have been more appropriate.

1.2 Is the study clear in what it seeks to do?

The design of qualitative research tends to be 'theory generative' rather than 'theory testing'; it is therefore unlikely that a research question will be found in the form of a hypothesis or null hypothesis in the way that you would expect in traditional quantitative research. Nevertheless, what the study is investigating should still be set out early and clearly. The research question should be set in context, with a summary of the background literature and the study's underpinning values and assumptions.

Section 2: study design

This section considers the robustness of the design of the research project.

2.1 How defensible/rigorous is the research design/methodology?

There are a large number of qualitative methodologies, and a tendency in many studies to 'mix' aspects of different methodologies or to use a generic qualitative method. From a qualitative perspective, this should not compromise the quality of the study if the research design captures appropriate data and has an appropriate plan of analysis for the subject under investigation.

Sampling in qualitative research can be purposive. Qualitative research is not experimental and does not purport to be generalisable, and therefore does not require a large or random sample. People are usually 'chosen' for qualitative research based on being key informers. The choice of sample and sampling method should be described, ideally including any shortcomings of the sample.

Section 3: data collection

3.1 How well was the data collection carried out?

Assess whether the methods of data collection are described with details of the following:

  • how the data were collected

  • how the data were recorded and transcribed (if verbal data)

  • how the data were stored

  • what records were kept of the data collection.

Were these appropriate methods of data collection to use, given the aims of the research?

Section 4: validity

Assessing the validity of qualitative research is very different from assessing that of quantitative research. Qualitative research is much more focused on demonstrating the causes of bias rather than eliminating them. The report should include sections discussing the reflexive position of the researcher (their 'role' in the research), the context in which the research was conducted and the reliability of the actual data.

4.1 Is the context clearly described?

It is important when gauging the validity of qualitative data to consider whether the data are plausible and realistic. To make an accurate assessment of this, it is important to describe the context of the research in terms of the physical context (for example, care home, day centre, school) and who else was there (for example, participants are likely to position themselves very differently, and thus to respond very differently, in a discussion with parents present compared with a discussion with peers present). The participants should be described in enough detail to allow some insight into their life and situation and any potential context bias considered by the authors (that is, interpretation of the influence of the setting).

4.2 Were the methods reliable?

It is important that the method used to collect the data is appropriate for the research question and that the data generated map well to the aims of the study. Ideally, more than 1 method should have been used to collect data.

Section 5: analysis

Qualitative data analysis is very different from quantitative analysis. This does not mean that it should not be systematic and rigorous; however, systematisation and rigour require different methods of assessment.

5.1 Are the data 'rich'?

Qualitative researchers use the adjective 'rich' to describe data that are in-depth, convincing, compelling and detailed enough that they can provide some insight into the research participants' experience. It is also important to know the 'context' of the data – where they came from, what prompted them, what they pertain to, and so on.

5.2 Is the analysis reliable?

The analysis of data can be made more reliable by the researchers putting checks in place. Sections of data should be coded by another researcher or, as a minimum, a second researcher should check the coding for consistency. Participants may also verify the transcripts of their interview (or other data collection, if appropriate). Negative or discrepant results should be highlighted and discussed.

5.3 Are the findings convincing?

The results of the research should be convincing or credible. Findings should be presented clearly and organised logically and the authors should consider and explain any contradictions. Extracts from original data should be included where possible to give a fuller sense of the findings. These data should be appropriately referenced – although you would expect data to be anonymised, they still need to be referenced in relevant ways (for example, if sex differences were important, then you would expect extracts to be marked male or female).

5.4 Are the conclusions adequate?

This section is self explanatory.

Section 6: ethics

6.1 Was the study approved by an ethics committee?

All qualitative research involves ethical considerations, and these should be considered within any research report. Ideally there should be a full discussion of ethics, although this is rare because of space constraints in peer-reviewed journals. Any qualitative research should be approved by a research ethics committee, and this should be stated in the report so that it is clear that every care was taken to protect research participants.

6.2 Is the role of the researcher clearly described?

The researcher should have considered their role in the research; for example, as a reader, interviewer or observer. This is often referred to as 'reflexivity'. The 'status' of the researcher can profoundly affect the data. For example, a middle-aged woman and an 18-year-old man may get different responses to questions about care, support needs and dignity when interviewing a group of older women. It is important to consider age, gender, ethnicity and 'insider' status (such as whether the interviewer or researcher is part of the group being researched or has the same condition or disability). The researcher can also profoundly influence the data by use of questions, opinions, judgements and so on, so it is important to know what the researcher's position is in this regard, and how the researcher introduced and talked about the research with the participants.

Section 7: overall assessment

Grade the study according to the list below:

++ All or most of the checklist criteria have been fulfilled, where they have not been fulfilled the conclusions are very unlikely to alter

+ Some of the checklist criteria have been fulfilled, where they have not been fulfilled, or not adequately described, the conclusions are unlikely to alter

− Few or no checklist criteria have been fulfilled and the conclusions are likely or very likely to alter

Most qualitative studies by their very nature will not be generalisable. However, where there is reason to suppose the results would have broader applicability they should be assessed for external validity. Qualitative studies that are rated for external validity should be prefixed with 'EV' (external validity).

Further reading

Barbour RS (2001) Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? British Medical Journal 322: 1115–7

Daly J, Willis K, Small R et al. (2007) A hierarchy of evidence for assessing qualitative health research. Journal of Clinical Epidemiology 60: 43–9

Mays N, Pope C (2000) Assessing quality in qualitative research. British Medical Journal 320: 50–2

Miller G, Dingwall R, editors (1997) Context and method in qualitative research. London: Sage

Shaw I, Gould N (2001) Qualitative research in social work. London: Sage