Cognitive testing is a technique used to test and improve survey questions. During a cognitive interview survey questions are administered and participants are asked about the cognitive processes they go through in answering those questions to uncover problems with those questions and to make recommendations for improvements. Cognitive testing is also known as cognitive interviewing but should not be confused with the cognitive interviewing used by the police with witnesses.
Survey questions are designed to be standardised and understood and answered in a consistent manner to produce reliable and unbiased data for analysis. A survey pilot can tell us about overt problems with questions – for example questions which participants don't answer or give 'don't know' answers to, questions which participants asked to be repeated, or questions where their answers are inconsistent with responses to other questions. However, it doesn't tell us about whether participants:
- understand the terms used in the question or the question as a whole,
- can accurately recall the information needed to answer the question,
- make appropriate judgements about which information to include in their answer,
- can fit the answer they want to give into the answer categories or format provided, or
- can navigate a self-completion questionnaire.
Cognitive interviews allow us to explore all these issues so that the final survey questions are understood as the researchers intended and produce data which can be compared between key groups of interest and over time.
During a cognitive interview, the survey questions are delivered in a mode and setting as close to the main survey as possible – a process termed as 'mode mimicking'. For example, if the questions are designed to be delivered in a face to face (CAPI) survey, the interviewer will read them out, use showcards where necessary, and record the responses provided. If the questionnaire is a self-completion online survey (CAWI), ideally the questions will be programmed, and tested with the participant on a laptop, tablet or mobile – even though it is likely that an interviewer would still be present to observe and to carry out the exploratory probing.
Three main techniques are then used to explore the question and answer process:
- Observation – looking for hesitation, confusion, requests for clarification;
- Think aloud – the participant is asked to verbalise the thought processes they go through when answering the questions;
Probing – the interviewer asks scripted or spontaneous probes to explore specific issues such as how participants understand a particular word or phrase, what time frame they are thinking about when recalling information, whether there are any missing answer options. Probing can either be carried out concurrently (after each question or a series of questions) or retrospectively, after the entire test questionnaire.
A cognitive interview is a qualitative interview but based around testing quantitative survey questions. The samples for testing should be purposive like qualitative research samples and should focus on the characteristics of the survey population which are likely to have a bearing on how the questions might be approached and answered. The findings should be reported in qualitative terms by identify themes and issues rather than reporting on the number of participants experiencing a problem.
The key output of cognitive testing should be a report of the problems identified, the implications of those problems for the survey data, and recommendations for improvements. Recommendations might involve suggested tweaks to the question wording (a replacement of a word or phrase for example, the addition of response categories), or they might suggest revising the measurement objectives or even dropping the item. Cognitive testing is usually carried out before questions are included in a survey for the first time. However, it can also be used to test questions on long running surveys if they need to adapt to changes in policy or society. The technique can also be used for testing research or other documents such as advance letters, diaries or information leaflets.
Ipsos Point Of View
In an ideal world all new survey questions would be tested using cognitive interviewing before they are administered in a survey. However, client timescales, budgets and the scale of the survey mean that it is not always possible or appropriate. Sometimes clients request cognitive interviewing in a brief, but it is something we can offer even if it is not requested if we feel it would add value. Consider offering cognitive interviews when the survey questions:
- will be included on a long-running survey over multiple waves or years,
- are on a challenging or new topic where there are few existing questions,
- have not previously been asked to the types of participants taking part in the survey (e.g. questions usually asked to adults being adapted to children),
- will be subject to intense scrutiny when the results are reported.
At Ipsos MORI cognitive interviews are usually carried out by members of the research team or freelance researchers. This means they will be well placed to ask spontaneous probes, to adjust the survey questions during fieldwork on the basis of feedback from earlier interviews, and generally be flexible in the conduct of the fieldwork. Analysis usually involves a meeting of those who carried out the interviews which allows ideas to be shared and emerging themes to be discussed. This should be supplemented by desk analysis of the written notes – an excel sheet with a participant per row and a column per question or element of a question is an effective and systematic way to do this.
For a practical guide on the background to cognitive interviewing and how to carry out cognitive interviewing problems from start to finish read: Collins, D. (ed) (2015) Cognitive Interviewing Practice. London: Sage. This book includes chapters by Margaret Blake and Michelle Mackie (Gray)
An older book which also provides an overview of the approach is: Willis, G. (2005) Cognitive Interviewing. Thousand Oaks, CA: Sage.
For a background to the psychology behind cognitive interviews read: Tourangeau, R., Rips, L., Rasinski, K. (2000) Psychology of Survey Response. Cambridge: Cambridge University Press.
For further reading, the Collins book (Cognitive Interviewing Practice) includes a wide range of useful references.