Ipsos Encyclopedia - Questionnaire Design
At the core of survey research is our goal to find out more about people - what people think, how they feel, what their experiences have been, what they plan to do, and what they have done. To achieve our goal, we ask questions of research participants and record how they respond. Our most important tool for understanding people is the questionnaire, which contains our questions and allows for relevant responses that will provide the data which can be used to give us a better understanding of people.
The quality of a questionnaire depends on the care we take in forming our questions, how we structure the questions and topics to unfold, and what response formats we provide participants to best represent their experiences. A questionnaire is often called our ‘instrument’ or ‘survey.’
Type of Question Task
There are two primary types of questions in questionnaires:
- Open-ended questions allow the participant to answer using their own words (such as ‘When you think of high quality products, which brand comes to mind?,’ ‘Why do you prefer hot beverages over cold beverages?’ or ‘What are the issues that are important for you in voting for a candidate?’). This is typically presented as an open field allowing any text that the respondent wants to type in (or is typed in by an interviewer). Open-ended questions are often used when we are not sure of the nature or number of responses that a group of people might have – it is often exploratory in nature.
- Closed-ended questions are the most common type of question in questionnaires. These questions present a set of responses that a participant can select from that can best describe their attitudes, feelings, or behaviours. The first kind of closed-ended question presents a fixed set of responses where each response describes different qualities or characteristics. This kind of question could be a multiple response question where the participant can select all responses that apply to them (such as ‘What features do you look for in purchasing your next vehicle?’ listing price, efficiency, speed, colour, etc.), or could be a single response question (“Which is the MOST important feature you look for in purchasing your next vehicle?” using the same set of qualitatively different responses). A second type of closed-ended question is a graded response ‘scale,’ which refers to a standard set of graded responses used to assess dimensions of thought – importance, satisfaction, evaluation, liking, etc. An example of a graded response scale might be using just 3 responses at a minimum – ‘Not important, Somewhat important, Very important.’ The ability to measure ideas on a graded scale provides us with a more powerful measurement tool and allows us to compare groups (such as Conservatives versus Liberals or Older versus Younger Participants) and to compare results across time (as might occur in brand-tracking studies). A graded scale also allows us to compare one variable with another variable (such as ‘how are political attitudes towards an issues (support, neutral, oppose) related to candidate choice’ or ‘how are product experiences (good, neutral, bad) related to future purchase?’).
Designing Good Questions
When designing a questionnaire, the best questions are:
- Short
- Easy to read on a smartphone
- Easy to understand by a person with limited education (reading level of an average 10 year-old child)
- Refers to only one idea (not double-barrelled or complex)
- Relevant to the experiences of most participants who are being asked the question
- Easy to select a response that best describes a participant’s experience which often depends on how we present the responses
Common Sections of a Questionnaire
When developing questions and responses, there are three main groupings of questions within a stand-alone questionnaire (as distinct from an omnibus, which asks about many different topics). These sections provide an order that participants come to expect as a sort of narrative or unfolding of the topics within the survey. The ordering of tasks and questions are often referred to as the ‘flow’ of the survey, which generally moves from general topics to specific topics.
- Screening Section. This section helps filter participants to those who have the qualifications or experiences of interest to our study. For example, if we are interested in how women experience pregnancy, we filter out people who have not experienced pregnancy directly. If we are interested in current cigarette smoking behaviour, we filter out people who are not current smokers. The screening section also often has basic demographic questions to ensure that people are representative of the group in which we are interested (in terms of gender, age, region of country, etc.). Note that we do not want more than two or three demographic items in the screening section – these are often very boring for respondents and lead to higher break-off rates, which can reduce the representativeness of your results from the survey.
- Main Body. The main body of the questionnaire has questions that assess aspects of the behaviour or experiences of interest (such as the frequency of smoking or number of cigarettes smoked, or frequency of purchase of products). The main body also will include questions about the attitudes, feelings, and intentions regarding the topic of interest (such as ratings of product or brand satisfaction, likelihood to purchase in the future). This section of the survey will have many graded response scales as a result.
- Demographics and Ancillary Analytic Variables. As the questionnaire concludes, we ask about demographics that weren't asked in the screening section (most demographics should come at the end of the questionnaire – it is very boring if all demographics come first). Often these additional demographics are used for weighting or planned analyses (such as political party identification, or marital status).
Obtaining Quality Data With Questionnaires
Good data starts with making sure your participants are real, live human beings and not bots or coming from click farms. These issues often arise when you need to focus on issues that have very low incidences of occurrence and need to obtain sample from less reliable sources. When assured of dealing with real participants, obtaining good and accurate information from your questionnaire requires attention to the details of the questions and responses, the flow of the survey, and continual attention to the needs and motivations of participants. The following are some of the issues we see that can reduce the accuracy and quality of your data:
- Questionnaires that take longer than 10 minutes without appropriate compensation (and longer than 20 minutes even with appropriate compensation is associated with higher rates of inattention).
- Asking participants to do complex cognitive tasks increases respondent burden (ranking items is harder than rating items, etc.), and more burden increases inattention and break-offs.
- Asking people to read a lot (high text density) or using more complex words (high reading level) increases respondent burden.
- Lots of open-ended questions requiring detailed explanations by participants is very burdensome
- Having lots of redundancy in your questions, using the same questions repeatedly (such as rating 40 different brands with the same 20 rating scales), or using the same agree-disagree scale for everything, can lead to participant boredom.
Ipsos Point of View
Ipsos has some of the best internationally-recognized survey methodologists who do research to find the ways to create questions and responses that will most accurately measure ideas of interest. The document linked below contains more detailed questionnaire design recommendations and may be a helpful starting point in your journey to improve your questionnaire design.
