Attention checks are an effective way to ensure data quality for your Connect study. We recommend that Connect researchers include 1-2 attention checks and one open-ended question for a study that lasts 5-10 minutes. For studies that are longer than 10 minutes, additional attention checks should be included. For example, in a 20-minute study, we recommend including 2-3 direct attention checks.
What Should your Attention Check Look Like?
Attention checks are designed to measure if the participant is paying attention. Responses to attention checks help researchers to protect their data from inattentive responses that can harm data quality.
The most direct type of attention check instructs a participant exactly how they should respond to a question (Gummer et al. 2021; Meade & Craig, 2012). These are called Instructed Response Items. An example of an instructed response item is shown below:
Another type of attention check that may be included is called a Nonsense Item (Meade & Craig, 2012). Nonsense items ask participants about impossible events (example: Have you ever had a fatal heart attack while watching TV?), nonexistent items (example: Do you know what the word ‘wutlett’ means?), or something the researcher is certain must be true for participants (example: Have you ever used the internet?). If participants fail these items, they are likely not paying attention.
The last type of attention checks we recommend are called Near Non-Existent Events where the researcher asks participants to respond to questions about events that have an extremely low probability of being true. Examples include questions such as “Are you currently employed as a petroleum engineer?” or “Do you currently reside in Ross, Indiana?”. Because the likelihood of these answers being true is very low (<1% of the US workforce is employed as a petroleum engineer and Ross, Indiana has a population of 400), participants who respond affirmatively to these questions are likely not paying attention.
No matter what type of attention check you choose to include, it is important that the question is clear and that there is only one correct answer choice. The question should be easy to pass for someone who is paying attention. To create clear attention checks, researchers should use a direct prompt (example: To show you’re paying attention, please select “strongly disagree” as your answer) and ask a question that does not rely on participants’ prior knowledge or working memory. Often researchers rely on questions that ask participants to recall information from previous parts of the study as an attention check (example: In the story you read earlier, what was the server’s name?). This type of question better assesses working memory than the 3 more direct types of attention checks mentioned above. Details that may seem pertinent to the researcher may not seem that way to participants. Unless explicitly instructed (e.g., “the name of the server in the following story is important, you will be asked to remember it”), it is not reasonable to expect participants to remember any particular detail, no matter how important it may seem.
Some other rules of thumb to consider when using attention checks are to:
- Use validated and previously tested attention checks rather than creating new ones.
- However, make sure to use variations of these questions in order to prevent participants from catching on to how to respond to attention checks that are overly used.
- Try to embed attention checks with similar measures so that they are less obvious. This can be accomplished by asking attention check questions that match the topic of the survey. For example, if your study is about prescription drug use, a question such as "How many doses of Psyclofyne have you taken in the last two weeks?" (Psyclofyne is a made-up drug) would blend in well.
- Explain the reason why you’re including attention checks in your study to avoid negative reactions from participants in response to these checks. At the beginning of your study, you may want to write a brief paragraph explaining why you are using attention checks or even include a question asking participants to pledge their commitment to data quality such as in the example below:
Open-Ended Questions
Along with direct attention checks, we recommend that you also include one open-ended question in your study. This question should be clear, specific, and related to the topic of your study. For example, if your study is on social media use, you could ask participants to describe in two sentences which social media site they enjoy the most and why. In combination with incorrect responses to attention checks, open-ended responses will allow you to look for participants who leave these questions blank, provide answers that are off topic, give answers that appear to be copy & pasted from another source, write answers that are bad grammar or complete gibberish. Any of these responses signal inattention or outright fraud.
In sum, we recommend using 1-2 direct attention checks for a 5-10 minute study and one open-ended response. Consider including more as your study gets longer than 10 minutes. Keep in mind that you should be looking for attentive participants, not perfection. If a participant misses one attention check but provides a thoughtful open-ended response, they may pass your data quality threshold. However, if they miss two attention checks and fail to respond to the open-ended response, they are likely providing inattentive responses. Before deciding whether or not to remove a participant from your dataset, we recommend looking at other things in combination with attention checks including inconsistency in responses and extreme speeding.
For more information on attention checks, please check out our blog:
Examples of Good (and Bad) Attention Check Questions in Surveys (cloudresearch.com)
Comments
0 comments
Please sign in to leave a comment.