Questionnaire Length: The Long and Short of Participant Engagement

Survey length in online interviews continues to be a bone of contention among people all along the marketing research continuum. While some within the industry recommend keeping questionnaires short to promote participant engagement, others often dismiss this advice. Short questionnaires, they assert, won’t unearth the depth and breadth of information that is needed.

As a sample provider offering guidance based on research, SSI suggests keeping online interview length at 20 minutes or less. Generally, when interview length increases, fatigue also increases and, conversely, attention span decreases potentially damaging data integrity. In effect, researchers who insist on longer questionnaires, sincerely believing they’ll get more information, may, in actuality, be sabotaging their efforts.

Research Design

In order to more fully understand the possible effects of survey length, fatigue and subsequent response quality, SSI recently fielded two surveys: one long and one short. This study replicated a ground-breaking study conducted in 2004 by Sandra Rathod and Andrea la Bruna which concluded, among other things, that data quality suffers as interview length increases.

The surveys, both then and now, utilized a block design and were divided into four blocks of questions, each representing a different subject matter. The blocks were randomized for each respondent so that the effect of survey length on response quality could be investigated by comparing whether the different order of the blocks led to different response patterns as the block position varied in the survey.

Fatigue Effects

One of the hypotheses of the 2004 study was that respondents would take less time and exert less effort later in the questionnaire than they would earlier in the questionnaire, due to fatigue. This hypothesis was proven in 2004: as the same block of questions was moved further back in the study, the time taken to complete it gradually reduced.

It could be argued that the decrease in block completion time was due to increased familiarity with the question set. It is true that the question blocks were similar in their construction and contained somewhat similar questions. However, additional evidence on panelist fatigue shows that at least some of the increased speed was due to fatigue.

Panelist Fatigue and Satisficing

One of the behavior outcomes of cognitive fatigue is satisficing – doing just enough work to satisfy the task. To see if this behavior was present in the 2004 study, researchers looked at a question, in each block, that it was possible to skip. This question offered a set of scales, presented in the form of sliders. The slider bar was positioned at the mid-point so it was possible to click on “next” without moving the slider and still leave some data behind. The likelihood of skipping the question rose as the skippable question was encountered further and further into the questionnaire.

In 2009, SSI found precisely the same pattern. The first time the skippable question was encountered it was more likely to be completed than when it was seen on subsequent occasions.  This was particularly true for the long survey. A reduction in elapsed survey times did not mitigate the effect. The long survey, at nearly 25 minutes, was still too long.


In both 2004 and 2009, the long survey proved itself too long. It fatigued the respondent and led to satisficing behavior. When questions could legitimately be skipped, they were. Perhaps the most unsettling finding was that the instances of cheating, deliberately telling a falsehood in order to skip an entire section, also increased as the survey progressed.

Following the 2004 study, researchers indicated that there is a “critical point in online survey response when the fatigue effects become significantly more pronounced. That critical time is around the 20 minute mark. If researchers work to keep surveys shorter, it will not only help ensure response quality, but it will also make for more motivated and responsive respondents.”

Today’s research confirms that interview lengths of 20 minutes or less can produce wonderful and engaged responses if well designed. The fact that there was much less satisficing and cheating in the short survey attests to this. If researchers work to keep surveys shorter, it will not only help ensure response quality, but it will also make for more motivated and engaged participants.

Note: For a copy of the white paper Questionnaire Length, Fatigue Effects and Response Quality Revisited, with the results of the studies cited in this article and to share your comments, go to

About the Author: Pete Cape is Global Knowledge Director for Survey Sampling International. SSI provides access to more than 6 million research respondents in 72 countries. Sources include SSI proprietary panel communities in 27 countries and a portfolio of managed affiliates. SSI can potentially access anyone online to give their opinions via a network of relationships with websites, panels, communities and social media groups.

Reblog this post [with Zemanta]

Ivana Taylor is the publisher of DIYMarketers and the President of Third Force, a strategic marketing firm. She's the marketing expert and book editor for Small Business Trends and a frequent contributor to AMEX Open Forum.

Tagged with: , , , , , , , ,
Posted in Best Practice, Newsletter
6 comments on “Questionnaire Length: The Long and Short of Participant Engagement
  1. the guy says:

    I agree, surveys have to be quick, if they’re more than 10 questions, my self-reflection becomes more and more inadequate and i just start checking…

  2. Greg Timpany says:

    I think 10 questions may be too limiting, but agree that if average response times are over the 10 – 15 minute mark that data quality begins to erode significantly. My thought is that longer questionnaires can be broken into shorter ones and respondents can be invited to participate in the subsequent studies. May require larger samples, but should help to increase data quality.

  3. Lisa says:

    I completely agree with the last post. It’s hard to stay focused on endless questions asking the same thing a different way.

  4. […] – that respondents “check out” after 15-20 mins. Our buddies from Survey Sampling say that again a couple of months […]

  5. […] – that respondents “check out” after 15-20 mins. Our buddies from Survey Sampling say that again a couple of months […]

  6. […] Questionnaire Length: The Long and Short of Participant Engagement = March 31, 2010 […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Get Embed With Us

March 18th, 11am PST / 2pm EST

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 16,827 other followers

Follow us on Twitter

FREE Training

Weekly Training Webinar
Seating is Limited!

%d bloggers like this: