Comments on: User Surveys – Do it Right or Not at All.

By: Gary Barber

Gary Barber — Sat, 20 Feb 2010 06:30:15 +0000

@dave This is where testing the survey, especially these responses is important.

You have to get them just right. You are really looking at responses that will mean or be weighted equally overall across the audience, ones with a good statistic spread. However they also have to give a preferential bias for each audience group.

Otherwise having alternative responses that are just additional filling is a waste of time. Maybe this is even the wrong style of question for the information you want to capture in this case.

What is survey experience, if over 5 surveys your boss has discovered that the data was biased, that mistakes were made. And on the subsequent survey they improved their skills and technique.

Then yes they are on the way.

But if they just repeated the same skill set with no real thought. Then reality is they aren’t gaining experience.

It’s the old adage: 10 years at a job = 1 years experiences repeated 10 times.

Sadly I know that the business world is not black and white and if I stand up here on a soapbox and say – “DON”T do it”, people will anyway. So better to educate them a little in the ways of doing it correctly.

By: Dave

Dave — Fri, 19 Feb 2010 14:28:47 +0000

I disagree with scrambling multiple choice response order, especially as advice for amateur survey designers. Scrambling assumes that the text of the response choices will absolutely be interpreted the way intended by the author.

What I see too often are choices like:
– it’s OK
– it’s acceptable
– it’s good
– it’s bad
– it’s great

When poorly written choices are scrambled, there’s no way to know how the author differentiates “OK” and “acceptable”. Of course, the ideal solution is to not have poorly written choices, but leaving the choices in order helps protects against problems from poorly written choices.

I would really, really like to see a stronger post that outright says “If you design your own survey and you don’t have years of task-specific training, you are introducing unacceptable bias and error and your survey data is mostly useless.” As this post stands, I can’t send this post to my boss because he’ll think that having done 5 surveys in the past counts as “survey experience”, which means he’s safe to keep doing what he’s doing.

By: Gary Barber

Gary Barber — Thu, 18 Feb 2010 02:00:03 +0000

@eric – but you don’t want 5 questions all likert scale, all the same number of responses in the matrix, you’ll just end up with the same results for the most part. At least its not a 25 question nightmare

By: eric

eric — Thu, 18 Feb 2010 01:50:41 +0000

Don’t have too many different types of question formats.

Eg. Q1 is multichoice, Q2 is likert scale, Q3 is open text, Q4 likert scale, Q5 “rank these” grid, Q5 yet-another-format, Q6 omgwtfbbq my brain just melted.