Biases, UX Validation and GOOB


Walk Sign

There is a big movement to support project teams getting out of the building (GOOB) to validate product assumptions and the like.

This is a great thing, the more users and customers project teams talk to the better. After-all something is better than nothing, right?

However I have seen this research done very poorly to the extent it is pointless.

Often the people going out have no experience, have not been taught the basics and haven’t even had a few “dry runs” on their approach or questions.

This can lead to all the information captured being worthless.   In fact it can be worse than that, sometimes it can re-enforces the belief that the team is approaching things the right way, when in fact they are not.

Bias to Watch for.

There are a whole series of tricks that our minds play on us (biases) that we can bring with us, even before we start talking to people.  And even then there are some that can come to light during questioning from the interviewee and interviewer.

Interviewer biases:

  • Confirmation Bias

    This is one of the major ones we always have to look for.  It’s the tendency to interpret information only that confirms your preconceptions.  In a way we force the conformity, by only examining what we think is right.  Often we make assumptions and don’t completely review the facts as they stand due to this bias.   Causality comes into play here.

  • Affective Heuristic

    This is where we make decisions for an outcome based on superficial evaluations, which aren’t relevant to the study at hand.   We look for confirmation by anchoring and diagnosis.

  • Diagnosis Bias

    This is our tendency to label people based on our first impressions.   Once we label like this it is very difficult for us to change this preconception.

  • Anchoring

    This is when we rely on one trait or information too heavily to support a finding, decision or hypothesis. We will start to change our perception and discount other traits in favour of one aspect that we are focusing on.   When we anchor we often extrapolate on an aspect of a person and that influences our approach to the subject.

  • Acquiescence Bias

    Avoid asking questions that are basically in a agree or disagree in format will lead to distinct patterning towards one outcome, usually agreeing.   People tend to agree with others as the path of least resistance.

  • Question Order Bias

    There can be a degree of bias from the way you ask the questions.  A rule of thumb is to use this order:  General before Specific, Positive before Negative, Behavioural before Attitude.

Interviewee biases:

  • Consistency Bias

    Some people may try and be consist in their answers.  This means the first answer will influence and taint the following responses, even if they are negative.

  • Memory / Error Bias

    Human memory isn’t that good, we tend to make mistakes and gloss over bad points in our memory or reduce the impact of them.

  • Availability Heuristic

    We can remember or imagine vivid, easy to picture yet uncommon events first. Also recent events are often recalled first over past events. This is the old fading memory trick.

  • Hindsight Bias

    Once we know about something, we find it hard to recall when we didn’t know about it. This does not help with learning from failures.

  • Hostility Bias

    Some people will just be hostile with an interviewer and provide negative responses.   If it gets to short angry replies, let them go.  There may also be prejudicial bias in play as well.

  • Acceptance Bias

    Sometimes people answer questions a certain way just to please the interviewer and to be accepted by them.  They will interpret what they believe they think the interviewer wants to hear, not their own views.

  • Social Acceptance Bias

    Interviewees may only provide information that is socially acceptable within a conservative society.  They will warp the truth or just offer up half-truths.   This is based on people wanting to conform and belong to a group.   Indirect questions usually uncover the real direction.

  • Mood Bias 

    People can be in an extreme emotion state, and you will not know it.   This will be reflected in their answers.  For example executives or medical professionals tend to give short, clipped answers, usually due to stress.

  • Overstatement Bias

    It’s not unusual for people to overstate their opinions on a subject.   This is particularly true with purchasing decisions.  Watch out for this one, combined with acceptance bias and it’s a game changer.

  • Reference / Framing Bias

    Watch for people extending a frame of reference from one question into an unrelated question.  As this will change the outcome, as they will force the replying to the perceived frame of reference.

  • Associative Bias

    People can sometime know the brand or even the company you are from, they may have an opinion on that company.  This association will influence their response and bias it.   It’s better to conduct questioning in an environment devoid of branding.

The Right Questions

Asking the right questions when you “Get out of the Building” is critical.   Usually you only get one chance at this.

You want:

  • Clear and Specific Questions

    Ensure the question is clear with a defined intent on what you are asking about.

  • One Question at a Time

    Only ask on question at a time, this way the interviewee can focus on the question and respond to it fully.

  • Open Questions

     These are questions that allow people to fill in the gaps, they allow people to tell a story or expand on a short answer.  Open questions can’t be answered with a yes or no reply.

  • Neutral Questions

    These are questions that don’t favour or suggest a direction.

  • Deeper Probing Questions

    These allow you to prompt the interviewee for more information.

  • Balanced Questions

    You need to consider non-sexist, non-discriminatory questions. Now I’m not saying that they need to be completely politically correct.  Just watch the way you say things.

You don’t want:

  • No Leading Questions

    These are one of the biggest issues, I have found.  Questions that give people a predirected response and presenting a certain viewpoint.

  • No Emotional Intent

     Ask people if they like or hate an idea or product.   To avoid this ask them instead what they want to achieve with the product.

  • No Jargon Loaded Questions

    Avoid asking questions that are filled with industry jargon, from UX, tech, to business or government speak.   Make it very clear and easy to define everything in a question.

  • No Short Answers

    If the interviewee is giving short answers, let them go, move on, you are just wasting both your time and theirs.

Any questions you prepare should be tested on people outside your team or better your audience before you get out of the building or consider the results valid.

Just treat this as case of micro iterative development, it’s just to ensure the question has no bias to start with, that it makes sense, and is in the language of the customer.

Probing Questions

Often the first response to question no matter how open can be limited.  It’s then that you need consider finding more depth by probing questions such as:

  • What did you do then?
  • Why did you do that?
  • Tell me how you did that?
  • Tell me more about that?
  • Why is this important to you?

Appearance and Body Language

Your appearance, and body language can severely influence the outcome of any questioning.

What you wear can be critical, in some ways you need to mirror the dress codes of the people you are questioning.

The tone of your voice, mannerisms, facial expressions, style of speech all influence people.    Also consider age, gender, race, and social status, like it or not these also can change peoples perceptions.

Body language is a big aspect of this, you need to practice non-aggressive, neutralising gestures and be very aware of the limits of peoples personal space as you approach them.

If you can control some of these aspect do so, as that first impression people get of you will set the level of bias you will encounter.

Don’t go Alone

I would recommend doing everything in pairs, anything bigger is a group and frankly a little intimidating to some people.

By doing research with someone else is of great benefit.  You will be able to have one person focus on the questions, and another focus on collecting the response, including any body language cues.   It’s these other cues that are often more important that what is said.

You will both notice different things at different times, so remember to debrief.

Beyond the Questions

General research such as face-to-face interviewing, surveying and questioning is great but there will be times that you need to consider more intense methods to validate your assumptions such as:

  • A day in the life studies
  • Ethnographic studies
  • Contextual enquiries
  • Shadowing

Don’t let this all stop you getting out of the building, just consider that maybe you need a little assistance in doing so,  after all you want to maximise your outcomes from your research.

Tags: , , , , , , , ,

Looks like there is no conversation here yet, why not start one.