I recently wrote about the difference between selling and learning when engaging customers, and mentioned that learning is affected by cognitive biases. This time, I’ll talk about the biases that affect customer interviews, and tools for overcoming them.
How to Avoid Fooling Yourself When Interviewing Customers
“If I had asked people what they wanted, they would have said faster horses.” Attributed to Henry Ford
Ford was unable to get useful answers from his customers. This is a common struggle. When you interview customers, both the interviewer and interviewee are influenced by what psychology calls cognitive biases. These are thought patterns that distort how we perceive and communicate about reality. They really get in your way! In customer interviews, most biases affect the customer, but it’s the interviewer who needs to find a way around them – or risk building an irrelevant product.
If you are unaware of such biases, they will be harder to avoid. So today I’ll discuss the most common biases in customer interviews, and what you can do about them.
Cognitive Biases Affecting Customer Interviews
Biases affecting the customer (interviewee):
People don’t know what they need or want.This was Ford’s main problem. Interviewees are bad at imagining hypothetical situations, and terrible at predicting their decisions and experience in them. They are generally not aware of this limitation. When you ask broad hypothetical questions, people’s answers are often very close to their existing reality, in scope and timeline. They fail to consider ideas that are significantly different from current solutions, or that occur on a longer time horizon.For those reasons, you can’t directly ask the most important customer research questions: What are your needs and challenges? What would you want to improve in the way you currently do things? What might be a good solution to your challenges? How would things change if you used my product, and would you like it? Again, these are, unfortunately, the questions you want to ask, but really shouldn’t. Some of this information can be discovered through indirect questions and other techniques – see the tools section.
Acquiescence (agreement) bias. Interviewees tend to answer questions with a bias towards agreement and positivity. This happens unconsciously, and often skews the data collected in interviews and surveys. The bias is worse when interviewing people with a vested interest in your success.How you phrase questions heavily influences this bias (the framing effect). The answer to a yes/no (agree/disagree) question will likely overstate the positive option. Similarly, when the interviewee perceives a specific answer as preferable to the interviewer, they become biased towards it: responding otherwise feels awkward, and they’re trying to help you! Somehow, this tendency to please skips teenagers, especially if the interviewer bears any resemblance to their parents.Specifically, interviewees often overestimate elements such as the price they’ll be willing to pay for your product, the value of product features, and their future willingness to overcome obstacles to using your product.
Anchoring. When interviewees have a reference point (an anchor), it influences their thinking (and responses to questions): They will answer questions in the context of the anchor, rather than in general.This is a problem when an interviewee is aware of your product or service, which becomes the anchor. They will focus their answers on needs and challenges they see as related to the offering, rather than provide fuller answers – the answers you need to develop more innovative products or features.
Biases affecting the interviewer:
Confirmation bias. Confirmation bias is a tendency to pay attention to details that support your beliefs, and discount those that challenge them. This is probably the most important bias on this list: you become biased towards “learning” that your assumptions were correct. In my previous post, I mentioned that the sales mindset is geared towards finding agreement, and therefore, when you’re selling, you become especially vulnerable to this bias. This is a key reason for distinguishing between sales and learning.
An important special case of confirmation bias is congruence bias, the tendency to focus on testing your hypothesis only, rather than competing ones.
Availability bias.Availability bias is a tendency to give too much weight to available details, even when you know they don’t represent reality. This bias is significant when your interviewees don’t fully represent the group you want to learn about. For instance, if you happen to interview only men, your product isn’t likely to meet the needs of women. Another common case is when you interview senior managers (e.g. the CRO), but the actual users will be members of their organization (e.g. CSMs). The CRO’s perspective is critical – they will decide whether to buy your product – but their understanding of the CSMs’ daily needs may fall short.
Tools for overcoming these biases
These tools improve learning by directly addressing some biases, and creating structures that reduce the risk of others.
Interview plan:Before an interview, create a plan, and guide the interview based on it. Plan the phrasing of questions, the sequencing of topics, and how the context is presented. Where possible, create guidelines for interpreting information that may come up. This is similar to preregistration in scientific study design, which effectively reduces bias. Every plan develops and changes during interviews, but having one that accounts for these challenges will improve your results.
Interviewee selection:Verify your interviewees represent the group you want to learn about, and that they are intended users of your product (and not primarily their managers).Where possible, select interviewees who are unfamiliar with your product. The acquiescence and anchoring biases will influence them less (verify they are not prepared or given context before the interview). For this reason, it is best to speak with each interviewee separately, so they don’t bias each other.When you interview people familiar with your offering, track this in your interview notes to account for the presence of those biases.
Focus on what people know:Focus interviews on understanding people’s current reality, to avoid their limitations in imagining the future. The better you understand their routines, the more relevant your products and services will be. Here are some good questions that help you learn what you need, and minimize potential bias:
Describe how you do X (or: describe your routine for Y).Then get more details about parts that interest you. If possible, “shadow” the interviewee for a typical day or ask them to demonstrate how they perform specific tasks. Pay attention to the methods and systems/tools they currently use, and specifically any that will be replaced by your product or relate to its use. Note the use of unsophisticated tools (such as manual note-taking) and general-purpose software (such as Excel). Track anything that seems inefficient, or ripe for automation, process improvement, etc.
What share of your time is allocated to each of the main tasks you perform? This is a way to identify inefficiencies. Estimates of time allocation tend to be inaccurate, but they give you a good idea of people’s perception of it (and where they may value efficiency gains). A more precise alternative is to leave people a take-home time-tracking survey (over a week, to account for variation), or to observe them directly.
What are your least favorite tasks / most tedious parts of your job? People find this question relatively easy to answer. It’s an indirect way to ask about their challenges, and where they could use tech solutions.
How do you allocate your budget? Where do you get the least value for your money?These are the financial versions of the previous two questions. They’re relevant for interviewees who own a budget – such as managers above a certain level, or small business owners. In most cases, the resources that people manage are time and money, but versions of these questions can be used for any other resource.
Sequencing:You will occasionally want to ask interviewees directly about your product ideas to get their feedback. It is best to sequence these questions, and the description of your product, as the last part of the interview. That way, the interviewee is not anchored in the earlier parts.
Show, don’t tell: When you discuss your product, interviewees will still be limited in imagining future possibilities. Their responses will be more valuable if they can see or feel it, so present mock-ups or demos if you have them. If you can only present concepts or general ideas, expect responses that don’t represent reality well. Any conclusions you draw from them about customer needs will require additional validation.
Ask open-ended questions: Ask open rather than closed questions to avoid the acquiescence and anchoring biases. The Magic-Wand question is a classic: “If you could change/fix anything about this, what would it be?” This one asks for people’s imagination, which is a risk, but its focus on current reality helps.
Multiple perspectives:Add 2-3 observers to an interview, who will write down their conclusions and insights separately before comparing notes. When you do this, it’s best if one person runs the interview, and the others are mostly silent, only taking notes. That way, they can observe the interviewer as well. They can notice biases the interviewer unintentionally expresses, and account for them in their evaluation of the interviewee’s responses.It's sometimes easier to record interviews for this purpose. Remember, however, that you must ask for consent to record before starting the interview. Some interviewees are uncomfortable with being recorded even when they consent, which could influence their responses.
These ideas are a starting point, that you’ll have to adapt to your company’s needs. If you study the biases and create a plan, you’ll be better prepared to interview customers than most early-stage founders. All plans, however, change when confronted with reality. To succeed over time, you’ll have to learn from each interview, and continuously evolve your approach. Good luck!
Thanks to Dustin Ramsay, who helped me develop my thinking about this topic. Subscribe to his new Substack for a deep dive into his approach to validating ideas through user research.
In my next several newsletters, I’ll cover starting startups: figuring out your team, idea, funding, and more. I’ll focus on insights from my work with second-time founders, and go deeper than most of the discussions you can find online. Please share this with anyone you know who’s just starting out – it will mean a lot to me!
I’d love to hear your feedback or suggestions — please respond directly to this email!
Comments