The art of formulating good questions
Take your time to turn your features and ideas into Kano survey questions. From good questions come good results.
Clearly define what you want to know. Make it easy for your customers to understand what you want to know from them. Don’t tell the customer how a feature works, just explain what it does for them. Do this very clearly and unequivocally.
What is it that you want to know?
Think hard about what you exactly want to know. It’s no use asking your customers questions that, when answered, won’t help you any further.
For every question you create, play around with the possible answers. Does every possible answer make sense?
Also have a look at every possible Kano category. If the feature your question is based on falls into one of the categories, what does that tell you? Does the category tell you what you wanted to know?
Say you did a survey containing this pair of questions:
When opening a new bank account, how would you feel if you were provided with a mobile banking service?
and
When opening a new bank account, how would you feel if you were not provided with a mobile banking service?
Suppose that after doing the survey, the feature appears to be a Must-Be feature. That means customers think getting access to the mobile banking service when opening an account is only natural. They’d be unsatisfied if they weren’t given access to your mobile banking service.
Is this what you wanted to know? Or were you in fact trying to find out how you should provide access to the mobile service? If that’s what you wanted to know, the survey results won’t tell you that.
Suppose the feature turns out to be a One-Dimensional feature, meaning the better the provisioning of the mobile banking service, the more satisfied the customers will be. I’ve seen teams interpret this as a blank check to start developing a whole slew of features for the mobile banking service.
But look again at the question and the category. The customers' responses indicate that the fact of being provided with a mobile banking service is what will contribute to their satisfaction with your service. So delivering access to the service should be as effortless as possible: the less paperwork the better, the more automatically it happens the better, and so on.
The survey result does not mean that customers want the mobile banking service as such to be hyper-performant. That’s another matter entirely and that was not what you asked.
Were the feature an Attractor, that would mean customers would be positively surprised by getting access to a mobile service. It means you should start working on the process of automatically giving access and emphasise it more in your marketing communication. It does not mean customers feel the mobile banking service as such is attractive. It only means customers feel that getting access to the service is attractive. If you wanted to know how customers felt about the mobile banking service, you didn’t ask the right question.
If the attribute falls into the Indifferent category, it would not mean customers do not care about the mobile banking service. It means customers don’t care about being given access to the service when opening a new account. Again, that’s a completely different matter.
You get the gist. Run through the possible results when formulating your questions to see whether the answers you’ll get are the answers you are looking for.
Focus on the outcome or presence, not on the how
There are three types of questions you can ask morning commuters about how they would feel about changes to their milkshake.
Technical questions (don't ask these)
You can ask customers how they would feel if extra fats were added to their milkshakes after the ice cream was aged for a few days. (Because that’s how you make ice cream thicker).
But:
The customer is not interested in how but which of his problems will be solved. If one asks about the technical solutions of a product, it can easily happen that the question is not correctly understood (Matzler, 1998).
You don’t want to ask technical questions. Customers don’t understand what the value is of aged ice-cream and added fats. They do understand however the value of a milkshake that takes a bit longer to finish or keeps them from getting hungry before lunchtime.
Ask outcome-based questions...
Focus on the outcome. Ask customers how they would feel if it would take longer for them to finish the milkshake.
Outcome questions are great questions to spur innovation. If you know what outcomes are most valuable to customers, you know in what context you should start ideating about product features.
Outcome-based questions are
measurable, controllable, actionable, devoid of solutions and stable over time (Ulwick, 2016)
So don't ask: "The brakes work really well", but "When braking, the car goes from a 100 to 0 mph in 3 seconds".
... or ask presence-based questions
You will have discussions with your team about the usefulness of a Kano study based on outcomes, because it does not really seem to help with deciding how to implement a feature.
"If we ask customers whether they would like their milkshakes to take longer to finish, how are we going to know whether we need to make the ice-cream thicker or provide smaller straws?", the team might object.
You can ask presence-based questions in your Kano survey. Just don't mix outcome-based questions with presence-based questions. To avoid confusion, your survey questions must maintain unity.
Maintaining unity in the types of questions
Before you write your questions, cluster your questions into desired outcomes and features.
If you’re trying to determine whether customer satisfaction will increase when it takes longer to finish the milkshake, that question should figure next to other desired outcomes like “one milkshake will make me feel filled until noon”. It should not stand alongside a presence-based question like "The milkshake contains bits of fruit".
Stick to the same hierarchy
If it turns out “the milkshake takes longer to finish” is a feature that contributes to customer satisfaction, you can look into it further with a separate survey. (Because you should never mix outcome-based questions with presence-based questions).
In that follow-up survey, you can go to a lower level in the hierarchy: the one where you try and find out what attributes of the milkshake that will make it longer to finish will contribute to customer satisfaction the most. A question about extra bits of fruit will figure next to questions regarding thickness, larger containers or thinner straws.
Never mix hierarchical levels. They will confuse the survey participant and lead to less useful answers.
Suppose you’re asking customers about milkshakes taking longer to finish and about thinner straws in the same survey. And suppose a customer does not like the idea of needing more time to finish a milkshake.
Her reasoning could be “If it would take longer to finish the milkshake — which I do not like, but anyway — I would expect that to be because the milkshake is thicker, and not because of a thinner straw”. So she'd answer positively to the presence of a thicker milkshake. But she could also answer that she doesn't care about a milkshake being thicker, because she dislikes the idea of needing more time to finish a milkshake. Her answer is not about the presence of thickness only, it's also about needing more time to finish it.
The team won’t know whether her answer was concerning the thickness of the milkshake, or about the time it takes to finish the milkshake. Worse still, they may not even know they don’t know.
To prevent confusion and avoid suboptimal outcomes, cluster your features and create a hierarchy. To get the best results from your Kano study, make sure your questions are from the same hierarchical level.
If you must use features from across different levels, put them in separate surveys. If you want to know whether your ideas of how you could increase the time to finish a milkshake and on how to make sure the customer is not hungry until noon are valuable, do two separate surveys.
Introduce your presence-based surveys with a statement like "We're making a milkshake that takes longer to finish". That way, the customer will know the context of the questions that follow.
TODO
Don't mix categories
The same applies to feature categories. Don't mix questions that pertain to the underlying milkshake quality of taking longer to finish with questions that have to do with fending off hunger until noon.
The best thing to do is create separate surveys per category. Sometimes that's impractical, however. There may be too few questions for a given category, or you only have one chance of surveying customers.
If you must ask questions from different categories in the same survey, make it very clear that the questions are clustered around a category. Use a heading, an introduction or a separator to make it clear to the customer that she's moving on to another mental construct.
Go from the abstract to the concrete
If you feel you’re still leaning on too many assumptions for your survey, move to a higher level of the hierarchy.
If you’re not sure whether a milkshake that takes longer to finish would increase customer satisfaction, do a survey on that level first. Don’t start with the details if you’re not sure about the overarching idea.
TODO
If you're doing a remote survey, i.e. you're not interviewing the participants but only asking them to complete the survey, the more concrete your questions are, the better.
Oral interviews give you the opportunity to clarify and explain broader concepts and more abstract questions. You don't have that luxury with remote surveys; all the clarity must come from your introduction, your questions and the possible answers. The more concrete they are, the easier they will be for the participant to understand.
Don’t ask two questions at once
Consider this set of questions:
Suppose that this turns out to be a “Must-Be” quality of the milkshake. What does that even mean? Do customers expect extra bits of fruit and a thinner straw? Or do they expect one or the other? Should the team start working on both, or on only one of the two?
This is what is known as a double-barrelled question. You're asking several questions at once, and the answers you'll get will be ambiguous.
Never be afraid to split up your questions or push them to a higher level of abstraction. (And then make sure your other questions are on that same level of abstraction too of course).
Don’t dismiss what seems like an obvious feature
In 2021, a group of researchers did a Kano study on the quality of transportation services at mega events (Chen, 2021). One pair of questions in the study read:
How would you feel if clear information were provided about the transportation services?
and
How would you feel if the information about the transportation services were unclear?
Surely, nobody would want information to be unclear? Why would the researchers include such an obvious feature?
Everyone at the transport company will probably agree that clarity of information is a necessary attribute of their service.
But when is clear clear enough? Is it ever clear enough? Should the company invest in mobile apps, integrations with map services like Google Maps, an open API, more ubiquitous signage? Should it keep looking for new and better ways to provide information to travelers?
The answer depends on how the feature determines customer satisfaction. Will travelers be satisfied as long as what they consider the norm is met? Or will increasing investments in providing information also increase satisfaction?
Remember that a Kano survey is not about finding out if a certain service attribute is desirable or not. It’s about determining what Kano category the attribute belongs to. And that may be less obvious.
If clarity of information turns out to be a must-be attribute, investments above a certain level will not contribute to higher customer satisfaction. If on the other hand, information clarity is a Performer attribute, the transport company will have to continuously invest in clarity of information. Merely matching expectations will not be enough.
When you’re thinking of excluding a feature from your Kano study, don’t look at how obvious the feature is. Instead, think about how obvious its Kano category is. More often than not, you won’t be able to tell, and you’ll want to ask your customers.
Functional and dysfunctional must not always be exact opposites
You're doing a Kano survey because you want to know how customers feel about certain attributes of your product or service. You don't always want or need to know how customers feel about the presence or absence of a feature. You may want to know how they feel about the level of performance of the feature.
Say you're responsible for customer services at a public transport company. For technological reasons, you're not always able to present precise information about arrival times to travelers. The issue could be fixed, but it will be costly.
One parameter in your decision will of course be: how does this impact customer satisfaction? In order to know, you want to know how customers will feel if the arrival times are sometimes wrong.
So instead of asking:
ask this:
Be clear and be specific
Compare these two examples from Mikulić and Prebezac (2011):
When opening a new bank account, how would you feel if you were provided with a mobile banking service?
When opening a new bank account, how would you feel if you were not provided with a mobile banking service?
and
When opening a new bank account, how would you feel if you were provided with a mobile banking service that works very well?
When opening a new bank account, how would you feel if you were provided with a mobile banking service that works very poorly?
For the first set of questions, the majority of answers (59%) were in the “Attract” category. In the second version that dropped to 22%.
Some (Grapentine, 2015) consider this a sign of the Kano model’s unreliability. But It isn’t. This study is just another indication that you should think hard about what you want to know and how you formulate your questions.
The initial set of questions considers a binary option: either the feature is present or it isn’t. You’re asking customers how they would feel about the mobile banking’s presence, not about its performance. You want to know whether you should even investigate the idea of a mobile banking service.
From the results (55% Attractive), it’s clear that a mobile banking service would be a competitive differentiator.
Had you used the second set of questions, you’d have been trying to find out how the service’s performance impacts customer satisfaction. That’s a different question; hence the different answers.
But to be honest, the second pair of questions is badly thought out. Everyone has a different idea about “a mobile banking service that works very well”. The question is unclear and not specific. It is far too open for interpretation.
Consider the previous car brakes example. TODO
Brakes very well versus brakes in 3 seconds
A litmus test of clarity and specificity is: can people point to it and agree it is what it says it is?
You can point to a mobile banking service and ask customers whether they too think if what you’re showing them is a mobile banking service. Everyone will agree it is indeed a mobile banking service. Therefore, it’s a good topic for a question. “How do you feel if a mobile banking service were present?” is a good question.
But what is “bad performance” or “good performance”? Some customers will think financial errors are a sign of bad performance, while others may consider a slow service frustrating enough to call it bad. The problem with this question is not its subjectivity. After all, you're doing a Kano survey to find out the subjective feelings of customers. Its problem is that it's too general. It is not precise enough to get useful answers for.
To get useful results, the researcher should have split up the "good performance" feature into its parts. A separate survey with questions on financial errors and the app's loading times would have delivered far better results.
A question like “How do you feel if the mobile app is fully operational after 3 seconds of loading time?” is a valid, clear and precise question. Always aim for clarity and precision. Avoid broad, sweeping questions.
Don’t bother with features that have no other value
Remember that the Kano model is a tool to help you decide what features to develop or improve based on how their impact on customer satisfaction. But customer satisfaction is not the only parameter in organisational decision-making. A feature’s contribution to business goals and the effort required to make, improve and maintain it are equally important factors.
Even if you don’t use a formal method to calculate it, this concept helps you decide which features or improvements you can leave out of your product roadmap.
Features that contribute little to your business goals and require a lot of effort – even if they are of high value to the customer – do not return enough of that value to your business. So there’s little use in pursuing them further.
You can safely remove such features from your survey.
How many questions go in a survey?
#TODO
Last updated
