Selecting survey participants

Who to invite

Once in a while, someone plasters social channels with an invitation to complete a Kano survey. That person is probably looking to convince management by numbers. Very bad decisions will come of this.

You don't care about a random person's attitude towards features of your product. Think about what you want to know. If you want to know how adding extra bits of fruit to a breakfast milkshake will impact a morning commuters' attitude towards it, ask morning commuters who habitually buy breakfast milkshakes. So don't ask moms who come in at 4PM to buy a milkshake for their six year olds.

You want to have survey participants that have a (future) relation with your product. As existing or potential users of your product or service, they must be able to see its value. Otherwise the answers you'll get will be irrelevant. Uninvolved participants will give you what they think is important to a customer of your product, you won't find out what actually is important to your customers.

Long-time users of a product or service will attach different values than first-time users. Mittal (2001) cites Kekre et al. (1995) who found that among software users usability and documentation were of higher importance for novice users, while reliability and capability were more important for long-time users.

The consumption goal (the reason why they use your product) is another important parameter. It's not only product usage frequency or length that matters. Race car drivers have different consumption goals than commuters. Professional writers attach different values to Microsoft Word's features than an office clerk. People who really have to lose weight will answer differently to a survey about appetite-suppressant capsules (Olive et al., 1995) than people who merely wants to lose a few pounds.

In the same article, Mittal raises an important point about customer self-selection. Suppose you have a restaurant. You've done some great Kano studies, and found out that:

• New customers value the prices

• Long-time customers value the different meat options on the menu

Now you're wondering whether to introduce a vegetarian menu, so you do a new study. It turns out new customers consider it an Attractive feature, but returning customers are mostly indifferent. You decide not to add a vegetarian menu, as it doesn't seem to contribute to customer retention. Vegetarion options on the menu won't seem to incite customers to return.

Doesn't it though? Long-time customers don't need new options on the menu, because what they value the most is currently on offer in your restaurant. That's what makes them returning customers. But you're only surveying people who stayed a customer. What possibly happened here is a case of customer self-selection; you didn't find out what customers who did not return value.

If you had surveyed people who have eaten at your restaurant only once or twice and never returned, the results could have been much different. (It's not easy to reach out to ex-customers, but that's another argument for doing Kano surveys with the same people repeatedly, across their customer life-cycle.)

When selecting survey participants, think about who you want to know what from. Parameters that can help you make this decision are:

• their level of expertise with your product or the complexity of their consumption goals;

• their position in the customer life-cycle (and don't forget about lost customers).

How many participants do you need?

Determining the optimal amount of survey participants is a field of study on its own. It is highly dependent on the subject of your survey, the population you want to survey and what attributes you believe will influence their answers.

But as a rule of thumb, 10 to 15 participants per distinctive, relevant population attribute is sufficient. You'll start to detect patterns even as early as the first 4 to 7 interviews.

If you are certain you can find participants who use your product for distinctively different reasons or have different levels of involvement (for example first-time and regular users), try and survey 10 to 15 of each.

Don't overthink this: you don't want to look at all the participant attributes and wonder whether you should add another 10 participants for each attribute you can think of. Work backwards and ask yourself: "what do I want to know"?

Do you want to know how first-time users evaluate the different features of your product? Then survey 10 to 15 first time users. Do you want to know whether there is a difference between frequent and infrequent users? Then survey 10 to 15 of each.

Again, don't overthink this. Most of the time, you'll find out what participant attributes are relevant after you've done your survey. If you can't distinguish between first-time and regular users, simply be aware of that when analysing the results of your study.

It's more beneficial to take your time during the survey to understand who the participant is and why she uses your product or service than fretting over segmentation criteria beforehand.