Trade-Off Analysis Part 1

MAXDIFF

In an earlier blog post, we explored the benefits of MaxDiff scaling as a form of trade-off analysis. However, we just scratched the surface of trade-off analysis, or choice-based modelling. In part one of our deeper dive into trade-off analysis we will focus on MaxDiff scaling and explore choice-based conjoint in part two.

We previously wrote about a business objective that addressed the need from a marketing team to identify the best set of features for their new product offering that launches in the next six months in three countries – US, UK and China. The first thing that may come to mind is to conduct a survey asking people to rate each feature on a scale from ‘least appealing’ to “most appealing”. Scales are easy for people to understand and you will be able to see which features are rated highly more often. The problems with this approach are:

  • Many features having similar ratings for “top box” or “top two box” ratings. This may leave you just as confused on what is important as before.
  • Given that you will be talking to people from different countries, you may experience scale bias (i.e. people in certain cultures responding more favorably).
  • The list of product features can be quite lengthy, up to 25 items.

Problem: Rank Importance of an Attribute List

Solution: MaxDiff Analysis

Maximum Difference Scaling, or MaxDiff, is a technique that provides a relative importance rating for each item in the list, free of scale bias, and drives differentiation in making trade-offs. MaxDiff can work with lists of 10 to 30 features in a way that does not make the exercise for respondents difficult. The results can also give you an idea of the degree of importance. Instead of just saying that Feature A is favored over Feature B, we can provide a sense of magnitude as well. The ranking exercise will force people to make trade-offs across multiple screens of three to five items. They will often be asked to rate the item they like “best” and the item they like “worst”. An experimental design sits behind the scenes, and versioning is distributed in such a way to create a balanced design. Once all respondents have all gone through the exercise, we can determine their share of preference for all features shown.

MaxDiff can also be leveraged in at least two other ways that are common and benefit from a trade-off or choice-based methodology.

  1. Line, message or menu optimization
  2. Segmentation of consumers

Problem: Line, Message or Menu Optimization

Solution: MaxDiff Analysis + TURF

The differentiation, lack of scale biases and engaging survey experience are all advantages of choice-based modelling that make a TURF or Total Unduplicated Reach and Frequency analysis more appealing. TURF can tell you which collection of features is likely to cover the most unique consumers within your target audience. Traditionally, TURF is executed with ratings on a list of messages, flavour/menu items or products on a five- or seven-point scale.  With traditional scales we often lose differentiation on large lists of items.

Utilizing the foundation of the MaxDiff analysis and all its benefits of trade-off scaling, a TURF analysis can be conducted leveraging the raw MaxDiff utilities. The binary coding of “reached” vs “not reached” can be determined based off a threshold above/below the average respondent utilities. MaxDiff+TURF gives added benefits of not just understanding feature importance and magnitude, but which combination of items reach the most unique respondents all within one survey exercise. Additionally, identifying the optimal number of items in the message, menu or line will help create the most efficient business outcomes.

Conducting the TURF analysis from MaxDiff utilities mitigates scales bias issues, provides greater discrimination and allows the researcher to reliably ask respondents to evaluate longer lists of items.

Problem: Identifying Sub-Groups of Consumers

Solution: MaxDiff Analysis + Segmentation

Through several use cases, ENGINE has seen the power of MaxDiff methodologies to identify segments of consumers. Utilities from Hierarchical Bayes estimation from the MaxDiff exercise can be used to identify sub-groups of consumers who may have different preferences or tastes. By not having extensive grids of attributes to evaluate with traditional scales, respondents are less likely to straight-line responses and often provide more meaningful responses through trade-off approaches. Showing smaller choice tasks, albeit somewhat repetitive, respondents are forced to make unique trade-offs on each screen and thus provide useful information on all attributes. Since each item is shown multiple times across the choice exercises, we can better understand implicit appeal of items as respondents provide impulse evaluations at each screen.

While utilizing MaxDiff utilities to create unique segments of your target population does create strong, differentiated segments it does create challenges in developing a typing tool to find these segments in future research. Due to the tradeoffs it forces respondents to make at each screen, the differentiation or choice-modelling is tough to replicate through traditional scales that are often used in typing tools. It’s not to say that you can’t later identify the segments discovered through a Maxdiff Segmentation, you just may need to run the MaxDiff exercise in each future study to be able to find your newly created segments.

Written by Mike Miller, Research Director at ENGINE Insights.