MaxDiff

The MaxDiff (Maximum Difference Scaling) question type is a statistical technique used to measure the relative importance or preference of multiple items.

MaxDiff is a methodologically-sound trade off analysis of a list of items, rather than a respondent ranking items on a scale directly, thus preventing some of the biases associated with traditional rating scales. Respondents are asked to select the best/worst or most important/least important item in multiple lists based on a model that has been selected. MaxDiff is often used to determine which features, attributes or benefits of a product or service are most important to the end user.

The model defines the number of answers presented to the respondent on each page, and the total number of pages. Each set of answers being compared appear on its own page. Once all pages are complete, a score, called a utility score, is allocated to each of the answers in the list. The utility score is then used to determine the most to least popular of the answers available.

There are a number of options for the MaxDiff question type:

• Answer list sorting – Determines the order of the answers presented to the respondent. If groups are used, this controls the answer order within each group (any answers not in a group will then be controlled by the group display property). Refer to Answer List Sorting for further details.

• Page sorting – determines the display order of the multiple pages for the MaxDiff question.

• Answer label position – determines the position of the answer label column when presented to the respondent.

• Model dimensions – determines the number of answers per page and the total number of pages, based on the number of answers defined for the question. To select the model dimensions, you must first define the answers in the Answers section, then come back to this option to determine the dimensions. The list only includes 'perfect' models which mean each answer is shown an equal number of times and pairs of answers are also shown an equal number of times.

Enter the attributes to be shown to the respondent in the Answers section. You must enter at least 3, but less than 30 answers, with a recommendation of at least 6.

Capturing and Presenting the Results

As mentioned above, once all pages for the MaxDiff question have been answered by a respondent, a utility score is calculated and allocated to each of the answers in the list. The utility score determines the most to least popular of answers available. It is calculated by the following method:

• Answers are allocated a score for each page presented to a respondent and stored as variables in the survey data (providing access to the raw data). The scores are based on:

  • Option not presented to the respondent: null

  • Option presented to the respondent but not selected: 0

  • Option presented to the respondent and selected as the best: 1

  • Option presented to the respondent and selected as the worst: -1

• At the end of the MaxDiff experience, when all questions have been answered, each answer is allocated a utility score based on the average score of the -1, 0, 1 values (excluding the null values).

The names of the variables are automatically generated based on the Quesion ID (QID) and the model dimensions (number of answers and number of pages). For example, if the MaxDiff question has a QID of q40 and 15 answers are defined, the utility scores would be q40_1 to q40_15. These variables can be used to chart or present the utility scores using a Studio or Viz dashboard.

In addition, the raw data for each page is also stored in another set of variables based on the QID, the page number and the answer code. For example, if the MaxDiff question has a QID of q40, 12 pages, and 15 answers defined, the raw data would be stored in q40_pg1_1 to q40_pg1_15 all the way through to q40_pg12_15. Remember that those answers not presented on a page will receive a score of 'null', so regardless of whether an answer is displayed, each answer receives a score of 'null', 0, 1 or -1 per page.

It is possible to access MaxDiff scores using JScript or JavaScript. The example shown below (compatible with both JScript and JavaScript survey engines) retrieves the highest and lowest scored rows from a MaxDiff question (md1), and the result stored in a Single Choice question type (with a question ID of 'highest' and 'lowest'). In the case of there being more than one row holding an equal highest or lowest score, the example below randomly chooses one at random.

Note that only the first three lines of code (highlighted in bold) need to be altered if you wish to use this script with your MaxDiff question type. The first line defines the question id for the MaxDiff question (in the example, md1), and the next two lines define the question IDs of the Single Choice questions that will store the highest and lowest codes.

var mdQid = "md1";  //MaxDiff question id
var highestQid = "highest";   //Single qid to store Highest scored item
var lowestQid = "lowest";     //Single qid to store Lowest scored item
var rows = f(mdQid).domainValues();  //retrieve an Array of row codes
var scores = f(mdQid).values();  //retrieve all Scores
var highest = Max(scores);  //find highest score
var lowest = Min(scores);  //find lowest score
var highestSet = set();  //create a new Set to store rows with highest score
var lowestSet = set();  //create a new Set to store rows with lowest score
for(var i=0; i<rows.length;i++)  //iterate through all row codes
{
  var currRow = rows[i];  //the current row code
  if(f(mdQid).item(currRow).get() == highest)
  {  //if current row is scored highest then add to highestSet
    highestSet.add(currRow);
  }
  if(f(mdQid).item(currRow).get() == lowest)
  {  //if current row is scored lowest then add to lowestSet
    lowestSet.add(currRow);
  }
}
//use custom function punchSelected() to switch on answer from Set into the lowest/highest questions
punchSelected(highestSet,highestQid);
punchSelected(lowestSet,lowestQid);
//custom function to select answer into the highest/lowest questions according to scores
//if more than 1 row has same highest/lowest score then selection is done randomly:
function punchSelected(aSet,targetQid)
{
  if(aSet.size()==1)
  {
    f(targetQid).set(aSet);
  }
  else if(aSet.size()>1)
  {
    var codes = aSet.members();
    var randomNumber = Math.random() * codes.length;
    var randomIndex = Math.floor(randomNumber);
    var selectedCode = codes[randomIndex];
    f(targetQid).set(selectedCode);
  }
}

Notes

• MaxDiff is only supported for surveys using responsive rendering.

• Once a MaxDiff question is live and capturing data, if the answer list or model is changed, the existing captured data becomes invalidated.

• Survey launch will fail if the number of answers is not within the supported range of 3 to 30, or if a valid model has not been selected.