Choice Modeling - Experimental Design

From Q
Jump to: navigation, search

Creates an experimental design for Choice Modeling. Several different algorithms are available. The characteristics and quality of a design may be assessed from the Balances and Overlaps diagnostic [1] [2] and via simulation [3].

The various algorithms available are discussed in detail on the blog [4] [5] [6]. See [7] for a discussion of alternative-specific designs.


The output for an example choice experiment with seven attributes using the Balanced Overlaps algorithm is shown below.

The following table shows the first 2 questions of an experimental design with 4 alternatives plus None.

An example of a prior for an Efficient design is shown below.

Choice design prior.PNG


Algorithm One of,

Random Randomly chooses levels, only ensuring alternatives are not identical within a question.
Shortcut The design is built with each alternative consisting of the least frequently used Level for each attribute. If levels are equally frequent, the least used level within the question is selected, or else a random choice is made.
Complete enumeration For each alternative, every possible alternative is evaluated and the one with the lowest cost selected. The cost of an alternative depends on its incremental impact upon the design in terms of a combination of single level balance, pairwise level balance and level overlap within questions.
Balanced overlap As per Complete enumeration except level overlaps are less strongly penalized.
Efficient The design is chosen using a recent algorithm [8] to optimize the D-error, so that the variance of the model parameter estimates is minimized.
Partial profiles Uses the same algorithm as 'Efficient' except that designs generated with this algorithm can have a specified number of attributes set constant.
Alternative specific - Random Creates a design with attributes that are specific to each alternative. Random levels are chosen for each attribute.
Alternative specific - Federov Creates a design with attributes that are specific to each alternative. The design is optimized to maximize the information about the model parameters, given the responses.

Questions per respondent The number of questions to show to each respondent.

Versions The number of versions of the experimental design (defaults to 1).

Alternatives are labeled by first attribute If checked, a labeled design is produced where each level of the first attribute is used exactly once in each question. Hence the number of alternatives per question (excluding None) is equal to the number of levels of the first attribute.

Alternatives per question (excluding None(s)) The number of alternatives for each respondent to choose between in each question.

None alternatives The number of None alternatives added to each question.

None alternative positions The position(s) of the None alternative(s) as a comma delimited list. For example '1, 3' means that there are two None alternatives per question, which are the first and third alternatives. If not supplied the None alternative(s) will be the last alternative(s) of each question, or else the number of supplied positions must match the number specified in None alternatives.

Attributes and levels The method for entering the attributes and their levels (without prior). One of,

Enter in spreadsheet Enter the labels of the attributes along the first row, with the levels for each attribute in the column below its label. For the Efficient and Partial Profiles algorithms, mean and standard deviation priors may be specified for the levels by adding columns labeled mean or sd containing the priors to the right of each attribute column.
Enter attributes individually Specify a comma delimited list of label followed by levels for each attribute. When a list is entered (press enter) another box appears for the next attribute.
For Alternative specific designs the Labels of alternatives is a comma delimited list of alternatives. Attributes per alternative is a comma delimited list of integers specifying the number of attributes of each alternative. The following fields should each contain a comma delimited list of label followed by levels for each attribute.
Enter number of levels per attribute A quick method for specifying a design as a comma delimited list of the number of levels per attribute, e.g., 2,2,3,3. Attributes are labeled Attribute1, Attribute2, etc and levels are labeled 1, 2, 3 etc.

Enter prohibited alternatives A table of prohibited alternatives with one prohibition per row. The columns contain levels of the attributes in the same order that the attributes have been specified in Attributes and levels. If a level is All or blank then all levels of the attribute are prohibited when in combination with the other specified attribute levels. Not available when Algorithm is Efficient or Shortcut.

Constant attributes The number of attributes to keep constant when the partial profiles algorithm is selected.

Extensive search Whether to use the extensive version of the partial profiles algorithm. This is many times slower but the resulting design is usually more optimal.

Maximum candidate questions When the Alternative specific - Federov algorithm is chosen, the design is selected from the enumeration of all possible questions. For design with many attributes and/or many levels this enumeration may be excessively large. Maximum candidate questions sets a limit on the number of questions to consider when building the design. A random sample of Maximum candidate questions are drawn, from which the final design is optimized. When this value is more than the enumeration it has no effect. When it is lower than the enumeration it increases the speed for a decrease in accuracy.

Sample size The anticipated number of responses. This number of random responses are used to calculate the standard errors.


A table showing the design. Note that additional diagnostics describing balances and overlaps and standard errors provide more information.


  1. Hoare, J. (2018, July 20). How Good is your Choice Model Experimental Design? [Blog post]. Accessed from
  2. Bock, T (2017, July 25). How to Check an Experimental Design (MaxDiff, Choice Modeling) [Blog post]. Accessed from
  3. Yap, J. (2018, September 18). How to Use Simulated Data to Check Choice Model Experimental Designs Using Displayr [Blog post]. Accessed from
  4. Hoare, J. (2018, September 5). Algorithms to Create your Choice Model Experimental Design [Blog post]. Accessed from
  5. Yap, J. (2018, September 12) The Partial Profiles Algorithm for Experimental Designs [Blog post]. Accessed from
  6. Yap, J. (2018, September 12) The Efficient Algorithm for Choice Model Experimental Designs [Blog post]. Accessed from
  7. Hoare, J. (2018, August 16). How to Create Alternative-Specific Choice Model Designs in Displayr [Blog post]. Accessed from
  8. D. P. Cuervo, R. Kessels, P. Goos and K. Sörensen (2016). An integrated algorithm for the optimal design of stated choice experiments with partial profiles. Transportation Research Part B 93.


form.setHeading('Choice Model Experimental Design');

var algorithm = form.comboBox({label: "Algorithm", prompt: "Select the algorithm to use to determine the experimental design.",
               alternatives: ["Random", "Shortcut", "Complete enumeration", "Balanced overlap", "Efficient", "Partial profiles", "Alternative specific - Random", "Alternative specific - Federov"],
               name: "formAlgorithm", default_value: "Balanced overlap"}).getValue();

var allow_control_groups = Q.fileFormatVersion() > 10.9; // Group controls for Displayr and later versions of Q

form.numericUpDown({name:"formQuestionsPerRespondent", label:"Questions per respondent", default_value: 10, minimum: 1, maximum: 1000,
                    prompt: "The number of questions to show each respondent."});
form.numericUpDown({name:"formVersions", prompt: "The number of different versions of the experiment to generate.",
                    label:"Versions", default_value: 1, minimum: 1});

var labeled_alternatives;
if (algorithm == "Partial profiles")
    labeled_alternatives = false;
else if (algorithm.substring(0, 3) == "Alt")
    labeled_alternatives = true;
    labeled_alternatives = form.checkBox({label: "Alternatives are labeled by first attribute", name: "formLabeledAlternatives", default_value: false}).getValue();
if (!labeled_alternatives)
    form.numericUpDown({name:"formAlternativesPerQuestion", label:"Alternatives per question (excluding None(s))", default_value: 3, minimum: 2});

var none_alternatives = form.numericUpDown({name:"formNoneAlternatives", label:"None alternatives", default_value: 0, minimum: 0,
                                            prompt: "The number of 'none' alternatives."}).getValue();
if (none_alternatives > 0)
    form.textBox({name: "formNonePositions", label: "None alternative positions", prompt: "Comma delimited list of the positions of None alternative(s).", required: false});

if (algorithm.substring(0, 3) == "Alt") {
    var input_type = "Enter attributes individually";
} else {
    var entry_options = ["Enter in spreadsheet", "Enter attributes individually", "Enter number of levels per attribute"];
    var input_type = form.comboBox({label: "Attributes and levels", 
                   alternatives: entry_options,
                   name: "formDataEntry", default_value: "Enter in spreadsheet"}).getValue();

if (input_type == "Enter in spreadsheet")
    form.dataEntry({name: "formPastedData", prompt: "Enter attribute names in first column and levels in subsequent columns.", label: "Add attributes and levels", edit_label: "Edit attributes and levels"});
else if (input_type == "Enter attributes individually" && !(algorithm.substring(0, 3) == "Alt")) {
    var i = 1;
    while (i == 1 || attribute != "") {
        attribute = form.textBox({name: "formAttribute" + i, label: "Attribute " + i, prompt: "Attribute name followed by list of levels, delimited by commas", required: i == 1}).getValue();
} else if (input_type == "Enter attributes individually" && (algorithm.substring(0, 3) == "Alt")) {
    var alternative_labels = form.textBox({name: "formSpecificAlternativeLabels", label: "Labels of alternatives", prompt: "Comma delimited list of alternative labels", required: true}).getValue();
    var alternative_attributes = form.textBox({name: "formSpecificAttributesPerAlternative", label: "Attributes per alternative", prompt: "Comma delimited list of the number of attributes per alternative", required: true}).getValue();
    var alternative_labels = alternative_labels.split(/[ ,]+/);
    var alternative_attributes = alternative_attributes.split(/[ ,]+/);
    for (i = 0; i < alternative_labels.length; ++i) {
        for (j = 0; j < Math.min(50, alternative_attributes[i]); ++j) {
            form.textBox({name: "formSpecificAttribute" + i + j, label: alternative_labels[i] + " attribute " + (j + 1), prompt: "Attribute label followed by list of levels, delimited by commas", required: true})
} else if (input_type == "Enter number of levels per attribute")
    form.textBox({name: "formUnlabeledAttrributes", label: "Attribute levels", prompt: "Comma delimited list of levels per attribute", required: true});

if (algorithm == "Random" || algorithm == "Complete enumeration" || algorithm == "Balanced overlap") {
    var has_prohibitions = form.checkBox({label: "Enter prohibited alternatives", name: "formHasProhibitions", default_value: false}).getValue();
    if (has_prohibitions)
        form.dataEntry({name: "formProhibitions", prompt: "Enter one prohibition per row with the levels or 'All' along the columns in the same order as the attributes."});

if (algorithm == "Partial profiles")
    var constant_attributes = form.numericUpDown({name:"formConstantAttributes", label:"Constant attributes", default_value: 0, minimum: 0,
                                                  prompt: "The number of attributes that will not vary for each question."}).getValue();
    if (constant_attributes > 0)
        form.checkBox({label: "Extensive search", name: "formExtensive", default_value: false});

if (algorithm == "Alternative specific - Federov")
    form.numericUpDown({name:"formMaxCandidates", label:"Maximum candidate questions", default_value: 100000, minimum: 10000, increment: 10000, maximum: 1000000000,
                        prompt: "Limit the number of candidates to reduce computation time for large designs."});

form.numericUpDown({name: "formStandardErrorIndividuals", label: "Sample size", 
                    prompt: "Number of random individual responses used to calculate standard errors",
                    default_value: 300, increment: 1, maximum:100000, minimum: 1});

if (get0("formDataEntry", ifnotfound = "") == "Enter number of levels per attribute") {
    levels.per.attribute <- as.numeric(ConvertCommaSeparatedStringToVector(formUnlabeledAttrributes))
    attribute.levels <- CreateExperiment(levels.per.attribute)$attribute.levels
} else if (get0("formDataEntry", ifnotfound = "") == "Enter attributes individually") {
    n.attributes <- 0
    while (get0(paste0("formAttribute", n.attributes + 1)) != "")
        n.attributes <- n.attributes + 1
    attribute.levels <- sapply(paste0("formAttribute", seq(n.attributes)), get0)
    attribute.levels <- sapply(attribute.levels, ConvertCommaSeparatedStringToVector, simplify = FALSE)
    names(attribute.levels) <- sapply(attribute.levels, function (x) x[1])
    attribute.levels <- sapply(attribute.levels, function (x) x[2:length(x)], simplify = FALSE)
} else if (substr(formAlgorithm, 1, 3) == "Alt") {
    attribute.levels <- list()
    alternative.labels <- ConvertCommaSeparatedStringToVector(formSpecificAlternativeLabels)
    alternative.attributes <- as.numeric(ConvertCommaSeparatedStringToVector(formSpecificAttributesPerAlternative))
    if (length(alternative.labels) != length(alternative.attributes))
        stop("The number of 'Labels of alternatives' must equal the length of the 'Attributes per alternative' list.")
    for (i in seq(0, length(alternative.labels) - 1)) {
        for (j in seq(0, alternative.attributes[i + 1] - 1)) {
            att.label.and.levels <- ConvertCommaSeparatedStringToVector(get0(paste0("formSpecificAttribute", i, j)))
            att.label <- att.label.and.levels[1]
            att.levels <- att.label.and.levels[-1]               
            attribute.levels[[alternative.labels[i + 1]]][[att.label]] <- att.levels
} else {   # spreadsheet, may contain prior and is processed within ChoiceModelDesign
    attribute.levels <- formPastedData
} <- ChoiceModelDesign(design.algorithm = formAlgorithm,
                         attribute.levels = attribute.levels,
                         n.questions = formQuestionsPerRespondent,
                         n.versions = formVersions,
                         alternatives.per.question = get0("formAlternativesPerQuestion", ifnotfound = 0),
                         prohibitions = get0("formProhibitions", ifnotfound = NULL),
                         none.alternatives = formNoneAlternatives,
                         none.positions = get0("formNonePositions", ifnotfound = ""),
                         labeled.alternatives = get0("formLabeledAlternatives", ifnotfound = FALSE),
                         n.constant.attributes = get0("formConstantAttributes", ifnotfound = 0),
                         extensive = get0("formExtensive", ifnotfound = FALSE),
                         max.subsample = get0("formMaxCandidates", ifnotfound = 0),
                         standard.error.respondents = formStandardErrorIndividuals)