Choice Modeling - Experimental Design

Several different algorithms are available. The characteristics and quality of a design may be assessed from the Balances and Overlaps of Design diagnostic

and via simulation .

The various algorithms available are discussed in detail on the blog . See for a discussion of alternative-specific designs.

How to Create

1. Add the object by selecting from the menu Anything > Advanced Analysis > Choice Modeling > Experimental DesignAutomate > Browse Online Library > Choice Modeling > Experimental Design
2. Provide further design inputs in the Inputs menu as described here

Example

Output Example 1:
The output for an example choice experiment with seven attributes using the Balanced Overlaps algorithm is shown below. The D-error shown account for any supplied priors, (i.e., if no priors are supplied, then it is D₀; if fixed priors (only prior means) are supplied, then it is D_p; if both prior means and standard deviations are supplied, then it is D_B, see this blog post for more information).

Output Example 2:
The following table shows the first 2 questions of an experimental design with 4 alternatives plus None.

Input Example 2:
An example illustrating the format in which to specify priors for an Efficient design is shown below. In this example, means and standard deviations are specified for levels in the "Brand" attribute whereas only means are specified for the "Price" attribute. Only attributes for which priors are desired need to be included. Note that prior means and standard deviations need to be zero for the first level of each attribute due to dummy coding of the levels.

Input Example 3:
An example illustrating the format in which to specify prohibitions is shown below. Each row in the table below corresponds to a prohibition. The first row prohibits alternatives containing both Hershey and Switzerland, while the second row prohibits alternatives containing both Dove and Belgium, and the third row prohibits alternatives containing both Godiva and $0.99. Note that attribute levels are placed in the columns corresponding to the order of the attributes in the design, i.e., the first column contains "Brand" levels, the second column contains "Price" levels etc.. Also note that the first row contains "All" while the other rows have blank cells instead. The two are equivalent and mean "all levels of the attribute corresponding to this column" should be included in the prohibition for the row.

Options

Algorithm One of,

Random Randomly chooses levels, only ensuring alternatives are not identical within a question.

Shortcut The design is built with each alternative consisting of the least frequently used Level for each attribute. If levels are equally frequent, the least used level within the question is selected, or else a random choice is made.

Complete enumeration For each alternative, every possible alternative is evaluated and the one with the lowest cost selected. The cost of an alternative depends on its incremental impact upon the design in terms of a combination of single level balance, pairwise level balance and level overlap within questions.

Balanced overlap As per Complete enumeration except level overlaps are less strongly penalized.

Efficient The design is chosen using a recent algorithm to optimize the D-error, so that the variance of the model parameter estimates is minimized.

Partial profiles Uses the same algorithm as 'Efficient' except that designs generated with this algorithm can have a specified number of attributes set constant.

Alternative specific - Random Creates a design with attributes that are specific to each alternative. Random levels are chosen for each attribute.

Alternative specific - Federov Creates a design with attributes that are specific to each alternative. The design is optimized to maximize the information about the model parameters, given the responses.

Questions per respondent The number of questions to show to each respondent.

Versions The number of versions of the experimental design (defaults to 1).

Alternatives are labeled by first attribute If checked, a labeled design is produced where each level of the first attribute is used exactly once in each question. Hence the number of alternatives per question (excluding None) is equal to the number of levels of the first attribute.

Alternatives per question (excluding None(s)) The number of alternatives for each respondent to choose between in each question.

None alternatives The number of None alternatives added to each question.

None alternative positions The position(s) of the None alternative(s) as a comma delimited list. For example '1, 3' means that there are two None alternatives per question, which are the first and third alternatives. If not supplied the None alternative(s) will be the last alternative(s) of each question, or else the number of supplied positions must match the number specified in None alternatives.

Attributes and levels The method for entering the attributes and their levels (without prior). One of,

Enter in spreadsheet Enter the labels of the attributes along the first row, with the levels for each attribute in the column below its label. For the Efficient and Partial Profiles algorithms, mean and standard deviation priors may be specified for the levels by adding columns labeled mean or sd containing the priors to the right of each attribute column (see example above).

Enter attributes individually Specify a comma delimited list of label followed by levels for each attribute. When a list is entered (press enter) another box appears for the next attribute.

For Alternative specific designs the Labels of alternatives is a comma delimited list of alternatives. Attributes per alternative is a comma delimited list of integers specifying the number of attributes of each alternative. The following fields should each contain a comma delimited list of label followed by levels for each attribute.

Enter number of levels per attribute A quick method for specifying a design as a comma delimited list of the number of levels per attribute, e.g., 2,2,3,3. Attributes are labeled Attribute1, Attribute2, etc and levels are labeled 1, 2, 3 etc.

Enter prohibited alternatives A table of prohibited alternatives with one prohibition per row. The columns contain levels of the attributes in the same order that the attributes have been specified in Attributes and levels. If a level is All or blank then all levels of the attribute are prohibited when in combination with the other specified attribute levels. Not available when Algorithm is Efficient or Shortcut. See example above.

Constant attributes The number of attributes to keep constant when the partial profiles algorithm is selected.

Extensive search Whether to use the extensive version of the partial profiles algorithm. This is many times slower but the resulting design is usually more optimal.

Maximum candidate questions When the Alternative specific - Federov algorithm is chosen, the design is selected from the enumeration of all possible questions multiplied by the number of versions. For a design with many attributes and/or many levels and versions this enumeration may be excessively large. Maximum candidate questions sets a limit on the number of questions to consider when building the design. A random sample of Maximum candidate questions are drawn, from which the final design is optimized. When this value is more than the enumeration it has no effect. When it is lower than the enumeration it increases the speed for a decrease in accuracy.

Sample size The anticipated number of responses. This number of random responses are used to calculate the standard errors.

DIAGNOSTICS

Balances and overlaps Creates an output containing level balances and overlap between levels of the design.

Numeric design Creates an output containing a numeric version of the design.

Standard errors Creates an output containing the parameter fits and their standard errors for the design.

Outputs

A table showing the design. Note that additional diagnostics describing balances and overlaps and standard errors provide more information. The A-error metric shown in Output Example 1 is proportional to the average standard error of the parameters from a multinomial logit analysis with the design. A lower A-error translates to lower average standard errors of the parameters.

References

Template:Reflist

Additional Properties

When using this feature you can obtain additional information that is stored by the R code which produces the output.

1. To do so, select Create > R Output.
2. In the R CODE, paste: item = YourReferenceName
3. Replace YourReferenceName with the reference name of your item. Find this in the Report tree or by selecting the item and then going to Properties > General > Name from the object inspector on the right.
4. Below the first line of code, you can paste in snippets from below or type in str(item) to see a list of available information.

For a more in depth discussion on extracting information from objects in R, checkout our blog post here.

Code

var heading_text = 'Choice Model Experimental Design';
if (!!form.setObjectInspectorTitle)
    form.setObjectInspectorTitle(heading_text)
else 
    form.setHeading(heading_text);


var algorithm = form.comboBox({label: "Algorithm", prompt: "Select the algorithm to use to determine the experimental design.",
               alternatives: ["Random", "Shortcut", "Complete enumeration", "Balanced overlap", "Efficient", "Partial profiles", "Alternative specific - Random", "Alternative specific - Federov"],
               name: "formAlgorithm", default_value: "Balanced overlap"}).getValue();

var allow_control_groups = Q.fileFormatVersion() > 10.9; // Group controls for Displayr and later versions of Q

form.numericUpDown({name:"formQuestionsPerRespondent", label:"Questions per respondent", default_value: 10, minimum: 1, maximum: Number.MAX_SAFE_INTEGER,
                    prompt: "The number of questions to show each respondent."});
form.numericUpDown({name:"formVersions", prompt: "The number of different versions of the experiment to generate.",
                    label:"Versions", default_value: 1, minimum: 1, maximum: Number.MAX_SAFE_INTEGER});

var labeled_alternatives;
if (algorithm == "Partial profiles")
    labeled_alternatives = false;
else if (algorithm.substring(0, 3) == "Alt")
    labeled_alternatives = true;
else
    labeled_alternatives = form.checkBox({label: "Alternatives are labeled by first attribute", name: "formLabeledAlternatives", default_value: false}).getValue();
if (!labeled_alternatives)
    form.numericUpDown({name:"formAlternativesPerQuestion", label:"Alternatives per question (excluding None(s))", default_value: 3, minimum: 2, maximum: Number.MAX_SAFE_INTEGER,});

var none_alternatives = form.numericUpDown({name:"formNoneAlternatives", label:"None alternatives", default_value: 0, minimum: 0, maximum: Number.MAX_SAFE_INTEGER,
                                            prompt: "The number of 'none' alternatives."}).getValue();
if (none_alternatives > 0)
    form.textBox({name: "formNonePositions", label: "None alternative positions", prompt: "Comma delimited list of the positions of None alternative(s).", required: false});

if (algorithm.substring(0, 3) == "Alt") {
    var input_type = "Enter attributes individually";
} else {
    var entry_options = ["Enter in spreadsheet", "Enter attributes individually", "Enter number of levels per attribute"];
    var input_type = form.comboBox({label: "Attributes and levels", 
                   alternatives: entry_options,
                   name: "formDataEntry", default_value: "Enter in spreadsheet"}).getValue();
}

if (input_type == "Enter in spreadsheet")
    form.dataEntry({name: "formPastedData", prompt: "Enter attribute names in first column and levels in subsequent columns.", label: "Add attributes and levels", edit_label: "Edit attributes and levels", required: true, large_data_error: "The data entered is too large. You may only enter data with up to 1000 rows and up to 100 columns."});

else if (input_type == "Enter attributes individually" && !(algorithm.substring(0, 3) == "Alt")) {
    var i = 1;
    while (i == 1 || attribute != "") {
        attribute = form.textBox({name: "formAttribute" + i, label: "Attribute " + i, prompt: "Attribute name followed by list of levels, delimited by commas", required: i == 1}).getValue();
        ++i;
    }
} else if (input_type == "Enter attributes individually" && (algorithm.substring(0, 3) == "Alt")) {
    var alternative_labels = form.textBox({name: "formSpecificAlternativeLabels", label: "Labels of alternatives", prompt: "Comma delimited list of alternative labels", required: true}).getValue();
    var alternative_attributes = form.textBox({name: "formSpecificAttributesPerAlternative", label: "Attributes per alternative", prompt: "Comma delimited list of the number of attributes per alternative", required: true}).getValue();
    var alternative_labels = alternative_labels.split(/[ ,]+/);
    var alternative_attributes = alternative_attributes.split(/[ ,]+/);
    for (i = 0; i < alternative_labels.length; ++i) {
        for (j = 0; j < Math.min(50, alternative_attributes[i]); ++j) {
            form.textBox({name: "formSpecificAttribute" + i + j, label: alternative_labels[i] + " attribute " + (j + 1), prompt: "Attribute label followed by list of levels, delimited by commas", required: true})
        }
    }
} else if (input_type == "Enter number of levels per attribute")
    form.textBox({name: "formUnlabeledAttrributes", label: "Attribute levels", prompt: "Comma delimited list of levels per attribute", required: true});

if (algorithm == "Random" || algorithm == "Complete enumeration" || algorithm == "Balanced overlap") {
    var has_prohibitions = form.checkBox({label: "Enter prohibited alternatives", name: "formHasProhibitions", default_value: false}).getValue();
    if (has_prohibitions)
        form.dataEntry({name: "formProhibitions", prompt: "Enter one prohibition per row with the levels or 'All' along the columns in the same order as the attributes.", required: true, large_data_error: "The data entered is too large. You may only enter data with up to 1000 rows and up to 100 columns."});

}

if (algorithm == "Partial profiles")
{
    var constant_attributes = form.numericUpDown({name:"formConstantAttributes", label:"Constant attributes", default_value: 0, minimum: 0, maximum: Number.MAX_SAFE_INTEGER,
                                                  prompt: "The number of attributes that will not vary for each question."}).getValue();
    if (constant_attributes > 0)
        form.checkBox({label: "Extensive search", name: "formExtensive", default_value: false});
}

if (algorithm == "Alternative specific - Federov")
    form.numericUpDown({name:"formMaxCandidates", label:"Maximum candidate questions", default_value: 100000, minimum: 10000, increment: 10000, maximum: Number.MAX_SAFE_INTEGER,
                        prompt: "Limit the number of candidates to reduce computation time for large designs."});

form.numericUpDown({name: "formStandardErrorIndividuals", label: "Sample size", 
                    prompt: "Number of random individual responses used to calculate standard errors",
                    default_value: 300, increment: 1, maximum: Number.MAX_SAFE_INTEGER, minimum: 1});

library(flipU)
library(flipChoice)

if (get0("formDataEntry", ifnotfound = "") == "Enter number of levels per attribute") {
    levels.per.attribute <- as.numeric(ConvertCommaSeparatedStringToVector(formUnlabeledAttrributes))
    attribute.levels <- CreateExperiment(levels.per.attribute)$attribute.levels
} else if (get0("formDataEntry", ifnotfound = "") == "Enter attributes individually") {
    n.attributes <- 0
    while (get0(paste0("formAttribute", n.attributes + 1)) != "")
        n.attributes <- n.attributes + 1
    attribute.levels <- sapply(paste0("formAttribute", seq(n.attributes)), get0)
    attribute.levels <- sapply(attribute.levels, ConvertCommaSeparatedStringToVector, simplify = FALSE)
    names(attribute.levels) <- sapply(attribute.levels, function (x) x[1])
    attribute.levels <- sapply(attribute.levels, function (x) x[2:length(x)], simplify = FALSE)
} else if (substr(formAlgorithm, 1, 3) == "Alt") {
    attribute.levels <- list()
    alternative.labels <- ConvertCommaSeparatedStringToVector(formSpecificAlternativeLabels)
    alternative.attributes <- as.numeric(ConvertCommaSeparatedStringToVector(formSpecificAttributesPerAlternative))
    if (length(alternative.labels) != length(alternative.attributes))
        stop("The number of 'Labels of alternatives' must equal the length of the 'Attributes per alternative' list.")
    for (i in seq(0, length(alternative.labels) - 1)) {
        for (j in seq(0, alternative.attributes[i + 1] - 1)) {
            att.label.and.levels <- ConvertCommaSeparatedStringToVector(get0(paste0("formSpecificAttribute", i, j)))
            att.label <- att.label.and.levels[1]
            att.levels <- att.label.and.levels[-1]               
            attribute.levels[[alternative.labels[i + 1]]][[att.label]] <- att.levels
        }
    }
} else {   # spreadsheet, may contain prior and is processed within ChoiceModelDesign
    attribute.levels <- formPastedData
}

choice.model.design <- ChoiceModelDesign(design.algorithm = formAlgorithm,
                         attribute.levels = attribute.levels,
                         n.questions = formQuestionsPerRespondent,
                         n.versions = formVersions,
                         alternatives.per.question = get0("formAlternativesPerQuestion", ifnotfound = 0),
                         prohibitions = get0("formProhibitions", ifnotfound = NULL),
                         none.alternatives = formNoneAlternatives,
                         none.positions = get0("formNonePositions", ifnotfound = ""),
                         labeled.alternatives = get0("formLabeledAlternatives", ifnotfound = FALSE),
                         n.constant.attributes = get0("formConstantAttributes", ifnotfound = 0),
                         extensive = get0("formExtensive", ifnotfound = FALSE),
                         max.subsample = get0("formMaxCandidates", ifnotfound = 0),
                         standard.error.respondents = formStandardErrorIndividuals)