Category:Experiments

An experiment involves the manipulation of one or more factors, where a factor is often referred to as an attribute or independent variable. For example, in an experiment testing the effect of different designs for packs of chips the package design is the factor. And, in an experiment that manipulates different prices for a new product then price is the factor. Each factor has two or more levels. For example, an experiment that tests five different price points has five levels.

Q has a very general method for analyzing experiments which is by using the Question Type of Experiment. This can be used to analyze most types of experiments used in survey analysis. However, it is reasonably complex and where an experiment is relatively simple then there are generally easier (and sometimes better) ways of analyzing it in Q. When trying to work out the best way to analyze an experiment in Q it is useful to work out which of the following descriptions applies to the experiment:

• Completely Randomized Single Factor Experiments, in which each respondent is randomly shown one and only one level of one factor. This is sometimes referred to as a monadic design.
• Repeated Measures Single Factor Experiments, in which each respondent is shown all of the levels. This is sometimes referred to as a sequential monadic design and as a split cell design.
• Ranking Experiments, in which the respondent ranks multiple objects. This includes Max-Diff and paired comparisons.
• Multifactor Experiments, such as choice modeling and conjoint analysis.

Checking your experiment

As the design of experiments is difficult, and the setup an Experiment in Q is also difficult, it is always advisable to perform lots of checks. Where there are egregious errors, Q will provide an error message. However, it is possible to get estimates of Coefficients even for experiments that are incorrectly designed and/or setup, so the following checks should be always be performed (but these checks alone are not sufficient for concluding that an experiment is valid):

• Create a SUMMARY table of the experiment, select all the values in the Coefficient column, press and review all the output. Note that if the Invalid task report contains data, it means that some of the data for some of the respondents is not being used (either because it contains errors, or, is missing). Any invalid tasks can be checked by reading through a respondent's data as it appears in the Data tab for the Experiment question. It can be useful to read through the data even when nothing is shown in the Invalid task report.
• Right-click on the table and select Statistics - Cells > Standard Error and check that the values are sensible (see Standard Errors in an Experiment or Ranking Question Are Large or NaN).

Case studies

These case studies involve the use of Question Types of Experiment and Ranking, which are generally most appropriate for the analysis of complex experiments.

These case studies are designed to be read in order, with each case study introducing more information about the interpretation of the outputs.

• Completely Randomized Single Factor Experiment Case Study. Please also refer to Completely Randomized Single Factor Experiment and Repeated Measures Single Factor Experiment for more detail about the analysis of single-factor experiments.
• Brand Price Trade-Off Experiment
• Discrete Choice Experiment Case Study
• Conjoint Analysis Case Study
• Max-Diff Case Study

Analyzing experiments

Once an Experiment has been set up in Q there are a variety of different ways of analyzing it. They key thing to understand is that an Experiment is, from Q's perspective, just another question.^{[note 1]} All the standard analyses that are done in Q with simpler question types can also be done with experiments. For example:

• When an Experiment question is selected in the Blue Drop-down Menu, the statistics most likely to be useful are displayed (e.g., the Coefficients from a multinomial logit model).
• Filters and weights can be applied. When weights are applied, Q automatically adjusts the Standard Error, t-Statistic and p to reflect the Effective Sample Size.
• Additional statistics can be selected by right-clicking on the table, selecting Statistics - Cells, holding the Ctrl button down and clicking the preferred options.
• If a Pick One question is selected in the brown drop-down menu, the automatic Tests of Statistical Significance shown on the table compare the coefficients between the groups represented by the columns.
• Specific tests of interest can be conducted by selecting the cells of interest and pressing . For example, if you want to see the standard set of outputs for a logit model, select all the coefficients in the table and press . This is discussed in more detail in the tutorials (shown at the top-right of this page).
• Categories can be moved, merged and deleted in tables and this causes the underlying models and significance tests to be automatically updated.
• Changing the attribute from numeric to categorical and vice versa (by right-clicking).
• Delete an attribute by deleting all of the attribute’s variables (by right-clicking).
• When the data for a project is updated, the question and the analyses will automatically update.
• Crosstabs of Experiments and Pick One questions can be mapped using Create > Maps.
• The tables can be exported to other applications.
• Parameters can be estimated for each respondent in the survey and these can then be used for further analyses (see Individual-Level Parameters).
• Use JavaScript to create interactions. This is best done by first creating a single variable and then using Use as Template for Replication. An example of a JavaScript interaction is:

if (Alt1_2 == 0 && Q30# == 24) 0; // NAB AND NAB is Transaction Bank
else if (Alt1_2 == 0 && Q30# != 24) 1; // NAB AND NAB is NOT Transaction Bank
else if (Alt1_2 == 1 && Q30# == 11) 2; // CBA Shown AND is Main Transaction Bank
else if (Alt1_2 == 2 && Q30# == 30) 3; // Westpac Shown AND is Main Transaction Bank
else if (Alt1_2 == 3 && Q30# == 4) 4; // ANZ Shown AND is Main Transaction Bank
else if (Alt1_2 == 4 && Q30# == 28) 5; // Suncorp Shown AND is Main Transaction Bank
else if (Alt1_2 == 5 && Q30# == 10) 6; // Bendigo Shown AND is Main Transaction Bank
else if (Alt1_2 == 6 && Q30# == 9) 7; // BankWest Shown AND is Main Transaction Bank
else if (Alt1_2 == 7 && Q30# == 1) 8; // Adelaide Bank Shown AND is Main Transaction Bank
else if (Alt1_2 == 8 && Q30# == 27) 9; // St George Shown AND is Main Transaction Bank
else if (Alt1_2<=3 ) 10 ; //Big 4 AND IS NOT Main Trainsaction
else 11; //regional AND IS NOT Main Transaction

Many of these analyses are illustrated in the earlier case studies.

Notes

Template:Reflist
Cite error: <ref> tags exist for a group named 'note', but no corresponding <references group='note'/> tag was found