Overview of Statistical Testing in Q
The basic process of significance testing in Q consists of the following steps. These steps will not necessarily be performed if a Rule has been applied which overrides them.
Contents
The user specifies the relationship to be tested
The user can specify which relationships are to be tested by:
- Creating tables manually, by selection weights filters and questions from the blue and drop-down menus. Each cell in the table is then tested.
- Selecting individual cells on the table and pressing .
- Creating large batches of tables using Basic Tables. This is equivalent to creating tables manually, except that many are done at the same time.
- Creating batches of tables using Smart Tables. As with Basic Tables, this causes lots of tables to be created with significance tests conducted in each cell of the table. However, it also causes each table to be tested (which is equivalent to selecting all the cells in a table and pressing ) and places the significant and insignificant tables in different groups.
- Regression. There are two quite different ways of doing this. It can be done by setting up an Experiment question (in which case each of the different ways of specifying relationships to test, as described in the previous bullet points, can be conducted), or, it can be done by selecting Create > Traditional Multivariate Statistics > Regression, in which case the p-values are reported (and the user is left to interpret results as being significant or not, based on the p-values or any other criteria that are considered applicable).
- Creating Segments, where for some models significance tests are presented for coefficients and information criteria, which are a type of significance test, are used to select the number of segments.
The p-Values is computed
Q computes the p-value for each relationship that is tested by taking into account:
- What the data means. In particular, Q takes into account how the data has been setup in terms of Question Type, Variable Type, Value Attributes, and which categories have been merged on the Outputs Tab.
- Various technical assumptions that the user has made about minimum sample sizes and how to deal with weights (see Statistical Assumptions).
The computed p-values for each cell can be shown by selecting p from Statistics - Cells.
The p-Value is corrected
If a Multiple Comparison Correction has been specified, the p-value is corrected to take this into account. When the False Discovery Rate Correction has been applied, which is the default, the Corrected p for each cell can be shown by selecting p from Statistics - Cells; with other types of Multiple Comparison Corrections then p itself is corrected.
Significance is determined
A determination is made about whether the result is or is not statistically significant. This is made by comparing p or Corrected p with the Overall significance level.
The result is displayed
The result is displayed as being significant in accordance with the chosen method of display (e.g., Arrows or Column Comparisons; see Ways of Showing Statistical Significance).