Common Questions About Statistical Testing in Q

How does Q determine which cells in a table to conduct tests on?

By default, Q conducts tests on all cells in a table. Even NET statistics have significance tests computed (although often the resulting significance tests are not valid and thus the cells are not shown as being significant). Please refer to Ways of Showing Statistical Significance for more information on the options that can be used to determine whether or not to show significance tests.

Why does significance change when categories are merged or deleted?

By Hiding, Removing and Merging categories on tables (see Table Context Menu you can change what Q does and does not automatically conclude as being statistically significant if you are using corrections for Multiple Comparison Corrections, which are applied by default in Q. The table below, which shows a Pick One – Multi question where Don’t Know and the one below has it removed. Not only do the numbers change when the Don’t Know is removed, but the conclusions about the statistical significance also change. For example, we can see that Pepsi Light and Neither like nor dislike is now significant.

Although it may seem counterintuitive that the results change, it is in keeping with statistical theory about how best to conduct large numbers of statistical tests. In most instances, it is inappropriate to delete categories just because they are not significant. Similarly, it is inappropriate to delete categories with the goal of changing conclusions regarding which results are and are not significant. The default approach to determining which cells are significant uses a method called the False Discovery Rate, which simultaneously looks at the p-values of all the cells in the table.