How Q Highlights Results as Being Significant
p-values are the most important determinant of whether a result is highlighted as being significant. However, they are not the only determinant. A result will only be shown as being significant if:
- The relevant formatting settings permit the result as being shown to be significant.
- The Corrected p-value is less than or equal to the specified Overall significance level for that table.
Each of these settings can be changed either for an entire project (in Project Options or a specific Table Options. Each is now discussed in more detail, for a more general overview of how statistical testing is conducted in Q you can go to this page.
Settings for showing statistical significance
By default, all tables and a number of charts show results as being significant using color-coding and arrows (where charts do not show this, it is where users typically do not show statistical significance). However, the color-coding and arrows may be turned off, and, there are a variety of other ways of showing and determining statistical significance (see Ways of Showing Statistical Significance).
Significance Level and the corrected p-value
Whenever statistical testing is conducted a threshold has to be set for determining if a particular result is “significant” or not. The most common threshold, and the default in Q, is to require results to have a p-value of less than or equal to 0.05 (this is the same thing as what is sometimes referred to as the 95% level of confidence). This value is set as the Overall significance level; the interpretation of this value is also determined by the Cell comparisons and Column comparisons settings.
Weight variance by
This setting determines how variances can be computed, and variance is an input into many statistical tests.
Equal variance in tests when sample size is less than
This setting determines whether to assume variances are homogemeous or differing between sub-groups when conducting t-tests and Z-tests.