Category:Tests Of Statistical Significance
Related Online Training modules  

Automatic Tests of Statistical Significance  
Significance Tests on Grids  
Type 1 Error  
Population Weights  
NonProportional Sampling Weights  
Planned Tests of Statistical Significance  
Column Comparisons  
ANOVA with Post Hocs  
ANOVA Repeated Measures with Post Hocs  
Generally it is best to access online training from within Q by selecting Help : Online Training 
Contents
Quick Links
 A general overview on how to perform a statistical test is here: Overview of Statistical Testing in Q
 A guide to what settings and statistical assumptions are available in Q is here: Statistical Assumptions
 Info on how Q displays significance testing is here: How Q Highlights Results as Being Significant
 To learn how to interpret significance testing results start here: Category: Reading Tables and Interpreting Significance Tests
 If your significant results are different than expected start here: Results Are Different to those from Another Program
 Many times results differ due to:
 Overlapping data  overlapping respondents are by default removed when doing column comparisons which may not be done in other softwares
 Multiple Comparison Correction  helps reduce false positives found by running lots of significance tests by adjusting the critical pvalue to make it harder to be found significant
 Many times results differ due to:
Popular How Tos and Frequently Asked Questions
 How to see details of the underlying test (Right click cell or cells and select Test Statistical Significance)
 How to Specify Columns to be Compared
 How To Test Between Adjacent Time Periods
 How to Change Significance Levels for Column Comparisons
 How To Compare A SubGroup Against The Total
 How to Replicate SPSS Significance Tests in Q
Statistical Significance Testing in Tables
There are two ways that Q can do statistical significance testing in a table:
 Cell comparisons  test a cell against it's complement, see also: Testing the Complement of a Cell
 Column comparisons  test columns against one another, see also: Interpreting Column Comparisons and How to Specify Columns to be Compared
How Q Determines Significance Testing
Project & tablespecific Statistical Assumption settings Q Rules that might override settings 
Select significance test to run based on the data and settings Calculate pvalues 
Testing the cell complement:

More detail can be found here: Overview of Statistical Testing in Q
Default Statistical Assumptions
These settings can be found under Edit > Project Options > Customize > Statistical Assumptions or modified for individual charts under Edit > Table Options > Statistical Assumptions.
 Show significance: Arrows and font colors will designate significant results in tables
 Overall significance level: testing will be done at the 95% confidence level and above
 Minimal sample size for testing: you must have at least 2 respondents in your sample to test
 Proportions: nonparametric tests will be done on categorical data
 Means: ttest will be done on numeric data and corrected with Bessel’s correction
 Correlations: default is Pearson
 Equal variance in tests when sample size is less than: if the sample size is less than 10 variance is assumed equal
 Cell comparisons: the complement of the cell with be tested
 Weights and significance: Automatically a mix of Taylor Series Linearization and Kish's Effective Sample Size Formula
 Date questions: tests compare across all dates rather than previous period
Statistical tests for categorical and numeric data:
Multiple comparison correction: False Discovery Rate is by default applied to help reduce for the number of false positives based on the entire table
 Significance levels and appearance: Arrows: get longer with increased significance. Colors: Blue = significantly higher. Red = significantly lower. Font: Letters for column comparisons become capitalized after .001 is reached.
 Multiple Comparison correction: False Discovery Rate is by default applied to help reduce for the number of false positives based on the number of columns within the row & column span
 Overlaps: Default is for Q to ignore the sample that overlaps between columns when respondents in columns are not mutually exclusive
 ANOVAType Test: ANOVA is not run before displaying significance
 Show redundant tests: show significance on one cell (the one with the higher value)
 Show as groups: Show letters for insignificant columns rather than significant
 Recycle column letters: each span begins labeling columns at A
 No test symbol:  is shown if a test isn’t performed due to settings
 Symbol for nonsignificant test: nothing is shown if a test comes back insignificant
Column comparisons: take affect only if Column Comparisons are selected
Troubleshooting
To work out which test has been conducted on the cell of a table:
 Check to see if any Rules have been applied. If they have, review their documentation.
 Select the cell or both cells if comparing columns and press (the same as right clicking and selecting Test Statistical Significance)
Other troubleshooting links:
 Category: Troubleshooting Significance Tests
 Interpreting "Inconsistent" Statistical Testing Results
 The Statistical Assumptions selected for this table are inappropriate
 Results Are Different to those from Another Program
Other technical resources
Technical Assumptions of Tests of Statistical Significance contains a general discussion about the use and interpretation of tests of statistical significance. Further reading: Market Research Analysis Software
Subcategories
This category has the following 2 subcategories, out of 2 total.
Pages in category ‘Tests Of Statistical Significance’
The following 134 pages are in this category, out of 134 total.
B
C
 Cell Comparisons
 ChiSquare Test for Compatibility of K Counts
 ChiSquare Test of a Frequency Distribution
 Cochran's Q
 Column Comparisons with Missing Data and Grid Questions
 Combined pValues
 Common Questions About Statistical Testing in Q
 Comparing Columns Across Spans
 Complex Samples Dependent TTest  Comparing a SubGroup Mean to Total
 Complex Samples Dependent TTest  Comparing a SubGroup Proportion to Total
 Complex Samples Dependent ZTest  Comparing a SubGroup Mean to Total
 Complex Samples Dependent ZTest  Comparing a SubGroup Proportion to Total
 Complex Samples tTest of a Mean
 Complex Samples tTest of a Proportion
 Complex Samples ZTest of a Mean
 Complex Samples ZTest of a Proportion
 Confidence Interval
 Correlations  Comparing Two Numeric Variables
 Crosstabs of Proportions (One Categorical Question by Another Categorical Question)
D
F
H
 How Q Highlights Results as Being Significant
 How to Change Significance Levels for Column Comparisons
 How To Compare A SubGroup Against The Total
 How to Conduct MANOVA Tests
 How to Conduct OneTailed Column Comparisons
 How To Create Banner Tables Using Pick One  Multi Or Grid Questions
 How to Include the Main NET Column in Column Comparisons
 How To Override Default Statistical Testing Settings
 How to Replicate SPSS Significance Tests in Q
 How to Replicate Survey Reporter Significance Tests
 How to Specify Columns to be Compared
 How to Specify Comparisons for ANOVABased Tests
 How To Test Against The NET/Total/Average
 How To Test Between Adjacent Time Periods
I
 Independent Complex Samples tTest  Comparing Two Means
 Independent Complex Samples TTest  Comparing Two Proportions
 Independent Complex Samples ZTest  Comparing Two Means
 Independent Complex Samples ZTest  Comparing Two Proportions
 Independent Sample Tests  Comparing Two Means
 Independent Sample Tests  Comparing Two Proportions
 Independent Samples  Quantum Column Means Test
 Independent Samples  Quantum Column Proportions Test
 Independent Samples  Survey Reporter Column Means Test
 Independent Samples  Survey Reporter Column Proportions Test
 Independent Samples tTest  Comparing Two Coefficients
 Independent Samples tTest  Comparing Two Means with Equal Variances
 Independent Samples tTest  Comparing Two Means with Unequal Variances
 Independent Samples tTest  Comparing Two Probability %
 Independent Samples TTest  Comparing Two Proportions
 Independent Samples TTest  Equal Variance
 Independent Samples TTest  Pooled Variance
 Independent Samples TTest  Unequal Variance
 Independent Samples ZTest  Comparing Two Means with Equal Variances
 Independent Samples ZTest  Comparing Two Means with Unequal Variances
 Independent Samples ZTest  Comparing Two Proportions
 Independent Samples ZTest  Comparing Two Proportions (Pooled)
 Independent Samples ZTest  Comparing Two Proportions (UnPooled)
 Information Criteria
 Interpretation of the Overall Significance Level by Q
 Interpreting Column Comparisons
M
 Mann–Whitney U
 Modifying Significance Tests
 Modifying Significance Tests Using Rules
 Multiple Comparison Correction
 Multiple Comparisons (Post Hoc Testing)
 Multiple Comparisons tTest (Fisher LSD)
 Multiple Comparisons tTest with Bonferroni Correction
 Multiple Comparisons tTest with False Discovery Rate Correction
 Multivariate Tests
O
P
 PValues
 Paired tTest of Means
 Paired tTest of Proportions
 Paired ZTest of Means
 Paired ZTest of Proportions
 Pearson's ChiSquare Test of Independence
 Pearson's Product Moment Correlation
 Pearson’s Chisquare for Canonical Correlation Analysis
 Planned ANOVAType Tests
 Planned Tests Of Statistical Significance
R
 Related Samples Tests  Comparing Two Means
 Related Samples Tests  Comparing Two Proportions
 Repeated Measures
 Repeated Measures ANOVA
 Repeated Measures ANOVA with Greenhouse & Geisser Epsilon Correction
 Repeated Measures ANOVA with Huynh and Feldt Epsilon Correction
 Repeated Measures ANOVA with Lower Bound Epsilon Correction
S
 Second Order RaoScott Test of Independence of a Contingency Table
 Setting Statistical Assumptions When Setting Up Projects
 Significance Tests Change When I Merge, Hide or Remove Categories
 Significance Tests on Trend Plots
 Simplified Independent Complex Samples TTest  Comparing Two Means
 Simplified Independent Complex Samples TTest  Comparing Two Proportions
 Spearman’s Correlation
 Standard Errors in an Experiment or Ranking Question Are Large or NaN
 Statistical Assumptions
 Statistical Tests for Experiment Questions
 Statistical Tests for Ranking Questions