# Significance Testing - Compute Column Comparisons on Grids with Lots of Missing Data

From Q

This rule runs an independent samples *z*-test on columns in a grid, testing at the 0.05 level and 0.10 level with no multiple comparison corrections. This rule is only provided for backwards compatibility. In general, it is preferable to instead use the Statistical Assumptions to control testing with missing data on grid questions (see Column Comparisons with Missing Data and Grid Questions) or using Significance Testing - Independent Samples Column Means and Proportions.

## Technical details

- You must set
**Show significance**to**Compare columns**in order for the results of this rule to be visible. - The reason that some people use this test instead of the in-built tests in Q is that, in general, this rule uses equal or higher sample sizes to those used in Q. For example, if 20 people have provided data on one cell and 20 on another, but only 2 people have provided data for both cells, then the sample size of 2 would be used by default in Q's tests but 40 in the tests performed using this rule. Although this may seem a clear benefit, the problem is that, in general, the cells are not independent so the assumptions implicit in the application of this rule are rarely met in practice (the default test in Q is a related sample test). Additionally, this rule does not apply any Multiple Comparison Correction nor any other of the settings in Statistical Assumptions.
- Testing is conducted on NET and SUM categories.
- See also Modifying Significance Tests Using Rules.

## How to apply this rule

### For the first time in a project

- Select the table(s)/chart(s) that you wish to apply the rule to.
- Start typing the name of the Rule into the
**Search features and data**box in the top right of the Q window. - Click on the Rule when it appears in the
**QScripts and Rules**section of the search results.

*OR*

- Select
**Automate > Browse Online Library**. - Choose this rule from the list.

### Additional applications of the rule

- Select a table or chart that has the rule
*and*any table(s)/chart(s) that you wish to apply the rule to. - Click on the
**Rules**tab (bottom-left of the table/chart). - Select the rule that you wish to apply.
- Click on the
**Apply**drop-down and choose your desired option. - Check
**New items**to have it automatically applied to new items that you create. Use**Edit > Project Options > Save as Template**to create a new project template that automatically uses this rule.

## Removing the rule

- Select the table(s)/chart(s) that you wish to remove the rule from.
- Press the
**Rules**tab (bottom-right corner). - Press
**Apply**next to the rule you wish to remove and choose the appropriate option.

## How to modify the rule

- Click on the
**Rules**tab (bottom-left of the table/chart). - Select the rule that you wish to modify.
- Click
**Edit Rule**and make the desired changes. Alternatively, you can use the JavaScript below to make your own rule (see Customizing Rules).

## JavaScript

form.setSummary('Column Comparisons on Grids with Lots of Missing Data'); //Independent samples z-test with unequal variance runProjectJavaScript() //Ensures that any JavaScript set for the entire project runs //get the relevant statistics var has_average = table.statistics.indexOf('Average') != -1;//checking if table has averages var is_proportion = table.statistics.indexOf('%') != -1;//checking if table has percentages if (!has_average && !is_proportion) form.ruleNotApplicable("JavaScript for this table requires the table to have percentages or an average"); if (table.numberColumns > 26) form.ruleNotApplicable("JavaScript for this table only works with 26 or fewer columns"); var se = table.get('Standard Error'); var cc = table.get('Column Comparisons'); var values = has_average ? table.get('Average') : table.get('%') // Set up controls for user input. form.setHeading('Column Comparisons on Grids with Lots of Missing Data:'); var label1 = form.newLabel('Lower t-stat:'); var numeric_lower = form.newNumericUpDown('Lower'); numeric_lower.setIncrement(.005); numeric_lower.setDefault(1.96);//test at approximately 0.05 level numeric_lower.setMaximum(5); var label2 = form.newLabel('Upper t-stat:'); var numeric_upper = form.newNumericUpDown('Upper'); numeric_upper.setIncrement(.005); numeric_upper.setDefault(1.645);//test at approximately 0.1 level numeric_upper.setMaximum(5); form.setInputControls([label1, numeric_lower, label2, numeric_upper]); var lower_tstat = numeric_lower.requireValue(); var upper_tstat = numeric_upper.requireValue(); form.setSummary('Column Comparisons on Grids with Lots of Missing Data, showing significant results with t-stats ' + lower_tstat + ' and ' + upper_tstat); //removing current Column Comparisons for (var row = 0; row < table.numberRows; row++) { for (var column = 0; column < table.numberColumns; column++) { cc[row][column] = ''; } } // Doing the testing var upper = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z']; var lower = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']; for (var r = 0; r < table.numberRows; r++) { for (var right_column = 1; right_column < table.numberColumns; right_column++) { for (var left_column = 0; left_column < right_column; left_column++) { var diff = values[r][left_column] - values[r][right_column]; var left_se = se[r][left_column]; var right_se = se[r][right_column]; var pooled_se = Math.sqrt(left_se*left_se + right_se*right_se) var t = diff/pooled_se; if (t < -lower_tstat) cc[r][right_column] += upper[left_column] + " "; else if (t < -upper_tstat) cc[r][right_column] += lower[left_column] + " "; else if (t > lower_tstat) cc[r][left_column] += upper[right_column] + " "; else if (t > upper_tstat) cc[r][left_column] += lower[right_column] + " "; } } } table.set('Column Comparisons', cc);

## See also

- User Input for Rules for technical information on Rules.
- Rule Online Library for other examples of Rules.
- Table JavaScript and Plot JavaScript for the JavaScript that is used to write custom rules.
- JavaScript for information about the JavaScript programming language.

Q Technical Reference

Q Technical Reference

Q Technical Reference > Creating And Modifying Tables

Q Technical Reference > Setting Up Data > Creating New Variables

Q Technical Reference > Updating and Automation > Automation Online Library

Q Technical Reference > Updating and Automation > JavaScript > Table JavaScript and Plot JavaScript

Rule Online Library