Text Analysis - Automatic Categorization - Unstructured Text
This tool allows you to use a partial categorization of text data to predict categories for text which was not categorized. This is useful if you have done some categorization on a subset of your data and you wish to apply this categorization to more text. For example, if text data has been categorized for the first wave of a study, and you wish to use the same categorization to subsequent waves of the same study. Alternatively, this tool may be used to determine appropriate categories and their labels automatically based on patterns observed in the data. However, this second use case is only supported for legacy reasons, and the generation of categories can be done in a better and more flexible way by using + Anything > Advanced Analysis > Text Analysis > Semi-Automatic Categorization
The categories for all cases can then be saved using Create > Text Analysis > Advanced > Save Variable(s) > Categories.
Example
To create an Unstructured Text output, go to Create > Text Analysis > Automatic Categorization > Unstructured Text.
With Existing Categorization
- Under Inputs > DATA SOURCE > Text variable select a Text variable.
- Under Inputs > CATEGORIES > Existing Categorization select a Categorical or Ordered categorical variable or Pick Any Question.
- Make any other selections or changes to the settings that you require.
- Ensure the Automatic box is checked, or click Calculate
The output below shows automatically generated categories for a question on how people feel about their cell phone provider. The text variable has 2090 responses, 895 of which were provided an existing categorization by a market researcher. Each row in the example column can be expanded to show all the text in the category and the performance metrics can be viewed in the main table and the cross validation performance visible in the table which appears in the footer.
Without Existing Categorization
- Under Inputs > CATEGORIES > Category creation choose Create new categorization
- Under Inputs > DATA SOURCE > Text variable select a Text variable.
- Make any other selections or changes to the settings that you require.
- Ensure the Automatic box is checked, or click Calculate
The output below shows automatically generated categories for a question on how people feel about Microsoft. Each row in the example column can be expanded to show all the text in the category.
Extracting a Table of Frequencies
To extract the table of frequencies from this output can be done by saving the results as a variable into your Data Set and then making a table from that. Take these steps:
- Select the Unstructured Text output.
- Go to Create > Text Analysis > Advanced > Save Variable(s) > Categories
A Question called "Categories from Text data" will be saved in your Data Set. This can then be used like any other variable to create tables and further outputs.
More Information
Coding of Unstructured Text Data
Categorization of Unstructured Text Data
Options
DATA SOURCE
Text variable The text variable to run automatic categorization on.
Truncate cases when characters exceed Truncate cases whose number of characters exceeds this number. This is done as the algorithm expects text data to consist of one sentence per case and may not function correctly when cases are too long. If any cases are truncated due to this setting, a warning will be shown, and it is likely that the data does not conform to this assumption (one sentence per case) and is not appropriate for this analysis.
CATEGORIES
Category creation Choose between Use existing categorization and Create new categorization.
Existing categorization A Pick One or Pick Any question containing an existing categorization to be used. This is hidden if you choose to create a new categorization.
- Categories to Ignore Comma separated list of categories to ignore in the input provided to Existing Categorization, typically used to remove the data reduction variables such as NET, Total and SUM.
Number of categories The number of categories to automatically generate. This is hidden if you are using an existing categorization.
TRANSLATE (GOOGLE CLOUD TRANSLATION)
Source language The language of the input text variable. To specify the language separately for each case, select "Specify with variable" (see Source language variable for more information). The input text is translated from the source language into English before categorization is performed. Translation is done using Google Cloud Translation and results may change over time as the translation algorithm improves.
Source language variable A text/categorical variable containing the language for each case. Languages should be referred to by the language names in Source language. This control appears when Source language is set to "Specify with variable".
Output language The language of the categories and example text to display in the output.
CROSS VALIDATION
These controls are only available when an existing categorization is provided. When provided, the cases in the data that have been given an existing categorization are interpreted as the full training data set and repeated random sub-sampling cross validation is performed on the full training data set. In particular, the full training data-set is split into a training and test datasets where the test data has its existing categorization removed. The categorization for the test data is then predicted from the available data in the training sub sample and performance table generated across all cross validation iterations. The controls below determine the size of the training data sub sample and how many cross validation iterations of this type are performed. If no existing categorization is provided, then these controls are hidden.
Sample size increment for cross-validation An integer that specifies how large the initial training data sub sample size will be and how much it will grow for each repeated random sub-sampling cross validation iteration that is performed. The default value is 50 meaning that the initial training sub sample will have 50 observations from the full training data set and 50 more observations will be added for each new cross validation iteration that is performed.
Number of cross-validation iterations Number of cross validations iterations to perform for the random sub-sampling cross validation procedure described above. The default is four iterations and the default value for the Sample size increment for cross-validation control above is 50. If n > 200 cases are provided for the existing categorization then this would result in 4 randomly generated training and cross validation splits of the full training data whereby the training sub samples will have 50, 100, 150 and 200 cases while the test sub samples will have n - 50, n - 100, n - 150 and n - 200 cases respectively over the four cross validation iterations.
SAVE VARIABLE(S)
Save categories Adds variables to the data set containing the categories. Where there are multiple input variables, multiple sets of variables are added for each.
Save first category Adds a variable to the data set containing the first category mentioned. Where there are multiple input categories, the first category of each will be saved as a separate variable.
Code
▶ Show Code
▶ Show Code