# Text Analysis - Automatic Categorization - Unstructured Text

Categorize a text variable containing unstructured text into single-response or multiple-response categories

Automatically categorize a text variable containing unstructured text into single-response or multiple-response categories. The user can allow the algorithm to determine appropriate categories and their labels automatically based on patterns observed in the data. Alternatively, the user can a provide partial categorization of cases in the data and the algorithm will predict which of the user specified categories the remaining cases belong to. These categories for all cases can then be saved using Create > Text Analysis > Advanced > Save Variable(s) > CategoriesInsert > Text Analysis > Advanced > Save Variable(s) > Categories.

## Example

To create an Unstructured Text output, go to Insert > Text Analysis > Automatic Categorization > Unstructured TextCreate > Text Analysis > Automatic Categorization > Unstructured Text.

### Without Existing Categorization

1. Under Inputs > DATA SOURCE > Text variable select a Text variable.
2. Make any other selections or changes to the settings that you require.
3. Ensure the Automatic box is checked, or click Calculate

The output below shows automatically generated categories for a question on how people feel about Microsoft. Each row in the example column can be expanded to show all the text in the category.

### With Existing Categorization

1. Under Inputs > DATA SOURCE > Text variable select a Text variable.
2. Under Inputs > CATEGORIES > Existing Categorization select a Nominal or Ordinal variable or Binary - Multi variable setCategorical or Ordered categorical variable or Pick Any Question.
3. Make any other selections or changes to the settings that you require.
4. Ensure the Automatic box is checked, or click Calculate

The output below shows automatically generated categories for a question on how people feel about their cell phone provider. The text variable has 2090 responses, 895 of which were provided an existing categorization by a market researcher. Each row in the example column can be expanded to show all the text in the category and the performance metrics can be viewed in the main table and the cross validation performance visible in the smaller table at the bottom.

### Extracting a Table of Frequencies

To extract the table of frequencies from this output can be done by saving the results as a variable into your Data Set and then making a table from that. Take these steps:

1. Select the Unstructured Text output.
2. Go to Insert > Text Analysis > Advanced > Save Variable(s) > CategoriesCreate > Text Analysis > Advanced > Save Variable(s) > Categories

A variable setQuestion called "Categories from Text data" will be saved in your Data Set. This can then be used like any other variable to create tables and further outputs.

## Options

### DATA SOURCE

Text variable The text variable to run automatic categorization on.

Truncate cases when characters exceed Truncate cases whose number of characters exceeds this number. This is done as the algorithm expects text data to consist of one sentence per case and may not function correctly when cases are too long. If any cases are truncated due to this setting, a warning will be shown, and it is likely that the data does not conform to this assumption (one sentence per case) and is not appropriate for this analysis.

Source language The language of the input text variable. To specify the language separately for each case, select "Specify with variable" (see Source language variable for more information). The input text is translated from the source language into English before categorization is performed. Translation is done using Google Cloud Translation and results may change over time as the translation algorithm improves.

Source language variable A text/categorical variable containing the language for each case. Languages should be referred to by the language names in Source language. This control appears when Source language is set to "Specify with variable".

Output language The language of the categories and example text to display in the output.

### CATEGORIES

Existing categorization A categorical, ordered-categorical variable or Pick Any question containing an existing categorization to be used.

Categories to Ignore Comma separated list of categories to ignore in the input provided to Existing Categorization, typically used to remove the data reduction variables such as NET, Total and SUM.

Number of categories The number of categories to automatically generate. This is hidden if an existing categorization is provided.

### CROSS VALIDATION

These controls are only available when an existing categorization is provided. When provided, the cases in the data that have been given an existing categorization are interpreted as the full training data set and repeated random sub-sampling cross validation is performed on the full training data set. In particular, the full training data-set is split into a training and test datasets where the test data has its existing categorization removed. The categorization for the test data is then predicted from the available data in the training sub sample and performance table generated across all cross validation iterations. The controls below determine the size of the training data sub sample and how many cross validation iterations of this type are performed. If no existing categorization is provided, then these controls are hidden.

Sample size increment for cross-validation An integer that specifies how large the initial training data sub sample size will be and how much it will grow for each repeated random sub-sampling cross validation iteration that is performed. The default value is 50 meaning that the initial training sub sample will have 50 observations from the full training data set and 50 more observations will be added for each new cross validation iteration that is performed.

Number of cross-validation iterations Number of cross validations iterations to perform for the random sub-sampling cross validation procedure described above. The default is four iterations and the default value for the Sample size increment for cross-validation control above is 50. If n > 200 cases are provided for the existing categorization then this would result in 4 randomly generated training and cross validation splits of the full training data whereby the training sub samples will have 50, 100, 150 and 200 cases while the test sub samples will have n - 50, n - 100, n - 150 and n - 200 cases respectively over the four cross validation iterations.

### SAVE VARIABLE(S)

Save categories Adds variables to the data set containing the categories. Where there are multiple input variables, multiple sets of variables are added for each.

Save first category Adds a variable to the data set containing the first category mentioned. Where there are multiple input categories, the first category of each will be saved as a separate variable.

## Code

let heading_text = "Unstructured Text Categorization";
if (!!form.setObjectInspectorTitle)
else

let allow_control_groups = Q.fileFormatVersion() > 10.9;

let languages = ["Afrikaans","Albanian","Amharic","Arabic","Armenian","Azerbaijani","Basque",
"Belarusian","Bengali","Bosnian","Bulgarian","Catalan","Cebuano","Chichewa",
"Danish","Dutch","English","Esperanto","Estonian","Filipino","Finnish","French",
"Frisian","Galician","Georgian","German","Greek","Gujarati","Haitian Creole",
"Hausa","Hawaiian","Hebrew","Hindi","Hmong","Hungarian","Icelandic","Igbo",
"Kinyarwanda","Korean","Kurdish (Kurmanji)","Kyrgyz","Lao","Latin","Latvian",
"Lithuanian","Luxembourgish","Macedonian","Malagasy","Malay","Malayalam","Maltese",
"Maori","Marathi","Mongolian","Myanmar (Burmese)","Nepali","Norwegian",
"Odia (Oriya)","Pashto","Persian","Polish","Portuguese","Punjabi","Romanian",
"Russian","Samoan","Scots Gaelic","Serbian","Sesotho","Shona","Sindhi","Sinhala",
"Slovak","Slovenian","Somali","Spanish","Sundanese","Swahili","Swedish","Tajik",
"Tamil","Tatar","Telugu","Thai","Turkish","Turkmen","Ukrainian","Urdu","Uyghur",
"Uzbek","Vietnamese","Welsh","Xhosa","Yiddish","Yoruba","Zulu"];

if (allow_control_groups)
form.group({label: "Data source", expanded: true});

form.dropBox({name: "formTextVar", label: "Text variable",
types: ["Variable: text, categorical", "R:character"],
prompt: "Select the text variable to analyze.", multi: false});

form.numericUpDown({name: "formMaxCaseChar",
label: "Truncate cases with characters exceeding",
default_value: 2000,
increment: 1000,
maximum: Number.MAX_SAFE_INTEGER,
minimum: 1,
prompt: "Truncate cases whose number of characters exceeds this number."})

if (allow_control_groups)
form.group({label: "Categories", expanded: true});

let selected_mode = form.comboBox({name: "formMode", label: "Category creation",
alternatives: ["Use existing categorization", "Create new categorization"],
default_value: "Use existing categorization",
prompt: "Predict categories for new cases using and exising categorization or create a new categorization from scratch"}).getValue();

let existing_cat;
if (selected_mode === "Use existing categorization") {
existing_cat = form.dropBox({name: "formExistingCat", label: "Existing categorization",
types: ["Variable: categorical, orderedcategorical", "Question: PickAny"],
prompt: "Optional variable containing a manual categorization",
multi: false, required: true}).getValue();

form.textBox({label: "Categories to ignore",
type: "text",
default_value: "NET, Total, SUM",
name: "formIgnore",
required: false,
prompt: "Specify categories to ignore when predicting the existing categorization"});

} else {
form.numericUpDown({name: "formCategories",
label: "Number of categories",
default_value: 10,
increment: 1,
maximum: Number.MAX_SAFE_INTEGER,
minimum: 1})
}

if (allow_control_groups)
form.group({label: "Filters & Weight"});

if (allow_control_groups)
form.group({label: "Translate (Google Cloud Translation)", expanded: true});

let source_language = form.comboBox({name: "formSourceLang",
label: "Source language",
alternatives: ["English",
"Specify with variable"].concat(languages),
default_value: "English",
prompt: "Specify the language of the input text variable."}).getValue();

if (source_language == "Specify with variable")
{
form.dropBox({name: "formSourceLangVar",
label: "Source language variable",
types: ["Variable: Text, Categorical, OrderedCategorical"],
prompt: "Variable containing source language for each case",
required: true});
}

form.comboBox({name: "formOutputLang",
label: "Output language",
alternatives: languages,
default_value: "English",
prompt: "Specify the output language"}).getValue();

if (selected_mode === "Use existing categorization") {
if (allow_control_groups)
form.group({label: "Cross Validation", expanded: true});

form.numericUpDown({name: "formCVIncrement",
label: "Sample size increment for cross-validation",
prompt: "Increment of the training data sample size across a repeated random sub-sampling cross validation",
default_value: 50,
increment: 50,
minimum: 50,
maximum: Number.MAX_SAFE_INTEGER})
form.numericUpDown({name: "formNCVs",
label: "Number of cross-validation iterations",
prompt: "Number of cross validations iterations to perform or number of times to increment the training data sample size",
default_value: 4,
increment: 1,
minimum: 1,
maximum: Number.MAX_SAFE_INTEGER})
}

if (allow_control_groups)
form.group({label: "Save Variable(s)", expanded: false});

library(flipTextAnalysis)
existing.categories <- !is.null(get0("formExistingCat", ifnotfound = NULL))
arguments <- list(min.frequency = max(5L, 1e-3 * length(formTextVar)),
raw.text.replacement = NULL,
subset = QFilter,
weights = QPopulationWeight,
seed = if (existing.categories) 12321L else 1223L,
source.language = get0("formSourceLangVar", ifnotfound = formSourceLang),
output.language = formOutputLang,
max.case.characters = formMaxCaseChar,
max.bag.size = if (existing.categories) 100L else Inf)

if (existing.categories)
{
vars <- formExistingCat
if (is.data.frame(vars))
vars <- formExistingCat[, !names(vars) %in% strsplit(formIgnore, ",")[[1L]], drop = FALSE]
arguments <- c(arguments,
categorization = list(vars),
x = list(formTextVar),
cross.validation.size = formCVIncrement,
n.cross.validations = formNCVs)
} else
arguments <- c(arguments,
x = list(formTextVar),
n.categories = min(formCategories, length(formTextVar)),
sentiment.weight = 2,
version = 0)

text.analysis.function <- if (existing.categories) TextClassifier else AutomaticCategorization

categorization <- TextAnalysis(text.analysis.function, arguments)