Text Analysis - Automatic Categorization - Unstructured Text

From Q
Jump to: navigation, search

Automatically categorize a text variable containing unstructured text into single-response or multiple-response categories. The user can allow the algorithm to determine appropriate categories and their labels automatically based on patterns observed in the data. Alternatively, the user can a provide partial categorization of cases in the data and the algorithm will predict which of the user specified categories the remaining cases belong to. These categories for all cases can then be saved using Create > Text Analysis > Advanced > Save Variable(s) > CategoriesInsert > Text Analysis > Advanced > Save Variable(s) > Categories.

Example

To create an Unstructured Text output, go to Insert > Text Analysis > Automatic Categorization > Unstructured TextCreate > Text Analysis > Automatic Categorization > Unstructured Text.

Without Existing Categorization

  1. Under Inputs > DATA SOURCE > Text variable select a Text variable.
  2. Make any other selections or changes to the settings that you require.
  3. Ensure the Automatic box is checked, or click Calculate

The output below shows automatically generated categories for a question on how people feel about Microsoft. Each row in the example column can be expanded to show all the text in the category.

TAUnstructured Text 2020-06-25.png

With Existing Categorization

  1. Under Inputs > DATA SOURCE > Text variable select a Text variable.
  2. Under Inputs > CATEGORIES > Existing Categorization select a Nominal or Ordinal variable or Binary - Multi variable setCategorical or Ordered categorical variable or Pick Any Question.
  3. Make any other selections or changes to the settings that you require.
  4. Ensure the Automatic box is checked, or click Calculate

The output below shows automatically generated categories for a question on how people feel about their cell phone provider. The text variable has 2090 responses, 895 of which were provided an existing categorization by a market researcher. Each row in the example column can be expanded to show all the text in the category and the performance metrics can be viewed in the main table and the cross validation performance visible in the smaller table at the bottom.

TAUnstructured Text TC 2020-06-25.png


Extracting a Table of Frequencies

To extract the table of frequencies from this output can be done by saving the results as a variable into your Data Set and then making a table from that. Take these steps:

  1. Select the Unstructured Text output.
  2. Go to Insert > Text Analysis > Advanced > Save Variable(s) > CategoriesCreate > Text Analysis > Advanced > Save Variable(s) > Categories

A variable setQuestion called "Categories from Text data" will be saved in your Data Set. This can then be used like any other variable to create tables and further outputs.


More Information

Automatic Coding of Unstructured Text Data
Automatic Categorization of Unstructured Text Data

Options

DATA SOURCE

Text variable The text variable to run automatic categorization on.

TRANSLATE (GOOGLE CLOUD TRANSLATION)

Source language The language of the input text variable. To specify the language separately for each case, select "Specify with variable" (see Source language variable for more information). The input text is translated from the source language into English before categorization is performed. Translation is done using Google Cloud Translation and results may change over time as the translation algorithm improves.

Source language variable A text/categorical variable containing the language for each case. Languages should be referred to by the language names in Source language. This control appears when Source language is set to "Specify with variable".

Output language The language of the categories and example text to display in the output.

CATEGORIES

Existing categorization A categorical, ordered-categorical variable or Pick Any question containing an existing categorization to be used.

Categories to Ignore Comma separated list of categories to ignore in the input provided to Existing Categorization, typically used to remove the data reduction variables such as NET, Total and SUM.

Number of categories The number of categories to automatically generate. This is hidden if an existing categorization is provided.

CROSS VALIDATION

These controls are only available when an existing categorization is provided. When provided, the cases in the data that have been given an existing categorization are interpreted as the full training data set and repeated random sub-sampling cross validation is performed on the full training data set. In particular, the full training data-set is split into a training and validation datasets where the validation data has its existing categorization removed. The categorization for the validation data is then predicted from the available data in the training sub sample and performance table generated across all cross validation iterations. The controls below determine the size of the training data sub sample and how many cross validation iterations of this type are performed. If no existing categorization is provided, then these controls are hidden.

Sample size increment for cross-validation An integer that specifies how large the initial training data sub sample size will be and how much it will grow for each repeated random sub-sampling cross validation iteration that is performed. The default value is 50 meaning that the initial training sub sample will have 50 observations from the full training data set and 50 more observations will be added for each new cross validation iteration that is performed.

Number of cross-validation iterations Number of cross validations iterations to perform for the random sub-sampling cross validation procedure described above. The default is four iterations and the default value for the Sample size increment for cross-validation control above is 50. If n > 200 cases are provided for the existing categorization then this would result in 4 randomly generated training and cross validation splits of the full training data whereby the training sub samples will have 50, 100, 150 and 200 cases while the validation sub samples will have n - 50, n - 100, n - 150 and n - 200 cases respectively over the four cross validation iterations.

SAVE VARIABLE(S)

Save categories Adds variables to the data set containing the categories. Where there are multiple input variables, multiple sets of variables are added for each.

Save first category Adds a variable to the data set containing the first category mentioned. Where there are multiple input categories, the first category of each will be saved as a separate variable.

Code

var allow_control_groups = Q.fileFormatVersion() > 10.9;

var languages = ["Afrikaans","Albanian","Amharic","Arabic","Armenian","Azerbaijani","Basque",
                 "Belarusian","Bengali","Bosnian","Bulgarian","Catalan","Cebuano","Chichewa",
                 "Chinese (Simplified)","Chinese (Traditional)","Corsican","Croatian","Czech",
                 "Danish","Dutch","English","Esperanto","Estonian","Filipino","Finnish","French",
                 "Frisian","Galician","Georgian","German","Greek","Gujarati","Haitian Creole",
                 "Hausa","Hawaiian","Hebrew","Hindi","Hmong","Hungarian","Icelandic","Igbo",
                 "Indonesian","Irish","Italian","Japanese","Javanese","Kannada","Kazakh","Khmer",
                 "Kinyarwanda","Korean","Kurdish (Kurmanji)","Kyrgyz","Lao","Latin","Latvian",
                 "Lithuanian","Luxembourgish","Macedonian","Malagasy","Malay","Malayalam","Maltese",
                 "Maori","Marathi","Mongolian","Myanmar (Burmese)","Nepali","Norwegian",
                 "Odia (Oriya)","Pashto","Persian","Polish","Portuguese","Punjabi","Romanian",
                 "Russian","Samoan","Scots Gaelic","Serbian","Sesotho","Shona","Sindhi","Sinhala",
                 "Slovak","Slovenian","Somali","Spanish","Sundanese","Swahili","Swedish","Tajik",
                 "Tamil","Tatar","Telugu","Thai","Turkish","Turkmen","Ukrainian","Urdu","Uyghur",
                 "Uzbek","Vietnamese","Welsh","Xhosa","Yiddish","Yoruba","Zulu"];

if (allow_control_groups)
    form.group({label: "Data source", expanded: true});

form.dropBox({name: "formTextVar", label: "Text variable",
              types: ["Variable: text, categorical", "R:character"],
              prompt: "Select the text variable to analyze.", multi: false});

if (allow_control_groups)
    form.group({label: "Translate (Google Cloud Translation)", expanded: true});

var source_language = form.comboBox({name: "formSourceLang",
                                     label: "Source language",
                                     alternatives: ["English",
                                                    "Specify with variable"].concat(languages),
                                     default_value: "English",
                                     prompt: "Specify the language of the input text variable."}).getValue();

if (source_language == "Specify with variable")
{
    form.dropBox({name: "formSourceLangVar",
                  label: "Source language variable",
                  types: ["Variable: Text, Categorical, OrderedCategorical"],
                  prompt: "Variable containing source language for each case",
                  required: true});
}

form.comboBox({name: "formOutputLang",
               label: "Output language",
               alternatives: languages,
               default_value: "English",
               prompt: "Specify the output language"}).getValue();

if (allow_control_groups)
    form.group({label: "Categories", expanded: true});

var existing_cat = form.dropBox({name: "formExistingCat", label: "Existing categorization",
              types: ["Variable: categorical, orderedcategorical", "Question: PickAny"],
              prompt: "Optional variable containing a manual categorization",
              multi: false, required: false}).getValue();
if (existing_cat === null)
    form.numericUpDown({name: "formCategories",
                    label: "Number of categories",
                    default_value: 10,
                    increment: 1,
                    maximum:99999999,
                    minimum: 1})
else {
    form.textBox({label: "Categories to ignore", 
              type: "text",
              default_value: "NET, Total, SUM", 
              name: "formIgnore",
              required: false,
              prompt: "Specify categories to ignore when predicting the existing categorization"});

    if (allow_control_groups)
        form.group({label: "Cross Validation", expanded: true});

    form.numericUpDown({name: "formCVIncrement",
                   label: "Sample size increment for cross-validation",
                   prompt: "Increment of the training data sample size across a repeated random sub-sampling cross validation",     
                   default_value: 50,
                   increment: 50,
                   minimum: 50,
                   maximum: 99999999})
    form.numericUpDown({name: "formNCVs",
                   label: "Number of cross-validation iterations",
                   prompt: "Number of cross validations iterations to perform or number of times to increment the training data sample size",     
                   default_value: 4,
                   increment: 1,
                   minimum: 1,
                   maximum: 99999999})
}
library(flipTextAnalysis)
categorization <- if (is.null(formExistingCat)) {
    AutomaticCategorization(formTextVar,
                            n.categories = min(formCategories, length(formTextVar)),
                            sentiment.weight = 2,
                            discard.phrases = NULL,
                            raw.text.replacement = NULL,
                            min.frequency = max(5, log(length(formTextVar))),
                            max.bag.size = Inf,
                            subset = QFilter,
                            weights = QPopulationWeight,
                            seed = 1223,
                            version = 0,
                            source.language = get0("formSourceLangVar", ifnotfound = formSourceLang),
                            output.language = formOutputLang)
} else {
    vars <- formExistingCat
    if(is.data.frame(vars)) {
        vars <- formExistingCat[, !names(vars) %in% strsplit(formIgnore, ",")[[1]], drop = FALSE]
        n.observed <- nrow(vars)
    } else 
        n.observed <- length(vars)
    TextClassifier(vars,
                   formTextVar,
                   discard.phrases = NULL,
                   raw.text.replacement = NULL,
                   min.frequency = 5,
                   max.bag.size = 100,
                   subset = QFilter,
                   weights = QPopulationWeight,
                   cross.validation.size = formCVIncrement,
                   n.cross.validations = formNCVs,
                   seed = 12321,
                   source.language = get0("formSourceLangVar", ifnotfound = formSourceLang),
                   output.language = formOutputLang)
}