.. _sec_textprediction_beginner:

Text Prediction - Quick Start
=============================


Here we introduce the ``TextPrediction`` task, which helps you
automatically train and deploy models for various Natural Language
Processing (NLP) problems. This tutorial presents two examples to
demonstrate how ``TextPrediction`` can be used for different NLP tasks
including:

-  `Sentiment
   Analysis <https://en.wikipedia.org/wiki/Sentiment_analysis>`__
-  `Sentence Similarity <https://arxiv.org/abs/1910.03940>`__

The general usage is similar to AutoGluon's ``TabularPrediction``
module. We treat NLP datasets as tables where certain columns contain
text fields and a special column contains the labels to predict. Here,
the labels can be discrete categories (classification) or numerical
values (regression). ``TextPrediction`` fits neural networks to your
data via transfer learning from pretrained NLP models like:
`BERT <https://arxiv.org/pdf/1810.04805.pdf>`__,
`ALBERT <https://arxiv.org/pdf/1909.11942.pdf>`__, and
`ELECTRA <https://openreview.net/pdf?id=r1xMH1BtvB>`__.
``TextPrediction`` also trains multiple models with different
hyperparameters and returns the best model, a process called
Hyperparameter Optimization (HPO).

.. code:: python

    import numpy as np
    import warnings
    warnings.filterwarnings('ignore')
    np.random.seed(123)

Sentiment Analysis
------------------

First, we consider the Stanford Sentiment Treebank
(`SST <https://nlp.stanford.edu/sentiment/>`__) dataset, which consists
of movie reviews and their associated sentiment. Given a new movie
review, the goal is to predict the sentiment reflected in the text (in
this case a **binary classification** problem, where reviews are labeled
as 1 if they convey a positive opinion and labeled as 0 otherwise).
Let's first load the data and view some examples, noting the labels are
stored in a column called **label**.

.. code:: python

    from autogluon.utils.tabular.utils.loaders.load_pd import load
    train_data = load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/train.parquet')
    dev_data = load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/dev.parquet')
    rand_idx = np.random.permutation(np.arange(len(train_data)))[:2000]
    train_data = train_data.iloc[rand_idx]
    train_data.head(10)


.. parsed-literal::
    :class: output

    Loaded data from: https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/train.parquet | Columns = 2 / 2 | Rows = 67349 -> 67349
    Loaded data from: https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/dev.parquet | Columns = 2 / 2 | Rows = 872 -> 872




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sentence</th>
          <th>label</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>2434</th>
          <td>goes by quickly</td>
          <td>1</td>
        </tr>
        <tr>
          <th>27796</th>
          <td>reading lines from a teleprompter</td>
          <td>0</td>
        </tr>
        <tr>
          <th>249</th>
          <td>degraded , handheld blair witch video-cam foot...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>12115</th>
          <td>reminds us how realistically nuanced a robert ...</td>
          <td>1</td>
        </tr>
        <tr>
          <th>50834</th>
          <td>indulges in the worst elements of all of them .</td>
          <td>0</td>
        </tr>
        <tr>
          <th>43622</th>
          <td>are nowhere near as vivid as the 19th-century ...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>3955</th>
          <td>throughout a film that is both gripping and co...</td>
          <td>1</td>
        </tr>
        <tr>
          <th>51011</th>
          <td>to see over and over again</td>
          <td>1</td>
        </tr>
        <tr>
          <th>31232</th>
          <td>that fails to match the freshness of the actre...</td>
          <td>0</td>
        </tr>
        <tr>
          <th>32153</th>
          <td>this is an undeniably intriguing film from an ...</td>
          <td>1</td>
        </tr>
      </tbody>
    </table>
    </div>



Above the data happen to be stored in a
`Parquet <https://databricks.com/glossary/what-is-parquet>`__ table
format, but you can also directly ``load()`` data from a
`CSV <https://en.wikipedia.org/wiki/Comma-separated_values>`__ file
instead. While here we load files from `AWS S3 cloud
storage <https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html>`__,
these could instead be local files on your machine. After loading,
``train_data`` is simply a `Pandas
DataFrame <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html>`__,
where each row represents a different training example (for machine
learning to be appropriate, the rows should be independent and
identically distributed).

To ensure this tutorial runs quickly, we simply call ``fit()`` with a
subset of 2000 training examples and limit its runtime to approximately
1 minute. To achieve reasonable performance in your applications, you
should set much longer ``time_limits`` (eg. 1 hour), or do not specify
``time_limits`` at all.

.. code:: python

    from autogluon import TextPrediction as task
    
    predictor = task.fit(train_data, label='label', 
                         time_limits=60,
                         ngpus_per_trial=1,
                         seed=123,
                         output_directory='./ag_sst')


.. parsed-literal::
    :class: output

    /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_15/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)
    NumPy-shape semantics has been activated in your code. This is required for creating and manipulating scalar and zero-size tensors, which were not supported in MXNet before, as in the official NumPy library. Please DO NOT manually deactivate this semantics while using `mxnet.numpy` and `mxnet.numpy_extension` modules.
    2020-12-08 20:17:00,613 - root - INFO - All Logs will be saved to ./ag_sst/ag_text_prediction.log
    2020-12-08 20:17:00,623 - root - INFO - Train Dataset:
    2020-12-08 20:17:00,624 - root - INFO - Columns:
    
    - Text(
       name="sentence"
       #total/missing=1600/0
       length, min/avg/max=4/51.93/259
    )
    - Categorical(
       name="label"
       #total/missing=1600/0
       num_class (total/non_special)=2/2
       categories=[0, 1]
       freq=[714, 886]
    )
    
    
    2020-12-08 20:17:00,624 - root - INFO - Tuning Dataset:
    2020-12-08 20:17:00,625 - root - INFO - Columns:
    
    - Text(
       name="sentence"
       #total/missing=400/0
       length, min/avg/max=4/55.12/218
    )
    - Categorical(
       name="label"
       #total/missing=400/0
       num_class (total/non_special)=2/2
       categories=[0, 1]
       freq=[167, 233]
    )
    
    
    2020-12-08 20:17:00,625 - root - INFO - Label columns=['label'], Feature columns=['sentence'], Problem types=['classification'], Label shapes=[2]
    2020-12-08 20:17:00,626 - root - INFO - Eval Metric=acc, Stop Metric=acc, Log Metrics=['f1', 'mcc', 'auc', 'acc', 'nll']



.. parsed-literal::
    :class: output

    HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=4.0), HTML(value='')))


.. parsed-literal::
    :class: output

    
      0%|          | 0/200 [00:00<?, ?it/s]
      0%|          | 1/200 [00:00<01:55,  1.72it/s]
      1%|          | 2/200 [00:00<01:29,  2.20it/s]
      2%|▏         | 3/200 [00:00<01:10,  2.80it/s]
      2%|▏         | 4/200 [00:00<00:56,  3.50it/s]
      2%|▎         | 5/200 [00:01<01:07,  2.91it/s]
      3%|▎         | 6/200 [00:01<00:53,  3.62it/s]
      4%|▎         | 7/200 [00:01<00:44,  4.36it/s]
      4%|▍         | 9/200 [00:01<00:36,  5.25it/s]
      5%|▌         | 10/200 [00:02<00:42,  4.48it/s]
      6%|▌         | 11/200 [00:02<00:35,  5.26it/s]
      6%|▋         | 13/200 [00:02<00:30,  6.16it/s]
      8%|▊         | 15/200 [00:03<00:35,  5.20it/s]
      8%|▊         | 17/200 [00:03<00:30,  6.10it/s]
     10%|▉         | 19/200 [00:03<00:26,  6.88it/s]
     10%|█         | 20/200 [00:03<00:43,  4.15it/s]
     11%|█         | 22/200 [00:04<00:34,  5.20it/s]
     12%|█▏        | 23/200 [00:04<00:29,  6.07it/s]
     12%|█▏        | 24/200 [00:04<00:25,  6.84it/s]
     12%|█▎        | 25/200 [00:04<00:34,  5.03it/s]
     14%|█▎        | 27/200 [00:04<00:28,  6.00it/s]
     14%|█▍        | 28/200 [00:04<00:25,  6.79it/s]
     15%|█▌        | 30/200 [00:05<00:30,  5.59it/s]
     16%|█▌        | 31/200 [00:05<00:26,  6.37it/s]
     16%|█▋        | 33/200 [00:05<00:23,  7.17it/s]
     18%|█▊        | 35/200 [00:06<00:25,  6.37it/s]
     18%|█▊        | 36/200 [00:06<00:23,  6.96it/s]
     18%|█▊        | 37/200 [00:06<00:21,  7.47it/s]
     20%|█▉        | 39/200 [00:06<00:19,  8.22it/s]
     20%|██        | 40/200 [00:06<00:28,  5.65it/s]
     21%|██        | 42/200 [00:06<00:23,  6.69it/s]
     22%|██▏       | 44/200 [00:07<00:20,  7.70it/s]
     22%|██▎       | 45/200 [00:07<00:27,  5.65it/s]
     23%|██▎       | 46/200 [00:07<00:24,  6.37it/s]
     24%|██▍       | 48/200 [00:07<00:20,  7.35it/s]
     25%|██▌       | 50/200 [00:08<00:22,  6.55it/s]
     26%|██▌       | 52/200 [00:08<00:19,  7.47it/s]
     27%|██▋       | 54/200 [00:08<00:17,  8.45it/s]
     28%|██▊       | 55/200 [00:08<00:24,  5.83it/s]
     28%|██▊       | 57/200 [00:08<00:21,  6.74it/s]
     30%|██▉       | 59/200 [00:09<00:18,  7.74it/s]
     30%|███       | 60/200 [00:09<00:30,  4.56it/s]
     31%|███       | 62/200 [00:09<00:25,  5.50it/s]
     32%|███▏      | 63/200 [00:09<00:21,  6.24it/s]
     32%|███▎      | 65/200 [00:10<00:26,  5.10it/s]
     34%|███▎      | 67/200 [00:10<00:21,  6.19it/s]
     34%|███▍      | 69/200 [00:10<00:18,  7.13it/s]
     35%|███▌      | 70/200 [00:11<00:24,  5.29it/s]
     36%|███▌      | 72/200 [00:11<00:20,  6.27it/s]
     37%|███▋      | 74/200 [00:11<00:17,  7.30it/s]
     38%|███▊      | 75/200 [00:11<00:28,  4.39it/s]
     38%|███▊      | 77/200 [00:11<00:22,  5.38it/s]
     40%|███▉      | 79/200 [00:12<00:19,  6.31it/s]
     40%|████      | 80/200 [00:12<00:24,  4.95it/s]
     41%|████      | 82/200 [00:12<00:19,  5.94it/s]
     42%|████▏     | 84/200 [00:12<00:16,  7.03it/s]
     42%|████▎     | 85/200 [00:13<00:21,  5.34it/s]
     44%|████▎     | 87/200 [00:13<00:17,  6.36it/s]
     44%|████▍     | 89/200 [00:13<00:15,  7.34it/s]
     45%|████▌     | 90/200 [00:13<00:26,  4.14it/s]
     46%|████▌     | 91/200 [00:14<00:22,  4.94it/s]
     46%|████▋     | 93/200 [00:14<00:18,  5.86it/s]
     48%|████▊     | 95/200 [00:14<00:20,  5.04it/s]
     48%|████▊     | 97/200 [00:14<00:16,  6.08it/s]
     50%|████▉     | 99/200 [00:15<00:14,  7.12it/s]
     50%|█████     | 100/200 [00:15<00:18,  5.43it/s]
     50%|█████     | 101/200 [00:15<00:15,  6.29it/s]
     52%|█████▏    | 103/200 [00:15<00:13,  7.27it/s]
     52%|█████▎    | 105/200 [00:16<00:14,  6.56it/s]
     53%|█████▎    | 106/200 [00:16<00:12,  7.31it/s]
     54%|█████▎    | 107/200 [00:16<00:12,  7.68it/s]
     55%|█████▍    | 109/200 [00:16<00:10,  8.58it/s]
     55%|█████▌    | 110/200 [00:16<00:15,  5.85it/s]
     56%|█████▌    | 112/200 [00:16<00:12,  6.78it/s]
     56%|█████▋    | 113/200 [00:17<00:11,  7.51it/s]
     57%|█████▊    | 115/200 [00:17<00:14,  5.71it/s]
     58%|█████▊    | 117/200 [00:17<00:12,  6.79it/s]
     59%|█████▉    | 118/200 [00:17<00:10,  7.50it/s]
     60%|██████    | 120/200 [00:18<00:12,  6.58it/s]
     61%|██████    | 122/200 [00:18<00:10,  7.43it/s]
     62%|██████▏   | 124/200 [00:18<00:09,  8.41it/s]
     62%|██████▎   | 125/200 [00:19<00:17,  4.33it/s]
     64%|██████▎   | 127/200 [00:19<00:13,  5.33it/s]
     64%|██████▍   | 129/200 [00:19<00:11,  6.37it/s]
     65%|██████▌   | 130/200 [00:19<00:13,  5.06it/s]
     66%|██████▌   | 132/200 [00:19<00:11,  6.16it/s]
     67%|██████▋   | 134/200 [00:20<00:09,  7.19it/s]
     68%|██████▊   | 135/200 [00:20<00:11,  5.45it/s]
     68%|██████▊   | 137/200 [00:20<00:09,  6.39it/s]
     70%|██████▉   | 139/200 [00:20<00:08,  7.35it/s]
     70%|███████   | 140/200 [00:20<00:11,  5.44it/s]
     71%|███████   | 142/200 [00:21<00:08,  6.48it/s]
     72%|███████▏  | 144/200 [00:21<00:07,  7.58it/s]
     73%|███████▎  | 146/200 [00:21<00:08,  6.70it/s]
     74%|███████▍  | 148/200 [00:21<00:06,  7.74it/s]
     75%|███████▌  | 150/200 [00:22<00:07,  6.72it/s]
     76%|███████▌  | 152/200 [00:22<00:06,  7.66it/s]
     77%|███████▋  | 154/200 [00:22<00:05,  8.42it/s]
     78%|███████▊  | 155/200 [00:22<00:07,  5.75it/s]
     78%|███████▊  | 157/200 [00:23<00:06,  6.70it/s]
     80%|███████▉  | 159/200 [00:23<00:05,  7.75it/s]
     80%|████████  | 160/200 [00:23<00:07,  5.46it/s]
     81%|████████  | 162/200 [00:23<00:05,  6.45it/s]
     82%|████████▏ | 164/200 [00:23<00:04,  7.50it/s]
     82%|████████▎ | 165/200 [00:24<00:06,  5.42it/s]
     84%|████████▎ | 167/200 [00:24<00:05,  6.44it/s]
     84%|████████▍ | 169/200 [00:24<00:04,  7.40it/s]
     85%|████████▌ | 170/200 [00:25<00:07,  4.25it/s]
     86%|████████▌ | 172/200 [00:25<00:05,  5.30it/s]
     87%|████████▋ | 174/200 [00:25<00:04,  6.40it/s]
     88%|████████▊ | 175/200 [00:25<00:04,  5.02it/s]
     88%|████████▊ | 177/200 [00:25<00:03,  6.07it/s]
     90%|████████▉ | 179/200 [00:25<00:02,  7.06it/s]
     90%|█████████ | 180/200 [00:26<00:03,  5.26it/s]
     91%|█████████ | 182/200 [00:26<00:02,  6.26it/s]
     92%|█████████▏| 184/200 [00:26<00:02,  7.10it/s]
     92%|█████████▎| 185/200 [00:26<00:02,  5.36it/s]
     94%|█████████▎| 187/200 [00:27<00:02,  6.39it/s]
     94%|█████████▍| 189/200 [00:27<00:01,  7.30it/s]
     95%|█████████▌| 190/200 [00:27<00:01,  5.12it/s]
     96%|█████████▌| 192/200 [00:27<00:01,  6.09it/s]
     97%|█████████▋| 194/200 [00:28<00:00,  7.08it/s]
     98%|█████████▊| 195/200 [00:28<00:00,  5.36it/s]
     98%|█████████▊| 197/200 [00:28<00:00,  6.48it/s]
    100%|█████████▉| 199/200 [00:28<00:00,  7.43it/s]
    
    100%|██████████| 200/200 [00:28<00:00,  6.92it/s]
      0%|          | 0/200 [00:00<?, ?it/s]
      0%|          | 1/200 [00:00<01:58,  1.67it/s]
      1%|          | 2/200 [00:00<01:31,  2.16it/s]
      2%|▏         | 3/200 [00:00<01:11,  2.76it/s]
      2%|▏         | 4/200 [00:01<00:57,  3.41it/s]
      2%|▎         | 5/200 [00:01<01:07,  2.89it/s]
      3%|▎         | 6/200 [00:01<00:53,  3.61it/s]
      4%|▎         | 7/200 [00:01<00:45,  4.27it/s]
      4%|▍         | 9/200 [00:01<00:37,  5.16it/s]
      5%|▌         | 10/200 [00:02<00:43,  4.39it/s]
      6%|▌         | 12/200 [00:02<00:34,  5.41it/s]
      7%|▋         | 14/200 [00:02<00:28,  6.47it/s]
      8%|▊         | 15/200 [00:03<00:45,  4.06it/s]
      8%|▊         | 16/200 [00:03<00:38,  4.78it/s]
      9%|▉         | 18/200 [00:03<00:31,  5.84it/s]
     10%|█         | 20/200 [00:03<00:36,  4.90it/s]
     11%|█         | 22/200 [00:04<00:30,  5.89it/s]
     12%|█▏        | 23/200 [00:04<00:27,  6.49it/s]
     12%|█▏        | 24/200 [00:04<00:24,  7.11it/s]
     12%|█▎        | 25/200 [00:04<00:32,  5.34it/s]
     13%|█▎        | 26/200 [00:04<00:28,  6.13it/s]
     14%|█▎        | 27/200 [00:04<00:25,  6.92it/s]
     14%|█▍        | 29/200 [00:04<00:22,  7.63it/s]
     15%|█▌        | 30/200 [00:05<00:38,  4.36it/s]
     16%|█▌        | 31/200 [00:05<00:32,  5.15it/s]
     16%|█▋        | 33/200 [00:05<00:27,  6.16it/s]
     17%|█▋        | 34/200 [00:05<00:25,  6.43it/s]
     18%|█▊        | 35/200 [00:06<00:32,  5.08it/s]
     18%|█▊        | 37/200 [00:06<00:27,  6.01it/s]
     20%|█▉        | 39/200 [00:06<00:23,  6.97it/s]
     20%|██        | 40/200 [00:06<00:30,  5.25it/s]
     20%|██        | 41/200 [00:06<00:26,  6.11it/s]
     22%|██▏       | 43/200 [00:07<00:22,  6.99it/s]
     22%|██▎       | 45/200 [00:07<00:24,  6.30it/s]
     23%|██▎       | 46/200 [00:07<00:22,  6.93it/s]
     24%|██▍       | 48/200 [00:07<00:19,  7.85it/s]
     25%|██▌       | 50/200 [00:08<00:22,  6.60it/s]
     26%|██▌       | 52/200 [00:08<00:19,  7.51it/s]
     27%|██▋       | 54/200 [00:08<00:17,  8.47it/s]
     28%|██▊       | 55/200 [00:09<00:32,  4.50it/s]
     28%|██▊       | 57/200 [00:09<00:25,  5.50it/s]
     29%|██▉       | 58/200 [00:09<00:22,  6.29it/s]
     30%|███       | 60/200 [00:09<00:27,  5.15it/s]
     31%|███       | 62/200 [00:10<00:22,  6.19it/s]
     32%|███▏      | 64/200 [00:10<00:18,  7.24it/s]
     32%|███▎      | 65/200 [00:10<00:25,  5.39it/s]
     34%|███▎      | 67/200 [00:10<00:20,  6.42it/s]
     34%|███▍      | 69/200 [00:10<00:17,  7.40it/s]
     35%|███▌      | 70/200 [00:11<00:30,  4.30it/s]
     36%|███▌      | 71/200 [00:11<00:25,  5.15it/s]
     36%|███▋      | 73/200 [00:11<00:20,  6.12it/s]
     38%|███▊      | 75/200 [00:12<00:24,  5.02it/s]
     38%|███▊      | 77/200 [00:12<00:20,  6.01it/s]
     40%|███▉      | 79/200 [00:12<00:17,  7.07it/s]
     40%|████      | 80/200 [00:12<00:22,  5.28it/s]
     41%|████      | 82/200 [00:12<00:18,  6.23it/s]
     42%|████▏     | 83/200 [00:13<00:16,  6.92it/s]
     42%|████▎     | 85/200 [00:13<00:20,  5.49it/s]
     44%|████▎     | 87/200 [00:13<00:17,  6.44it/s]
     44%|████▍     | 89/200 [00:14<00:15,  7.35it/s]
     45%|████▌     | 90/200 [00:14<00:26,  4.22it/s]
     46%|████▌     | 92/200 [00:14<00:20,  5.22it/s]
     47%|████▋     | 94/200 [00:14<00:16,  6.35it/s]
     48%|████▊     | 95/200 [00:15<00:20,  5.02it/s]
     48%|████▊     | 97/200 [00:15<00:17,  6.01it/s]
     50%|████▉     | 99/200 [00:15<00:14,  7.03it/s]
     50%|█████     | 100/200 [00:15<00:18,  5.27it/s]
     51%|█████     | 102/200 [00:15<00:15,  6.26it/s]
     52%|█████▏    | 103/200 [00:16<00:13,  6.96it/s]
     52%|█████▎    | 105/200 [00:16<00:14,  6.34it/s]
     53%|█████▎    | 106/200 [00:16<00:13,  6.96it/s]
     54%|█████▍    | 108/200 [00:16<00:11,  7.90it/s]
     55%|█████▌    | 110/200 [00:17<00:15,  5.96it/s]
     56%|█████▌    | 112/200 [00:17<00:12,  6.99it/s]
     57%|█████▋    | 114/200 [00:17<00:10,  7.84it/s]
     57%|█████▊    | 115/200 [00:18<00:19,  4.32it/s]
     58%|█████▊    | 117/200 [00:18<00:15,  5.35it/s]
     60%|█████▉    | 119/200 [00:18<00:12,  6.39it/s]
     60%|██████    | 120/200 [00:18<00:15,  5.03it/s]
     61%|██████    | 122/200 [00:18<00:12,  6.13it/s]
     62%|██████▏   | 123/200 [00:18<00:11,  6.92it/s]
     62%|██████▎   | 125/200 [00:19<00:11,  6.34it/s]
     64%|██████▎   | 127/200 [00:19<00:10,  7.17it/s]
     64%|██████▍   | 129/200 [00:19<00:08,  8.16it/s]
     65%|██████▌   | 130/200 [00:19<00:12,  5.75it/s]
     66%|██████▌   | 132/200 [00:20<00:10,  6.73it/s]
     67%|██████▋   | 134/200 [00:20<00:08,  7.62it/s]
     68%|██████▊   | 135/200 [00:20<00:11,  5.43it/s]
     68%|██████▊   | 137/200 [00:20<00:09,  6.47it/s]
     70%|██████▉   | 139/200 [00:21<00:08,  7.47it/s]
     70%|███████   | 140/200 [00:21<00:13,  4.32it/s]
     71%|███████   | 142/200 [00:21<00:10,  5.35it/s]
     72%|███████▏  | 144/200 [00:21<00:08,  6.36it/s]
     72%|███████▎  | 145/200 [00:22<00:10,  5.07it/s]
     74%|███████▎  | 147/200 [00:22<00:08,  6.07it/s]
     74%|███████▍  | 149/200 [00:22<00:07,  7.15it/s]
     75%|███████▌  | 150/200 [00:22<00:09,  5.31it/s]
     76%|███████▌  | 152/200 [00:22<00:07,  6.28it/s]
     77%|███████▋  | 154/200 [00:23<00:06,  7.22it/s]
     78%|███████▊  | 155/200 [00:23<00:11,  4.08it/s]
     78%|███████▊  | 157/200 [00:23<00:08,  5.12it/s]
     80%|███████▉  | 159/200 [00:23<00:06,  6.20it/s]
     80%|████████  | 160/200 [00:24<00:08,  4.91it/s]
     81%|████████  | 162/200 [00:24<00:06,  6.00it/s]
     82%|████████▏ | 164/200 [00:24<00:05,  7.11it/s]
     83%|████████▎ | 166/200 [00:24<00:05,  6.36it/s]
     84%|████████▍ | 168/200 [00:25<00:04,  7.36it/s]
     85%|████████▌ | 170/200 [00:25<00:04,  6.62it/s]
     86%|████████▌ | 172/200 [00:25<00:03,  7.54it/s]
     87%|████████▋ | 174/200 [00:25<00:03,  8.43it/s]
     88%|████████▊ | 175/200 [00:26<00:04,  5.84it/s]
     88%|████████▊ | 177/200 [00:26<00:03,  6.80it/s]
     90%|████████▉ | 179/200 [00:26<00:02,  7.73it/s]
     90%|█████████ | 180/200 [00:26<00:03,  5.52it/s]
     91%|█████████ | 182/200 [00:26<00:02,  6.46it/s]
     92%|█████████▏| 184/200 [00:27<00:02,  7.51it/s]
     92%|█████████▎| 185/200 [00:27<00:02,  5.49it/s]
     94%|█████████▎| 187/200 [00:27<00:01,  6.55it/s]
     94%|█████████▍| 189/200 [00:27<00:01,  7.57it/s]
     95%|█████████▌| 190/200 [00:28<00:01,  5.53it/s]
     96%|█████████▌| 192/200 [00:28<00:01,  6.54it/s]
     97%|█████████▋| 194/200 [00:28<00:00,  7.40it/s]
     98%|█████████▊| 195/200 [00:28<00:01,  4.33it/s]
     98%|█████████▊| 197/200 [00:29<00:00,  5.35it/s]
    100%|█████████▉| 199/200 [00:29<00:00,  6.36it/s]
    100%|██████████| 200/200 [00:29<00:00,  6.77it/s]


.. parsed-literal::
    :class: output

    


.. parsed-literal::
    :class: output

    100%|██████████| 200/200 [00:29<00:00,  5.71it/s]
    100%|██████████| 200/200 [00:29<00:00,  6.74it/s]

Above we specify that: the **label** column of our DataFrame contains
the label-values to predict, AutoGluon should run for 60 seconds, each
training run of an individual model (with particular hyperparameters)
should run on 1 GPU, a particular random seed should be used to
facilitate reproducibility, and that trained models should be saved in
the **ag\_sst** folder.

Now you can use ``predictor.evaluate()`` to evaluate the trained model
on some separate test data.

.. code:: python

    dev_score = predictor.evaluate(dev_data, metrics='acc')
    print('Total Time = {}s'.format(predictor.results['total_time']))
    print('Accuracy = {:.2f}%'.format(dev_score['acc'] * 100))


.. parsed-literal::
    :class: output

    /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_15/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)


.. parsed-literal::
    :class: output

    Total Time = 108.48764729499817s
    Accuracy = 82.91%


And you can easily obtain predictions from these models.

.. code:: python

    sentence1 = "it's a charming and often affecting journey." 
    sentence2 = "It's slow, very, very, very slow."
    predictions = predictor.predict({'sentence': [sentence1, sentence2]})
    print('"Sentence":', sentence1, '"Predicted Sentiment":', predictions[0])
    print('"Sentence":', sentence2, '"Predicted Sentiment":', predictions[1])



.. parsed-literal::
    :class: output

    "Sentence": it's a charming and often affecting journey. "Predicted Sentiment": 1
    "Sentence": It's slow, very, very, very slow. "Predicted Sentiment": 0


For classification tasks, you can ask for predicted class-probabilities
instead of predicted classes.

.. code:: python

    probs = predictor.predict_proba({'sentence': [sentence1, sentence2]})
    print('"Sentence":', sentence1, '"Predicted Class-Probabilities":', probs[0])
    print('"Sentence":', sentence2, '"Predicted Class-Probabilities":', probs[1])



.. parsed-literal::
    :class: output

    "Sentence": it's a charming and often affecting journey. "Predicted Class-Probabilities": [2.8325178e-04 9.9971670e-01]
    "Sentence": It's slow, very, very, very slow. "Predicted Class-Probabilities": [0.92579746 0.07420251]


Sentence Similarity
-------------------

Next, let's use AutoGluon to train a model for evaluating how
semantically similar two sentences are. We use the `Semantic Textual
Similarity
Benchmark <http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark>`__
dataset for illustration.

.. code:: python

    train_data = load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sts/train.parquet')[['sentence1', 'sentence2', 'score']]
    dev_data = load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sts/dev.parquet')[['sentence1', 'sentence2', 'score']]
    train_data.head(10)




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>sentence1</th>
          <th>sentence2</th>
          <th>score</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>A plane is taking off.</td>
          <td>An air plane is taking off.</td>
          <td>5.00</td>
        </tr>
        <tr>
          <th>1</th>
          <td>A man is playing a large flute.</td>
          <td>A man is playing a flute.</td>
          <td>3.80</td>
        </tr>
        <tr>
          <th>2</th>
          <td>A man is spreading shreded cheese on a pizza.</td>
          <td>A man is spreading shredded cheese on an uncoo...</td>
          <td>3.80</td>
        </tr>
        <tr>
          <th>3</th>
          <td>Three men are playing chess.</td>
          <td>Two men are playing chess.</td>
          <td>2.60</td>
        </tr>
        <tr>
          <th>4</th>
          <td>A man is playing the cello.</td>
          <td>A man seated is playing the cello.</td>
          <td>4.25</td>
        </tr>
        <tr>
          <th>5</th>
          <td>Some men are fighting.</td>
          <td>Two men are fighting.</td>
          <td>4.25</td>
        </tr>
        <tr>
          <th>6</th>
          <td>A man is smoking.</td>
          <td>A man is skating.</td>
          <td>0.50</td>
        </tr>
        <tr>
          <th>7</th>
          <td>The man is playing the piano.</td>
          <td>The man is playing the guitar.</td>
          <td>1.60</td>
        </tr>
        <tr>
          <th>8</th>
          <td>A man is playing on a guitar and singing.</td>
          <td>A woman is playing an acoustic guitar and sing...</td>
          <td>2.20</td>
        </tr>
        <tr>
          <th>9</th>
          <td>A person is throwing a cat on to the ceiling.</td>
          <td>A person throws a cat on the ceiling.</td>
          <td>5.00</td>
        </tr>
      </tbody>
    </table>
    </div>



In this data, the **score** column contains numerical values (which we'd
like to predict) that are human-annotated similarity scores for each
given pair of sentences.

.. code:: python

    print('Min score=', min(train_data['score']), ', Max score=', max(train_data['score']))


.. parsed-literal::
    :class: output

    Min score= 0.0 , Max score= 5.0


.. parsed-literal::
    :class: output

    /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_15/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
      and should_run_async(code)


Let's train a regression model to predict these scores with
``task.fit()``. Note that we only need to specify the label column and
AutoGluon automatically determines the type of prediction problem and an
appropriate loss function. Once again, you should increase the short
``time_limits`` below to obtain reasonable performance in your own
applications.

.. code:: python

    predictor_sts = task.fit(train_data, label='score',
                             time_limits='1min', ngpus_per_trial=1, seed=123,
                             output_directory='./ag_sts')


.. parsed-literal::
    :class: output

    2020-12-08 20:19:01,791 - root - INFO - All Logs will be saved to ./ag_sts/ag_text_prediction.log
    2020-12-08 20:19:01,810 - root - INFO - Train Dataset:
    2020-12-08 20:19:01,810 - root - INFO - Columns:
    
    - Text(
       name="sentence1"
       #total/missing=4599/0
       length, min/avg/max=16/57.68/367
    )
    - Text(
       name="sentence2"
       #total/missing=4599/0
       length, min/avg/max=15/57.47/265
    )
    - Numerical(
       name="score"
       #total/missing=4599/0
       shape=()
    )
    
    
    2020-12-08 20:19:01,811 - root - INFO - Tuning Dataset:
    2020-12-08 20:19:01,811 - root - INFO - Columns:
    
    - Text(
       name="sentence1"
       #total/missing=1150/0
       length, min/avg/max=17/57.81/315
    )
    - Text(
       name="sentence2"
       #total/missing=1150/0
       length, min/avg/max=16/57.79/311
    )
    - Numerical(
       name="score"
       #total/missing=1150/0
       shape=()
    )
    
    
    2020-12-08 20:19:01,811 - root - INFO - Label columns=['score'], Feature columns=['sentence1', 'sentence2'], Problem types=['regression'], Label shapes=[()]
    2020-12-08 20:19:01,812 - root - INFO - Eval Metric=mse, Stop Metric=mse, Log Metrics=['mse', 'rmse', 'mae']



.. parsed-literal::
    :class: output

    HBox(children=(HTML(value=''), FloatProgress(value=0.0, max=4.0), HTML(value='')))


.. parsed-literal::
    :class: output

    
      0%|          | 0/576 [00:00<?, ?it/s]
      0%|          | 1/576 [00:00<05:50,  1.64it/s]
      0%|          | 2/576 [00:00<04:37,  2.07it/s]
      1%|          | 3/576 [00:00<03:42,  2.57it/s]
      1%|          | 4/576 [00:01<03:01,  3.15it/s]
      1%|          | 5/576 [00:01<02:32,  3.73it/s]
      1%|          | 6/576 [00:01<02:18,  4.11it/s]
      1%|          | 7/576 [00:01<01:54,  4.95it/s]
      1%|▏         | 8/576 [00:01<01:39,  5.70it/s]
      2%|▏         | 9/576 [00:01<01:27,  6.50it/s]
      2%|▏         | 10/576 [00:01<01:23,  6.81it/s]
      2%|▏         | 11/576 [00:02<01:24,  6.72it/s]
      2%|▏         | 12/576 [00:02<01:27,  6.46it/s]
      2%|▏         | 13/576 [00:02<01:24,  6.63it/s]
      2%|▏         | 14/576 [00:02<01:22,  6.80it/s]
      3%|▎         | 15/576 [00:03<04:33,  2.05it/s]
      3%|▎         | 16/576 [00:03<03:28,  2.68it/s]
      3%|▎         | 17/576 [00:04<02:45,  3.39it/s]
      3%|▎         | 18/576 [00:04<02:16,  4.09it/s]
      3%|▎         | 19/576 [00:04<01:58,  4.71it/s]
      3%|▎         | 20/576 [00:04<01:45,  5.26it/s]
      4%|▎         | 21/576 [00:04<01:35,  5.81it/s]
      4%|▍         | 22/576 [00:04<01:38,  5.64it/s]
      4%|▍         | 23/576 [00:04<01:31,  6.06it/s]
      4%|▍         | 24/576 [00:04<01:21,  6.75it/s]
      4%|▍         | 25/576 [00:05<01:14,  7.38it/s]
      5%|▍         | 26/576 [00:05<01:12,  7.62it/s]
      5%|▍         | 28/576 [00:05<01:06,  8.20it/s]
      5%|▌         | 29/576 [00:05<01:06,  8.29it/s]
      5%|▌         | 30/576 [00:06<04:14,  2.14it/s]
      5%|▌         | 31/576 [00:06<03:21,  2.70it/s]
      6%|▌         | 32/576 [00:07<02:40,  3.39it/s]
      6%|▌         | 33/576 [00:07<02:11,  4.12it/s]
      6%|▌         | 34/576 [00:07<01:57,  4.63it/s]
      6%|▋         | 36/576 [00:07<01:39,  5.41it/s]
      6%|▋         | 37/576 [00:07<01:34,  5.68it/s]
      7%|▋         | 38/576 [00:07<01:26,  6.25it/s]
      7%|▋         | 39/576 [00:07<01:18,  6.80it/s]
      7%|▋         | 40/576 [00:08<01:12,  7.39it/s]
      7%|▋         | 41/576 [00:08<01:13,  7.30it/s]
      7%|▋         | 42/576 [00:08<01:09,  7.72it/s]
      7%|▋         | 43/576 [00:08<01:06,  8.04it/s]
      8%|▊         | 44/576 [00:08<01:09,  7.68it/s]
      8%|▊         | 45/576 [00:09<04:19,  2.04it/s]
      8%|▊         | 46/576 [00:10<03:19,  2.65it/s]
      8%|▊         | 47/576 [00:10<02:36,  3.38it/s]
      8%|▊         | 48/576 [00:10<02:09,  4.09it/s]
      9%|▊         | 49/576 [00:10<01:56,  4.51it/s]
      9%|▊         | 50/576 [00:10<01:39,  5.28it/s]
      9%|▉         | 51/576 [00:10<01:27,  6.03it/s]
      9%|▉         | 52/576 [00:10<01:17,  6.77it/s]
      9%|▉         | 53/576 [00:10<01:11,  7.32it/s]
     10%|▉         | 55/576 [00:11<01:05,  7.99it/s]
     10%|▉         | 56/576 [00:11<01:03,  8.19it/s]
     10%|▉         | 57/576 [00:11<01:30,  5.71it/s]
     10%|█         | 58/576 [00:11<01:19,  6.49it/s]
     10%|█         | 59/576 [00:11<01:12,  7.10it/s]
     10%|█         | 60/576 [00:12<03:47,  2.26it/s]
     11%|█         | 61/576 [00:12<03:00,  2.85it/s]
     11%|█         | 62/576 [00:13<02:21,  3.63it/s]
     11%|█         | 63/576 [00:13<01:55,  4.46it/s]
     11%|█         | 64/576 [00:13<01:39,  5.16it/s]
     11%|█▏        | 65/576 [00:13<01:26,  5.88it/s]
     11%|█▏        | 66/576 [00:13<01:18,  6.46it/s]
     12%|█▏        | 67/576 [00:13<01:12,  7.04it/s]
     12%|█▏        | 68/576 [00:13<01:05,  7.71it/s]
     12%|█▏        | 69/576 [00:13<01:01,  8.25it/s]
     12%|█▏        | 70/576 [00:13<00:59,  8.53it/s]
     12%|█▏        | 71/576 [00:14<01:06,  7.60it/s]
     12%|█▎        | 72/576 [00:14<01:03,  7.91it/s]
     13%|█▎        | 73/576 [00:14<01:01,  8.19it/s]
     13%|█▎        | 74/576 [00:14<01:00,  8.36it/s]
     13%|█▎        | 75/576 [00:15<03:57,  2.11it/s]
     13%|█▎        | 76/576 [00:15<03:04,  2.71it/s]
     13%|█▎        | 77/576 [00:15<02:25,  3.44it/s]
     14%|█▎        | 78/576 [00:16<01:57,  4.22it/s]
     14%|█▎        | 79/576 [00:16<01:39,  4.99it/s]
     14%|█▍        | 80/576 [00:16<01:27,  5.68it/s]
     14%|█▍        | 81/576 [00:16<01:17,  6.36it/s]
     14%|█▍        | 82/576 [00:16<01:10,  7.04it/s]
     14%|█▍        | 83/576 [00:16<01:05,  7.54it/s]
     15%|█▍        | 84/576 [00:16<01:01,  7.99it/s]
     15%|█▍        | 86/576 [00:16<00:57,  8.49it/s]
     15%|█▌        | 87/576 [00:17<00:57,  8.56it/s]
     15%|█▌        | 89/576 [00:17<00:54,  9.01it/s]
     16%|█▌        | 90/576 [00:18<03:24,  2.38it/s]
     16%|█▌        | 91/576 [00:18<02:38,  3.06it/s]
     16%|█▌        | 92/576 [00:18<02:07,  3.79it/s]
     16%|█▌        | 93/576 [00:18<01:45,  4.59it/s]
     16%|█▋        | 94/576 [00:18<01:28,  5.47it/s]
     16%|█▋        | 95/576 [00:18<01:17,  6.18it/s]
     17%|█▋        | 96/576 [00:19<01:10,  6.81it/s]
     17%|█▋        | 98/576 [00:19<01:04,  7.46it/s]
     17%|█▋        | 99/576 [00:19<01:01,  7.70it/s]
     17%|█▋        | 100/576 [00:19<00:57,  8.26it/s]
     18%|█▊        | 101/576 [00:19<00:55,  8.53it/s]
     18%|█▊        | 102/576 [00:19<00:55,  8.62it/s]
     18%|█▊        | 103/576 [00:19<00:55,  8.55it/s]
     18%|█▊        | 104/576 [00:19<00:54,  8.73it/s]
     18%|█▊        | 105/576 [00:21<03:20,  2.35it/s]
     18%|█▊        | 106/576 [00:21<02:36,  3.01it/s]
     19%|█▊        | 107/576 [00:21<02:05,  3.72it/s]
     19%|█▉        | 108/576 [00:21<01:44,  4.48it/s]
     19%|█▉        | 109/576 [00:21<01:28,  5.26it/s]
     19%|█▉        | 110/576 [00:21<01:17,  5.98it/s]
     19%|█▉        | 111/576 [00:21<01:10,  6.58it/s]
     19%|█▉        | 112/576 [00:21<01:05,  7.06it/s]
     20%|█▉        | 113/576 [00:22<01:02,  7.42it/s]
     20%|█▉        | 114/576 [00:22<00:58,  7.87it/s]
     20%|█▉        | 115/576 [00:22<00:57,  7.96it/s]
     20%|██        | 116/576 [00:22<00:55,  8.27it/s]
     20%|██        | 118/576 [00:22<00:52,  8.71it/s]
     21%|██        | 120/576 [00:24<02:13,  3.41it/s]
     21%|██        | 121/576 [00:24<01:49,  4.16it/s]
     21%|██        | 122/576 [00:24<01:31,  4.94it/s]
     21%|██▏       | 123/576 [00:24<01:20,  5.62it/s]
     22%|██▏       | 124/576 [00:24<01:11,  6.31it/s]
     22%|██▏       | 125/576 [00:24<01:05,  6.92it/s]
     22%|██▏       | 127/576 [00:24<00:58,  7.71it/s]
     22%|██▏       | 128/576 [00:24<01:00,  7.36it/s]
     22%|██▏       | 129/576 [00:25<00:57,  7.79it/s]
     23%|██▎       | 130/576 [00:25<01:02,  7.17it/s]
     23%|██▎       | 131/576 [00:25<00:56,  7.83it/s]
     23%|██▎       | 132/576 [00:25<00:54,  8.15it/s]
     23%|██▎       | 134/576 [00:25<00:51,  8.63it/s]
     23%|██▎       | 135/576 [00:26<03:09,  2.33it/s]
     24%|██▎       | 136/576 [00:26<02:27,  2.99it/s]
     24%|██▍       | 138/576 [00:27<01:56,  3.76it/s]
     24%|██▍       | 139/576 [00:27<01:35,  4.60it/s]
     24%|██▍       | 140/576 [00:27<01:21,  5.37it/s]
     24%|██▍       | 141/576 [00:27<01:10,  6.19it/s]
     25%|██▍       | 142/576 [00:27<01:05,  6.65it/s]
     25%|██▍       | 143/576 [00:27<01:00,  7.16it/s]
     25%|██▌       | 144/576 [00:27<00:58,  7.40it/s]
     25%|██▌       | 145/576 [00:27<00:55,  7.78it/s]
     25%|██▌       | 146/576 [00:28<00:53,  8.03it/s]
     26%|██▌       | 147/576 [00:28<00:51,  8.28it/s]
     26%|██▌       | 148/576 [00:28<00:51,  8.38it/s]
     26%|██▌       | 149/576 [00:28<00:51,  8.26it/s]
     26%|██▌       | 150/576 [00:29<03:28,  2.04it/s]
     26%|██▌       | 151/576 [00:29<02:41,  2.64it/s]
     26%|██▋       | 152/576 [00:29<02:05,  3.38it/s]
     27%|██▋       | 153/576 [00:30<01:41,  4.19it/s]
     27%|██▋       | 154/576 [00:30<01:23,  5.03it/s]
     27%|██▋       | 155/576 [00:30<01:13,  5.73it/s]
     27%|██▋       | 156/576 [00:30<01:06,  6.35it/s]
     27%|██▋       | 157/576 [00:30<00:59,  6.99it/s]
     27%|██▋       | 158/576 [00:30<00:54,  7.61it/s]
     28%|██▊       | 159/576 [00:30<00:53,  7.73it/s]
     28%|██▊       | 160/576 [00:30<00:51,  8.08it/s]
     28%|██▊       | 161/576 [00:30<00:51,  8.00it/s]
     28%|██▊       | 162/576 [00:31<00:49,  8.44it/s]
     28%|██▊       | 163/576 [00:31<00:47,  8.65it/s]
     28%|██▊       | 164/576 [00:31<00:46,  8.93it/s]
     29%|██▊       | 165/576 [00:32<03:18,  2.07it/s]
     29%|██▉       | 166/576 [00:32<02:31,  2.71it/s]
     29%|██▉       | 168/576 [00:32<01:59,  3.42it/s]
     30%|██▉       | 170/576 [00:33<01:35,  4.26it/s]
     30%|██▉       | 171/576 [00:33<01:21,  4.99it/s]
     30%|██▉       | 172/576 [00:33<01:13,  5.53it/s]
     30%|███       | 173/576 [00:33<01:04,  6.22it/s]
     30%|███       | 174/576 [00:33<00:59,  6.72it/s]
     30%|███       | 175/576 [00:33<00:56,  7.09it/s]
     31%|███       | 176/576 [00:33<00:54,  7.35it/s]
     31%|███       | 177/576 [00:34<00:52,  7.66it/s]
     31%|███       | 178/576 [00:34<00:49,  8.10it/s]
     31%|███       | 179/576 [00:34<00:48,  8.20it/s]
     31%|███▏      | 180/576 [00:35<03:12,  2.06it/s]
     31%|███▏      | 181/576 [00:35<02:29,  2.65it/s]
     32%|███▏      | 183/576 [00:35<01:55,  3.40it/s]
     32%|███▏      | 184/576 [00:36<01:33,  4.20it/s]
     32%|███▏      | 186/576 [00:36<01:16,  5.08it/s]
     32%|███▏      | 187/576 [00:36<01:05,  5.91it/s]
     33%|███▎      | 188/576 [00:36<00:58,  6.65it/s]
     33%|███▎      | 189/576 [00:36<00:53,  7.30it/s]
     33%|███▎      | 190/576 [00:36<00:50,  7.69it/s]
     33%|███▎      | 191/576 [00:36<00:48,  7.89it/s]
     33%|███▎      | 192/576 [00:36<00:46,  8.31it/s]
     34%|███▎      | 193/576 [00:36<00:45,  8.46it/s]
     34%|███▎      | 194/576 [00:37<00:46,  8.13it/s]
     34%|███▍      | 195/576 [00:38<03:09,  2.01it/s]
     34%|███▍      | 196/576 [00:38<02:25,  2.61it/s]
     34%|███▍      | 197/576 [00:38<01:53,  3.35it/s]
     35%|███▍      | 199/576 [00:38<01:30,  4.17it/s]
     35%|███▍      | 201/576 [00:39<01:15,  5.00it/s]
     35%|███▌      | 202/576 [00:39<01:03,  5.85it/s]
     35%|███▌      | 203/576 [00:39<00:59,  6.30it/s]
     36%|███▌      | 205/576 [00:39<00:52,  7.02it/s]
     36%|███▌      | 206/576 [00:39<00:49,  7.49it/s]
     36%|███▌      | 207/576 [00:39<00:47,  7.80it/s]
     36%|███▌      | 208/576 [00:39<00:44,  8.26it/s]
     36%|███▋      | 209/576 [00:39<00:42,  8.61it/s]
     36%|███▋      | 210/576 [00:41<02:38,  2.31it/s]
     37%|███▋      | 211/576 [00:41<02:03,  2.96it/s]
     37%|███▋      | 212/576 [00:41<01:38,  3.71it/s]
     37%|███▋      | 213/576 [00:41<01:23,  4.37it/s]
     37%|███▋      | 214/576 [00:41<01:09,  5.24it/s]
     37%|███▋      | 215/576 [00:41<00:59,  6.03it/s]
     38%|███▊      | 216/576 [00:41<00:52,  6.79it/s]
     38%|███▊      | 217/576 [00:41<00:49,  7.24it/s]
     38%|███▊      | 218/576 [00:42<00:47,  7.54it/s]
     38%|███▊      | 219/576 [00:42<00:47,  7.57it/s]
     38%|███▊      | 220/576 [00:42<00:46,  7.59it/s]
     38%|███▊      | 221/576 [00:42<00:45,  7.81it/s]
     39%|███▊      | 222/576 [00:42<00:43,  8.12it/s]
     39%|███▉      | 224/576 [00:42<00:41,  8.54it/s]
     39%|███▉      | 225/576 [00:44<02:49,  2.07it/s]
     39%|███▉      | 226/576 [00:44<02:10,  2.69it/s]
     39%|███▉      | 227/576 [00:44<01:41,  3.44it/s]
     40%|███▉      | 229/576 [00:44<01:21,  4.27it/s]
     40%|███▉      | 230/576 [00:44<01:08,  5.06it/s]
     40%|████      | 231/576 [00:44<00:58,  5.90it/s]
     40%|████      | 232/576 [00:44<00:53,  6.45it/s]
     40%|████      | 233/576 [00:44<00:49,  6.89it/s]
     41%|████      | 234/576 [00:45<00:48,  7.10it/s]
     41%|████      | 235/576 [00:45<00:43,  7.76it/s]
     41%|████      | 236/576 [00:45<00:41,  8.17it/s]
     41%|████      | 237/576 [00:45<00:40,  8.36it/s]
     41%|████▏     | 239/576 [00:45<00:38,  8.71it/s]
     42%|████▏     | 240/576 [00:46<02:24,  2.33it/s]
     42%|████▏     | 241/576 [00:46<01:56,  2.88it/s]
     42%|████▏     | 242/576 [00:47<01:31,  3.65it/s]
     42%|████▏     | 243/576 [00:47<01:14,  4.47it/s]
     42%|████▏     | 244/576 [00:47<01:03,  5.20it/s]
     43%|████▎     | 245/576 [00:47<00:55,  5.95it/s]
     43%|████▎     | 246/576 [00:47<00:49,  6.62it/s]
     43%|████▎     | 247/576 [00:47<00:45,  7.25it/s]
     43%|████▎     | 248/576 [00:47<00:42,  7.64it/s]
     43%|████▎     | 249/576 [00:47<00:41,  7.92it/s]
     44%|████▎     | 251/576 [00:48<00:37,  8.56it/s]
     44%|████▍     | 252/576 [00:48<00:38,  8.45it/s]
     44%|████▍     | 253/576 [00:48<00:37,  8.64it/s]
     44%|████▍     | 254/576 [00:48<00:37,  8.67it/s]
     44%|████▍     | 255/576 [00:49<02:18,  2.31it/s]
     44%|████▍     | 256/576 [00:49<01:48,  2.95it/s]
     45%|████▍     | 257/576 [00:49<01:26,  3.71it/s]
     45%|████▍     | 258/576 [00:49<01:09,  4.56it/s]
     45%|████▍     | 259/576 [00:50<00:59,  5.34it/s]
     45%|████▌     | 260/576 [00:50<00:52,  6.02it/s]
     45%|████▌     | 261/576 [00:50<00:46,  6.74it/s]
     45%|████▌     | 262/576 [00:50<00:43,  7.27it/s]
     46%|████▌     | 263/576 [00:50<00:41,  7.51it/s]
     46%|████▌     | 264/576 [00:50<00:39,  7.97it/s]
     46%|████▌     | 265/576 [00:50<00:39,  7.95it/s]
     46%|████▋     | 267/576 [00:50<00:36,  8.39it/s]
     47%|████▋     | 268/576 [00:51<00:34,  8.80it/s]
     47%|████▋     | 269/576 [00:51<00:33,  9.09it/s]
     47%|████▋     | 270/576 [00:52<02:11,  2.32it/s]
     47%|████▋     | 271/576 [00:52<01:41,  3.00it/s]
     47%|████▋     | 272/576 [00:52<01:22,  3.70it/s]
     47%|████▋     | 273/576 [00:52<01:08,  4.40it/s]
     48%|████▊     | 274/576 [00:52<00:57,  5.25it/s]
     48%|████▊     | 275/576 [00:52<00:49,  6.02it/s]
     48%|████▊     | 276/576 [00:52<00:44,  6.68it/s]
     48%|████▊     | 278/576 [00:53<00:40,  7.32it/s]
     48%|████▊     | 279/576 [00:53<00:38,  7.66it/s]
     49%|████▊     | 280/576 [00:53<00:36,  8.12it/s]
     49%|████▉     | 281/576 [00:53<00:35,  8.42it/s]
     49%|████▉     | 282/576 [00:53<00:34,  8.53it/s]
     49%|████▉     | 283/576 [00:53<00:32,  8.92it/s]
     49%|████▉     | 285/576 [00:55<01:19,  3.68it/s]
     50%|████▉     | 286/576 [00:55<01:05,  4.44it/s]
     50%|████▉     | 287/576 [00:55<00:56,  5.16it/s]
     50%|█████     | 288/576 [00:55<00:49,  5.80it/s]
     50%|█████     | 289/576 [00:55<00:44,  6.51it/s]
     50%|█████     | 290/576 [00:55<00:41,  6.89it/s]
     51%|█████     | 291/576 [00:55<00:38,  7.33it/s]
     51%|█████     | 292/576 [00:55<00:36,  7.81it/s]
     51%|█████     | 293/576 [00:55<00:34,  8.31it/s]
     51%|█████     | 294/576 [00:56<00:32,  8.55it/s]
     51%|█████▏    | 296/576 [00:56<00:31,  8.85it/s]
     52%|█████▏    | 297/576 [00:56<00:30,  9.01it/s]
     52%|█████▏    | 298/576 [00:56<00:32,  8.64it/s]
     52%|█████▏    | 299/576 [00:56<00:32,  8.64it/s]
     52%|█████▏    | 300/576 [00:57<01:59,  2.30it/s]
     52%|█████▏    | 302/576 [00:57<01:31,  2.99it/s]
     53%|█████▎    | 304/576 [00:58<01:11,  3.80it/s]
     53%|█████▎    | 305/576 [00:58<00:59,  4.58it/s]
     53%|█████▎    | 306/576 [00:58<00:51,  5.29it/s]
     53%|█████▎    | 307/576 [00:58<00:44,  6.06it/s]
     53%|█████▎    | 308/576 [00:58<00:40,  6.57it/s]
     54%|█████▎    | 309/576 [00:58<00:37,  7.08it/s]
     54%|█████▍    | 310/576 [00:58<00:36,  7.36it/s]
     54%|█████▍    | 311/576 [00:58<00:34,  7.74it/s]
     54%|█████▍    | 313/576 [00:59<00:31,  8.34it/s]
     55%|█████▍    | 314/576 [01:00<00:50,  5.19it/s]


.. parsed-literal::
    :class: output

    


.. parsed-literal::
    :class: output

     55%|█████▍    | 314/576 [01:02<00:51,  5.04it/s]


We again evaluate our trained model's performance on some separate test
data. Below we choose to compute the following metrics: RMSE, Pearson
Correlation, and Spearman Correlation.

.. code:: python

    dev_score = predictor_sts.evaluate(dev_data, metrics=['rmse', 'pearsonr', 'spearmanr'])
    print('Best Config = {}'.format(predictor_sts.results['best_config']))
    print('Total Time = {}s'.format(predictor_sts.results['total_time']))
    print('RMSE = {:.2f}'.format(dev_score['rmse']))
    print('PEARSONR = {:.4f}'.format(dev_score['pearsonr']))
    print('SPEARMANR = {:.4f}'.format(dev_score['spearmanr']))


.. parsed-literal::
    :class: output

    Best Config = {'search_space▁optimization.lr': 5.5e-05}
    Total Time = 138.23879075050354s
    RMSE = 0.81
    PEARSONR = 0.8440
    SPEARMANR = 0.8467


Let's use our model to predict the similarity score among these
sentences:

-  'The child is riding a horse.'
-  'The young boy is riding a horse.'
-  'The young man is riding a horse.'
-  'The young man is riding a bicycle.'

.. code:: python

    sentences = ['The child is riding a horse.',
                 'The young boy is riding a horse.',
                 'The young man is riding a horse.',
                 'The young man is riding a bicycle.']
    
    score1 = predictor_sts.predict({'sentence1': [sentences[0]],
                                    'sentence2': [sentences[1]]})
    
    score2 = predictor_sts.predict({'sentence1': [sentences[0]],
                                    'sentence2': [sentences[2]]})
    
    score3 = predictor_sts.predict({'sentence1': [sentences[0]],
                                    'sentence2': [sentences[3]]})
    print(score1, score2, score3)


.. parsed-literal::
    :class: output

    [3.382951] [2.845128] [2.061431]


Save and Load
-------------

Finally we demonstrate how to easily save and load a trained
TextPrediction model.

.. code:: python

    predictor_sts.save('saved_dir')
    predictor_sts_new = task.load('saved_dir')
    
    score3 = predictor_sts_new.predict({'sentence1': [sentences[0]],
                                        'sentence2': [sentences[3]]})
    print(score3)


.. parsed-literal::
    :class: output

    [2.061431]


**Note:** ``TextPrediction`` depends on the
`GluonNLP <https://gluon-nlp.mxnet.io/>`__ package. Due to an ongoing
upgrade of GluonNLP, we are currently using a custom version of the
package:
`autogluon-contrib-nlp <https://github.com/sxjscience/autogluon-contrib-nlp.git>`__.
In a future release, AutoGluon will switch to using the official
GluonNLP, but the APIs demonstrated here will remain the same.