FAQ

How can I get the most accurate predictions?

See “Maximizing predictive performance” in the Quick Start Tutorial.

Can I run AutoGluon Tabular on Mac/Windows?

Yes! The only functionality that may not work is hyperparameter tuning with the NN model (this should be resolved in the next MXNet update).

What machine is best for running AutoGluon Tabular?

As an open-source library, AutoGluon can be run on any machine including your laptop. Currently TabularPredictor does not benefit much from GPUs, so CPU machines are fine (in contrast, TextPredictor/ImagePredictor/ObjectDetector do greatly benefit from GPUs). Most Tabular issues arise due to lack of memory, so we recommend running on a machine with as much memory as possible. For example if using AWS instances for Tabular: we recommend M5 instances, where a m5.24xlarge machine should be able to handle most datasets.

How can I reduce the time required for training?

Specify the time_limit argument in fit() to the number of seconds you are willing to wait (longer time limits generally result in superior predictive performance). You may also try other settings of the presets argument in fit(), and can also subsample your data for a quick trial run via train_data.sample(n=SUBSAMPLE_SIZE). If a particular type of model is taking much longer to train on your data than the other types of models, you can tell AutoGluon not to train any models of this particular type by specifying its short-name in the excluded_model_types argument of fit().

Since many of the strategies to reduce memory usage also reduce training times, also check out: “If you encounter memory issues” in the In Depth Tutorial.

How can I reduce the time required for prediction?

See “Accelerating inference” in the In Depth Tutorial.

How does AutoGluon Tabular work internally?

Details are provided in the following paper:

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data. Arxiv, 2020.

How to view more detailed logs of what is happening during fit?

Specify the argument verbosity = 4 in fit().

What model is AutoGluon using for prediction?

See “Prediction options” in the In Depth Tutorial.

Which classes do predicted probabilities correspond to?

This should become obvious if you look at the pandas DataFrame column names after specifying the as_pandas argument like this:

predictor.predict_proba(test_data, as_pandas=True)

For multiclass classification:

predictor.class_labels

is a list of classes whose order corresponds to columns of predict_proba() output when it is a Numpy array. For binary classification, this is the order of classes when predict_proba(as_multiclass=True) is called.

You can see which class AutoGluon treats as the positive class in binary classification via:

predictor.positive_class

The positive class can also be retrieved via predictor.class_labels[-1]. The output of predict_proba(as_multiclass=False) for binary classification is the probability of the positive class.

How can I use AutoGluon for interpretability?

See “Interpretability (feature importance)” in the In Depth Tutorial, which allows you to quantify how much each feature contributes to AutoGluon’s predictive accuracy.

Additionally, you can explain particular AutoGluon predictions using Shapely values. Notebooks demonstrating this are provided at: https://github.com/awslabs/autogluon/tree/master/examples/tabular/interpret. Handling of multiclass classification tasks and data with categorical features are demonstrated in the notebook “SHAP with AutoGluon-Tabular and Categorical Features” contained in this folder.

How can I perform inference on a file that won’t fit in memory?

The Tabular Dataset API works with pandas Dataframes, which supports chunking data into sizes that fit in memory. Here’s an example of one such chunk-based inference:

from autogluon.tabular import TabularDataset, TabularPredictor
import pandas as pd
import requests

train_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')
predictor = TabularPredictor(label='class').fit(train_data.sample(n=100, random_state=0), hyperparameters={'GBM': {}})

# Get the test dataset, if you are working with local data then omit the next two lines
r = requests.get('https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv', allow_redirects=True)
open('test.csv', 'wb').write(r.content)
reader = pd.read_csv('test.csv', chunksize=1024)
y_pred = []
y_true = []
for df_chunk in reader:
    y_pred.append(predictor.predict(df_chunk))
    y_true.append(df_chunk['class'])
y_pred = pd.concat(y_pred, axis=0, ignore_index=True)
y_true = pd.concat(y_true, axis=0, ignore_index=True)
predictor.evaluate_predictions(y_true=y_true, y_pred=y_pred)
No path specified. Models will be saved in: "AutogluonModels/ag-20210429_010445/"
Beginning AutoGluon training ...
AutoGluon will save models to "AutogluonModels/ag-20210429_010445/"
AutoGluon Version:  0.2.0b20210429
Train Data Rows:    100
Train Data Columns: 14
Preprocessing data ...
AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).
    2 unique label values:  [' >50K', ' <=50K']
    If 'binary' is not the correct problem_type, please manually specify the problem_type argument in fit() (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])
Selected class <--> label mapping:  class 1 =  >50K, class 0 =  <=50K
    Note: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive ( >50K) vs negative ( <=50K) class.
    To explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init.
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
    Available Memory:                    27027.9 MB
    Train Data (Original)  Memory Usage: 0.06 MB (0.0% of available memory)
    Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
    Stage 1 Generators:
            Fitting AsTypeFeatureGenerator...
    Stage 2 Generators:
            Fitting FillNaFeatureGenerator...
    Stage 3 Generators:
            Fitting IdentityFeatureGenerator...
            Fitting CategoryFeatureGenerator...
                    Fitting CategoryMemoryMinimizeFeatureGenerator...
    Stage 4 Generators:
            Fitting DropUniqueFeatureGenerator...
    Types of features in original data (raw dtype, special dtypes):
            ('int', [])    : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
            ('object', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]
    Types of features in processed data (raw dtype, special dtypes):
            ('category', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]
            ('int', [])      : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
    0.1s = Fit runtime
    14 features in original data used to generate 14 features in processed data.
    Train Data (Processed) Memory Usage: 0.01 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.07s ...
AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'
    To change this, specify the eval_metric argument of fit()
Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 80, Val Rows: 20
Fitting model: LightGBM ...
/var/lib/jenkins/workspace/workspace/autogluon-tutorial-tabular-v3/venv/lib/python3.7/site-packages/fsspec/__init__.py:47: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  for spec in entry_points.get("fsspec.specs", []):
    0.8      = Validation accuracy score
    0.74s    = Training runtime
    0.01s    = Validation runtime
Fitting model: WeightedEnsemble_L2 ...
    0.8      = Validation accuracy score
    0.0s     = Training runtime
    0.0s     = Validation runtime
AutoGluon training complete, total runtime = 0.86s ...
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20210429_010445/")
Evaluation: accuracy on test data: 0.7628211690039922
Evaluations on test data:
{
    "accuracy": 0.7628211690039922,
    "balanced_accuracy": 0.515967102411626,
    "mcc": 0.09302340489249436,
    "f1": 0.08455156064796523,
    "precision": 0.5023474178403756,
    "recall": 0.04616048317515099
}
{'accuracy': 0.7628211690039922,
 'balanced_accuracy': 0.515967102411626,
 'mcc': 0.09302340489249436,
 'f1': 0.08455156064796523,
 'precision': 0.5023474178403756,
 'recall': 0.04616048317515099}

Here we split the test data into chunks of up to 1024 rows each, but you may select a larger size as long as it fits into your system’s memory. Further Reading

How can I skip some particular models?

To avoid training certain models, specify these in the excluded_model_types argument. For example, here’s how to call fit() without training K Nearest Neighbor (KNN), Random Forest (RF), or ExtraTrees (XT) models:

task.fit(..., excluded_model_types=['KNN','RF','XT'])

How can I add my own custom model to the set of models that AutoGluon trains, tunes, and ensembles?

See this example in the source code: examples/tabular/example_custom_model_tabular.py

How can I add my own custom data preprocessing or feature engineering?

Note that the TabularDataset object is essentially a pandas DataFrame and you can transform your training data however you wish before calling fit(). Note that any transformations you perform yourself must also be applied to all future test data before calling predict(), and AutoGluon will still perform its default processing on your transformed data inside fit().

To solely use custom data preprocessing and automatically apply your custom transformations to both the train data and all future data encountered during inference, you should instead create a custom FeatureGenerator. Follow this example in the source code: examples/tabular/example_custom_feature_generator.py

How can I differently weight the importance of training examples?

You can specify the sample_weight and weight_evaluation arguments when initializing a TabularPredictor.

I’m receiving C++ warning spam during training or inference

Warning message: [W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)

This can happen from downstream PyTorch dependencies (OpenMP) when using a specific environment. If you are using PyTorch 1.7, Mac OS X, Python 3.6/3.7, and using the PyTorch DataLoader, then you may get this warning spam. We have only seen this occur with the TabTransformer model. Reference open torch issue.

The recommended workaround from the torch issue to suppress this warning is to set an environment variable used in OpenMP. This has been tested on Torch 1.7, Python 3.6, and Mac OS X.

export OMP_NUM_THREADS=1

Issues not addressed here

First search if your issue is addressed in the tutorials, examples, documentation, or Github issues. If it is not there, please open a new Github Issue and clearly state your issue. If you have a bug, please include: your code (call fit(..., verbosity=4) which will print more details), the output printed during the code execution, and information about your operating system, Python version, and installed packages (output of pip freeze).