FAQ¶

How can I get the most accurate predictions?¶

See “Maximizing predictive performance” in the Quick Start Tutorial.

Can I run AutoGluon Tabular on Mac/Windows?¶

Yes!

Can I use GPUs for model training?¶

Yes! Most of the models used by AutoGluon support GPU training, including LightGBM, CatBoost, XGBoost, MXNet Neural Network, and FastAI Neural Network.

To enable GPU training, specify in predictor.fit the argument ag_args_fit={'num_gpus': 1}. This will enable GPU training for all models that support it. Multi-GPU training is still experimental.

For most of these models, CUDA will have to be installed and some models may need special installations such as LightGBM and MXNet to be compatible with GPU training. Refer to installation instructions for more details.

What machine is best for running AutoGluon Tabular?¶

As an open-source library, AutoGluon can be run on any machine including your laptop. Currently it is not necessary to use a GPU to train TabularPredictor so CPU machines are fine (in contrast, TextPredictor/ImagePredictor/ObjectDetector require GPUs). Most Tabular issues arise due to lack of memory, so we recommend running on a machine with as much memory as possible. For example if using AWS instances for Tabular: we recommend M5 instances, where a m5.24xlarge machine should be able to handle most datasets.

How can I resolve memory issues?¶

See “If you encounter memory issues” in the In Depth Tutorial.

How can I resolve disk space issues?¶

See “If you encounter disk space issues” in the In Depth Tutorial.

How can I reduce the time required for training?¶

Specify the time_limit argument in predictor.fit to the number of seconds you are willing to wait (longer time limits generally result in superior predictive performance). You may also try other settings of the presets argument in predictor.fit, and can also subsample your data for a quick trial run via train_data.sample(n=SUBSAMPLE_SIZE). If a particular type of model is taking much longer to train on your data than the other types of models, you can tell AutoGluon not to train any models of this particular type by specifying its short-name in the excluded_model_types argument of fit().

Since many of the strategies to reduce memory usage also reduce training times, also check out: “If you encounter memory issues” in the In Depth Tutorial.

How can I reduce the time required for prediction?¶

See “Accelerating inference” in the In Depth Tutorial.

How does AutoGluon Tabular work internally?¶

Details are provided in the following paper:

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data. Arxiv, 2020.

How can I view more detailed logs of what is happening during fit?¶

Specify the argument verbosity = 4 in fit().

What model is AutoGluon using for prediction?¶

See “Prediction options” in the In Depth Tutorial.

Which classes do predicted probabilities correspond to?¶

This should become obvious if you look at the pandas DataFrame column names from the prediction probability output:

predictor.predict_proba(test_data)

For binary and multiclass classification:

predictor.class_labels

is a list of classes whose order corresponds to columns of predict_proba(as_pandas=False) output when it is a Numpy array.

You can see which class AutoGluon treats as the positive class in binary classification via:

predictor.positive_class

The positive class can also be retrieved via predictor.class_labels[-1]. The output of predict_proba(as_multiclass=False) for binary classification is the probability of the positive class.

How can I use AutoGluon for interpretability?¶

See “Interpretability (feature importance)” in the In Depth Tutorial, which allows you to quantify how much each feature contributes to AutoGluon’s predictive accuracy.

Additionally, you can explain particular AutoGluon predictions using Shapely values. Notebooks demonstrating this are provided at: https://github.com/awslabs/autogluon/tree/master/examples/tabular/interpret. We recommend starting with the notebook “SHAP with AutoGluon-Tabular” contained in this folder, which demonstrates handling of multiclass classification tasks and data with categorical features.

How can I perform inference on a file that won’t fit in memory?¶

The Tabular Dataset API works with pandas DataFrames, which supports chunking data into sizes that fit in memory. Here’s an example of one such chunk-based inference:

from autogluon.tabular import TabularDataset, TabularPredictor
import pandas as pd
import requests

train_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')
predictor = TabularPredictor(label='class').fit(train_data.sample(n=100, random_state=0), hyperparameters={'GBM': {}})

# Get the test dataset, if you are working with local data then omit the next two lines
r = requests.get('https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv', allow_redirects=True)
open('test.csv', 'wb').write(r.content)
reader = pd.read_csv('test.csv', chunksize=1024)
y_pred = []
y_true = []
for df_chunk in reader:
    y_pred.append(predictor.predict(df_chunk))
    y_true.append(df_chunk['class'])
y_pred = pd.concat(y_pred, axis=0, ignore_index=True)
y_true = pd.concat(y_true, axis=0, ignore_index=True)
predictor.evaluate_predictions(y_true=y_true, y_pred=y_pred)

No path specified. Models will be saved in: "AutogluonModels/ag-20220728_210716/"
Beginning AutoGluon training ...
AutoGluon will save models to "AutogluonModels/ag-20220728_210716/"
AutoGluon Version:  0.5.2b20220728
Python Version:     3.8.10
Operating System:   Linux
Train Data Rows:    100
Train Data Columns: 14
Label Column: class
Preprocessing data ...
AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).
    2 unique label values:  [' >50K', ' <=50K']
    If 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])
Selected class <--> label mapping:  class 1 =  >50K, class 0 =  <=50K
    Note: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive ( >50K) vs negative ( <=50K) class.
    To explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init.
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
    Available Memory:                    31846.93 MB
    Train Data (Original)  Memory Usage: 0.06 MB (0.0% of available memory)
    Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
    Stage 1 Generators:
            Fitting AsTypeFeatureGenerator...
                    Note: Converting 1 features to boolean dtype as they only contain 2 unique values.
    Stage 2 Generators:
            Fitting FillNaFeatureGenerator...
    Stage 3 Generators:
            Fitting IdentityFeatureGenerator...
            Fitting CategoryFeatureGenerator...
                    Fitting CategoryMemoryMinimizeFeatureGenerator...
    Stage 4 Generators:
            Fitting DropUniqueFeatureGenerator...
    Types of features in original data (raw dtype, special dtypes):
            ('int', [])    : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
            ('object', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]
    Types of features in processed data (raw dtype, special dtypes):
            ('category', [])  : 7 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]
            ('int', [])       : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
            ('int', ['bool']) : 1 | ['sex']
    0.1s = Fit runtime
    14 features in original data used to generate 14 features in processed data.
    Train Data (Processed) Memory Usage: 0.01 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.09s ...
AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'
    To change this, specify the eval_metric parameter of Predictor()
Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 80, Val Rows: 20
Fitting 1 L1 models ...
Fitting model: LightGBM ...
    0.8      = Validation score   (accuracy)
    0.58s    = Training   runtime
    0.01s    = Validation runtime
Fitting model: WeightedEnsemble_L2 ...
    0.8      = Validation score   (accuracy)
    0.0s     = Training   runtime
    0.0s     = Validation runtime
AutoGluon training complete, total runtime = 0.7s ... Best model: "WeightedEnsemble_L2"
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20220728_210716/")
Evaluation: accuracy on test data: 0.7628211690039922
Evaluations on test data:
{
    "accuracy": 0.7628211690039922,
    "balanced_accuracy": 0.515967102411626,
    "mcc": 0.09302340489249436,
    "f1": 0.08455156064796523,
    "precision": 0.5023474178403756,
    "recall": 0.04616048317515099
}

{'accuracy': 0.7628211690039922,
 'balanced_accuracy': 0.515967102411626,
 'mcc': 0.09302340489249436,
 'f1': 0.08455156064796523,
 'precision': 0.5023474178403756,
 'recall': 0.04616048317515099}

Here we split the test data into chunks of up to 1024 rows each, but you may select a larger size as long as it fits into your system’s memory. Further Reading

How can I skip some particular models?¶

To avoid training certain models, specify these in the excluded_model_types argument. For example, here’s how to call fit() without training K Nearest Neighbor (KNN), Random Forest (RF), or ExtraTrees (XT) models:

task.fit(..., excluded_model_types=['KNN','RF','XT'])

How can I add my own custom model to the set of models that AutoGluon trains, tunes, and ensembles?¶

To learn how to add your own custom models to the set that AutoGluon trains, tunes, and ensembles, review Adding a custom model to AutoGluon.

How can I reproduce the results of model XYZ without using AutoGluon?¶

Exactly reproducing the results of AutoGluon models without using AutoGluon can be challenging (Ex: Trying to replicate the results of AutoGluon’s LightGBM without depending on AutoGluon). AutoGluon uses a variety of techniques not present in the model frameworks it builds upon such as data preprocessing, edge case handling, custom early stopping, and custom architectures. These are all used to enhance the performance and simplify usage beyond what exists in the original model framework (such as LightGBM, XGBoost, FastAI, etc.). This is a core benefit of AutoML frameworks and is inherently complicated to replicate (you would have to reimplement much of what AutoGluon does behind the scenes). If you still wish to try to replicate results, the logic can be derived from the source code.

Why was XYZ design decision chosen?¶

Usually the answer boils down to “because it performed better on average than the alternatives we tested across a wide variety of benchmark datasets” and/or “because it is simple and easy to implement while likely performing similarly to more complex methods” and/or “because it worked well enough and we haven’t investigated further options”. AutoML is inherently an empirically oriented field rather than a theoretical one. This is because theoretically explaining why the various AutoML components work together better than numerous alternatives in practice is not particularly feasible nor generalizable.

How can I add my own custom data preprocessing or feature engineering?¶

Note that the TabularDataset object is essentially a pandas DataFrame and you can transform your training data however you wish before calling fit(). Note that any transformations you perform yourself must also be applied to all future test data before calling predict(), and AutoGluon will still perform its default processing on your transformed data inside fit().

To solely use custom data preprocessing and automatically apply your custom transformations to both the train data and all future data encountered during inference, you should instead create a custom FeatureGenerator. Follow this example in the source code: examples/tabular/example_custom_feature_generator.py

How can I differently weight the importance of training examples?¶

You can specify the sample_weight and weight_evaluation arguments when initializing a TabularPredictor.

I’m receiving C++ warning spam during training or inference¶

Warning message: [W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)

This can happen from downstream PyTorch dependencies (OpenMP) when using a specific environment. If you are using PyTorch 1.7, Mac OS X, Python 3.6/3.7, and using the PyTorch DataLoader, then you may get this warning spam. We have only seen this occur with the TabTransformer model. Reference open torch issue.

The recommended workaround from the torch issue to suppress this warning is to set an environment variable used in OpenMP. This has been tested on Torch 1.7, Python 3.6, and Mac OS X.

export OMP_NUM_THREADS=1

How to limit the number of cores AutoGluon will use¶

Although it is generally recommended to let AutoGluon to use all the cores, you can limit it by setting the ag_fit_args:

predictor = TabularPredictor(...).fit(..., ag_fit_args={'num_cpus': NUM_CORES_YOU_WANT})

You can also limit the number of cores used by a specific model:

# We use 1 core for CatBoost model, 4 cores for XGBoost model, and all cores for lightGBM model here.
predictor = TabularPredictor(...).fit(..., hyperparameters= {'CAT': {'ag_args_fit': {'num_cpus': 1}}, 'XGB': {'ag_args_fit': {'num_cpus': 4}}, 'GBM': {}},)

My custom metric is raising `_pickle.PicklingError: Can't pickle` errors¶

See “Ensuring Metric is Serializable” in Custom Metric Tutorial.

No space left error on SageMaker Notebook¶

If you are using AutoGluon on SageMaker Notebook, it is likely that you will encounter such error: OSError: [Errno 28] No space left on device. This is because the default disk size of a SageMaker Notebook instance is 5 GiB regardless of the type. AutoGluon training on some large datasets could end up with artifacts that’s larger than 5GiB.

To address it, either cleanup your workspace, or 1) shutdown your Notebook instance 2) choose your Notebook instance 3) update the Volume size in GB field under Edit

Issues not addressed here¶

First search if your issue is addressed in the tutorials, examples, documentation, or Github issues (search both Closed and Open issues). If it is not there, please open a new Github Issue and clearly state your issue. If you have a bug, please include: your code (call fit(..., verbosity=4) which will print more details), the output printed during the code execution, and information about your operating system, Python version, and installed packages (output of pip freeze).