.. _sec_tabularquick: Predicting Columns in a Table - Quick Start =========================================== Via a simple ``fit()`` call, AutoGluon can produce highly-accurate models to predict the values in one column of a data table based on the rest of the columns' values. Use AutoGluon with tabular data for both classification and regression problems. This tutorial demonstrates how to use AutoGluon to produce a classification model that predicts whether or not a person's income exceeds $50,000. To start, import AutoGluon's TabularPredictor and TabularDataset classes: .. code:: python from autogluon.tabular import TabularDataset, TabularPredictor Load training data from a `CSV file `__ into an AutoGluon Dataset object. This object is essentially equivalent to a `Pandas DataFrame `__ and the same methods can be applied to both. .. code:: python train_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv') subsample_size = 500 # subsample subset of data for faster demo, try setting this to much larger values train_data = train_data.sample(n=subsample_size, random_state=0) train_data.head() .. raw:: html
age workclass fnlwgt education education-num marital-status occupation relationship race sex capital-gain capital-loss hours-per-week native-country class
6118 51 Private 39264 Some-college 10 Married-civ-spouse Exec-managerial Wife White Female 0 0 40 United-States >50K
23204 58 Private 51662 10th 6 Married-civ-spouse Other-service Wife White Female 0 0 8 United-States <=50K
29590 40 Private 326310 Some-college 10 Married-civ-spouse Craft-repair Husband White Male 0 0 44 United-States <=50K
18116 37 Private 222450 HS-grad 9 Never-married Sales Not-in-family White Male 0 2339 40 El-Salvador <=50K
33964 62 Private 109190 Bachelors 13 Married-civ-spouse Exec-managerial Husband White Male 15024 0 40 United-States >50K
Note that we loaded data from a CSV file stored in the cloud (`AWS s3 bucket `__), but you can you specify a local file-path instead if you have already downloaded the CSV file to your own machine (e.g., using `wget `__). Each row in the table ``train_data`` corresponds to a single training example. In this particular dataset, each row corresponds to an individual person, and the columns contain various characteristics reported during a census. Let's first use these features to predict whether the person's income exceeds $50,000 or not, which is recorded in the ``class`` column of this table. .. code:: python label = 'class' print("Summary of class variable: \n", train_data[label].describe()) .. parsed-literal:: :class: output Summary of class variable: count 500 unique 2 top <=50K freq 365 Name: class, dtype: object Now use AutoGluon to train multiple models: .. code:: python save_path = 'agModels-predictClass' # specifies folder to store trained models predictor = TabularPredictor(label=label, path=save_path).fit(train_data) .. parsed-literal:: :class: output Beginning AutoGluon training ... AutoGluon will save models to "agModels-predictClass/" AutoGluon Version: 0.3.0b20210827 Train Data Rows: 500 Train Data Columns: 14 Preprocessing data ... AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed). 2 unique label values: [' >50K', ' <=50K'] If 'binary' is not the correct problem_type, please manually specify the problem_type argument in fit() (You may specify problem_type as one of: ['binary', 'multiclass', 'regression']) Selected class <--> label mapping: class 1 = >50K, class 0 = <=50K Note: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive ( >50K) vs negative ( <=50K) class. To explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init. Using Feature Generators to preprocess the data ... Fitting AutoMLPipelineFeatureGenerator... Available Memory: 22171.99 MB Train Data (Original) Memory Usage: 0.29 MB (0.0% of available memory) Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features. Stage 1 Generators: Fitting AsTypeFeatureGenerator... Stage 2 Generators: Fitting FillNaFeatureGenerator... Stage 3 Generators: Fitting IdentityFeatureGenerator... Fitting CategoryFeatureGenerator... Fitting CategoryMemoryMinimizeFeatureGenerator... Stage 4 Generators: Fitting DropUniqueFeatureGenerator... Types of features in original data (raw dtype, special dtypes): ('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...] ('object', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...] Types of features in processed data (raw dtype, special dtypes): ('category', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...] ('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...] 0.1s = Fit runtime 14 features in original data used to generate 14 features in processed data. Train Data (Processed) Memory Usage: 0.03 MB (0.0% of available memory) Data preprocessing and feature engineering runtime = 0.08s ... AutoGluon will gauge predictive performance using evaluation metric: 'accuracy' To change this, specify the eval_metric argument of fit() Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 400, Val Rows: 100 Fitting 13 L1 models ... Fitting model: KNeighborsUnif ... 0.73 = Validation score (accuracy) 0.0s = Training runtime 0.1s = Validation runtime Fitting model: KNeighborsDist ... 0.65 = Validation score (accuracy) 0.0s = Training runtime 0.1s = Validation runtime Fitting model: LightGBMXT ... 0.83 = Validation score (accuracy) 0.52s = Training runtime 0.01s = Validation runtime Fitting model: LightGBM ... 0.85 = Validation score (accuracy) 0.15s = Training runtime 0.01s = Validation runtime Fitting model: RandomForestGini ... 0.84 = Validation score (accuracy) 0.54s = Training runtime 0.11s = Validation runtime Fitting model: RandomForestEntr ... 0.82 = Validation score (accuracy) 0.51s = Training runtime 0.11s = Validation runtime Fitting model: CatBoost ... 0.86 = Validation score (accuracy) 0.61s = Training runtime 0.01s = Validation runtime Fitting model: ExtraTreesGini ... 0.82 = Validation score (accuracy) 0.61s = Training runtime 0.11s = Validation runtime Fitting model: ExtraTreesEntr ... 0.82 = Validation score (accuracy) 0.5s = Training runtime 0.11s = Validation runtime Fitting model: NeuralNetFastAI ... No improvement since epoch 2: early stopping 0.81 = Validation score (accuracy) 5.44s = Training runtime 0.02s = Validation runtime Fitting model: XGBoost ... 0.87 = Validation score (accuracy) 0.19s = Training runtime 0.01s = Validation runtime Fitting model: NeuralNetMXNet ... 0.84 = Validation score (accuracy) 5.28s = Training runtime 0.02s = Validation runtime Fitting model: LightGBMLarge ... 0.83 = Validation score (accuracy) 0.39s = Training runtime 0.01s = Validation runtime Fitting model: WeightedEnsemble_L2 ... 0.87 = Validation score (accuracy) 0.25s = Training runtime 0.0s = Validation runtime AutoGluon training complete, total runtime = 16.71s ... TabularPredictor saved. To load, use: predictor = TabularPredictor.load("agModels-predictClass/") Next, load separate test data to demonstrate how to make predictions on new examples at inference time: .. code:: python test_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv') y_test = test_data[label] # values to predict test_data_nolab = test_data.drop(columns=[label]) # delete label column to prove we're not cheating test_data_nolab.head() .. parsed-literal:: :class: output Loaded data from: https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv | Columns = 15 / 15 | Rows = 9769 -> 9769 .. raw:: html
age workclass fnlwgt education education-num marital-status occupation relationship race sex capital-gain capital-loss hours-per-week native-country
0 31 Private 169085 11th 7 Married-civ-spouse Sales Wife White Female 0 0 20 United-States
1 17 Self-emp-not-inc 226203 12th 8 Never-married Sales Own-child White Male 0 0 45 United-States
2 47 Private 54260 Assoc-voc 11 Married-civ-spouse Exec-managerial Husband White Male 0 1887 60 United-States
3 21 Private 176262 Some-college 10 Never-married Exec-managerial Own-child White Female 0 0 30 United-States
4 17 Private 241185 12th 8 Never-married Prof-specialty Own-child White Male 0 0 20 United-States
We use our trained models to make predictions on the new data and then evaluate performance: .. code:: python predictor = TabularPredictor.load(save_path) # unnecessary, just demonstrates how to load previously-trained predictor from file y_pred = predictor.predict(test_data_nolab) print("Predictions: \n", y_pred) perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=True) .. parsed-literal:: :class: output Evaluation: accuracy on test data: 0.8374449790152523 Evaluations on test data: { "accuracy": 0.8374449790152523, "balanced_accuracy": 0.7430558394221018, "mcc": 0.5243657567117436, "f1": 0.621904761904762, "precision": 0.69394261424017, "recall": 0.5634167385677308 } .. parsed-literal:: :class: output Predictions: 0 <=50K 1 <=50K 2 <=50K 3 <=50K 4 <=50K ... 9764 <=50K 9765 <=50K 9766 <=50K 9767 <=50K 9768 <=50K Name: class, Length: 9769, dtype: object We can also evaluate the performance of each individual trained model on our (labeled) test data: .. code:: python predictor.leaderboard(test_data, silent=True) .. raw:: html
model score_test score_val pred_time_test pred_time_val fit_time pred_time_test_marginal pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 CatBoost 0.843894 0.86 0.017315 0.009343 0.609264 0.017315 0.009343 0.609264 1 True 7
1 RandomForestGini 0.841949 0.84 0.215034 0.106803 0.542569 0.215034 0.106803 0.542569 1 True 5
2 RandomForestEntr 0.840618 0.82 0.215525 0.106933 0.506189 0.215525 0.106933 0.506189 1 True 6
3 LightGBM 0.839799 0.85 0.017673 0.007375 0.150319 0.017673 0.007375 0.150319 1 True 4
4 LightGBMXT 0.839390 0.83 0.012838 0.007391 0.520103 0.012838 0.007391 0.520103 1 True 3
5 XGBoost 0.837445 0.87 0.048777 0.005044 0.192974 0.048777 0.005044 0.192974 1 True 11
6 WeightedEnsemble_L2 0.837445 0.87 0.050234 0.005529 0.444304 0.001457 0.000485 0.251330 2 True 14
7 ExtraTreesGini 0.835705 0.82 0.216772 0.106596 0.607523 0.216772 0.106596 0.607523 1 True 8
8 ExtraTreesEntr 0.833453 0.82 0.215174 0.106517 0.504193 0.215174 0.106517 0.504193 1 True 9
9 LightGBMLarge 0.827823 0.83 0.017230 0.007733 0.386852 0.017230 0.007733 0.386852 1 True 13
10 NeuralNetMXNet 0.825264 0.84 1.720702 0.022219 5.275552 1.720702 0.022219 5.275552 1 True 12
11 NeuralNetFastAI 0.809090 0.81 0.175964 0.021670 5.436273 0.175964 0.021670 5.436273 1 True 10
12 KNeighborsUnif 0.725970 0.73 0.104152 0.104346 0.004471 0.104152 0.104346 0.004471 1 True 1
13 KNeighborsDist 0.695158 0.65 0.104839 0.102856 0.003461 0.104839 0.102856 0.003461 1 True 2
Now you're ready to try AutoGluon on your own tabular datasets! As long as they're stored in a popular format like CSV, you should be able to achieve strong predictive performance with just 2 lines of code: :: from autogluon.tabular import TabularPredictor predictor = TabularPredictor(label=).fit(train_data=) **Note:** This simple call to ``fit()`` is intended for your first prototype model. In a subsequent section, we'll demonstrate how to maximize predictive performance by additionally specifying two ``fit()`` arguments: ``presets`` and ``eval_metric``. Description of fit(): --------------------- Here we discuss what happened during ``fit()``. Since there are only two possible values of the ``class`` variable, this was a binary classification problem, for which an appropriate performance metric is *accuracy*. AutoGluon automatically infers this as well as the type of each feature (i.e., which columns contain continuous numbers vs. discrete categories). AutogGluon can also automatically handle common issues like missing data and rescaling feature values. We did not specify separate validation data and so AutoGluon automatically choses a random training/validation split of the data. The data used for validation is seperated from the training data and is used to determine the models and hyperparameter-values that produce the best results. Rather than just a single model, AutoGluon trains multiple models and ensembles them together to ensure superior predictive performance. By default, AutoGluon tries to fit various types of models including neural networks and tree ensembles. Each type of model has various hyperparameters, which traditionally, the user would have to specify. AutoGluon automates this process. AutoGluon automatically and iteratively tests values for hyperparameters to produce the best performance on the validation data. This involves repeatedly training models under different hyperparameter settings and evaluating their performance. This process can be computationally-intensive, so ``fit()`` can parallelize this process across multiple threads (and machines if distributed resources are available). To control runtimes, you can specify various arguments in fit() as demonstrated in the subsequent **In-Depth** tutorial. For tabular problems, ``fit()`` returns a ``Predictor`` object. For classification, you can easily output predicted class probabilities instead of predicted classes: .. code:: python pred_probs = predictor.predict_proba(test_data_nolab) pred_probs.head(5) .. raw:: html
<=50K >50K
0 0.982107 0.017893
1 0.988337 0.011663
2 0.573505 0.426495
3 0.998272 0.001728
4 0.990299 0.009701
Besides inference, this object can also summarize what happened during fit. .. code:: python results = predictor.fit_summary(show_plot=True) .. parsed-literal:: :class: output *** Summary of fit() *** Estimated performance of each model: model score_val pred_time_val fit_time pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order 0 XGBoost 0.87 0.005044 0.192974 0.005044 0.192974 1 True 11 1 WeightedEnsemble_L2 0.87 0.005529 0.444304 0.000485 0.251330 2 True 14 2 CatBoost 0.86 0.009343 0.609264 0.009343 0.609264 1 True 7 3 LightGBM 0.85 0.007375 0.150319 0.007375 0.150319 1 True 4 4 NeuralNetMXNet 0.84 0.022219 5.275552 0.022219 5.275552 1 True 12 5 RandomForestGini 0.84 0.106803 0.542569 0.106803 0.542569 1 True 5 6 LightGBMXT 0.83 0.007391 0.520103 0.007391 0.520103 1 True 3 7 LightGBMLarge 0.83 0.007733 0.386852 0.007733 0.386852 1 True 13 8 ExtraTreesEntr 0.82 0.106517 0.504193 0.106517 0.504193 1 True 9 9 ExtraTreesGini 0.82 0.106596 0.607523 0.106596 0.607523 1 True 8 10 RandomForestEntr 0.82 0.106933 0.506189 0.106933 0.506189 1 True 6 11 NeuralNetFastAI 0.81 0.021670 5.436273 0.021670 5.436273 1 True 10 12 KNeighborsUnif 0.73 0.104346 0.004471 0.104346 0.004471 1 True 1 13 KNeighborsDist 0.65 0.102856 0.003461 0.102856 0.003461 1 True 2 Number of models trained: 14 Types of models trained: {'RFModel', 'CatBoostModel', 'NNFastAiTabularModel', 'LGBModel', 'KNNModel', 'XGBoostModel', 'TabularNeuralNetModel', 'XTModel', 'WeightedEnsembleModel'} Bagging used: False Multi-layer stack-ensembling used: False Feature Metadata (Processed): (raw dtype, special dtypes): ('category', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...] ('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...] *** End of fit() summary *** .. parsed-literal:: :class: output /var/lib/jenkins/workspace/workspace/autogluon-tutorial-tabular-v3/core/src/autogluon/core/utils/plots.py:138: UserWarning: AutoGluon summary plots cannot be created because bokeh is not installed. To see plots, please do: "pip install bokeh==2.0.1" warnings.warn('AutoGluon summary plots cannot be created because bokeh is not installed. To see plots, please do: "pip install bokeh==2.0.1"') From this summary, we can see that AutoGluon trained many different types of models as well as an ensemble of the best-performing models. The summary also describes the actual models that were trained during fit and how well each model performed on the held-out validation data. We can view what properties AutoGluon automatically inferred about our prediction task: .. code:: python print("AutoGluon infers problem type is: ", predictor.problem_type) print("AutoGluon identified the following types of features:") print(predictor.feature_metadata) .. parsed-literal:: :class: output AutoGluon infers problem type is: binary AutoGluon identified the following types of features: ('category', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...] ('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...] AutoGluon correctly recognized our prediction problem to be a **binary classification** task and decided that variables such as ``age`` should be represented as integers, whereas variables such as ``workclass`` should be represented as categorical objects. The ``feature_metadata`` attribute allows you to see the inferred data type of each predictive variable after preprocessing (this is it's *raw* dtype; some features may also be associated with additional *special* dtypes if produced via feature-engineering, e.g. numerical representations of a datetime/text column). We can evaluate the performance of each individual trained model on our (labeled) test data: .. code:: python predictor.leaderboard(test_data, silent=True) .. raw:: html
model score_test score_val pred_time_test pred_time_val fit_time pred_time_test_marginal pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 CatBoost 0.843894 0.86 0.016767 0.009343 0.609264 0.016767 0.009343 0.609264 1 True 7
1 RandomForestGini 0.841949 0.84 0.215670 0.106803 0.542569 0.215670 0.106803 0.542569 1 True 5
2 RandomForestEntr 0.840618 0.82 0.216254 0.106933 0.506189 0.216254 0.106933 0.506189 1 True 6
3 LightGBM 0.839799 0.85 0.017996 0.007375 0.150319 0.017996 0.007375 0.150319 1 True 4
4 LightGBMXT 0.839390 0.83 0.013044 0.007391 0.520103 0.013044 0.007391 0.520103 1 True 3
5 XGBoost 0.837445 0.87 0.048457 0.005044 0.192974 0.048457 0.005044 0.192974 1 True 11
6 WeightedEnsemble_L2 0.837445 0.87 0.050021 0.005529 0.444304 0.001565 0.000485 0.251330 2 True 14
7 ExtraTreesGini 0.835705 0.82 0.215991 0.106596 0.607523 0.215991 0.106596 0.607523 1 True 8
8 ExtraTreesEntr 0.833453 0.82 0.215198 0.106517 0.504193 0.215198 0.106517 0.504193 1 True 9
9 LightGBMLarge 0.827823 0.83 0.017329 0.007733 0.386852 0.017329 0.007733 0.386852 1 True 13
10 NeuralNetMXNet 0.825264 0.84 1.906982 0.022219 5.275552 1.906982 0.022219 5.275552 1 True 12
11 NeuralNetFastAI 0.809090 0.81 0.172021 0.021670 5.436273 0.172021 0.021670 5.436273 1 True 10
12 KNeighborsUnif 0.725970 0.73 0.104445 0.104346 0.004471 0.104445 0.104346 0.004471 1 True 1
13 KNeighborsDist 0.695158 0.65 0.104696 0.102856 0.003461 0.104696 0.102856 0.003461 1 True 2
When we call ``predict()``, AutoGluon automatically predicts with the model that displayed the best performance on validation data (i.e. the weighted-ensemble). We can instead specify which model to use for predictions like this: :: predictor.predict(test_data, model='LightGBM') Above the scores of predictive performance were based on a default evaluation metric (accuracy for binary classification). Performance in certain applications may be measured by different metrics than the ones AutoGluon optimizes for by default. If you know the metric that counts in your application, you should specify it as demonstrated in the next section. Maximizing predictive performance --------------------------------- **Note:** You should not call ``fit()`` with entirely default arguments if you are benchmarking AutoGluon-Tabular or hoping to maximize its accuracy! To get the best predictive accuracy with AutoGluon, you should generally use it like this: .. code:: python time_limit = 60 # for quick demonstration only, you should set this to longest time you are willing to wait (in seconds) metric = 'roc_auc' # specify your evaluation metric here predictor = TabularPredictor(label, eval_metric=metric).fit(train_data, time_limit=time_limit, presets='best_quality') predictor.leaderboard(test_data, silent=True) .. parsed-literal:: :class: output No path specified. Models will be saved in: "AutogluonModels/ag-20210827_223230/" Presets specified: ['best_quality'] Beginning AutoGluon training ... Time limit = 60s AutoGluon will save models to "AutogluonModels/ag-20210827_223230/" AutoGluon Version: 0.3.0b20210827 Train Data Rows: 500 Train Data Columns: 14 Preprocessing data ... AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed). 2 unique label values: [' >50K', ' <=50K'] If 'binary' is not the correct problem_type, please manually specify the problem_type argument in fit() (You may specify problem_type as one of: ['binary', 'multiclass', 'regression']) Selected class <--> label mapping: class 1 = >50K, class 0 = <=50K Note: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive ( >50K) vs negative ( <=50K) class. To explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init. Using Feature Generators to preprocess the data ... Fitting AutoMLPipelineFeatureGenerator... Available Memory: 18821.7 MB Train Data (Original) Memory Usage: 0.29 MB (0.0% of available memory) Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features. Stage 1 Generators: Fitting AsTypeFeatureGenerator... Stage 2 Generators: Fitting FillNaFeatureGenerator... Stage 3 Generators: Fitting IdentityFeatureGenerator... Fitting CategoryFeatureGenerator... Fitting CategoryMemoryMinimizeFeatureGenerator... Stage 4 Generators: Fitting DropUniqueFeatureGenerator... Types of features in original data (raw dtype, special dtypes): ('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...] ('object', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...] Types of features in processed data (raw dtype, special dtypes): ('category', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...] ('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...] 0.1s = Fit runtime 14 features in original data used to generate 14 features in processed data. Train Data (Processed) Memory Usage: 0.03 MB (0.0% of available memory) Data preprocessing and feature engineering runtime = 0.08s ... AutoGluon will gauge predictive performance using evaluation metric: 'roc_auc' This metric expects predicted probabilities rather than predicted class labels, so you'll need to use predict_proba() instead of predict() To change this, specify the eval_metric argument of fit() Fitting 13 L1 models ... Fitting model: KNeighborsUnif_BAG_L1 ... Training model for up to 59.92s of the 59.92s of remaining time. 0.5196 = Validation score (roc_auc) 0.0s = Training runtime 0.1s = Validation runtime Fitting model: KNeighborsDist_BAG_L1 ... Training model for up to 59.81s of the 59.8s of remaining time. 0.537 = Validation score (roc_auc) 0.0s = Training runtime 0.1s = Validation runtime Fitting model: LightGBMXT_BAG_L1 ... Training model for up to 59.69s of the 59.69s of remaining time. 0.8814 = Validation score (roc_auc) 0.87s = Training runtime 0.04s = Validation runtime Fitting model: LightGBM_BAG_L1 ... Training model for up to 58.76s of the 58.76s of remaining time. 0.867 = Validation score (roc_auc) 0.87s = Training runtime 0.04s = Validation runtime Fitting model: RandomForestGini_BAG_L1 ... Training model for up to 57.83s of the 57.83s of remaining time. 0.8868 = Validation score (roc_auc) 0.5s = Training runtime 0.09s = Validation runtime Fitting model: RandomForestEntr_BAG_L1 ... Training model for up to 57.22s of the 57.22s of remaining time. 0.8853 = Validation score (roc_auc) 0.5s = Training runtime 0.09s = Validation runtime Fitting model: CatBoost_BAG_L1 ... Training model for up to 56.61s of the 56.6s of remaining time. 0.8831 = Validation score (roc_auc) 2.84s = Training runtime 0.05s = Validation runtime Fitting model: ExtraTreesGini_BAG_L1 ... Training model for up to 53.7s of the 53.7s of remaining time. 0.8865 = Validation score (roc_auc) 0.61s = Training runtime 0.1s = Validation runtime Fitting model: ExtraTreesEntr_BAG_L1 ... Training model for up to 52.98s of the 52.98s of remaining time. 0.8853 = Validation score (roc_auc) 0.6s = Training runtime 0.1s = Validation runtime Fitting model: NeuralNetFastAI_BAG_L1 ... Training model for up to 52.26s of the 52.26s of remaining time. 0.8663 = Validation score (roc_auc) 7.03s = Training runtime 0.08s = Validation runtime Fitting model: XGBoost_BAG_L1 ... Training model for up to 45.09s of the 45.09s of remaining time. 0.8674 = Validation score (roc_auc) 2.23s = Training runtime 0.02s = Validation runtime Fitting model: NeuralNetMXNet_BAG_L1 ... Training model for up to 42.77s of the 42.77s of remaining time. Ran out of time, stopping training early. (Stopping on epoch 82) 0.8544 = Validation score (roc_auc) 31.17s = Training runtime 0.19s = Validation runtime Fitting model: LightGBMLarge_BAG_L1 ... Training model for up to 11.36s of the 11.36s of remaining time. 0.8417 = Validation score (roc_auc) 1.69s = Training runtime 0.04s = Validation runtime Completed 1/20 k-fold bagging repeats ... Fitting model: WeightedEnsemble_L2 ... Training model for up to 59.92s of the 9.59s of remaining time. 0.8984 = Validation score (roc_auc) 1.33s = Training runtime 0.0s = Validation runtime AutoGluon training complete, total runtime = 51.75s ... TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20210827_223230/") .. raw:: html
model score_test score_val pred_time_test pred_time_val fit_time pred_time_test_marginal pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 CatBoost_BAG_L1 0.902600 0.883105 0.053186 0.045541 2.839869 0.053186 0.045541 2.839869 1 True 7
1 LightGBMXT_BAG_L1 0.900161 0.881380 0.128378 0.037216 0.867347 0.128378 0.037216 0.867347 1 True 3
2 WeightedEnsemble_L2 0.897033 0.898366 9.839681 0.725015 45.456237 0.003944 0.001099 1.330617 2 True 14
3 LightGBM_BAG_L1 0.892347 0.866991 0.069976 0.037416 0.865634 0.069976 0.037416 0.865634 1 True 4
4 XGBoost_BAG_L1 0.891801 0.867357 0.190188 0.024917 2.227235 0.190188 0.024917 2.227235 1 True 11
5 RandomForestEntr_BAG_L1 0.887290 0.885317 0.216599 0.092428 0.502344 0.216599 0.092428 0.502344 1 True 6
6 RandomForestGini_BAG_L1 0.886345 0.886778 0.216649 0.092822 0.501374 0.216649 0.092822 0.501374 1 True 5
7 NeuralNetFastAI_BAG_L1 0.883346 0.866261 0.846453 0.077986 7.031628 0.846453 0.077986 7.031628 1 True 10
8 ExtraTreesEntr_BAG_L1 0.880050 0.885287 0.219953 0.095194 0.603429 0.219953 0.095194 0.603429 1 True 9
9 ExtraTreesGini_BAG_L1 0.880042 0.886464 0.215863 0.095158 0.605531 0.215863 0.095158 0.605531 1 True 8
10 LightGBMLarge_BAG_L1 0.873756 0.841684 0.085549 0.037466 1.691110 0.085549 0.037466 1.691110 1 True 13
11 NeuralNetMXNet_BAG_L1 0.870533 0.854368 7.938655 0.187572 31.174097 7.938655 0.187572 31.174097 1 True 12
12 KNeighborsDist_BAG_L1 0.525998 0.536956 0.104589 0.101812 0.002880 0.104589 0.101812 0.002880 1 True 2
13 KNeighborsUnif_BAG_L1 0.514970 0.519604 0.104071 0.101973 0.002973 0.104071 0.101973 0.002973 1 True 1
This command implements the following strategy to maximize accuracy: - Specify the argument ``presets='best_quality'``, which allows AutoGluon to automatically construct powerful model ensembles based on `stacking/bagging `__, and will greatly improve the resulting predictions if granted sufficient training time. The default value of ``presets`` is ``'medium_quality_faster_train'``, which produces *less* accurate models but facilitates faster prototyping. With ``presets``, you can flexibly prioritize predictive accuracy vs. training/inference speed. For example, if you care less about predictive performance and want to quickly deploy a basic model, consider using: ``presets=['good_quality_faster_inference_only_refit', 'optimize_for_deployment']``. - Provide the ``eval_metric`` if you know what metric will be used to evaluate predictions in your application. Some other non-default metrics you might use include things like: ``'f1'`` (for binary classification), ``'roc_auc'`` (for binary classification), ``'log_loss'`` (for classification), ``'mean_absolute_error'`` (for regression), ``'median_absolute_error'`` (for regression). You can also define your own custom metric function, see examples in the folder: ``autogluon/core/metrics/`` - Include all your data in ``train_data`` and do not provide ``tuning_data`` (AutoGluon will split the data more intelligently to fit its needs). - Do not specify the ``hyperparameter_tune_kwargs`` argument (counterintuitively, hyperparameter tuning is not the best way to spend a limited training time budgets, as model ensembling is often superior). We recommend you only use ``hyperparameter_tune_kwargs`` if your goal is to deploy a single model rather than an ensemble. - Do not specify ``hyperparameters`` argument (allow AutoGluon to adaptively select which models/hyperparameters to use). - Set ``time_limit`` to the longest amount of time (in seconds) that you are willing to wait. AutoGluon's predictive performance improves the longer ``fit()`` is allowed to run. Regression (predicting numeric table columns): ---------------------------------------------- To demonstrate that ``fit()`` can also automatically handle regression tasks, we now try to predict the numeric ``age`` variable in the same table based on the other features: .. code:: python age_column = 'age' print("Summary of age variable: \n", train_data[age_column].describe()) .. parsed-literal:: :class: output Summary of age variable: count 500.00000 mean 39.65200 std 13.52393 min 17.00000 25% 29.00000 50% 38.00000 75% 49.00000 max 85.00000 Name: age, dtype: float64 We again call ``fit()``, imposing a time-limit this time (in seconds), and also demonstrate a shorthand method to evaluate the resulting model on the test data (which contain labels): .. code:: python predictor_age = TabularPredictor(label=age_column, path="agModels-predictAge").fit(train_data, time_limit=60) performance = predictor_age.evaluate(test_data) .. parsed-literal:: :class: output Beginning AutoGluon training ... Time limit = 60s AutoGluon will save models to "agModels-predictAge/" AutoGluon Version: 0.3.0b20210827 Train Data Rows: 500 Train Data Columns: 14 Preprocessing data ... AutoGluon infers your prediction problem is: 'regression' (because dtype of label-column == int and many unique label-values observed). Label info (max, min, mean, stddev): (85, 17, 39.652, 13.52393) If 'regression' is not the correct problem_type, please manually specify the problem_type argument in fit() (You may specify problem_type as one of: ['binary', 'multiclass', 'regression']) Using Feature Generators to preprocess the data ... Fitting AutoMLPipelineFeatureGenerator... Available Memory: 15952.1 MB Train Data (Original) Memory Usage: 0.32 MB (0.0% of available memory) Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features. Stage 1 Generators: Fitting AsTypeFeatureGenerator... Stage 2 Generators: Fitting FillNaFeatureGenerator... Stage 3 Generators: Fitting IdentityFeatureGenerator... Fitting CategoryFeatureGenerator... Fitting CategoryMemoryMinimizeFeatureGenerator... Stage 4 Generators: Fitting DropUniqueFeatureGenerator... Types of features in original data (raw dtype, special dtypes): ('int', []) : 5 | ['fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week'] ('object', []) : 9 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...] Types of features in processed data (raw dtype, special dtypes): ('category', []) : 9 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...] ('int', []) : 5 | ['fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week'] 0.1s = Fit runtime 14 features in original data used to generate 14 features in processed data. Train Data (Processed) Memory Usage: 0.03 MB (0.0% of available memory) Data preprocessing and feature engineering runtime = 0.08s ... AutoGluon will gauge predictive performance using evaluation metric: 'root_mean_squared_error' To change this, specify the eval_metric argument of fit() Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 400, Val Rows: 100 Fitting 11 L1 models ... Fitting model: KNeighborsUnif ... Training model for up to 59.92s of the 59.92s of remaining time. -15.6869 = Validation score (root_mean_squared_error) 0.0s = Training runtime 0.1s = Validation runtime Fitting model: KNeighborsDist ... Training model for up to 59.81s of the 59.81s of remaining time. -15.1801 = Validation score (root_mean_squared_error) 0.0s = Training runtime 0.1s = Validation runtime Fitting model: LightGBMXT ... Training model for up to 59.7s of the 59.7s of remaining time. -11.8147 = Validation score (root_mean_squared_error) 0.22s = Training runtime 0.01s = Validation runtime Fitting model: LightGBM ... Training model for up to 59.47s of the 59.47s of remaining time. -11.9295 = Validation score (root_mean_squared_error) 0.19s = Training runtime 0.01s = Validation runtime Fitting model: RandomForestMSE ... Training model for up to 59.26s of the 59.26s of remaining time. -11.6512 = Validation score (root_mean_squared_error) 0.5s = Training runtime 0.11s = Validation runtime Fitting model: CatBoost ... Training model for up to 58.63s of the 58.63s of remaining time. -11.7221 = Validation score (root_mean_squared_error) 0.36s = Training runtime 0.01s = Validation runtime Fitting model: ExtraTreesMSE ... Training model for up to 58.26s of the 58.26s of remaining time. -11.3754 = Validation score (root_mean_squared_error) 0.5s = Training runtime 0.11s = Validation runtime Fitting model: NeuralNetFastAI ... Training model for up to 57.64s of the 57.64s of remaining time. -12.0237 = Validation score (root_mean_squared_error) 0.64s = Training runtime 0.02s = Validation runtime Fitting model: XGBoost ... Training model for up to 56.97s of the 56.97s of remaining time. -12.2744 = Validation score (root_mean_squared_error) 0.54s = Training runtime 0.01s = Validation runtime Fitting model: NeuralNetMXNet ... Training model for up to 56.41s of the 56.4s of remaining time. -12.7285 = Validation score (root_mean_squared_error) 7.38s = Training runtime 0.04s = Validation runtime Fitting model: LightGBMLarge ... Training model for up to 48.98s of the 48.98s of remaining time. -12.1676 = Validation score (root_mean_squared_error) 0.49s = Training runtime 0.01s = Validation runtime Fitting model: WeightedEnsemble_L2 ... Training model for up to 59.92s of the 47.89s of remaining time. -11.1971 = Validation score (root_mean_squared_error) 0.26s = Training runtime 0.0s = Validation runtime AutoGluon training complete, total runtime = 12.38s ... TabularPredictor saved. To load, use: predictor = TabularPredictor.load("agModels-predictAge/") Evaluation: root_mean_squared_error on test data: -10.616278908377607 Note: Scores are always higher_is_better. This metric score can be multiplied by -1 to get the metric value. Evaluations on test data: { "root_mean_squared_error": -10.616278908377607, "mean_squared_error": -112.70537786046324, "mean_absolute_error": -8.512377751350696, "r2": 0.3975678769947065, "pearsonr": 0.638919284019519, "median_absolute_error": -7.332328796386719 } Note that we didn't need to tell AutoGluon this is a regression problem, it automatically inferred this from the data and reported the appropriate performance metric (RMSE by default). To specify a particular evaluation metric other than the default, set the ``eval_metric`` argument of ``fit()`` and AutoGluon will tailor its models to optimize your metric (e.g. ``eval_metric = 'mean_absolute_error'``). For evaluation metrics where higher values are worse (like RMSE), AutoGluon may sometimes flips their sign and print them as negative values during training (as it internally assumes higher values are better). We can call leaderboard to see the per-model performance: .. code:: python predictor_age.leaderboard(test_data, silent=True) .. raw:: html
model score_test score_val pred_time_test pred_time_val fit_time pred_time_test_marginal pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 WeightedEnsemble_L2 -10.616279 -11.197146 0.354660 0.128426 1.943013 0.002306 0.000425 0.263079 2 True 12
1 CatBoost -10.623061 -11.722121 0.017722 0.009980 0.359344 0.017722 0.009980 0.359344 1 True 6
2 ExtraTreesMSE -10.667834 -11.375412 0.116410 0.106769 0.500232 0.116410 0.106769 0.500232 1 True 7
3 RandomForestMSE -10.749669 -11.651186 0.117847 0.106993 0.504859 0.117847 0.106993 0.504859 1 True 5
4 LightGBMXT -10.753344 -11.814712 0.033235 0.007874 0.219299 0.033235 0.007874 0.219299 1 True 3
5 LightGBM -10.972156 -11.929546 0.020159 0.008266 0.192255 0.020159 0.008266 0.192255 1 True 4
6 XGBoost -11.109788 -12.274424 0.052586 0.005434 0.538617 0.052586 0.005434 0.538617 1 True 9
7 LightGBMLarge -11.598649 -12.167606 0.041439 0.008702 0.491021 0.041439 0.008702 0.491021 1 True 11
8 NeuralNetMXNet -11.693356 -12.728461 0.900187 0.038729 7.379090 0.900187 0.038729 7.379090 1 True 10
9 NeuralNetFastAI -11.898842 -12.023682 0.183358 0.015799 0.641085 0.183358 0.015799 0.641085 1 True 8
10 KNeighborsUnif -14.902058 -15.686937 0.103126 0.102736 0.003440 0.103126 0.102736 0.003440 1 True 1
11 KNeighborsDist -15.771259 -15.180149 0.103994 0.102687 0.003121 0.103994 0.102687 0.003121 1 True 2
**Data Formats:** AutoGluon can currently operate on data tables already loaded into Python as pandas DataFrames, or those stored in files of `CSV format `__ or `Parquet format `__. If your data live in multiple tables, you will first need to join them into a single table whose rows correspond to statistically independent observations (datapoints) and columns correspond to different features (aka. variables/covariates). Refer to the `TabularPredictor documentation <../../api/autogluon.predictor.html#autogluon.tabular.TabularPredictor.fit>`__ to see all of the available methods/options.