{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "5e7d4675", "metadata": {}, "source": [ "# AutoGluon Tabular - Essential Functionality\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/autogluon/autogluon/blob/master/docs/tutorials/tabular/tabular-essentials.ipynb)\n", "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/autogluon/autogluon/blob/master/docs/tutorials/tabular/tabular-essentials.ipynb)\n", "\n", "\n", "Via a simple `fit()` call, AutoGluon can produce highly-accurate models to predict the values in one column of a data table based on the rest of the columns' values. Use AutoGluon with tabular data for both classification and regression problems. This tutorial demonstrates how to use AutoGluon to produce a classification model that predicts whether or not a person's income exceeds $50,000.\n", "\n", "## TabularPredictor\n", "\n", "To start, import AutoGluon's [TabularPredictor](../../api/autogluon.tabular.TabularPredictor.rst) and [TabularDataset](../../api/autogluon.core.TabularDataset.rst) classes:" ] }, { "cell_type": "code", "execution_count": null, "id": "aa00faab-252f-44c9-b8f7-57131aa8251c", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "!pip install autogluon.tabular[all]\n" ], "outputs": [] }, { "cell_type": "code", "execution_count": null, "id": "b48e2768", "metadata": {}, "source": [ "from autogluon.tabular import TabularDataset, TabularPredictor" ], "outputs": [] }, { "cell_type": "markdown", "id": "cef2fc39", "metadata": {}, "source": [ "Load training data from a [CSV file](https://en.wikipedia.org/wiki/Comma-separated_values) into an AutoGluon Dataset object. This object is essentially equivalent to a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) and the same methods can be applied to both." ] }, { "cell_type": "code", "execution_count": null, "id": "671f5ff7", "metadata": {}, "source": [ "train_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')\n", "subsample_size = 500 # subsample subset of data for faster demo, try setting this to much larger values\n", "train_data = train_data.sample(n=subsample_size, random_state=0)\n", "train_data.head()" ], "outputs": [] }, { "cell_type": "markdown", "id": "0ac3f9f5", "metadata": {}, "source": [ "Note that we loaded data from a CSV file stored in the cloud. You can also specify a local file-path instead if you have already downloaded the CSV file to your own machine (e.g., using [wget](https://www.gnu.org/software/wget/)).\n", "Each row in the table `train_data` corresponds to a single training example. In this particular dataset, each row corresponds to an individual person, and the columns contain various characteristics reported during a census.\n", "\n", "Let's first use these features to predict whether the person's income exceeds $50,000 or not, which is recorded in the `class` column of this table." ] }, { "cell_type": "code", "execution_count": null, "id": "7fbae4f4", "metadata": {}, "source": [ "label = 'class'\n", "print(f\"Unique classes: {list(train_data[label].unique())}\")" ], "outputs": [] }, { "cell_type": "markdown", "id": "e2808c11", "metadata": {}, "source": [ "AutoGluon works with raw data, meaning you don't need to perform any data preprocessing before fitting AutoGluon. We actively recommend that you avoid performing operations such as missing value imputation or one-hot-encoding, as AutoGluon has dedicated logic to handle these situations automatically. You can learn more about AutoGluon's preprocessing in the [Feature Engineering Tutorial](tabular-feature-engineering.ipynb).\n", "\n", "### Training\n", "\n", "Now we initialize and fit AutoGluon's TabularPredictor in one line of code:" ] }, { "cell_type": "code", "execution_count": null, "id": "93ed52d4", "metadata": { "tags": [ "hide-output" ] }, "source": [ "predictor = TabularPredictor(label=label).fit(train_data)" ], "outputs": [] }, { "cell_type": "markdown", "id": "1088b80f", "metadata": {}, "source": [ "That's it! We now have a TabularPredictor that is able to make predictions on new data.\n", "\n", "### Prediction\n", "\n", "Next, load separate test data to demonstrate how to make predictions on new examples at inference time:" ] }, { "cell_type": "code", "execution_count": null, "id": "38907743", "metadata": {}, "source": [ "test_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv')\n", "test_data.head()" ], "outputs": [] }, { "cell_type": "markdown", "id": "01bd6e65", "metadata": {}, "source": [ "We can now use our trained models to make predictions on the new data:" ] }, { "cell_type": "code", "execution_count": null, "id": "388da91b", "metadata": {}, "source": [ "y_pred = predictor.predict(test_data)\n", "y_pred.head() # Predictions" ], "outputs": [] }, { "cell_type": "code", "execution_count": null, "source": [ "y_pred_proba = predictor.predict_proba(test_data)\n", "y_pred_proba.head() # Prediction Probabilities" ], "metadata": { "collapsed": false }, "id": "1f2ea44baed01439", "outputs": [] }, { "cell_type": "markdown", "source": [ "### Evaluation\n", "\n", "Next, we can [evaluate](../../api/autogluon.tabular.TabularPredictor.evaluate.rst) the predictor on the (labeled) test data:" ], "metadata": { "collapsed": false }, "id": "c1ac16b755097c93" }, { "cell_type": "code", "execution_count": null, "source": [ "predictor.evaluate(test_data)" ], "metadata": { "collapsed": false }, "id": "ccfb48acf364b609", "outputs": [] }, { "cell_type": "markdown", "id": "ec141019", "metadata": {}, "source": [ "We can also [evaluate each model individually](../../api/autogluon.tabular.TabularPredictor.leaderboard.rst):" ] }, { "cell_type": "code", "execution_count": null, "id": "0630d00d", "metadata": {}, "source": [ "predictor.leaderboard(test_data)" ], "outputs": [] }, { "cell_type": "markdown", "source": [ "### Loading a Trained Predictor\n", "\n", "Finally, we can load the predictor in a new session (or new machine) by calling [TabularPredictor.load()](../../api/autogluon.tabular.TabularPredictor.load.rst) and specifying the location of the predictor artifact on disk." ], "metadata": { "collapsed": false }, "id": "ae35bc029d386579" }, { "cell_type": "code", "execution_count": null, "source": [ "predictor.path # The path on disk where the predictor is saved" ], "metadata": { "collapsed": false }, "id": "85fcbc65e9dd2cfd", "outputs": [] }, { "cell_type": "code", "execution_count": null, "source": [ "# Load the predictor by specifying the path it is saved to on disk.\n", "# You can control where it is saved to by setting the `path` parameter during init\n", "predictor = TabularPredictor.load(predictor.path)" ], "metadata": { "collapsed": false }, "id": "3710a0faca8d4af1", "outputs": [] }, { "cell_type": "markdown", "id": "6a32a595", "metadata": {}, "source": [ "```{warning}\n", "\n", "`TabularPredictor.load()` uses the `pickle` module implicitly, which is known to be insecure. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never load data that could have come from an untrusted source, or that could have been tampered with. **Only load data you trust.**\n", "\n", "```\n", "\n", "Now you're ready to try AutoGluon on your own tabular datasets!\n", "As long as they're stored in a popular format like CSV, you should be able to achieve strong predictive performance with just 2 lines of code:" ] }, { "cell_type": "markdown", "id": "ee1650bf", "metadata": {}, "source": [ "```\n", "from autogluon.tabular import TabularPredictor\n", "predictor = TabularPredictor(label=).fit(train_data=)\n", "```\n" ] }, { "cell_type": "markdown", "id": "255b4558", "metadata": {}, "source": [ "**Note:** This simple call to [TabularPredictor.fit()](../../api/autogluon.tabular.TabularPredictor.fit.rst) is intended for your first prototype model. In a subsequent section, we'll demonstrate how to maximize predictive performance by additionally specifying the `presets` parameter to `fit()` and the `eval_metric` parameter to `TabularPredictor()`.\n", "\n", "## Description of fit()\n", "\n", "Here we discuss what happened during `fit()`.\n", "\n", "Since there are only two possible values of the `class` variable, this was a binary classification problem, for which an appropriate performance metric is _accuracy_. AutoGluon automatically infers this as well as the type of each feature (i.e., which columns contain continuous numbers vs. discrete categories). AutoGluon can also automatically handle common issues like missing data and rescaling feature values.\n", "\n", "We did not specify separate validation data and so AutoGluon automatically chose a random training/validation split of the data. The data used for validation is separated from the training data and is used to determine the models and hyperparameter-values that produce the best results. Rather than just a single model, AutoGluon trains multiple models and ensembles them together to obtain superior predictive performance.\n", "\n", "By default, AutoGluon tries to fit [various types of models](../../api/autogluon.tabular.models.rst) including neural networks and tree ensembles. Each type of model has various hyperparameters, which traditionally, the user would have to specify. AutoGluon automates this process.\n", "\n", "AutoGluon automatically and iteratively tests values for hyperparameters to produce the best performance on the validation data. This involves repeatedly training models under different hyperparameter settings and evaluating their performance. This process can be computationally-intensive, so `fit()` parallelizes this process across multiple threads using [Ray](https://www.ray.io/). To control runtimes, you can specify various arguments in `fit()` such as `time_limit` as demonstrated in the subsequent **[In-Depth Tutorial](tabular-indepth.ipynb)**." ] }, { "cell_type": "markdown", "id": "75f84eca", "metadata": {}, "source": [ "We can view what properties AutoGluon automatically inferred about our prediction task:" ] }, { "cell_type": "code", "execution_count": null, "id": "f4074d3a", "metadata": {}, "source": [ "print(\"AutoGluon infers problem type is: \", predictor.problem_type)\n", "print(\"AutoGluon identified the following types of features:\")\n", "print(predictor.feature_metadata)" ], "outputs": [] }, { "cell_type": "markdown", "id": "14fde02c", "metadata": {}, "source": [ "AutoGluon correctly recognized our prediction problem to be a **binary classification** task and decided that variables such as `age` should be represented as integers, whereas variables such as `workclass` should be represented as categorical objects. The `feature_metadata` attribute allows you to see the inferred data type of each predictive variable after preprocessing (this is its _raw_ dtype; some features may also be associated with additional _special_ dtypes if produced via feature-engineering, e.g. numerical representations of a datetime/text column)." ] }, { "cell_type": "markdown", "source": [ "To transform the data into AutoGluon's internal representation, we can do the following:" ], "metadata": { "collapsed": false }, "id": "27f0ef525a7db211" }, { "cell_type": "code", "execution_count": null, "source": [ "test_data_transform = predictor.transform_features(test_data)\n", "test_data_transform.head()" ], "metadata": { "collapsed": false }, "id": "addae3bd40b4318a", "outputs": [] }, { "cell_type": "markdown", "source": [ "Notice how the data is purely numeric after pre-processing (although categorical features will still be treated as categorical downstream).\n", "\n", "To better understand our trained predictor, we can estimate the overall importance of each feature via [TabularPredictor.feature_importance()](../../api/autogluon.tabular.TabularPredictor.feature_importance.rst):" ], "metadata": { "collapsed": false }, "id": "5a608ba782bef998" }, { "cell_type": "code", "execution_count": null, "source": [ "predictor.feature_importance(test_data)" ], "metadata": { "collapsed": false }, "id": "567ebed45b3ba83c", "outputs": [] }, { "cell_type": "markdown", "id": "ef4f97a2", "metadata": {}, "source": [ "The `importance` column is an estimate for the amount the evaluation metric score would drop if the feature were removed from the data.\n", "Negative values of `importance` mean that it is likely to improve the results if re-fit with the feature removed.\n", "\n", "When we call `predict()`, AutoGluon automatically predicts with the model that displayed the best performance on validation data (i.e. the weighted-ensemble)." ] }, { "cell_type": "code", "execution_count": null, "source": [ "predictor.model_best" ], "metadata": { "collapsed": false }, "id": "79066cd8f9a34ee8", "outputs": [] }, { "cell_type": "markdown", "source": [ "We can instead specify which model to use for predictions like this:" ], "metadata": { "collapsed": false }, "id": "fb0ca088eaf1e452" }, { "cell_type": "markdown", "id": "90aab2e2", "metadata": {}, "source": [ "```\n", "predictor.predict(test_data, model='LightGBM')\n", "```\n", "\n", "You can get the list of trained models via `.leaderboard()` or `.model_names()`:" ] }, { "cell_type": "code", "execution_count": null, "source": [ "predictor.model_names()" ], "metadata": { "collapsed": false }, "id": "6eaa4acf8afdf20a", "outputs": [] }, { "cell_type": "markdown", "id": "d0a30ee6", "metadata": {}, "source": [ "The scores of predictive performance above were based on a default evaluation metric (accuracy for binary classification). Performance in certain applications may be measured by different metrics than the ones AutoGluon optimizes for by default. If you know the metric that counts in your application, you should specify it via the `eval_metric` argument as demonstrated in the next section.\n", "\n", "## Presets\n", "\n", "AutoGluon comes with a variety of presets that can be specified in the call to `.fit` via the `presets` argument. `medium_quality` is used by default to encourage initial prototyping, but for serious usage, the other presets should be used instead.\n", "\n", "| Preset | Model Quality | Use Cases | Fit Time (Ideal) | Inference Time (Relative to medium_quality) | Disk Usage |\n", "| :------------- |:-------------------------------------------------------| :------------------------------------------------------------------------------------------------------------------------------------------------------ |:-----------------| :------------------------------------------ | :--------- |\n", "| best_quality | State-of-the-art (SOTA), much better than high_quality | When accuracy is what matters | 16x+ | 32x+ | 16x+ |\n", "| high_quality | Better than good_quality | When a very powerful, portable solution with fast inference is required: Large-scale batch inference | 16x+ | 4x | 2x |\n", "| good_quality | Stronger than any other AutoML Framework | When a powerful, highly portable solution with very fast inference is required: Billion-scale batch inference, sub-100ms online-inference, edge-devices | 16x | 2x | 0.1x |\n", "| medium_quality | Competitive with other top AutoML Frameworks | Initial prototyping, establishing a performance baseline | 1x | 1x | 1x |\n", "\n", "We recommend users to start with `medium_quality` to get a sense of the problem and identify any data related issues. If `medium_quality` is taking too long to train, consider subsampling the training data during this prototyping phase. \n", "Once you are comfortable, next try `best_quality`. Make sure to specify at least 16x the `time_limit` value as used in `medium_quality`. Once finished, you should have a very powerful solution that is often stronger than `medium_quality`. \n", "Make sure to consider holding out test data that AutoGluon never sees during training to ensure that the models are performing as expected in terms of performance. \n", "Once you evaluate both `best_quality` and `medium_quality`, check if either satisfies your needs. If neither do, consider trying `high_quality` and/or `good_quality`. \n", "If none of the presets satisfy requirements, refer to [Predicting Columns in a Table - In Depth](tabular-indepth.ipynb) for more advanced AutoGluon options.\n", "\n", "## Maximizing predictive performance\n", "\n", "**Note:** You should not call `fit()` with entirely default arguments if you are benchmarking AutoGluon-Tabular or hoping to maximize its accuracy!\n", "To get the best predictive accuracy with AutoGluon, you should generally use it like this:" ] }, { "cell_type": "code", "execution_count": null, "id": "358b121a", "metadata": { "tags": [ "hide-output" ] }, "source": [ "time_limit = 60 # for quick demonstration only, you should set this to longest time you are willing to wait (in seconds)\n", "metric = 'roc_auc' # specify your evaluation metric here\n", "predictor = TabularPredictor(label, eval_metric=metric).fit(train_data, time_limit=time_limit, presets='best_quality')" ], "outputs": [] }, { "cell_type": "code", "execution_count": null, "source": [ "predictor.leaderboard(test_data)" ], "metadata": { "collapsed": false }, "id": "b45474df26853911", "outputs": [] }, { "cell_type": "markdown", "id": "cf8a57a7", "metadata": {}, "source": [ "This command implements the following strategy to maximize accuracy:\n", "\n", "- Specify the argument `presets='best_quality'`, which allows AutoGluon to automatically construct powerful model ensembles based on [stacking/bagging](https://arxiv.org/abs/2003.06505), and will greatly improve the resulting predictions if granted sufficient training time. The default value of `presets` is `'medium_quality'`, which produces _less_ accurate models but facilitates faster prototyping. With `presets`, you can flexibly prioritize predictive accuracy vs. training/inference speed. For example, if you care less about predictive performance and want to quickly deploy a basic model, consider using: `presets=['good_quality', 'optimize_for_deployment']`.\n", "\n", "- Provide the parameter `eval_metric` to `TabularPredictor()` if you know what metric will be used to evaluate predictions in your application. Some other non-default metrics you might use include things like: `'f1'` (for binary classification), `'roc_auc'` (for binary classification), `'log_loss'` (for classification), `'mean_absolute_error'` (for regression), `'median_absolute_error'` (for regression). You can also define your own custom metric function. For more information refer to [Adding a custom metric to AutoGluon](advanced/tabular-custom-metric.ipynb).\n", "\n", "- Include all your data in `train_data` and do not provide `tuning_data` (AutoGluon will split the data more intelligently to fit its needs).\n", "\n", "- Do not specify the `hyperparameter_tune_kwargs` argument (counterintuitively, hyperparameter tuning is not the best way to spend a limited training time budgets, as model ensembling is often superior). We recommend you only use `hyperparameter_tune_kwargs` if your goal is to deploy a single model rather than an ensemble.\n", "\n", "- Do not specify the `hyperparameters` argument (allow AutoGluon to adaptively select which models/hyperparameters to use).\n", "\n", "- Set `time_limit` to the longest amount of time (in seconds) that you are willing to wait. AutoGluon's predictive performance improves the longer `fit()` is allowed to run.\n", "\n", "## Regression (predicting numeric table columns):\n", "\n", "To demonstrate that `fit()` can also automatically handle regression tasks, we now try to predict the numeric `age` variable in the same table based on the other features:" ] }, { "cell_type": "code", "execution_count": null, "id": "ce850e3e", "metadata": {}, "source": [ "age_column = 'age'\n", "train_data[age_column].head()" ], "outputs": [] }, { "cell_type": "markdown", "id": "3ba6bfd5", "metadata": {}, "source": [ "We again call `fit()`, imposing a time-limit this time (in seconds), and also demonstrate a shorthand method to evaluate the resulting model on the test data (which contain labels):" ] }, { "cell_type": "code", "execution_count": null, "id": "36e8f913", "metadata": { "tags": [ "hide-output" ] }, "source": [ "predictor_age = TabularPredictor(label=age_column, path=\"agModels-predictAge\").fit(train_data, time_limit=60)" ], "outputs": [] }, { "cell_type": "code", "execution_count": null, "source": [ "predictor_age.evaluate(test_data)" ], "metadata": { "collapsed": false }, "id": "d4564a06d1766c76", "outputs": [] }, { "cell_type": "markdown", "id": "46af4e18", "metadata": {}, "source": [ "Note that we didn't need to tell AutoGluon this is a regression problem, it automatically inferred this from the data and reported the appropriate performance metric (RMSE by default). To specify a particular evaluation metric other than the default, set the `eval_metric` parameter of [TabularPredictor()](../../api/autogluon.tabular.TabularPredictor.rst) and AutoGluon will tailor its models to optimize your metric (e.g. `eval_metric = 'mean_absolute_error'`). For evaluation metrics where higher values are worse (like RMSE), AutoGluon will flip their sign and print them as negative values during training (as it internally assumes higher values are better). You can even specify a custom metric by following the [Custom Metric Tutorial](advanced/tabular-custom-metric.ipynb).\n", "\n", "We can call leaderboard to see the per-model performance:" ] }, { "cell_type": "code", "execution_count": null, "id": "6a20746a", "metadata": {}, "source": [ "predictor_age.leaderboard(test_data)" ], "outputs": [] }, { "cell_type": "markdown", "id": "9d692ceb", "metadata": {}, "source": [ "**Data Formats:** AutoGluon can currently operate on data tables already loaded into Python as pandas DataFrames, or those stored in files of [CSV format](https://en.wikipedia.org/wiki/Comma-separated_values) or [Parquet format](https://databricks.com/glossary/what-is-parquet). If your data lives in multiple tables, you will first need to join them into a single table whose rows correspond to statistically independent observations (datapoints) and columns correspond to different features (aka. variables/covariates).\n", "\n", "Refer to the [TabularPredictor documentation](../../api/autogluon.tabular.TabularPredictor.rst) to see all of the available methods/options.\n", "\n", "## Advanced Usage\n", "\n", "For more advanced usage examples of AutoGluon, refer to the [In Depth Tutorial](tabular-indepth.ipynb)\n", "\n", "If you are interested in deployment optimization, refer to the [Deployment Optimization Tutorial](advanced/tabular-deployment.ipynb).\n", "\n", "For adding custom models to AutoGluon, refer to the [Custom Model](advanced/tabular-custom-model.ipynb) and [Custom Model Advanced](advanced/tabular-custom-model-advanced.ipynb) tutorials." ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }