.. _sec_tabularprediction_multimodal: Multimodal Data Tables: Tabular, Text, and Image ================================================ **Tip**: Prior to reading this tutorial, it is recommended to have a basic understanding of the TabularPredictor API covered in :ref:`sec_tabularquick`. In this tutorial, we will train a multi-modal ensemble using data that contains image, text, and tabular features. Note: A GPU is required for this tutorial in order to train the image and text models. Additionally, GPU installations are required for MXNet and Torch with appropriate CUDA versions. The PetFinder Dataset --------------------- We will be using the `PetFinder dataset `__. The PetFinder dataset provides information about shelter animals that appear on their adoption profile with the goal to predict the adoption rate of the animal. The end goal is for rescue shelters to use the predicted adoption rate to identify animals whose profiles could be improved so that they can find a home. Each animal’s adoption profile contains a variety of information, such as pictures of the animal, a text description of the animal, and various tabular features such as age, breed, name, color, and more. To get started, we first need to download the dataset. Datasets that contain images require more than a CSV file, so the dataset is packaged in a zip file in S3. We will first download it and unzip the contents: .. code:: python download_dir = './ag_petfinder_tutorial' zip_file = 'https://automl-mm-bench.s3.amazonaws.com/petfinder_kaggle.zip' .. code:: python from autogluon.core.utils.loaders import load_zip load_zip.unzip(zip_file, unzip_dir=download_dir) .. parsed-literal:: :class: output Downloading ./ag_petfinder_tutorial/file.zip from https://automl-mm-bench.s3.amazonaws.com/petfinder_kaggle.zip... .. parsed-literal:: :class: output 100%|██████████| 2.00G/2.00G [00:43<00:00, 45.5MiB/s] Now that the data is download and unzipped, let’s take a look at the contents: .. code:: python import os os.listdir(download_dir) .. parsed-literal:: :class: output ['file.zip', 'petfinder_processed'] ‘file.zip’ is the original zip file we downloaded, and ‘petfinder_processed’ is a directory containing the dataset files. .. code:: python dataset_path = download_dir + '/petfinder_processed' os.listdir(dataset_path) .. parsed-literal:: :class: output ['train.csv', 'train_images', 'test.csv', 'test_images', 'dev.csv'] Here we can see the train, test, and dev CSV files, as well as two directories: ‘test_images’ and ‘train_images’ which contain the image JPG files. Note: We will be using the dev data as testing data as dev contains the ground truth labels for showing scores via ``predictor.leaderboard``. Let’s take a peek at the first 10 files inside of the ‘train_images’ directory: .. code:: python os.listdir(dataset_path + '/train_images')[:10] .. parsed-literal:: :class: output ['d765ae877-1.jpg', '756025f7c-2.jpg', 'e1a2d9477-4.jpg', '6d18707ee-2.jpg', '96607bca0-5.jpg', 'fde58f7fa-10.jpg', 'be7b65c23-3.jpg', 'dd36ab692-3.jpg', '2d8db1c19-2.jpg', '53037f091-2.jpg'] As expected, these are the images we will be training with alongside the other features. Next, we will load the train and dev CSV files: .. code:: python import pandas as pd train_data = pd.read_csv(f'{dataset_path}/train.csv', index_col=0) test_data = pd.read_csv(f'{dataset_path}/dev.csv', index_col=0) .. code:: python train_data.head(3) .. raw:: html
Type Name Age Breed1 Breed2 Gender Color1 Color2 Color3 MaturitySize ... Quantity Fee State RescuerID VideoAmt Description PetID PhotoAmt AdoptionSpeed Images
10721 1 Elbi 2 307 307 2 5 0 0 3 ... 1 0 41336 e9a86209c54f589ba72c345364cf01aa 0 I'm looking for people to adopt my dog e4b90955c 4.0 4 train_images/e4b90955c-1.jpg;train_images/e4b9...
13114 2 Darling 4 266 0 1 1 0 0 2 ... 1 0 41401 01f954cdf61526daf3fbeb8a074be742 0 Darling was born at the back lane of Jalan Alo... a0c1384d1 5.0 3 train_images/a0c1384d1-1.jpg;train_images/a0c1...
13194 1 Wolf 3 307 0 1 1 2 0 2 ... 1 0 41332 6e19409f2847326ce3b6d0cec7e42f81 0 I found Wolf about a month ago stuck in a drai... cf357f057 7.0 4 train_images/cf357f057-1.jpg;train_images/cf35...

3 rows × 25 columns

Looking at the first 3 examples, we can tell that there is a variety of tabular features, a text description (‘Description’), and an image path (‘Images’). For the PetFinder dataset, we will try to predict the speed of adoption for the animal (‘AdoptionSpeed’), grouped into 5 categories. This means that we are dealing with a multi-class classification problem. .. code:: python label = 'AdoptionSpeed' image_col = 'Images' Preparing the image column -------------------------- Let’s take a look at what a value in the image column looks like: .. code:: python train_data[image_col].iloc[0] .. parsed-literal:: :class: output 'train_images/e4b90955c-1.jpg;train_images/e4b90955c-2.jpg;train_images/e4b90955c-3.jpg;train_images/e4b90955c-4.jpg' Currently, AutoGluon only supports one image per row. Since the PetFinder dataset contains one or more images per row, we first need to preprocess the image column to only contain the first image of each row. .. code:: python train_data[image_col] = train_data[image_col].apply(lambda ele: ele.split(';')[0]) test_data[image_col] = test_data[image_col].apply(lambda ele: ele.split(';')[0]) train_data[image_col].iloc[0] .. parsed-literal:: :class: output 'train_images/e4b90955c-1.jpg' AutoGluon loads images based on the file path provided by the image column. Here we update the path to point to the correct location on disk: .. code:: python def path_expander(path, base_folder): path_l = path.split(';') return ';'.join([os.path.abspath(os.path.join(base_folder, path)) for path in path_l]) train_data[image_col] = train_data[image_col].apply(lambda ele: path_expander(ele, base_folder=dataset_path)) test_data[image_col] = test_data[image_col].apply(lambda ele: path_expander(ele, base_folder=dataset_path)) train_data[image_col].iloc[0] .. parsed-literal:: :class: output '/home/ci/autogluon/docs/_build/eval/tutorials/tabular_prediction/ag_petfinder_tutorial/petfinder_processed/train_images/e4b90955c-1.jpg' .. code:: python train_data.head(3) .. raw:: html
Type Name Age Breed1 Breed2 Gender Color1 Color2 Color3 MaturitySize ... Quantity Fee State RescuerID VideoAmt Description PetID PhotoAmt AdoptionSpeed Images
10721 1 Elbi 2 307 307 2 5 0 0 3 ... 1 0 41336 e9a86209c54f589ba72c345364cf01aa 0 I'm looking for people to adopt my dog e4b90955c 4.0 4 /home/ci/autogluon/docs/_build/eval/tutorials/...
13114 2 Darling 4 266 0 1 1 0 0 2 ... 1 0 41401 01f954cdf61526daf3fbeb8a074be742 0 Darling was born at the back lane of Jalan Alo... a0c1384d1 5.0 3 /home/ci/autogluon/docs/_build/eval/tutorials/...
13194 1 Wolf 3 307 0 1 1 2 0 2 ... 1 0 41332 6e19409f2847326ce3b6d0cec7e42f81 0 I found Wolf about a month ago stuck in a drai... cf357f057 7.0 4 /home/ci/autogluon/docs/_build/eval/tutorials/...

3 rows × 25 columns

Analyzing an example row ------------------------ Now that we have preprocessed the image column, let’s take a look at an example row of data and display the text description and the picture. .. code:: python example_row = train_data.iloc[1] example_row .. parsed-literal:: :class: output Type 2 Name Darling Age 4 Breed1 266 Breed2 0 Gender 1 Color1 1 Color2 0 Color3 0 MaturitySize 2 FurLength 1 Vaccinated 2 Dewormed 2 Sterilized 2 Health 1 Quantity 1 Fee 0 State 41401 RescuerID 01f954cdf61526daf3fbeb8a074be742 VideoAmt 0 Description Darling was born at the back lane of Jalan Alo... PetID a0c1384d1 PhotoAmt 5.0 AdoptionSpeed 3 Images /home/ci/autogluon/docs/_build/eval/tutorials/... Name: 13114, dtype: object .. code:: python example_row['Description'] .. parsed-literal:: :class: output 'Darling was born at the back lane of Jalan Alor and was foster by a feeder. All his siblings had died of accident. His mother and grandmother had just been spayed. Darling make a great condo/apartment cat. He love to play a lot. He would make a great companion for someone looking for a cat to love.' .. code:: python example_image = example_row['Images'] from IPython.display import Image, display pil_img = Image(filename=example_image) display(pil_img) .. figure:: output_tabular-multimodal_e625cb_24_0.jpg The PetFinder dataset is fairly large. For the purposes of the tutorial, we will sample 300 rows for training. Training on large multi-modal datasets can be very computationally intensive, especially if using the ``best_quality`` preset in AutoGluon. When prototyping, it is recommended to sample your data to get an idea of which models are worth training, then gradually train with larger amounts of data and longer time limits as you would with any other machine learning algorithm. .. code:: python train_data = train_data.sample(300, random_state=0) Constructing the FeatureMetadata -------------------------------- Next, let’s see what AutoGluon infers the feature types to be by constructing a FeatureMetadata object from the training data: .. code:: python from autogluon.tabular import FeatureMetadata feature_metadata = FeatureMetadata.from_df(train_data) print(feature_metadata) .. parsed-literal:: :class: output ('float', []) : 1 | ['PhotoAmt'] ('int', []) : 19 | ['Type', 'Age', 'Breed1', 'Breed2', 'Gender', ...] ('object', []) : 4 | ['Name', 'RescuerID', 'PetID', 'Images'] ('object', ['text']) : 1 | ['Description'] Notice that FeatureMetadata automatically identified the column ‘Description’ as text, so we don’t need to manually specify that it is text. In order to leverage images, we need to tell AutoGluon which column contains the image path. We can do this by specifying a FeatureMetadata object and adding the ‘image_path’ special type to the image column. We later pass this custom FeatureMetadata to TabularPredictor.fit. .. code:: python feature_metadata = feature_metadata.add_special_types({image_col: ['image_path']}) print(feature_metadata) .. parsed-literal:: :class: output ('float', []) : 1 | ['PhotoAmt'] ('int', []) : 19 | ['Type', 'Age', 'Breed1', 'Breed2', 'Gender', ...] ('object', []) : 3 | ['Name', 'RescuerID', 'PetID'] ('object', ['image_path']) : 1 | ['Images'] ('object', ['text']) : 1 | ['Description'] Specifying the hyperparameters ------------------------------ Next, we need to specify the models we want to train with. This is done via the ``hyperparameters`` argument to TabularPredictor.fit. AutoGluon has a predefined config that works well for multimodal datasets called ‘multimodal’. We can access it via: .. code:: python from autogluon.tabular.configs.hyperparameter_configs import get_hyperparameter_config hyperparameters = get_hyperparameter_config('multimodal') hyperparameters .. parsed-literal:: :class: output {'NN_TORCH': {}, 'GBM': [{}, {'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, 'GBMLarge'], 'CAT': {}, 'XGB': {}, 'AG_AUTOMM': {}, 'VW': {}} This hyperparameter config will train a variety of Tabular models as well as finetune an Electra BERT text model, and a ResNet image model. Fitting with TabularPredictor ----------------------------- Now we will train a TabularPredictor on the dataset, using the feature metadata and hyperparameters we defined prior. This TabularPredictor will leverage tabular, text, and image features all at once. .. code:: python from autogluon.tabular import TabularPredictor predictor = TabularPredictor(label=label).fit( train_data=train_data, hyperparameters=hyperparameters, feature_metadata=feature_metadata, time_limit=600, ) .. parsed-literal:: :class: output No path specified. Models will be saved in: "AutogluonModels/ag-20230306_121043/" Beginning AutoGluon training ... Time limit = 600s AutoGluon will save models to "AutogluonModels/ag-20230306_121043/" AutoGluon Version: 0.7.0b20230306 Python Version: 3.8.13 Operating System: Linux Platform Machine: x86_64 Platform Version: #1 SMP Tue Nov 30 00:17:50 UTC 2021 Train Data Rows: 300 Train Data Columns: 24 Label Column: AdoptionSpeed Preprocessing data ... AutoGluon infers your prediction problem is: 'multiclass' (because dtype of label-column == int, but few unique label-values observed). 5 unique label values: [2, 3, 4, 0, 1] If 'multiclass' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression']) Warning: Updated label_count_threshold from 10 to 8 to avoid cutting too many classes. Train Data Class Count: 5 Using Feature Generators to preprocess the data ... Fitting AutoMLPipelineFeatureGenerator... Available Memory: 31476.37 MB Train Data (Original) Memory Usage: 0.3 MB (0.0% of available memory) Stage 1 Generators: Fitting AsTypeFeatureGenerator... Note: Converting 2 features to boolean dtype as they only contain 2 unique values. Stage 2 Generators: Fitting FillNaFeatureGenerator... Stage 3 Generators: Fitting IdentityFeatureGenerator... Fitting IdentityFeatureGenerator... Fitting RenameFeatureGenerator... Fitting CategoryFeatureGenerator... Fitting CategoryMemoryMinimizeFeatureGenerator... Fitting TextSpecialFeatureGenerator... Fitting BinnedFeatureGenerator... Fitting DropDuplicatesFeatureGenerator... Fitting TextNgramFeatureGenerator... Fitting CountVectorizer for text features: ['Description'] CountVectorizer fit with vocabulary size = 96 Fitting IdentityFeatureGenerator... Fitting IsNanFeatureGenerator... Stage 4 Generators: Fitting DropUniqueFeatureGenerator... Unused Original Features (Count: 1): ['PetID'] These features were not used to generate any of the output features. Add a feature generator compatible with these features to utilize them. Features can also be unused if they carry very little information, such as being categorical but having almost entirely unique values or being duplicates of other features. These features do not need to be present at inference time. ('object', []) : 1 | ['PetID'] Types of features in original data (raw dtype, special dtypes): ('float', []) : 1 | ['PhotoAmt'] ('int', []) : 18 | ['Type', 'Age', 'Breed1', 'Breed2', 'Gender', ...] ('object', []) : 2 | ['Name', 'RescuerID'] ('object', ['image_path']) : 1 | ['Images'] ('object', ['text']) : 1 | ['Description'] Types of features in processed data (raw dtype, special dtypes): ('category', []) : 2 | ['Name', 'RescuerID'] ('category', ['text_as_category']) : 1 | ['Description'] ('float', []) : 1 | ['PhotoAmt'] ('int', []) : 16 | ['Age', 'Breed1', 'Breed2', 'Gender', 'Color1', ...] ('int', ['binned', 'text_special']) : 20 | ['Description.char_count', 'Description.word_count', 'Description.capital_ratio', 'Description.lower_ratio', 'Description.digit_ratio', ...] ('int', ['bool']) : 2 | ['Type', 'VideoAmt'] ('int', ['text_ngram']) : 97 | ['__nlp__.active', '__nlp__.adopt', '__nlp__.adoption', '__nlp__.adorable', '__nlp__.all', ...] ('object', ['image_path']) : 1 | ['Images'] ('object', ['text']) : 1 | ['Description_raw_text'] 0.3s = Fit runtime 23 features in original data used to generate 141 features in processed data. Train Data (Processed) Memory Usage: 0.29 MB (0.0% of available memory) Data preprocessing and feature engineering runtime = 0.35s ... AutoGluon will gauge predictive performance using evaluation metric: 'accuracy' To change this, specify the eval_metric parameter of Predictor() Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 240, Val Rows: 60 Fitting 8 L1 models ... Fitting model: LightGBM ... Training model for up to 599.65s of the 599.65s of remaining time. 0.4 = Validation score (accuracy) 1.15s = Training runtime 0.01s = Validation runtime Fitting model: LightGBMXT ... Training model for up to 598.48s of the 598.48s of remaining time. 0.4 = Validation score (accuracy) 1.32s = Training runtime 0.01s = Validation runtime Fitting model: CatBoost ... Training model for up to 597.09s of the 597.09s of remaining time. 0.4167 = Validation score (accuracy) 4.14s = Training runtime 0.01s = Validation runtime Fitting model: XGBoost ... Training model for up to 592.94s of the 592.94s of remaining time. 0.3667 = Validation score (accuracy) 1.85s = Training runtime 0.01s = Validation runtime Fitting model: NeuralNetTorch ... Training model for up to 591.06s of the 591.05s of remaining time. 0.3167 = Validation score (accuracy) 1.66s = Training runtime 0.02s = Validation runtime Fitting model: VowpalWabbit ... Training model for up to 589.37s of the 589.37s of remaining time. 0.25 = Validation score (accuracy) 0.42s = Training runtime 0.02s = Validation runtime Fitting model: LightGBMLarge ... Training model for up to 588.69s of the 588.69s of remaining time. 0.3833 = Validation score (accuracy) 1.17s = Training runtime 0.01s = Validation runtime Fitting model: MultiModalPredictor ... Training model for up to 587.5s of the 587.5s of remaining time. /home/ci/opt/venv/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Downloading: "https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_base_patch4_window7_224_22kto1k.pth" to /home/ci/.cache/torch/hub/checkpoints/swin_base_patch4_window7_224_22kto1k.pth Configuration saved in AutogluonModels/ag-20230306_121043/models/MultiModalPredictor/automm_model/hf_text/config.json tokenizer config file saved in AutogluonModels/ag-20230306_121043/models/MultiModalPredictor/automm_model/hf_text/tokenizer_config.json Special tokens file saved in AutogluonModels/ag-20230306_121043/models/MultiModalPredictor/automm_model/hf_text/special_tokens_map.json 0.3167 = Validation score (accuracy) 336.83s = Training runtime 2.75s = Validation runtime Fitting model: WeightedEnsemble_L2 ... Training model for up to 360.0s of the 247.81s of remaining time. 0.4167 = Validation score (accuracy) 0.2s = Training runtime 0.0s = Validation runtime AutoGluon training complete, total runtime = 352.4s ... Best model: "WeightedEnsemble_L2" TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20230306_121043/") After the predictor is fit, we can take a look at the leaderboard and see the performance of the various models: .. code:: python leaderboard = predictor.leaderboard(test_data) .. parsed-literal:: :class: output loading file vocab.txt loading file tokenizer.json loading file added_tokens.json loading file special_tokens_map.json loading file tokenizer_config.json loading configuration file /home/ci/autogluon/docs/_build/eval/tutorials/tabular_prediction/AutogluonModels/ag-20230306_121043/models/MultiModalPredictor/automm_model/hf_text/config.json Model config ElectraConfig { "_name_or_path": "/home/ci/autogluon/docs/_build/eval/tutorials/tabular_prediction/AutogluonModels/ag-20230306_121043/models/MultiModalPredictor/automm_model/hf_text", "architectures": [ "ElectraForPreTraining" ], "attention_probs_dropout_prob": 0.1, "classifier_dropout": null, "embedding_size": 768, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-12, "max_position_embeddings": 512, "model_type": "electra", "num_attention_heads": 12, "num_hidden_layers": 12, "pad_token_id": 0, "position_embedding_type": "absolute", "summary_activation": "gelu", "summary_last_dropout": 0.1, "summary_type": "first", "summary_use_proj": true, "transformers_version": "4.26.1", "type_vocab_size": 2, "use_cache": true, "vocab_size": 30522 } loading file vocab.txt loading file tokenizer.json loading file added_tokens.json loading file special_tokens_map.json loading file tokenizer_config.json Load pretrained checkpoint: /home/ci/autogluon/docs/_build/eval/tutorials/tabular_prediction/AutogluonModels/ag-20230306_121043/models/MultiModalPredictor/automm_model/model.ckpt Token indices sequence length is longer than the specified maximum sequence length for this model (587 > 512). Running this sequence through the model will result in indexing errors Token indices sequence length is longer than the specified maximum sequence length for this model (1394 > 512). Running this sequence through the model will result in indexing errors .. parsed-literal:: :class: output model score_test score_val pred_time_test pred_time_val fit_time pred_time_test_marginal pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order 0 MultiModalPredictor 0.315105 0.316667 46.225593 2.754488 336.832549 46.225593 2.754488 336.832549 1 True 8 1 XGBoost 0.299433 0.366667 0.123081 0.006268 1.849957 0.123081 0.006268 1.849957 1 True 4 2 LightGBM 0.296099 0.400000 0.035668 0.005602 1.152767 0.035668 0.005602 1.152767 1 True 1 3 NeuralNetTorch 0.292431 0.316667 0.050611 0.019907 1.663402 0.050611 0.019907 1.663402 1 True 5 4 CatBoost 0.291764 0.416667 0.022264 0.009497 4.137492 0.022264 0.009497 4.137492 1 True 3 5 WeightedEnsemble_L2 0.291764 0.416667 0.025394 0.010112 4.334439 0.003131 0.000615 0.196947 2 True 9 6 LightGBMXT 0.291430 0.400000 0.172012 0.007753 1.324705 0.172012 0.007753 1.324705 1 True 2 7 LightGBMLarge 0.275425 0.383333 0.352103 0.005250 1.171773 0.352103 0.005250 1.171773 1 True 7 8 VowpalWabbit 0.260754 0.250000 0.708939 0.024707 0.415922 0.708939 0.024707 0.415922 1 True 6 That’s all it takes to train with image, text, and tabular data (at the same time) using AutoGluon! For an in-depth tutorial on text + tabular multimodal functionality, refer to :ref:`sec_tabularprediction_text_multimodal`. For more tutorials, refer to :ref:`sec_tabularquick` and :ref:`sec_tabularadvanced`.