Multimodal Prediction

For problems on multimodal data tables that contain image, text, and tabular data, AutoGluon provides MultiModalPredictor (abbreviated as AutoMM) that automatically selects, fuses, and tunes foundation models from popular packages like timm, huggingface/transformers, CLIP, MMDetection etc.

You can not only use AutoMM to solve standard NLP/Vision tasks such as sentiment classification, intent detection, paraphrase detection, image classification, but also use it for multimodal problems that involve image, text, tabular features, object bounding boxes, named entities, etc. Moreover, AutoMM can be used as a basic model in the multi-layer stack-ensemble of AutoGluon Tabular, and is powering up the FT-Transformer in TabularPredictor.

Here are some example use-cases of AutoMM:

• Multilingual text classification [Tutorial]

• Predicting pets’ popularity based on their description, photo, and other metadata. [Tutorial] [Example].

• Predicting the price of book. [Tutorial].

• Scoring student’s essays. [Example].

• Image classification. [Tutorial].

• Object detection. [Tutorial] [Example].

• Extracting named entities. [Tutorial].

• Search for relevant text / image via text queries. [Tutorial].

• Document Classification (Experimental). [Tutorial].

In the following, we decomposed the functionalities of AutoMM and prepared step-by-step guide for each functionality.

Text Data

AutoMM for Text Prediction - Quick Starttext_prediction/beginner_text.html

How to train high-quality text prediction models with MultiModalPredictor.

AutoMM for Text Prediction - Multilingual Problemstext_prediction/multilingual_text.html

How to use MultiModalPredictor to build models on datasets with languages other than English.

Named Entity Recognition with AutoMM - Quick Starttext_prediction/ner.html

How to use MultiModalPredictor for entity extraction.

Image Data – Classification / Regression

AutoMM for Image Classification - Quick Startimage_prediction/beginner_image_cls.html

How to train image classification models with MultiModalPredictor.

Zero-Shot Image Classification with CLIPimage_prediction/clip_zeroshot.html

How to enable zero-shot image classification in AutoMM via pretrained CLIP model.

Image Data – Object Detection

Quick Start on a Tiny COCO Format Datasetobject_detection/quick_start/quick_start_coco.html

How to train high quality object detection model with MultiModalPredictor in under 5 minutes on COCO format dataset.

Prepare COCO2017 Datasetobject_detection/data_preparation/prepare_coco17.html

How to prepare COCO2017 dataset for object detection.

Prepare Pascal VOC Datasetobject_detection/data_preparation/prepare_voc.html

How to prepare Pascal VOC dataset for object detection.

Prepare Watercolor Datasetobject_detection/data_preparation/prepare_watercolor.html

How to prepare Watercolor dataset for object detection.

Convert VOC Format Dataset to COCO Formatobject_detection/data_preparation/voc_to_coco.html

How to convert a dataset from VOC format to COCO format for object detection.

Object Detection with DataFrameobject_detection/data_preparation/object_detection_with_dataframe.html

How to use pd.DataFrame format for object detection

Fast Finetune on COCO Format Datasetobject_detection/finetune/detection_fast_finetune_coco.html

How to fast finetune a pretrained model on a dataset in COCO format.

High Performance Finetune on COCO Format Datasetobject_detection/finetune/detection_high_performance_finetune_coco.html

How to finetune a pretrained model on a dataset in COCO format with high performance.

Evaluate Pretrained YOLOv3 on COCO Format Datasetobject_detection/evaluation/detection_eval_yolov3_coco.html

How to evaluate the very fast pretrained YOLOv3 model on dataset in COCO format.

Evaluate Pretrained Faster R-CNN on COCO Format Datasetobject_detection/evaluation/detection_eval_fasterrcnn_coco.html

How to evaluate the pretrained Faster R-CNN model with high performance on dataset in COCO format.

Evaluate Pretrained Deformable DETR on COCO Format Datasetobject_detection/evaluation/detection_eval_ddetr_coco.html

How to evaluate the pretrained Deformable DETR model with higher performance on dataset in COCO format

Evaluate Pretrained Faster R-CNN on VOC Format Datasetobject_detection/evaluation/detection_eval_fasterrcnn_voc.html

How to evaluate the pretrained Faster R-CNN model on dataset in VOC format

Document Data

AutoMM for Scanned Document Classificationdocument/document_classification.html

How to use MultiModalPredictor to build a scanned document classifier.

Matching

Text-to-text Matching with AutoMM - Quick Startmatching/text2text_matching.html

How to use AutoMM for text to text matching.

Semantic Textual Search with AutoGluon Multimodal Matchingmatching/semantic_search.html

How to use semantic embeddings to improve search ranking performance.

Extract Image/Text Embeddings in AutoMM for Matching Problemsmatching/clip_embedding.html

How to use CLIP to extract embeddings for retrieval problem.

Multimodal Data

AutoMM for Text + Tabular - Quick Startmultimodal_prediction/multimodal_text_tabular.html

How MultiModalPredictor can be applied to multimodal data tables with a mix of text, numerical, and categorical columns. Here, we train a model to predict the price of books.

AutoMM for Image + Text + Tabular - Quick Startmultimodal_prediction/beginner_multimodal.html

How to use MultiModalPredictor to train a model that predicts the adoption speed of pets.

AutoMM for Entity Extraction with Text and Image - Quick Startmultimodal_prediction/multimodal_ner.html

How to use MultiModalPredictor to train a model for multimodal named entity recognition.

Advanced Topics

Single GPU Billion-scale Model Training via Parameter-Efficient Finetuningadvanced_topics/efficient_finetuning_basic.html

How to take advantage of larger foundation models with the help of parameter-efficient finetuning. In the tutorial, we will use combine IA^3, BitFit, and gradient checkpointing to finetune FLAN-T5-XL.

HPO in AutoMMadvanced_topics/hyperparameter_optimization.html

How to do hyperparameter optimization in AutoMM.

Knowledge Distillation in AutoMMadvanced_topics/model_distillation.html

How to do knowledge distillation in AutoMM.

Customize AutoMMadvanced_topics/customization.html

How to customize AutoMM configurations.

AutoMM Presetsadvanced_topics/presets.html

How to use AutoMM presets.

Few Shot Learning with FusionSVMadvanced_topics/few_shot_learning.html

How to use SVM combined with feature extraction for few shot learning.

Focal Lossadvanced_topics/focal_loss.html

How to use focal loss in AutoMM.

FAQ

Frequently Asked Questionsadvanced_topics/multimodal_faq.html

Frequently Asked Questions