AutoMM Problem Types And Metrics

Open In Colab Open In SageMaker Studio Lab

AutoGluon Multimodal supports various problem types for different machine learning tasks. In this tutorial, we will introduce each problem type, its supported modalities, and evaluation metrics.

import warnings

warnings.filterwarnings('ignore')

Lets first write a helper function to print problem type information in a formatted way.

from autogluon.multimodal.constants import *
from autogluon.multimodal.problem_types import PROBLEM_TYPES_REG

def print_problem_type_info(name: str, props):
    """Helper function to print problem type information"""
    print(f"\n=== {name} ===")
    
    print("\nSupported Input Modalities:")
    # Convert set to sorted list for complete display
    for modality in sorted(list(props.supported_modality_type)):
        print(f"- {modality}")
        
    if hasattr(props, 'supported_evaluation_metrics') and props.supported_evaluation_metrics:
        print("\nEvaluation Metrics:")
        # Convert to sorted list to ensure complete and consistent display
        for metric in sorted(list(props.supported_evaluation_metrics)):
            if metric == props.fallback_evaluation_metric:
                print(f"- {metric} (default)")
            else:
                print(f"- {metric}")
                
    if hasattr(props, 'support_zero_shot'):
        print("\nSpecial Capabilities:")
        print(f"- Zero-shot prediction: {'Supported' if props.support_zero_shot else 'Not supported'}")
        print(f"- Training support: {'Supported' if props.support_fit else 'Not supported'}")

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[3], line 2
      1 from autogluon.multimodal.constants import *
----> 2 from autogluon.multimodal.problem_types import PROBLEM_TYPES_REG
      4 def print_problem_type_info(name: str, props):
      5     """Helper function to print problem type information"""

ModuleNotFoundError: No module named 'autogluon.multimodal.problem_types'

Classification

AutoGluon supports two types of classification:

  • Binary Classification (2 classes)

  • Multiclass Classification (3+ classes)

# Classification
binary_props = PROBLEM_TYPES_REG.get(BINARY)
multiclass_props = PROBLEM_TYPES_REG.get(MULTICLASS)
print_problem_type_info("Binary Classification", binary_props)
print_problem_type_info("Multiclass Classification", multiclass_props)

=== Binary Classification ===

Supported Input Modalities:
- categorical
- image
- image_base64_str
- image_bytearray
- numerical
- text

Evaluation Metrics:
- acc
- accuracy
- average_precision
- balanced_accuracy
- f1
- f1_macro
- f1_micro
- f1_weighted
- log_loss
- mcc
- nll
- pac
- pac_score
- precision
- precision_macro
- precision_micro
- precision_weighted
- quadratic_kappa
- recall
- recall_macro
- recall_micro
- recall_weighted
- roc_auc (default)

Special Capabilities:
- Zero-shot prediction: Not supported
- Training support: Supported

=== Multiclass Classification ===

Supported Input Modalities:
- categorical
- image
- image_base64_str
- image_bytearray
- numerical
- text

Evaluation Metrics:
- acc
- accuracy (default)
- balanced_accuracy
- f1_macro
- f1_micro
- f1_weighted
- log_loss
- mcc
- nll
- pac
- pac_score
- precision_macro
- precision_micro
- precision_weighted
- quadratic_kappa
- recall_macro
- recall_micro
- recall_weighted
- roc_auc_ovo
- roc_auc_ovo_macro
- roc_auc_ovo_weighted
- roc_auc_ovr
- roc_auc_ovr_macro
- roc_auc_ovr_micro
- roc_auc_ovr_weighted

Special Capabilities:
- Zero-shot prediction: Not supported
- Training support: Supported

Regression

Regression problems support predicting numerical values from various input modalities.

# Regression
regression_props = PROBLEM_TYPES_REG.get(REGRESSION)
print_problem_type_info("Regression", regression_props)

=== Regression ===

Supported Input Modalities:
- categorical
- image
- image_base64_str
- image_bytearray
- numerical
- text

Evaluation Metrics:
- mae
- mape
- mean_absolute_error
- mean_absolute_percentage_error
- mean_squared_error
- median_absolute_error
- mse
- pearsonr
- r2
- rmse (default)
- root_mean_squared_error
- smape
- spearmanr
- symmetric_mean_absolute_percentage_error

Special Capabilities:
- Zero-shot prediction: Not supported
- Training support: Supported

Object Detection

Object detection identifies and localizes objects in images using bounding boxes.

# Object Detection
object_detection_props = PROBLEM_TYPES_REG.get(OBJECT_DETECTION)
print_problem_type_info("Object Detection", object_detection_props)

=== Object Detection ===

Supported Input Modalities:
- image

Evaluation Metrics:
- map (default)
- map_50
- map_75
- map_large
- map_medium
- map_small
- mar_1
- mar_10
- mar_100
- mar_large
- mar_medium
- mar_small
- mean_average_precision

Special Capabilities:
- Zero-shot prediction: Supported
- Training support: Supported

Semantic Segmentation

Semantic segmentation performs pixel-level classification of images.

# Semantic Segmentation
segmentation_props = PROBLEM_TYPES_REG.get(SEMANTIC_SEGMENTATION)
print_problem_type_info("Semantic Segmentation", segmentation_props)

=== Semantic Segmentation ===

Supported Input Modalities:
- image

Evaluation Metrics:
- ber
- iou (default)
- sm

Special Capabilities:
- Zero-shot prediction: Supported
- Training support: Supported

Similarity Matching Problems

AutoGluon supports three types of similarity matching:

  • Text-to-Text Similarity

  • Image-to-Image Similarity

  • Image-to-Text Similarity

Check Matching Tutorials for more details

similarity_types = [
    (TEXT_SIMILARITY, "Text Similarity"),
    (IMAGE_SIMILARITY, "Image Similarity"),
    (IMAGE_TEXT_SIMILARITY, "Image-Text Similarity")
]

print("\n=== Similarity Matching ===")
for type_key, type_name in similarity_types:
    props = PROBLEM_TYPES_REG.get(type_key)
    print(f"\n{type_name}:")
    print("Input Requirements:")
    for modality in props.supported_modality_type:
        print(f"- {modality}")
    print(f"Zero-shot prediction: {'Supported' if props.support_zero_shot else 'Not supported'}")

=== Similarity Matching ===

Text Similarity:
Input Requirements:
- text
Zero-shot prediction: Supported

Image Similarity:
Input Requirements:
- image
Zero-shot prediction: Supported

Image-Text Similarity:
Input Requirements:
- text
- image
Zero-shot prediction: Supported

Named Entity Recognition (NER)

NER identifies and classifies named entities (like person names, locations, organizations) in text.

# Named Entity Recognition
ner_props = PROBLEM_TYPES_REG.get(NER)
print_problem_type_info("Named Entity Recognition", ner_props)

=== Named Entity Recognition ===

Supported Input Modalities:
- categorical
- image
- numerical
- text
- text_ner

Evaluation Metrics:
- ner_token_f1
- overall_f1 (default)

Special Capabilities:
- Zero-shot prediction: Not supported
- Training support: Supported

Feature Extraction

Feature extraction converts raw data into meaningful feature vector.

# Feature Extraction
feature_extraction_props = PROBLEM_TYPES_REG.get(FEATURE_EXTRACTION)
print_problem_type_info("Feature Extraction", feature_extraction_props)

=== Feature Extraction ===

Supported Input Modalities:
- image
- text

Special Capabilities:
- Zero-shot prediction: Supported
- Training support: Not supported

Few-shot Classification

Few-shot classification learns to classify from a small number of examples per class.

# Few-shot Classification
few_shot_props = PROBLEM_TYPES_REG.get(FEW_SHOT_CLASSIFICATION)
print_problem_type_info("Few-shot Classification", few_shot_props)

=== Few-shot Classification ===

Supported Input Modalities:
- image
- text

Evaluation Metrics:
- acc
- accuracy (default)
- balanced_accuracy
- f1_macro
- f1_micro
- f1_weighted
- log_loss
- mcc
- nll
- pac
- pac_score
- precision_macro
- precision_micro
- precision_weighted
- quadratic_kappa
- recall_macro
- recall_micro
- recall_weighted
- roc_auc_ovo
- roc_auc_ovo_macro
- roc_auc_ovo_weighted
- roc_auc_ovr
- roc_auc_ovr_macro
- roc_auc_ovr_micro
- roc_auc_ovr_weighted

Special Capabilities:
- Zero-shot prediction: Not supported
- Training support: Supported

Other Examples

You may go to AutoMM Examples to explore other examples about AutoMM.

Customization

To learn how to customize AutoMM, please refer to Customize AutoMM.