{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d53b656e-aa3b-433d-8131-02083a985090",
   "metadata": {},
   "source": [
    "# AutoMM Presets\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/autogluon/autogluon/blob/master/docs/tutorials/multimodal/advanced_topics/presets.ipynb)\n",
    "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/autogluon/autogluon/blob/master/docs/tutorials/multimodal/advanced_topics/presets.ipynb)\n",
    "\n",
    "\n",
    "It is well-known that we usually need to set hyperparameters before the learning process begins. Deep learning models, e.g., pretrained foundation models, can have anywhere from a few hyperparameters to a few hundred. The hyperparameters can impact training speed, final model performance, and inference latency. However, choosing the proper hyperparameters may be challenging for many users with limited expertise.\n",
    "\n",
    "In this tutorial, we will introduce the easy-to-use presets in AutoMM. Our presets can condense the complex hyperparameter setups into simple strings. More specifically, AutoMM supports three presets: `medium_quality`, `high_quality`, and `best_quality`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa00faab-252f-44c9-b8f7-57131aa8251c",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "!pip install autogluon.multimodal\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b02ccb8d-7f3f-4152-a526-ddcbf90e1a1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55496174-9331-484c-aafe-ce0659123def",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "For demonstration, we use a subsampled Stanford Sentiment Treebank ([SST](https://nlp.stanford.edu/sentiment/)) dataset, which consists of movie reviews and their associated sentiment. \n",
    "Given a new movie review, the goal is to predict the sentiment reflected in the text (in this case, a **binary classification**, where reviews are \n",
    "labeled as 1 if they conveyed a positive opinion and 0 otherwise).\n",
    "To get started, let's download and prepare the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4d9f14f-7948-4679-ad4c-3413d8f18e4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from autogluon.core.utils.loaders import load_pd\n",
    "\n",
    "train_data = load_pd.load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/train.parquet')\n",
    "test_data = load_pd.load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/dev.parquet')\n",
    "subsample_size = 1000  # subsample data for faster demo, try setting this to larger values\n",
    "train_data = train_data.sample(n=subsample_size, random_state=0)\n",
    "train_data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "016851a2-929a-478a-95d3-e3bda3ba2abb",
   "metadata": {},
   "source": [
    "## Medium Quality\n",
    "In some situations, we prefer fast training and inference to the prediction quality. `medium_quality` is designed for this purpose.\n",
    "Among the three presets, `medium_quality` has the smallest model size. Now let's fit the predictor using the `medium_quality` preset. Here we set a tight time budget for a quick demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa28985d-0f12-4e73-9b43-03b40b8fb543",
   "metadata": {},
   "outputs": [],
   "source": [
    "from autogluon.multimodal import MultiModalPredictor\n",
    "\n",
    "predictor = MultiModalPredictor(label='label', eval_metric='acc', presets=\"medium_quality\")\n",
    "predictor.fit(\n",
    "    train_data=train_data,\n",
    "    time_limit=20, # seconds\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48910313-b3f4-4b7c-a2b8-b84b701ed791",
   "metadata": {},
   "source": [
    "Then we can evaluate the predictor on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44726017-f55a-498f-b7f8-e0e5d9d0095f",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = predictor.evaluate(test_data, metrics=[\"roc_auc\"])\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4848b7bc-5478-4db8-9e88-f045c6e2080c",
   "metadata": {},
   "source": [
    "## High Quality\n",
    "If you want to balance the prediction quality and training/inference speed, you can try the `high_quality` preset, which uses a larger model than `medium_quality`. Accordingly, we need to increase the time limit since larger models require more time to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7792727-c0cb-4bff-8734-da4114709ccc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from autogluon.multimodal import MultiModalPredictor\n",
    "\n",
    "predictor = MultiModalPredictor(label='label', eval_metric='acc', presets=\"high_quality\")\n",
    "predictor.fit(\n",
    "    train_data=train_data,\n",
    "    time_limit=20, # seconds\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db2d0477-d379-4fd5-9377-d2f208212067",
   "metadata": {},
   "source": [
    "Although `high_quality` requires more training time than `medium_quality`, it also brings performance gains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee39944e-291a-40dd-a76c-3459d89ee781",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = predictor.evaluate(test_data, metrics=[\"roc_auc\"])\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "172e85e5-8105-406b-a720-2ccbb520fe80",
   "metadata": {},
   "source": [
    "## Best Quality\n",
    "If you want the best performance and don't care about the training/inference cost, give it a try for the `best_quality` preset. High-end GPUs with large memory are preferred in this case. Compared to `high_quality`, it requires much longer training time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3715091d-86c8-43aa-85de-c0816897cef1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from autogluon.multimodal import MultiModalPredictor\n",
    "\n",
    "predictor = MultiModalPredictor(label='label', eval_metric='acc', presets=\"best_quality\")\n",
    "predictor.fit(train_data=train_data, time_limit=180)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cfd8fd1-8c44-4d99-bdd7-60bdd1427958",
   "metadata": {},
   "source": [
    "We can see that `best_quality` achieves better performance than `high_quality`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "792396bb-7706-422a-914f-b63fd65407cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = predictor.evaluate(test_data, metrics=[\"roc_auc\"])\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1189267-6633-4c0d-bf8d-2bdcf666cb11",
   "metadata": {},
   "source": [
    "## HPO Presets\n",
    "The above three presets all use the default hyperparameters, which might not be optimal for your tasks. Fortunately, we also support hyperparameter optimization (HPO) with simple presets. To perform HPO, you can add a postfix `_hpo` in the three presets, resulting in `medium_quality_hpo`, `high_quality_hpo`, and `best_quality_hpo`.\n",
    "\n",
    "## Display Presets\n",
    "In case you want to see each preset's inside details, we provide you with a util function to get the hyperparameter setups. For example, here are hyperparameters of preset `high_quality`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96308d11-bf93-49ab-90a5-4772d3b0bd87",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from autogluon.multimodal.utils.presets import get_presets\n",
    "\n",
    "hyperparameters, hyperparameter_tune_kwargs = get_presets(problem_type=\"default\", presets=\"high_quality\")\n",
    "print(f\"hyperparameters: {json.dumps(hyperparameters, sort_keys=True, indent=4)}\")\n",
    "print(f\"hyperparameter_tune_kwargs: {json.dumps(hyperparameter_tune_kwargs, sort_keys=True, indent=4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ab0d124-19a4-428a-ba52-ce4f68adc833",
   "metadata": {},
   "source": [
    "The HPO presets make several hyperparameters tunable such as model backbone, batch size, learning rate, max epoch, and optimizer type. Below are the details of preset `high_quality_hpo`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "693ab3e9-e3e5-4aa8-b74b-264e00e7e1cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import yaml\n",
    "from autogluon.multimodal.utils.presets import get_presets\n",
    "\n",
    "hyperparameters, hyperparameter_tune_kwargs = get_presets(problem_type=\"default\", presets=\"high_quality_hpo\")\n",
    "print(f\"hyperparameters: {yaml.dump(hyperparameters, allow_unicode=True, default_flow_style=False)}\")\n",
    "print(f\"hyperparameter_tune_kwargs: {json.dumps(hyperparameter_tune_kwargs, sort_keys=True, indent=4)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "599f45fc-666e-4603-9226-098f47b2cbf1",
   "metadata": {},
   "source": [
    "## Other Examples\n",
    "You may go to [AutoMM Examples](https://github.com/autogluon/autogluon/tree/master/examples/automm) to explore other examples about AutoMM."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e0bb8b-2f72-4413-8758-39033ac4278d",
   "metadata": {},
   "source": [
    "## Customization\n",
    "To learn how to customize AutoMM, please refer to [Customize AutoMM](../advanced_topics/customization.ipynb)."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}