{ "cells": [ { "cell_type": "markdown", "id": "f2ffe44a", "metadata": {}, "source": [ "# Hyperparameter Optimization in AutoMM\n", "\n", "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/autogluon/autogluon/blob/stable/docs/tutorials/multimodal/advanced_topics/hyperparameter_optimization.ipynb)\n", "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/autogluon/autogluon/blob/stable/docs/tutorials/multimodal/advanced_topics/hyperparameter_optimization.ipynb)\n", "\n", "Hyperparameter optimization (HPO) is a method that helps solve the challenge of tuning hyperparameters of machine learning models. ML algorithms have multiple complex hyperparameters that generate an enormous search space, and the search space in deep learning methods is even larger than traditional ML algorithms. Tuning on a massive search space is a tough challenge, but AutoMM provides various options for you to guide the fitting process based on your domain knowledge and the constraint on computing resources.\n", "\n", "## Create Image Dataset\n", "\n", "In this tutorial, we are going to again use the subset of the Shopee-IET dataset from Kaggle for demonstration purpose. Each image contains a clothing item and the corresponding label specifies its clothing category. Our subset of the data contains the following possible labels: `BabyPants`, `BabyShirt`, `womencasualshoes`, `womenchiffontop`.\n", "\n", "We can load a dataset by downloading a url data automatically:" ] }, { "cell_type": "code", "execution_count": null, "id": "aa00faab-252f-44c9-b8f7-57131aa8251c", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "!pip install autogluon.multimodal\n" ] }, { "cell_type": "code", "execution_count": null, "id": "364d104f", "metadata": {}, "outputs": [], "source": [ "import warnings\n", "warnings.filterwarnings('ignore')\n", "from datetime import datetime\n", "\n", "from autogluon.multimodal.utils.misc import shopee_dataset\n", "download_dir = './ag_automm_tutorial_hpo'\n", "train_data, test_data = shopee_dataset(download_dir)\n", "train_data = train_data.sample(frac=0.5)\n", "print(train_data)" ] }, { "cell_type": "markdown", "id": "a8c1f74b", "metadata": {}, "source": [ "There are in total 400 data points in this dataset. The `image` column stores the path to the actual image, and the `label` column stands for the label class. \n", "\n", "\n", "## The Regular Model Fitting\n", "\n", "Recall that if we are to use the default settings predefined by Autogluon, we can simply fit the model using `MultiModalPredictor` with three lines of code:" ] }, { "cell_type": "code", "execution_count": null, "id": "c8c7e903", "metadata": {}, "outputs": [], "source": [ "from autogluon.multimodal import MultiModalPredictor\n", "predictor_regular = MultiModalPredictor(label=\"label\")\n", "start_time = datetime.now()\n", "predictor_regular.fit(\n", " train_data=train_data,\n", " hyperparameters = {\"model.timm_image.checkpoint_name\": \"ghostnet_100\"}\n", ")\n", "end_time = datetime.now()\n", "elapsed_seconds = (end_time - start_time).total_seconds()\n", "elapsed_min = divmod(elapsed_seconds, 60)\n", "print(\"Total fitting time: \", f\"{int(elapsed_min[0])}m{int(elapsed_min[1])}s\")" ] }, { "cell_type": "markdown", "id": "fa513a16", "metadata": {}, "source": [ "Let's check out the test accuracy of the fitted model:" ] }, { "cell_type": "code", "execution_count": null, "id": "0fed0f46", "metadata": {}, "outputs": [], "source": [ "scores = predictor_regular.evaluate(test_data, metrics=[\"accuracy\"])\n", "print('Top-1 test acc: %.3f' % scores[\"accuracy\"])" ] }, { "cell_type": "markdown", "id": "f3b185e2", "metadata": {}, "source": [ "## Use HPO During Model Fitting\n", "\n", "If you would like more control over the fitting process, you can specify various options for hyperparameter optimizations(HPO) in `MultiModalPredictor` by simply adding more options in `hyperparameter` and `hyperparameter_tune_kwargs`.\n", "\n", "There are a few options we can have in MultiModalPredictor. We use [Ray Tune](https://docs.ray.io/en/latest/tune/index.html) `tune` library in the backend, so we need to pass in a [Tune search space](https://docs.ray.io/en/latest/tune/api/search_space.html) or an [AutoGluon search space](https://auto.gluon.ai/stable/api/autogluon.common.space.html) which will be converted to Tune search space.\n", "\n", "1. Defining the search space of various `hyperparameter` values for the training of neural networks:\n", "\n", "\n", " \n", "2. Defining the search strategy for HPO with `hyperparameter_tune_kwargs`. You can pass in a string or initialize a `ray.tune.schedulers.TrialScheduler` object.\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "Let's work on HPO with combinations of different learning rates and backbone models:" ] }, { "cell_type": "code", "execution_count": null, "id": "89f33e03", "metadata": {}, "outputs": [], "source": [ "from ray import tune\n", "\n", "predictor_hpo = MultiModalPredictor(label=\"label\")\n", "\n", "hyperparameters = {\n", " \"optimization.learning_rate\": tune.uniform(0.00005, 0.001),\n", " \"model.timm_image.checkpoint_name\": tune.choice([\"ghostnet_100\",\n", " \"mobilenetv3_large_100\"])\n", "}\n", "hyperparameter_tune_kwargs = {\n", " \"searcher\": \"bayes\", # random\n", " \"scheduler\": \"ASHA\",\n", " \"num_trials\": 2,\n", " \"num_to_keep\": 3,\n", "}\n", "start_time_hpo = datetime.now()\n", "predictor_hpo.fit(\n", " train_data=train_data,\n", " hyperparameters=hyperparameters,\n", " hyperparameter_tune_kwargs=hyperparameter_tune_kwargs,\n", " )\n", "end_time_hpo = datetime.now()\n", "elapsed_seconds_hpo = (end_time_hpo - start_time_hpo).total_seconds()\n", "elapsed_min_hpo = divmod(elapsed_seconds_hpo, 60)\n", "print(\"Total fitting time: \", f\"{int(elapsed_min_hpo[0])}m{int(elapsed_min_hpo[1])}s\")" ] }, { "cell_type": "markdown", "id": "5e98eefe", "metadata": {}, "source": [ "Let's check out the test accuracy of the fitted model after HPO:" ] }, { "cell_type": "code", "execution_count": null, "id": "337ec1bc", "metadata": {}, "outputs": [], "source": [ "scores_hpo = predictor_hpo.evaluate(test_data, metrics=[\"accuracy\"])\n", "print('Top-1 test acc: %.3f' % scores_hpo[\"accuracy\"])" ] }, { "cell_type": "markdown", "id": "0cf0ac03", "metadata": {}, "source": [ "From the training log, you should be able to see the current best trial as below:" ] }, { "cell_type": "markdown", "id": "fb644dff", "metadata": {}, "source": [ "```\n", "Current best trial: 47aef96a with val_accuracy=0.862500011920929 and parameters={'optimization.learning_rate': 0.0007195214018085505, 'model.timm_image.checkpoint_name': 'ghostnet_100'}\n", "```\n" ] }, { "cell_type": "markdown", "id": "54909e81", "metadata": {}, "source": [ "After our simple 2-trial HPO run, we got a better test accuracy, by searching different learning rates and models, compared to the out-of-box solution provided in the previous section. HPO helps select the combination of hyperparameters with highest validation accuracy. \n", "\n", "## Other Examples\n", "\n", "You may go to [AutoMM Examples](https://github.com/autogluon/autogluon/tree/master/examples/automm) to explore other examples about AutoMM.\n", "\n", "## Customization\n", "To learn how to customize AutoMM, please refer to [Customize AutoMM](customization.ipynb)." ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }