Image Prediction - Search Space and Hyperparameter Optimization (HPO)

While the Image Prediction - Quick Start introduced basic usage of AutoGluon fit, evaluate, predict with default configurations, this tutorial dives into the various options that you can specify for more advanced control over the fitting process.

These options include: - Defining the search space of various hyperparameter values for the training of neural networks - Specifying how to search through your choosen hyperparameter space - Specifying how to schedule jobs to train a network under a particular hyperparameter configuration.

The advanced functionalities of AutoGluon enable you to use your external knowledge about your particular prediction problem and computing resources to guide the training process. If properly used, you may be able to achieve superior performance within less training time.

Tip: If you are new to AutoGluon, review Image Prediction - Quick Start to learn the basics of the AutoGluon API.

Since our task is to classify images, we will use AutoGluon to produce an ImagePredictor:

import autogluon.core as ag
from autogluon.vision import ImagePredictor

Create AutoGluon Dataset

Let’s first create the dataset using the same subset of the Shopee-IET dataset as the Image Prediction - Quick Start tutorial. Recall that there’s no validation split in original data, a 90/10 train/validation split is automatically performed when fit with train_data.

train_data, _, test_data = ImagePredictor.Dataset.from_folders('https://autogluon.s3.amazonaws.com/datasets/shopee-iet.zip')
data/
├── test/
└── train/

Specify which Networks to Try

We start with specifying the pretrained neural network candidates. Given such a list, AutoGluon tries to train different networks from this list to identify the best-performing candidate. This is an example of a autogluon.core.space.Categorical search space, in which there are a limited number of values to choose from.

model = ag.Categorical('resnet18_v1b', 'mobilenetv3_small')

# you may choose more than 70+ available model in the model zoo provided by GluonCV:
model_list = ImagePredictor.list_models()

Specify the training hyper-parameters

Similarly, we can manually specify many crucial hyper-parameters, with specific value or search space(autogluon.core.space).

batch_size = 8
lr = ag.Categorical(1e-2, 1e-3)

Search Algorithms

In AutoGluon, autogluon.core.searcher supports different search search strategies for both hyperparameter optimization and architecture search. Beyond simply specifying the space of hyperparameter configurations to search over, you can also tell AutoGluon what strategy it should employ to actually search through this space. This process of finding good hyperparameters from a given search space is commonly referred to as hyperparameter optimization (HPO) or hyperparameter tuning. autogluon.core.scheduler orchestrates how individual training jobs are scheduled. We currently support FIFO (standard) and Hyperband scheduling, along with search by random sampling or Bayesian optimization. These basic techniques are rendered surprisingly powerful by AutoGluon’s support of asynchronous parallel execution.

Bayesian Optimization

Here is an example of using Bayesian Optimization using autogluon.core.searcher.GPFIFOSearcher.

Bayesian Optimization fits a probabilistic surrogate model to estimate the function that relates each hyperparameter configuration to the resulting performance of a model trained under this hyperparameter configuration. Our implementation makes use of a Gaussian process surrogate model along with expected improvement as acquisition function. It has been developed specifically to support asynchronous parallel evaluations.

hyperparameters={'model': model, 'batch_size': batch_size, 'lr': lr, 'epochs': 2}
predictor = ImagePredictor()
predictor.fit(train_data, search_strategy='bayesopt', time_limit=60*10, hyperparameters=hyperparameters,
              hyperparameter_tune_kwargs={'num_trials': 2})
print('Top-1 val acc: %.3f' % predictor.fit_summary()['valid_acc'])
INFO:root:Reset labels to [0, 1, 2, 3]
WARNING:gluoncv.auto.tasks.image_classification:The number of requested GPUs is greater than the number of available GPUs.Reduce the number to 1
INFO:gluoncv.auto.tasks.image_classification:Randomly split train_data into train[726]/validation[74] splits.
INFO:gluoncv.auto.tasks.image_classification:Starting HPO experiments
  0%|          | 0/2 [00:00<?, ?it/s]
INFO:ImageClassificationEstimator:modified configs(<old> != <new>): {
INFO:ImageClassificationEstimator:root.valid.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.valid.batch_size 128 != 8
INFO:ImageClassificationEstimator:root.train.epochs    10 != 2
INFO:ImageClassificationEstimator:root.train.data_dir  ~/.mxnet/datasets/imagenet != auto
INFO:ImageClassificationEstimator:root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto
INFO:ImageClassificationEstimator:root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto
INFO:ImageClassificationEstimator:root.train.early_stop_max_value 1.0 != inf
INFO:ImageClassificationEstimator:root.train.rec_val   ~/.mxnet/datasets/imagenet/rec/val.rec != auto
INFO:ImageClassificationEstimator:root.train.num_training_samples 1281167 != -1
INFO:ImageClassificationEstimator:root.train.batch_size 128 != 8
INFO:ImageClassificationEstimator:root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto
INFO:ImageClassificationEstimator:root.train.lr        0.1 != 0.01
INFO:ImageClassificationEstimator:root.train.early_stop_patience -1 != 10
INFO:ImageClassificationEstimator:root.train.early_stop_baseline 0.0 != -inf
INFO:ImageClassificationEstimator:root.train.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.img_cls.model   resnet50_v1 != resnet18_v1b
INFO:ImageClassificationEstimator:}
INFO:ImageClassificationEstimator:Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_0/config.yaml
INFO:ImageClassificationEstimator:Start training from [Epoch 0]
INFO:ImageClassificationEstimator:Epoch[0] Batch [49]       Speed: 226.157379 samples/sec   accuracy=0.380000       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [99]       Speed: 267.660952 samples/sec   accuracy=0.480000       lr=0.010000
INFO:ImageClassificationEstimator:[Epoch 0] training: accuracy=0.480000
INFO:ImageClassificationEstimator:[Epoch 0] speed: 242 samples/sec  time cost: 4.748112
INFO:ImageClassificationEstimator:[Epoch 0] validation: top1=0.756250 top5=1.000000
INFO:ImageClassificationEstimator:[Epoch 0] Current best top-1: 0.756250 vs previous 0.000000, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_0/best_checkpoint.pkl
INFO:ImageClassificationEstimator:Epoch[1] Batch [49]       Speed: 249.422034 samples/sec   accuracy=0.612500       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[1] Batch [99]       Speed: 266.434935 samples/sec   accuracy=0.658750       lr=0.010000
INFO:ImageClassificationEstimator:[Epoch 1] training: accuracy=0.658750
INFO:ImageClassificationEstimator:[Epoch 1] speed: 254 samples/sec  time cost: 4.578471
INFO:ImageClassificationEstimator:[Epoch 1] validation: top1=0.813750 top5=1.000000
INFO:ImageClassificationEstimator:[Epoch 1] Current best top-1: 0.813750 vs previous 0.756250, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_0/best_checkpoint.pkl
INFO:ImageClassificationEstimator:Applying the state from the best checkpoint...
INFO:ImageClassificationEstimator:modified configs(<old> != <new>): {
INFO:ImageClassificationEstimator:root.valid.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.valid.batch_size 128 != 8
INFO:ImageClassificationEstimator:root.train.epochs    10 != 2
INFO:ImageClassificationEstimator:root.train.data_dir  ~/.mxnet/datasets/imagenet != auto
INFO:ImageClassificationEstimator:root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto
INFO:ImageClassificationEstimator:root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto
INFO:ImageClassificationEstimator:root.train.early_stop_max_value 1.0 != inf
INFO:ImageClassificationEstimator:root.train.rec_val   ~/.mxnet/datasets/imagenet/rec/val.rec != auto
INFO:ImageClassificationEstimator:root.train.num_training_samples 1281167 != -1
INFO:ImageClassificationEstimator:root.train.batch_size 128 != 8
INFO:ImageClassificationEstimator:root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto
INFO:ImageClassificationEstimator:root.train.lr        0.1 != 0.01
INFO:ImageClassificationEstimator:root.train.early_stop_patience -1 != 10
INFO:ImageClassificationEstimator:root.train.early_stop_baseline 0.0 != -inf
INFO:ImageClassificationEstimator:root.train.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.img_cls.model   resnet50_v1 != mobilenetv3_small
INFO:ImageClassificationEstimator:}
INFO:ImageClassificationEstimator:Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_1/config.yaml
INFO:ImageClassificationEstimator:Start training from [Epoch 0]
INFO:ImageClassificationEstimator:Epoch[0] Batch [49]       Speed: 134.162507 samples/sec   accuracy=0.317500       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [99]       Speed: 152.444575 samples/sec   accuracy=0.392500       lr=0.010000
INFO:ImageClassificationEstimator:[Epoch 0] training: accuracy=0.392500
INFO:ImageClassificationEstimator:[Epoch 0] speed: 141 samples/sec  time cost: 7.973307
INFO:ImageClassificationEstimator:[Epoch 0] validation: top1=0.685000 top5=1.000000
INFO:ImageClassificationEstimator:[Epoch 0] Current best top-1: 0.685000 vs previous 0.000000, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_1/best_checkpoint.pkl
INFO:ImageClassificationEstimator:Epoch[1] Batch [49]       Speed: 138.371348 samples/sec   accuracy=0.602500       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[1] Batch [99]       Speed: 155.496951 samples/sec   accuracy=0.611250       lr=0.010000
INFO:ImageClassificationEstimator:[Epoch 1] training: accuracy=0.611250
INFO:ImageClassificationEstimator:[Epoch 1] speed: 144 samples/sec  time cost: 7.773564
INFO:ImageClassificationEstimator:[Epoch 1] validation: top1=0.786250 top5=1.000000
INFO:ImageClassificationEstimator:[Epoch 1] Current best top-1: 0.786250 vs previous 0.685000, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_1/best_checkpoint.pkl
INFO:ImageClassificationEstimator:Applying the state from the best checkpoint...
INFO:ImageClassificationEstimator:modified configs(<old> != <new>): {
INFO:ImageClassificationEstimator:root.valid.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.valid.batch_size 128 != 8
INFO:ImageClassificationEstimator:root.train.epochs    10 != 2
INFO:ImageClassificationEstimator:root.train.data_dir  ~/.mxnet/datasets/imagenet != auto
INFO:ImageClassificationEstimator:root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto
INFO:ImageClassificationEstimator:root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto
INFO:ImageClassificationEstimator:root.train.early_stop_max_value 1.0 != inf
INFO:ImageClassificationEstimator:root.train.rec_val   ~/.mxnet/datasets/imagenet/rec/val.rec != auto
INFO:ImageClassificationEstimator:root.train.num_training_samples 1281167 != -1
INFO:ImageClassificationEstimator:root.train.batch_size 128 != 8
INFO:ImageClassificationEstimator:root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto
INFO:ImageClassificationEstimator:root.train.lr        0.1 != 0.01
INFO:ImageClassificationEstimator:root.train.early_stop_patience -1 != 10
INFO:ImageClassificationEstimator:root.train.early_stop_baseline 0.0 != -inf
INFO:ImageClassificationEstimator:root.train.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.img_cls.model   resnet50_v1 != resnet18_v1b
INFO:ImageClassificationEstimator:}
INFO:ImageClassificationEstimator:Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_0/config.yaml
INFO:ImageClassificationEstimator:Start training from [Epoch 0]
INFO:ImageClassificationEstimator:Epoch[0] Batch [49]       Speed: 225.972784 samples/sec   accuracy=0.365000       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [99]       Speed: 262.029976 samples/sec   accuracy=0.466250       lr=0.010000
INFO:ImageClassificationEstimator:[Epoch 0] training: accuracy=0.466250
INFO:ImageClassificationEstimator:[Epoch 0] speed: 240 samples/sec  time cost: 4.798971
INFO:ImageClassificationEstimator:[Epoch 0] validation: top1=0.745000 top5=1.000000
INFO:ImageClassificationEstimator:[Epoch 0] Current best top-1: 0.745000 vs previous 0.000000, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_0/best_checkpoint.pkl
INFO:ImageClassificationEstimator:Epoch[1] Batch [49]       Speed: 246.467547 samples/sec   accuracy=0.582500       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[1] Batch [99]       Speed: 262.081305 samples/sec   accuracy=0.620000       lr=0.010000
INFO:ImageClassificationEstimator:[Epoch 1] training: accuracy=0.620000
INFO:ImageClassificationEstimator:[Epoch 1] speed: 251 samples/sec  time cost: 4.625876
INFO:ImageClassificationEstimator:[Epoch 1] validation: top1=0.838750 top5=1.000000
INFO:ImageClassificationEstimator:[Epoch 1] Current best top-1: 0.838750 vs previous 0.745000, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-image-classification-v3/docs/_build/eval/tutorials/image_prediction/027f5dad/.trial_0/best_checkpoint.pkl
INFO:ImageClassificationEstimator:Applying the state from the best checkpoint...
INFO:gluoncv.auto.tasks.image_classification:Finished, total runtime is 48.52 s
INFO:gluoncv.auto.tasks.image_classification:{ 'best_config': { 'estimator': <class 'gluoncv.auto.estimators.image_classification.image_classification.ImageClassificationEstimator'>,
                   'gpus': [0],
                   'img_cls': { 'batch_norm': False,
                                'last_gamma': False,
                                'model': 'resnet18_v1b',
                                'use_gn': False,
                                'use_pretrained': True,
                                'use_se': False},
                   'train': { 'batch_size': 8,
                              'crop_ratio': 0.875,
                              'data_dir': 'auto',
                              'dtype': 'float32',
                              'early_stop_baseline': -inf,
                              'early_stop_max_value': inf,
                              'early_stop_min_delta': 0.001,
                              'early_stop_patience': 10,
                              'epochs': 2,
                              'hard_weight': 0.5,
                              'input_size': 224,
                              'label_smoothing': False,
                              'log_interval': 50,
                              'lr': 0.01,
                              'lr_decay': 0.1,
                              'lr_decay_epoch': '40, 60',
                              'lr_decay_period': 0,
                              'lr_mode': 'step',
                              'mixup': False,
                              'mixup_alpha': 0.2,
                              'mixup_off_epoch': 0,
                              'mode': '',
                              'momentum': 0.9,
                              'no_wd': False,
                              'num_training_samples': -1,
                              'num_workers': 8,
                              'output_lr_mult': 0.1,
                              'pretrained_base': True,
                              'rec_train': 'auto',
                              'rec_train_idx': 'auto',
                              'rec_val': 'auto',
                              'rec_val_idx': 'auto',
                              'resume_epoch': 0,
                              'start_epoch': 0,
                              'teacher': None,
                              'temperature': 20,
                              'transfer_lr_mult': 0.01,
                              'use_rec': False,
                              'warmup_epochs': 0,
                              'warmup_lr': 0.0,
                              'wd': 0.0001},
                   'valid': {'batch_size': 8, 'num_workers': 8}},
  'total_time': 48.5236701965332,
  'train_acc': 0.62,
  'valid_acc': 0.83875}
Top-1 val acc: 0.839

The BO searcher can be configured by search_options, see autogluon.core.searcher.GPFIFOSearcher. Load the test dataset and evaluate:

top1, top5 = predictor.evaluate(test_data)
print('Test acc on hold-out data:', top1)
Test acc on hold-out data: 0.8

Note that num_trials=2 above is only used to speed up the tutorial. In normal practice, it is common to only use time_limit and drop num_trials.

Hyperband Early Stopping

AutoGluon currently supports scheduling trials in serial order and with early stopping (e.g., if the performance of the model early within training already looks bad, the trial may be terminated early to free up resources). Here is an example of using an early stopping scheduler autogluon.core.scheduler.HyperbandScheduler. scheduler_options is used to configure the scheduler. In this example, we run Hyperband with a single bracket, and stop/go decisions are made after 1 and 2 epochs (grace_period, grace_period * reduction_factor):

hyperparameters.update({
  'search_strategy': 'hyperband',
  'grace_period': 1
  })

The fit, evaluate and predict processes are exactly the same, so we will skip training to save some time.

Bayesian Optimization and Hyperband

While Hyperband scheduling is normally driven by a random searcher, AutoGluon also provides Hyperband together with Bayesian optimization. The tuning of expensive DL models typically works best with this combination.

hyperparameters.update({
  'search_strategy': 'bayesopt_hyperband',
  'grace_period': 1
  })

For a comparison of different search algorithms and scheduling strategies, see Search Algorithms. For more options using fit, see autogluon.vision.ImagePredictor.