.. _sec_imgadvanced: Image Prediction - Search Space and Hyperparameter Optimization (HPO) ===================================================================== While the :ref:`sec_imgquick` introduced basic usage of AutoGluon ``fit``, ``evaluate``, ``predict`` with default configurations, this tutorial dives into the various options that you can specify for more advanced control over the fitting process. These options include: - Defining the search space of various hyperparameter values for the training of neural networks - Specifying how to search through your chosen hyperparameter space - Specifying how to schedule jobs to train a network under a particular hyperparameter configuration. The advanced functionalities of AutoGluon enable you to use your external knowledge about your particular prediction problem and computing resources to guide the training process. If properly used, you may be able to achieve superior performance within less training time. **Tip**: If you are new to AutoGluon, review :ref:`sec_imgquick` to learn the basics of the AutoGluon API. Since our task is to classify images, we will use AutoGluon to produce an `ImagePredictor <../../api/autogluon.predictor.html#autogluon.vision.ImagePredictor>`__: .. code:: python import autogluon.core as ag from autogluon.vision import ImagePredictor, ImageDataset .. parsed-literal:: :class: output /home/ci/opt/venv/lib/python3.8/site-packages/gluoncv/__init__.py:40: UserWarning: Both `mxnet==1.9.1` and `torch==1.12.1+cu102` are installed. You might encounter increased GPU memory footprint if both framework are used at the same time. warnings.warn(f'Both `mxnet=={mx.__version__}` and `torch=={torch.__version__}` are installed. ' INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkn_zq46 INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkn_zq46/_remote_module_non_scriptable.py INFO:root:Generating grammar tables from /usr/lib/python3.8/lib2to3/Grammar.txt INFO:root:Generating grammar tables from /usr/lib/python3.8/lib2to3/PatternGrammar.txt Create AutoGluon Dataset ------------------------ Let’s first create the dataset using the same subset of the ``Shopee-IET`` dataset as the :ref:`sec_imgquick` tutorial. Recall that there’s no validation split in original data, a 90/10 train/validation split is automatically performed when ``fit`` with ``train_data``. .. code:: python train_data, _, test_data = ImageDataset.from_folders('https://autogluon.s3.amazonaws.com/datasets/shopee-iet.zip') .. parsed-literal:: :class: output data/ ├── test/ └── train/ Specify which Networks to Try ----------------------------- We start with specifying the pretrained neural network candidates. Given such a list, AutoGluon tries to train different networks from this list to identify the best-performing candidate. This is an example of a :class:`autogluon.core.space.Categorical` search space, in which there are a limited number of values to choose from. .. code:: python model = ag.Categorical('resnet18_v1b', 'mobilenetv3_small') # you may choose more than 70+ available model in the model zoo provided by GluonCV: model_list = ImagePredictor.list_models() Specify the training hyper-parameters ------------------------------------- Similarly, we can manually specify many crucial hyper-parameters, with specific value or search space (``autogluon.core.space``). .. code:: python batch_size = 8 lr = ag.Categorical(1e-2, 1e-3) Search Algorithms ----------------- In AutoGluon, ``autogluon.core.searcher`` supports different search strategies for both hyperparameter optimization and architecture search. Beyond simply specifying the space of hyperparameter configurations to search over, you can also tell AutoGluon what strategy it should employ to actually search through this space. This process of finding good hyperparameters from a given search space is commonly referred to as *hyperparameter optimization* (HPO) or *hyperparameter tuning*. ``autogluon.core.scheduler`` orchestrates how individual training jobs are scheduled. We currently support random search. Random Search ~~~~~~~~~~~~~ Here is an example of using random search using :class:`autogluon.core.searcher.LocalRandomSearcher`. .. code:: python hyperparameters={'model': model, 'batch_size': batch_size, 'lr': lr, 'epochs': 2} predictor = ImagePredictor() predictor.fit(train_data, time_limit=60*10, hyperparameters=hyperparameters, hyperparameter_tune_kwargs={'num_trials': 2}) print('Top-1 val acc: %.3f' % predictor.fit_summary()['valid_acc']) .. parsed-literal:: :class: output AutoGluon ImagePredictor will be deprecated in v0.7. Please use AutoGluon MultiModalPredictor instead for more functionalities and better support. Visit https://auto.gluon.ai/stable/tutorials/multimodal/index.html for more details! ImagePredictor sets accuracy as default eval_metric for classification problems. Reset labels to [0, 1, 2, 3] Randomly split train_data into train[720]/validation[80] splits. The number of requested GPUs is greater than the number of available GPUs.Reduce the number to 1 Starting HPO experiments .. parsed-literal:: :class: output 0%| | 0/2 [00:00 != ): { INFO:ImageClassificationEstimator:root.train.early_stop_patience -1 != 10 INFO:ImageClassificationEstimator:root.train.epochs 10 != 2 INFO:ImageClassificationEstimator:root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto INFO:ImageClassificationEstimator:root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto INFO:ImageClassificationEstimator:root.train.data_dir ~/.mxnet/datasets/imagenet != auto INFO:ImageClassificationEstimator:root.train.lr 0.1 != 0.01 INFO:ImageClassificationEstimator:root.train.num_workers 4 != 8 INFO:ImageClassificationEstimator:root.train.num_training_samples 1281167 != -1 INFO:ImageClassificationEstimator:root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto INFO:ImageClassificationEstimator:root.train.early_stop_max_value 1.0 != inf INFO:ImageClassificationEstimator:root.train.early_stop_baseline 0.0 != -inf INFO:ImageClassificationEstimator:root.train.batch_size 128 != 8 INFO:ImageClassificationEstimator:root.train.rec_val ~/.mxnet/datasets/imagenet/rec/val.rec != auto INFO:ImageClassificationEstimator:root.valid.num_workers 4 != 8 INFO:ImageClassificationEstimator:root.valid.batch_size 128 != 8 INFO:ImageClassificationEstimator:root.img_cls.model resnet50_v1 != resnet18_v1b INFO:ImageClassificationEstimator:} INFO:ImageClassificationEstimator:Saved config to /home/ci/autogluon/docs/_build/eval/tutorials/image_prediction/4b352609/.trial_0/config.yaml INFO:root:Model file not found. Downloading. .. parsed-literal:: :class: output Downloading /home/ci/.mxnet/models/resnet18_v1b-2d9d980c.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/resnet18_v1b-2d9d980c.zip... .. parsed-literal:: :class: output 0%| | 0/42432 [00:00 != ): { INFO:ImageClassificationEstimator:root.train.early_stop_patience -1 != 10 INFO:ImageClassificationEstimator:root.train.epochs 10 != 2 INFO:ImageClassificationEstimator:root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto INFO:ImageClassificationEstimator:root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto INFO:ImageClassificationEstimator:root.train.data_dir ~/.mxnet/datasets/imagenet != auto INFO:ImageClassificationEstimator:root.train.lr 0.1 != 0.001 INFO:ImageClassificationEstimator:root.train.num_workers 4 != 8 INFO:ImageClassificationEstimator:root.train.num_training_samples 1281167 != -1 INFO:ImageClassificationEstimator:root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto INFO:ImageClassificationEstimator:root.train.early_stop_max_value 1.0 != inf INFO:ImageClassificationEstimator:root.train.early_stop_baseline 0.0 != -inf INFO:ImageClassificationEstimator:root.train.batch_size 128 != 8 INFO:ImageClassificationEstimator:root.train.rec_val ~/.mxnet/datasets/imagenet/rec/val.rec != auto INFO:ImageClassificationEstimator:root.valid.num_workers 4 != 8 INFO:ImageClassificationEstimator:root.valid.batch_size 128 != 8 INFO:ImageClassificationEstimator:root.img_cls.model resnet50_v1 != resnet18_v1b INFO:ImageClassificationEstimator:} INFO:ImageClassificationEstimator:Saved config to /home/ci/autogluon/docs/_build/eval/tutorials/image_prediction/4b352609/.trial_1/config.yaml INFO:ImageClassificationEstimator:Start training from [Epoch 0] INFO:ImageClassificationEstimator:Epoch[0] Batch [49] Speed: 223.436015 samples/sec accuracy=0.252500 lr=0.001000 INFO:ImageClassificationEstimator:[Epoch 0] training: accuracy=0.261111 INFO:ImageClassificationEstimator:[Epoch 0] speed: 236 samples/sec time cost: 3.014910 INFO:ImageClassificationEstimator:[Epoch 0] validation: top1=0.412500 top5=1.000000 INFO:ImageClassificationEstimator:[Epoch 0] Current best top-1: 0.412500 vs previous -inf, saved to /home/ci/autogluon/docs/_build/eval/tutorials/image_prediction/4b352609/.trial_1/best_checkpoint.pkl INFO:ImageClassificationEstimator:Epoch[1] Batch [49] Speed: 244.022933 samples/sec accuracy=0.277500 lr=0.001000 INFO:ImageClassificationEstimator:[Epoch 1] training: accuracy=0.313889 INFO:ImageClassificationEstimator:[Epoch 1] speed: 248 samples/sec time cost: 2.867769 INFO:ImageClassificationEstimator:[Epoch 1] validation: top1=0.562500 top5=1.000000 INFO:ImageClassificationEstimator:[Epoch 1] Current best top-1: 0.562500 vs previous 0.412500, saved to /home/ci/autogluon/docs/_build/eval/tutorials/image_prediction/4b352609/.trial_1/best_checkpoint.pkl INFO:ImageClassificationEstimator:Applying the state from the best checkpoint... ============================================================================= WARNING: Using MXNet models in ImagePredictor is deprecated as of v0.4.0 and may contain various bugs and issues! In v0.6.0, ImagePredictor will no longer support training MXNet models. Please consider switching to specifying Torch models instead. Users should ensure they update their code that depends on ImagePredictor when upgrading to future AutoGluon releases. For more information, refer to this GitHub issue: https://github.com/autogluon/autogluon/issues/1560 ============================================================================= Finished, total runtime is 28.34 s { 'best_config': { 'estimator': , 'gpus': [0], 'img_cls': { 'batch_norm': False, 'last_gamma': False, 'model': 'resnet18_v1b', 'use_gn': False, 'use_pretrained': True, 'use_se': False}, 'train': { 'batch_size': 8, 'crop_ratio': 0.875, 'data_dir': 'auto', 'dtype': 'float32', 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'early_stop_min_delta': 0.001, 'early_stop_patience': 10, 'epochs': 2, 'hard_weight': 0.5, 'input_size': 224, 'label_smoothing': False, 'log_interval': 50, 'lr': 0.01, 'lr_decay': 0.1, 'lr_decay_epoch': '40, 60', 'lr_decay_period': 0, 'lr_mode': 'step', 'mixup': False, 'mixup_alpha': 0.2, 'mixup_off_epoch': 0, 'mode': '', 'momentum': 0.9, 'no_wd': False, 'num_training_samples': -1, 'num_workers': 8, 'output_lr_mult': 0.1, 'pretrained_base': True, 'rec_train': 'auto', 'rec_train_idx': 'auto', 'rec_val': 'auto', 'rec_val_idx': 'auto', 'resume_epoch': 0, 'start_epoch': 0, 'teacher': None, 'temperature': 20, 'transfer_lr_mult': 0.01, 'use_rec': False, 'warmup_epochs': 0, 'warmup_lr': 0.0, 'wd': 0.0001}, 'valid': {'batch_size': 8, 'num_workers': 8}}, 'total_time': 28.343422174453735, 'train_acc': 0.3138888888888889, 'valid_acc': 0.5625} .. parsed-literal:: :class: output Top-1 val acc: 0.562 Load the test dataset and evaluate: .. code:: python results = predictor.evaluate(test_data) print('Test acc on hold-out data:', results) .. parsed-literal:: :class: output Test acc on hold-out data: {'top1': 0.775, 'top5': 1.0} Note that ``num_trials=2`` above is only used to speed up the tutorial. In normal practice, it is common to only use ``time_limit`` and drop ``num_trials``.