Searchable Objects

When defining custom Python objects such as network architectures, or specialized optimizers, it may be hard to decide what values to set for all of their attributes. AutoGluon provides an API that allows you to instead specify a search space of possible values to consider for such attributes, within which the optimal value will be automatically searched for at runtime. This tutorial demonstrates how easy this is to do, without having to modify your existing code at all!

Example for Constructing a Network

This tutorial covers an example of selecting a neural network’s architecture as a hyperparameter optimization (HPO) task. If you are interested in efficient neural architecture search (NAS), please refer to this other tutorial instead: sec_proxyless_ .

CIFAR ResNet in GluonCV

GluonCV provides CIFARResNet, which allow user to specify how many layers at each stage. For example, we can construct a CIFAR ResNet with only 1 layer per stage:

from gluoncv.model_zoo.cifarresnet import CIFARResNetV1, CIFARBasicBlockV1

layers = [1, 1, 1]
channels = [16, 16, 32, 64]
net = CIFARResNetV1(CIFARBasicBlockV1, layers, channels)

We can visualize the network:

import autogluon.core as ag
from autogluon.vision.utils import plot_network

plot_network(net, (1, 3, 32, 32))
../../_images/output_object_d3e86d_3_0.svg

Searchable Network Architecture Using AutoGluon Object

autogluon.obj() enables customized search space to any user defined class. It can also be used within autogluon.Categorical() if you have multiple networks to choose from.

@ag.obj(
    nstage1=ag.space.Int(2, 4),
    nstage2=ag.space.Int(2, 4),
)
class MyCifarResNet(CIFARResNetV1):
    def __init__(self, nstage1, nstage2):
        nstage3 = 9 - nstage1 - nstage2
        layers = [nstage1, nstage2, nstage3]
        channels = [16, 16, 32, 64]
        super().__init__(CIFARBasicBlockV1, layers=layers, channels=channels)

Create one network instance and print the configuration space:

mynet=MyCifarResNet()
print(mynet.cs)
Configuration space object:
  Hyperparameters:
    nstage1, Type: UniformInteger, Range: [2, 4], Default: 3
    nstage2, Type: UniformInteger, Range: [2, 4], Default: 3

We can also overwrite existing search spaces:

mynet1 = MyCifarResNet(nstage1=1,
                       nstage2=ag.space.Int(5, 10))
print(mynet1.cs)
Configuration space object:
  Hyperparameters:
    nstage2, Type: UniformInteger, Range: [5, 10], Default: 8

Decorate Existing Class

We can also use autogluon.obj() to easily decorate any existing classes. For example, if we want to search learning rate and weight decay for Adam optimizer, we only need to add a decorator:

from mxnet import optimizer as optim
@ag.obj()
class Adam(optim.Adam):
    pass

Then we can create an instance:

myoptim = Adam(learning_rate=ag.Real(1e-2, 1e-1, log=True), wd=ag.Real(1e-5, 1e-3, log=True))
print(myoptim.cs)
Configuration space object:
  Hyperparameters:
    learning_rate, Type: UniformFloat, Range: [0.01, 0.1], Default: 0.0316227766, on log-scale
    wd, Type: UniformFloat, Range: [1e-05, 0.001], Default: 0.0001, on log-scale

Launch Experiments Using AutoGluon Object

AutoGluon Object is compatible with Fit API in AutoGluon tasks, and also works with user-defined training scripts using autogluon.autogluon_register_args(). We can start fitting:

from autogluon.vision import ImagePredictor
classifier = ImagePredictor().fit('cifar10', hyperparameters={'net': mynet, 'optimizer': myoptim, 'epochs': 1}, ngpus_per_trial=1)
INFO:root:time_limit=auto set to time_limit=7200.
INFO:gluoncv.auto.tasks.image_classification:Starting fit without HPO
INFO:ImageClassificationEstimator:modified configs(<old> != <new>): {
INFO:ImageClassificationEstimator:root.img_cls.model   resnet50_v1 != resnet50_v1b
INFO:ImageClassificationEstimator:root.train.batch_size 128 != 16
INFO:ImageClassificationEstimator:root.train.rec_val   ~/.mxnet/datasets/imagenet/rec/val.rec != auto
INFO:ImageClassificationEstimator:root.train.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto
INFO:ImageClassificationEstimator:root.train.early_stop_max_value 1.0 != inf
INFO:ImageClassificationEstimator:root.train.epochs    10 != 1
INFO:ImageClassificationEstimator:root.train.num_training_samples 1281167 != -1
INFO:ImageClassificationEstimator:root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto
INFO:ImageClassificationEstimator:root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto
INFO:ImageClassificationEstimator:root.train.lr        0.1 != 0.01
INFO:ImageClassificationEstimator:root.train.data_dir  ~/.mxnet/datasets/imagenet != auto
INFO:ImageClassificationEstimator:root.train.early_stop_baseline 0.0 != -inf
INFO:ImageClassificationEstimator:root.train.early_stop_patience -1 != 10
INFO:ImageClassificationEstimator:root.valid.batch_size 128 != 16
INFO:ImageClassificationEstimator:root.valid.num_workers 4 != 8
INFO:ImageClassificationEstimator:}
INFO:ImageClassificationEstimator:Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/0c9861a9/.trial_0/config.yaml
INFO:ImageClassificationEstimator:Start training from [Epoch 0]
INFO:ImageClassificationEstimator:Epoch[0] Batch [49]       Speed: 101.533861 samples/sec   accuracy=0.181250       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [99]       Speed: 104.030060 samples/sec   accuracy=0.243125       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [149]      Speed: 103.350313 samples/sec   accuracy=0.285000       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [199]      Speed: 102.873391 samples/sec   accuracy=0.319688       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [249]      Speed: 102.532888 samples/sec   accuracy=0.347000       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [299]      Speed: 101.820300 samples/sec   accuracy=0.362500       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [349]      Speed: 102.061057 samples/sec   accuracy=0.384643       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [399]      Speed: 101.985035 samples/sec   accuracy=0.400781       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [449]      Speed: 102.025438 samples/sec   accuracy=0.415278       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [499]      Speed: 102.199182 samples/sec   accuracy=0.429250       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [549]      Speed: 102.404000 samples/sec   accuracy=0.439886       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [599]      Speed: 102.506414 samples/sec   accuracy=0.450208       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [649]      Speed: 102.461578 samples/sec   accuracy=0.459135       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [699]      Speed: 102.389020 samples/sec   accuracy=0.468214       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [749]      Speed: 102.292600 samples/sec   accuracy=0.476083       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [799]      Speed: 102.412564 samples/sec   accuracy=0.483438       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [849]      Speed: 102.572666 samples/sec   accuracy=0.491176       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [899]      Speed: 102.489056 samples/sec   accuracy=0.499306       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [949]      Speed: 102.275477 samples/sec   accuracy=0.505395       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [999]      Speed: 102.046847 samples/sec   accuracy=0.512000       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1049]     Speed: 101.620944 samples/sec   accuracy=0.516310       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1099]     Speed: 101.648537 samples/sec   accuracy=0.521364       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1149]     Speed: 101.668775 samples/sec   accuracy=0.526739       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1199]     Speed: 101.204937 samples/sec   accuracy=0.530208       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1249]     Speed: 101.491400 samples/sec   accuracy=0.535350       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1299]     Speed: 101.250430 samples/sec   accuracy=0.539856       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1349]     Speed: 101.384916 samples/sec   accuracy=0.543657       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1399]     Speed: 101.077534 samples/sec   accuracy=0.547545       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1449]     Speed: 101.228379 samples/sec   accuracy=0.551078       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1499]     Speed: 100.984856 samples/sec   accuracy=0.553458       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1549]     Speed: 100.860253 samples/sec   accuracy=0.555685       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1599]     Speed: 100.612682 samples/sec   accuracy=0.558867       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1649]     Speed: 100.483437 samples/sec   accuracy=0.562045       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1699]     Speed: 100.190070 samples/sec   accuracy=0.565625       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1749]     Speed: 99.764934 samples/sec    accuracy=0.568679       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1799]     Speed: 99.228630 samples/sec    accuracy=0.571215       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1849]     Speed: 99.170826 samples/sec    accuracy=0.574155       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1899]     Speed: 98.793152 samples/sec    accuracy=0.576645       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1949]     Speed: 98.236238 samples/sec    accuracy=0.578814       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1999]     Speed: 97.969785 samples/sec    accuracy=0.581500       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2049]     Speed: 97.465967 samples/sec    accuracy=0.584207       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2099]     Speed: 96.801682 samples/sec    accuracy=0.586577       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2149]     Speed: 96.294462 samples/sec    accuracy=0.588895       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2199]     Speed: 96.543644 samples/sec    accuracy=0.590625       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2249]     Speed: 98.358359 samples/sec    accuracy=0.592528       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2299]     Speed: 99.478762 samples/sec    accuracy=0.594429       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2349]     Speed: 100.215290 samples/sec   accuracy=0.596090       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2399]     Speed: 100.570066 samples/sec   accuracy=0.598151       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2449]     Speed: 100.968675 samples/sec   accuracy=0.600230       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2499]     Speed: 101.417558 samples/sec   accuracy=0.601650       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2549]     Speed: 101.543619 samples/sec   accuracy=0.603235       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2599]     Speed: 101.517497 samples/sec   accuracy=0.604832       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2649]     Speed: 101.374634 samples/sec   accuracy=0.606792       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2699]     Speed: 101.503696 samples/sec   accuracy=0.608634       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2749]     Speed: 101.424084 samples/sec   accuracy=0.610432       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2799]     Speed: 101.569153 samples/sec   accuracy=0.611741       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2849]     Speed: 101.823455 samples/sec   accuracy=0.613289       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2899]     Speed: 101.518796 samples/sec   accuracy=0.614978       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2949]     Speed: 101.355330 samples/sec   accuracy=0.616419       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2999]     Speed: 101.476032 samples/sec   accuracy=0.618354       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3049]     Speed: 101.429382 samples/sec   accuracy=0.620246       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3099]     Speed: 101.073567 samples/sec   accuracy=0.621895       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3149]     Speed: 101.054584 samples/sec   accuracy=0.623393       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3199]     Speed: 101.037209 samples/sec   accuracy=0.624648       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3249]     Speed: 100.945866 samples/sec   accuracy=0.626288       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3299]     Speed: 100.923923 samples/sec   accuracy=0.627860       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3349]     Speed: 100.978708 samples/sec   accuracy=0.629160       lr=0.010000
INFO:ImageClassificationEstimator:[Epoch 0] training: accuracy=0.629889
INFO:ImageClassificationEstimator:[Epoch 0] speed: 100 samples/sec  time cost: 555.823344
INFO:ImageClassificationEstimator:[Epoch 0] validation: top1=0.899000 top5=0.998667
INFO:ImageClassificationEstimator:[Epoch 0] Current best top-1: 0.899000 vs previous 0.000000, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/0c9861a9/.trial_0/best_checkpoint.pkl
INFO:ImageClassificationEstimator:Applying the state from the best checkpoint...
INFO:gluoncv.auto.tasks.image_classification:Finished, total runtime is 566.16 s
INFO:gluoncv.auto.tasks.image_classification:{ 'best_config': { 'batch_size': 16,
                   'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
),
                   'custom_optimizer': <__main__.Adam object at 0x7f8aede10dd0>,
                   'dist_ip_addrs': None,
                   'early_stop_baseline': -inf,
                   'early_stop_max_value': inf,
                   'early_stop_patience': 10,
                   'epochs': 1,
                   'estimator': <class 'gluoncv.auto.estimators.image_classification.image_classification.ImageClassificationEstimator'>,
                   'final_fit': False,
                   'gpus': [0],
                   'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/0c9861a9',
                   'lr': 0.01,
                   'model': 'resnet50_v1b',
                   'ngpus_per_trial': 1,
                   'nthreads_per_trial': 128,
                   'num_trials': 1,
                   'num_workers': 8,
                   'scheduler': 'local',
                   'search_strategy': 'random',
                   'searcher': 'random',
                   'seed': 87,
                   'time_limits': 7200,
                   'wall_clock_tick': 1619665595.0251148},
  'total_time': 548.4526901245117,
  'train_acc': 0.6298888888888889,
  'valid_acc': 0.899}
print(classifier.fit_summary())
{'train_acc': 0.6298888888888889, 'valid_acc': 0.899, 'total_time': 548.4526901245117, 'best_config': {'model': 'resnet50_v1b', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/0c9861a9', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
), 'custom_optimizer': <__main__.Adam object at 0x7f8aede10dd0>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 87, 'final_fit': False, 'estimator': <class 'gluoncv.auto.estimators.image_classification.image_classification.ImageClassificationEstimator'>, 'wall_clock_tick': 1619665595.0251148}, 'fit_history': {'train_acc': 0.6298888888888889, 'valid_acc': 0.899, 'total_time': 548.4526901245117, 'best_config': {'model': 'resnet50_v1b', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/0c9861a9', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
), 'custom_optimizer': <__main__.Adam object at 0x7f8aede10dd0>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 87, 'final_fit': False, 'estimator': <class 'gluoncv.auto.estimators.image_classification.image_classification.ImageClassificationEstimator'>, 'wall_clock_tick': 1619665595.0251148}}}