Searchable Objects

When defining custom Python objects such as network architectures, or specialized optimizers, it may be hard to decide what values to set for all of their attributes. AutoGluon provides an API that allows you to instead specify a search space of possible values to consider for such attributes, within which the optimal value will be automatically searched for at runtime. This tutorial demonstrates how easy this is to do, without having to modify your existing code at all!

Example for Constructing a Network

This tutorial covers an example of selecting a neural network’s architecture as a hyperparameter optimization (HPO) task. If you are interested in efficient neural architecture search (NAS), please refer to this other tutorial instead: sec_proxyless_ .

CIFAR ResNet in GluonCV

GluonCV provides CIFARResNet, which allow user to specify how many layers at each stage. For example, we can construct a CIFAR ResNet with only 1 layer per stage:

from gluoncv.model_zoo.cifarresnet import CIFARResNetV1, CIFARBasicBlockV1

layers = [1, 1, 1]
channels = [16, 16, 32, 64]
net = CIFARResNetV1(CIFARBasicBlockV1, layers, channels)

We can visualize the network:

import autogluon.core as ag
from autogluon.vision.utils import plot_network

plot_network(net, (1, 3, 32, 32))
../../_images/output_object_d3e86d_3_0.svg

Searchable Network Architecture Using AutoGluon Object

autogluon.obj() enables customized search space to any user defined class. It can also be used within autogluon.Categorical() if you have multiple networks to choose from.

@ag.obj(
    nstage1=ag.space.Int(2, 4),
    nstage2=ag.space.Int(2, 4),
)
class MyCifarResNet(CIFARResNetV1):
    def __init__(self, nstage1, nstage2):
        nstage3 = 9 - nstage1 - nstage2
        layers = [nstage1, nstage2, nstage3]
        channels = [16, 16, 32, 64]
        super().__init__(CIFARBasicBlockV1, layers=layers, channels=channels)

Create one network instance and print the configuration space:

mynet=MyCifarResNet()
print(mynet.cs)
Configuration space object:
  Hyperparameters:
    nstage1, Type: UniformInteger, Range: [2, 4], Default: 3
    nstage2, Type: UniformInteger, Range: [2, 4], Default: 3

We can also overwrite existing search spaces:

mynet1 = MyCifarResNet(nstage1=1,
                       nstage2=ag.space.Int(5, 10))
print(mynet1.cs)
Configuration space object:
  Hyperparameters:
    nstage2, Type: UniformInteger, Range: [5, 10], Default: 8

Decorate Existing Class

We can also use autogluon.obj() to easily decorate any existing classes. For example, if we want to search learning rate and weight decay for Adam optimizer, we only need to add a decorator:

from mxnet import optimizer as optim
@ag.obj()
class Adam(optim.Adam):
    pass

Then we can create an instance:

myoptim = Adam(learning_rate=ag.Real(1e-2, 1e-1, log=True), wd=ag.Real(1e-5, 1e-3, log=True))
print(myoptim.cs)
Configuration space object:
  Hyperparameters:
    learning_rate, Type: UniformFloat, Range: [0.01, 0.1], Default: 0.0316227766, on log-scale
    wd, Type: UniformFloat, Range: [1e-05, 0.001], Default: 0.0001, on log-scale

Launch Experiments Using AutoGluon Object

AutoGluon Object is compatible with Fit API in AutoGluon tasks, and also works with user-defined training scripts using autogluon.autogluon_register_args(). We can start fitting:

from autogluon.vision import ImagePredictor
classifier = ImagePredictor().fit('cifar10', hyperparameters={'net': mynet, 'optimizer': myoptim, 'epochs': 1}, ngpus_per_trial=1)
INFO:root:time_limit=auto set to time_limit=7200.
INFO:gluoncv.auto.tasks.image_classification:Starting fit without HPO
INFO:ImageClassificationEstimator:modified configs(<old> != <new>): {
INFO:ImageClassificationEstimator:root.img_cls.model   resnet50_v1 != resnet50_v1b
INFO:ImageClassificationEstimator:root.train.early_stop_baseline 0.0 != -inf
INFO:ImageClassificationEstimator:root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto
INFO:ImageClassificationEstimator:root.train.epochs    10 != 1
INFO:ImageClassificationEstimator:root.train.num_training_samples 1281167 != -1
INFO:ImageClassificationEstimator:root.train.lr        0.1 != 0.01
INFO:ImageClassificationEstimator:root.train.early_stop_patience -1 != 10
INFO:ImageClassificationEstimator:root.train.batch_size 128 != 16
INFO:ImageClassificationEstimator:root.train.data_dir  ~/.mxnet/datasets/imagenet != auto
INFO:ImageClassificationEstimator:root.train.early_stop_max_value 1.0 != inf
INFO:ImageClassificationEstimator:root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto
INFO:ImageClassificationEstimator:root.train.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto
INFO:ImageClassificationEstimator:root.train.rec_val   ~/.mxnet/datasets/imagenet/rec/val.rec != auto
INFO:ImageClassificationEstimator:root.valid.num_workers 4 != 8
INFO:ImageClassificationEstimator:root.valid.batch_size 128 != 16
INFO:ImageClassificationEstimator:}
INFO:ImageClassificationEstimator:Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/3d06fc69/.trial_0/config.yaml
INFO:ImageClassificationEstimator:Start training from [Epoch 0]
INFO:ImageClassificationEstimator:Epoch[0] Batch [49]       Speed: 98.277190 samples/sec    accuracy=0.147500       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [99]       Speed: 100.961702 samples/sec   accuracy=0.223750       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [149]      Speed: 100.130260 samples/sec   accuracy=0.270833       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [199]      Speed: 99.726518 samples/sec    accuracy=0.312188       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [249]      Speed: 99.050539 samples/sec    accuracy=0.338000       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [299]      Speed: 98.704300 samples/sec    accuracy=0.358750       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [349]      Speed: 98.056224 samples/sec    accuracy=0.378750       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [399]      Speed: 97.617914 samples/sec    accuracy=0.391094       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [449]      Speed: 96.955558 samples/sec    accuracy=0.406389       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [499]      Speed: 96.517793 samples/sec    accuracy=0.421375       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [549]      Speed: 95.847380 samples/sec    accuracy=0.433750       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [599]      Speed: 95.520561 samples/sec    accuracy=0.442812       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [649]      Speed: 94.993512 samples/sec    accuracy=0.453558       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [699]      Speed: 94.520986 samples/sec    accuracy=0.461964       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [749]      Speed: 94.071722 samples/sec    accuracy=0.469500       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [799]      Speed: 93.676169 samples/sec    accuracy=0.479531       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [849]      Speed: 93.647335 samples/sec    accuracy=0.488015       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [899]      Speed: 93.677145 samples/sec    accuracy=0.492986       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [949]      Speed: 93.690152 samples/sec    accuracy=0.501711       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [999]      Speed: 93.715645 samples/sec    accuracy=0.507938       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1049]     Speed: 93.662787 samples/sec    accuracy=0.513452       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1099]     Speed: 93.861704 samples/sec    accuracy=0.519091       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1149]     Speed: 93.886134 samples/sec    accuracy=0.523967       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1199]     Speed: 93.712441 samples/sec    accuracy=0.527552       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1249]     Speed: 93.806723 samples/sec    accuracy=0.532700       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1299]     Speed: 93.789475 samples/sec    accuracy=0.538077       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1349]     Speed: 93.655937 samples/sec    accuracy=0.541852       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1399]     Speed: 93.551808 samples/sec    accuracy=0.545491       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1449]     Speed: 93.566871 samples/sec    accuracy=0.549095       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1499]     Speed: 93.830775 samples/sec    accuracy=0.552625       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1549]     Speed: 93.648229 samples/sec    accuracy=0.556371       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1599]     Speed: 93.669368 samples/sec    accuracy=0.559297       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1649]     Speed: 93.759384 samples/sec    accuracy=0.562083       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1699]     Speed: 93.696457 samples/sec    accuracy=0.565478       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1749]     Speed: 93.618592 samples/sec    accuracy=0.568214       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1799]     Speed: 93.690035 samples/sec    accuracy=0.571528       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1849]     Speed: 93.605656 samples/sec    accuracy=0.575068       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1899]     Speed: 93.622928 samples/sec    accuracy=0.577237       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1949]     Speed: 93.470358 samples/sec    accuracy=0.580160       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [1999]     Speed: 93.412577 samples/sec    accuracy=0.582969       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2049]     Speed: 93.757262 samples/sec    accuracy=0.585366       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2099]     Speed: 93.666570 samples/sec    accuracy=0.587649       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2149]     Speed: 93.761755 samples/sec    accuracy=0.589651       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2199]     Speed: 93.689258 samples/sec    accuracy=0.591847       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2249]     Speed: 93.733656 samples/sec    accuracy=0.594028       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2299]     Speed: 93.829279 samples/sec    accuracy=0.596386       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2349]     Speed: 93.699380 samples/sec    accuracy=0.598457       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2399]     Speed: 93.697371 samples/sec    accuracy=0.600729       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2449]     Speed: 93.582043 samples/sec    accuracy=0.602449       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2499]     Speed: 93.677260 samples/sec    accuracy=0.604150       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2549]     Speed: 93.648556 samples/sec    accuracy=0.605784       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2599]     Speed: 93.490788 samples/sec    accuracy=0.607115       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2649]     Speed: 93.753272 samples/sec    accuracy=0.609104       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2699]     Speed: 93.711310 samples/sec    accuracy=0.611157       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2749]     Speed: 93.721484 samples/sec    accuracy=0.612795       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2799]     Speed: 93.854786 samples/sec    accuracy=0.614531       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2849]     Speed: 93.634593 samples/sec    accuracy=0.616360       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2899]     Speed: 93.580996 samples/sec    accuracy=0.617845       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2949]     Speed: 93.663019 samples/sec    accuracy=0.619428       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [2999]     Speed: 93.703004 samples/sec    accuracy=0.621375       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3049]     Speed: 93.620692 samples/sec    accuracy=0.622889       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3099]     Speed: 93.664047 samples/sec    accuracy=0.624597       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3149]     Speed: 93.502001 samples/sec    accuracy=0.626250       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3199]     Speed: 93.435517 samples/sec    accuracy=0.627383       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3249]     Speed: 93.589255 samples/sec    accuracy=0.628846       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3299]     Speed: 93.771007 samples/sec    accuracy=0.629943       lr=0.010000
INFO:ImageClassificationEstimator:Epoch[0] Batch [3349]     Speed: 93.807438 samples/sec    accuracy=0.631567       lr=0.010000
INFO:ImageClassificationEstimator:[Epoch 0] training: accuracy=0.632093
INFO:ImageClassificationEstimator:[Epoch 0] speed: 94 samples/sec   time cost: 594.130483
INFO:ImageClassificationEstimator:[Epoch 0] validation: top1=0.904500 top5=0.997333
INFO:ImageClassificationEstimator:[Epoch 0] Current best top-1: 0.904500 vs previous 0.000000, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/3d06fc69/.trial_0/best_checkpoint.pkl
INFO:ImageClassificationEstimator:Applying the state from the best checkpoint...
INFO:gluoncv.auto.tasks.image_classification:Finished, total runtime is 607.63 s
INFO:gluoncv.auto.tasks.image_classification:{ 'best_config': { 'batch_size': 16,
                   'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
),
                   'custom_optimizer': <__main__.Adam object at 0x7fdc0807d450>,
                   'dist_ip_addrs': None,
                   'early_stop_baseline': -inf,
                   'early_stop_max_value': inf,
                   'early_stop_patience': 10,
                   'epochs': 1,
                   'estimator': <class 'gluoncv.auto.estimators.image_classification.image_classification.ImageClassificationEstimator'>,
                   'final_fit': False,
                   'gpus': [0],
                   'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/3d06fc69',
                   'lr': 0.01,
                   'model': 'resnet50_v1b',
                   'ngpus_per_trial': 1,
                   'nthreads_per_trial': 128,
                   'num_trials': 1,
                   'num_workers': 8,
                   'scheduler': 'local',
                   'search_strategy': 'random',
                   'searcher': 'random',
                   'seed': 141,
                   'time_limits': 7200,
                   'wall_clock_tick': 1618181769.6101558},
  'total_time': 584.5839202404022,
  'train_acc': 0.6320925925925925,
  'valid_acc': 0.9045}
print(classifier.fit_summary())
{'train_acc': 0.6320925925925925, 'valid_acc': 0.9045, 'total_time': 584.5839202404022, 'best_config': {'model': 'resnet50_v1b', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/3d06fc69', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
), 'custom_optimizer': <__main__.Adam object at 0x7fdc0807d450>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 141, 'final_fit': False, 'estimator': <class 'gluoncv.auto.estimators.image_classification.image_classification.ImageClassificationEstimator'>, 'wall_clock_tick': 1618181769.6101558}, 'fit_history': {'train_acc': 0.6320925925925925, 'valid_acc': 0.9045, 'total_time': 584.5839202404022, 'best_config': {'model': 'resnet50_v1b', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/3d06fc69', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
), 'custom_optimizer': <__main__.Adam object at 0x7fdc0807d450>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 141, 'final_fit': False, 'estimator': <class 'gluoncv.auto.estimators.image_classification.image_classification.ImageClassificationEstimator'>, 'wall_clock_tick': 1618181769.6101558}}}