Searchable Objects

When defining custom Python objects such as network architectures, or specialized optimizers, it may be hard to decide what values to set for all of their attributes. AutoGluon provides an API that allows you to instead specify a search space of possible values to consider for such attributes, within which the optimal value will be automatically searched for at runtime. This tutorial demonstrates how easy this is to do, without having to modify your existing code at all!

Example for Constructing a Network

This tutorial covers an example of selecting a neural network’s architecture as a hyperparameter optimization (HPO) task. If you are interested in efficient neural architecture search (NAS), please refer to this other tutorial instead: sec_proxyless_ .

CIFAR ResNet in GluonCV

GluonCV provides CIFARResNet, which allow user to specify how many layers at each stage. For example, we can construct a CIFAR ResNet with only 1 layer per stage:

import pickle
from gluoncv.model_zoo.cifarresnet import CIFARResNetV1, CIFARBasicBlockV1

layers = [1, 1, 1]
channels = [16, 16, 32, 64]
net = CIFARResNetV1(CIFARBasicBlockV1, layers, channels)
/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/venv/lib/python3.9/site-packages/gluoncv/__init__.py:40: UserWarning: Both mxnet==1.7.0 and torch==1.7.1+cu101 are installed. You might encounter increased GPU memory footprint if both framework are used at the same time.
  warnings.warn(f'Both mxnet=={mx.__version__} and torch=={torch.__version__} are installed. '

We can visualize the network:

import autogluon.core as ag
from autogluon.vision.utils import plot_network

plot_network(net, (1, 3, 32, 32))
../../_images/output_object_d3e86d_3_0.svg

Searchable Network Architecture Using AutoGluon Object

autogluon.obj() enables customized search space to any user defined class. It can also be used within autogluon.Categorical() if you have multiple networks to choose from.

@ag.obj(
    nstage1=ag.space.Int(2, 4),
    nstage2=ag.space.Int(2, 4),
)
class MyCifarResNet(CIFARResNetV1):
    def __init__(self, nstage1, nstage2):
        nstage3 = 9 - nstage1 - nstage2
        layers = [nstage1, nstage2, nstage3]
        channels = [16, 16, 32, 64]
        super().__init__(CIFARBasicBlockV1, layers=layers, channels=channels)

Create one network instance and print the configuration space:

mynet=MyCifarResNet()
print(mynet.cs)
Configuration space object:
  Hyperparameters:
    nstage1, Type: UniformInteger, Range: [2, 4], Default: 3
    nstage2, Type: UniformInteger, Range: [2, 4], Default: 3

We can also overwrite existing search spaces:

mynet1 = MyCifarResNet(nstage1=1,
                       nstage2=ag.space.Int(5, 10))
print(mynet1.cs)
Configuration space object:
  Hyperparameters:
    nstage2, Type: UniformInteger, Range: [5, 10], Default: 8

Decorate Existing Class

We can also use autogluon.obj() to easily decorate any existing classes. For example, if we want to search learning rate and weight decay for Adam optimizer, we only need to add a decorator:

from mxnet import optimizer as optim
@ag.obj()
class Adam(optim.Adam):
    pass

Then we can create an instance:

myoptim = Adam(learning_rate=ag.Real(1e-2, 1e-1, log=True), wd=ag.Real(1e-5, 1e-3, log=True))
print(myoptim.cs)
Configuration space object:
  Hyperparameters:
    learning_rate, Type: UniformFloat, Range: [0.01, 0.1], Default: 0.0316227766, on log-scale
    wd, Type: UniformFloat, Range: [1e-05, 0.001], Default: 0.0001, on log-scale

Launch Experiments Using AutoGluon Object

AutoGluon Object is compatible with Fit API in AutoGluon tasks, and also works with user-defined training scripts using autogluon.autogluon_register_args(). We can start fitting:

from autogluon.vision import ImagePredictor
classifier = ImagePredictor().fit('cifar10', hyperparameters={'net': mynet, 'optimizer': myoptim, 'epochs': 1}, ngpus_per_trial=1)
time_limit=auto set to time_limit=7200.
Starting fit without HPO
modified configs(<old> != <new>): {
root.img_cls.model   resnet50_v1 != resnet50
root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto
root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto
root.train.data_dir  ~/.mxnet/datasets/imagenet != auto
root.train.num_workers 4 != 8
root.train.early_stop_max_value 1.0 != inf
root.train.early_stop_patience -1 != 10
root.train.batch_size 128 != 16
root.train.epochs    10 != 1
root.train.lr        0.1 != 0.01
root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto
root.train.rec_val   ~/.mxnet/datasets/imagenet/rec/val.rec != auto
root.train.num_training_samples 1281167 != -1
root.train.early_stop_baseline 0.0 != -inf
root.valid.batch_size 128 != 16
root.valid.num_workers 4 != 8
}
Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/62d5d7b7/.trial_0/config.yaml
Start training from [Epoch 0]
Epoch[0] Batch [49] Speed: 72.414085 samples/sec    accuracy=0.143750       lr=0.010000
Epoch[0] Batch [99] Speed: 72.354038 samples/sec    accuracy=0.148750       lr=0.010000
Epoch[0] Batch [149]        Speed: 71.834712 samples/sec    accuracy=0.153333       lr=0.010000
Epoch[0] Batch [199]        Speed: 71.321075 samples/sec    accuracy=0.162500       lr=0.010000
Epoch[0] Batch [249]        Speed: 70.593249 samples/sec    accuracy=0.162000       lr=0.010000
Epoch[0] Batch [299]        Speed: 70.002843 samples/sec    accuracy=0.168958       lr=0.010000
Epoch[0] Batch [349]        Speed: 69.348837 samples/sec    accuracy=0.171250       lr=0.010000
Epoch[0] Batch [399]        Speed: 68.763088 samples/sec    accuracy=0.171406       lr=0.010000
Epoch[0] Batch [449]        Speed: 68.229280 samples/sec    accuracy=0.175000       lr=0.010000
Epoch[0] Batch [499]        Speed: 67.536539 samples/sec    accuracy=0.176250       lr=0.010000
Epoch[0] Batch [549]        Speed: 67.018200 samples/sec    accuracy=0.175341       lr=0.010000
Epoch[0] Batch [599]        Speed: 66.499460 samples/sec    accuracy=0.177187       lr=0.010000
Epoch[0] Batch [649]        Speed: 66.653317 samples/sec    accuracy=0.179327       lr=0.010000
Epoch[0] Batch [699]        Speed: 69.429192 samples/sec    accuracy=0.182054       lr=0.010000
Epoch[0] Batch [749]        Speed: 70.987083 samples/sec    accuracy=0.184083       lr=0.010000
Epoch[0] Batch [799]        Speed: 71.695102 samples/sec    accuracy=0.187500       lr=0.010000
Epoch[0] Batch [849]        Speed: 71.935022 samples/sec    accuracy=0.189191       lr=0.010000
Epoch[0] Batch [899]        Speed: 72.376935 samples/sec    accuracy=0.191042       lr=0.010000
Epoch[0] Batch [949]        Speed: 72.488943 samples/sec    accuracy=0.191711       lr=0.010000
Epoch[0] Batch [999]        Speed: 72.645692 samples/sec    accuracy=0.193000       lr=0.010000
Epoch[0] Batch [1049]       Speed: 72.651850 samples/sec    accuracy=0.193690       lr=0.010000
Epoch[0] Batch [1099]       Speed: 72.629598 samples/sec    accuracy=0.192955       lr=0.010000
Epoch[0] Batch [1149]       Speed: 72.579226 samples/sec    accuracy=0.194022       lr=0.010000
Epoch[0] Batch [1199]       Speed: 72.534169 samples/sec    accuracy=0.195365       lr=0.010000
Epoch[0] Batch [1249]       Speed: 72.483888 samples/sec    accuracy=0.196750       lr=0.010000
Epoch[0] Batch [1299]       Speed: 72.364752 samples/sec    accuracy=0.198702       lr=0.010000
Epoch[0] Batch [1349]       Speed: 72.060039 samples/sec    accuracy=0.199815       lr=0.010000
Epoch[0] Batch [1399]       Speed: 72.163914 samples/sec    accuracy=0.200357       lr=0.010000
Epoch[0] Batch [1449]       Speed: 72.044811 samples/sec    accuracy=0.201250       lr=0.010000
Epoch[0] Batch [1499]       Speed: 71.851551 samples/sec    accuracy=0.201667       lr=0.010000
Epoch[0] Batch [1549]       Speed: 71.830444 samples/sec    accuracy=0.203266       lr=0.010000
Epoch[0] Batch [1599]       Speed: 71.754657 samples/sec    accuracy=0.204727       lr=0.010000
Epoch[0] Batch [1649]       Speed: 71.498605 samples/sec    accuracy=0.205455       lr=0.010000
Epoch[0] Batch [1699]       Speed: 71.158299 samples/sec    accuracy=0.206838       lr=0.010000
Epoch[0] Batch [1749]       Speed: 70.874066 samples/sec    accuracy=0.207500       lr=0.010000
Epoch[0] Batch [1799]       Speed: 70.371031 samples/sec    accuracy=0.208299       lr=0.010000
Epoch[0] Batch [1849]       Speed: 69.910600 samples/sec    accuracy=0.208885       lr=0.010000
Epoch[0] Batch [1899]       Speed: 69.279672 samples/sec    accuracy=0.209539       lr=0.010000
Epoch[0] Batch [1949]       Speed: 68.683950 samples/sec    accuracy=0.210160       lr=0.010000
Epoch[0] Batch [1999]       Speed: 68.103362 samples/sec    accuracy=0.210813       lr=0.010000
Epoch[0] Batch [2049]       Speed: 67.449686 samples/sec    accuracy=0.211585       lr=0.010000
Epoch[0] Batch [2099]       Speed: 66.966393 samples/sec    accuracy=0.212530       lr=0.010000
Epoch[0] Batch [2149]       Speed: 66.497472 samples/sec    accuracy=0.213169       lr=0.010000
Epoch[0] Batch [2199]       Speed: 65.935094 samples/sec    accuracy=0.214006       lr=0.010000
Epoch[0] Batch [2249]       Speed: 65.611746 samples/sec    accuracy=0.215000       lr=0.010000
Epoch[0] Batch [2299]       Speed: 65.481156 samples/sec    accuracy=0.215625       lr=0.010000
Epoch[0] Batch [2349]       Speed: 65.577602 samples/sec    accuracy=0.216144       lr=0.010000
Epoch[0] Batch [2399]       Speed: 65.636968 samples/sec    accuracy=0.216302       lr=0.010000
Epoch[0] Batch [2449]       Speed: 65.619597 samples/sec    accuracy=0.217270       lr=0.010000
Epoch[0] Batch [2499]       Speed: 65.631218 samples/sec    accuracy=0.217600       lr=0.010000
Epoch[0] Batch [2549]       Speed: 65.613621 samples/sec    accuracy=0.218235       lr=0.010000
Epoch[0] Batch [2599]       Speed: 65.602320 samples/sec    accuracy=0.218822       lr=0.010000
Epoch[0] Batch [2649]       Speed: 65.592080 samples/sec    accuracy=0.219552       lr=0.010000
Epoch[0] Batch [2699]       Speed: 65.486370 samples/sec    accuracy=0.220394       lr=0.010000
Epoch[0] Batch [2749]       Speed: 65.504651 samples/sec    accuracy=0.221000       lr=0.010000
Epoch[0] Batch [2799]       Speed: 65.466724 samples/sec    accuracy=0.221830       lr=0.010000
Epoch[0] Batch [2849]       Speed: 65.485180 samples/sec    accuracy=0.222237       lr=0.010000
Epoch[0] Batch [2899]       Speed: 65.408956 samples/sec    accuracy=0.223039       lr=0.010000
Epoch[0] Batch [2949]       Speed: 65.380457 samples/sec    accuracy=0.223665       lr=0.010000
Epoch[0] Batch [2999]       Speed: 65.278788 samples/sec    accuracy=0.224312       lr=0.010000
Epoch[0] Batch [3049]       Speed: 65.380504 samples/sec    accuracy=0.225676       lr=0.010000
Epoch[0] Batch [3099]       Speed: 65.691806 samples/sec    accuracy=0.226028       lr=0.010000
Epoch[0] Batch [3149]       Speed: 65.790143 samples/sec    accuracy=0.226429       lr=0.010000
Epoch[0] Batch [3199]       Speed: 65.813476 samples/sec    accuracy=0.227227       lr=0.010000
Epoch[0] Batch [3249]       Speed: 66.120451 samples/sec    accuracy=0.227885       lr=0.010000
Epoch[0] Batch [3299]       Speed: 66.252677 samples/sec    accuracy=0.228201       lr=0.010000
Epoch[0] Batch [3349]       Speed: 66.383845 samples/sec    accuracy=0.228694       lr=0.010000
[Epoch 0] training: accuracy=0.228833
[Epoch 0] speed: 68 samples/sec     time cost: 786.801392
[Epoch 0] validation: top1=0.326333 top5=0.864000
[Epoch 0] Current best top-1: 0.326333 vs previous -inf, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/62d5d7b7/.trial_0/best_checkpoint.pkl
Unable to pickle object due to the reason: Can't pickle <class '__main__.MyCifarResNet'>: it's not the same object as __main__.MyCifarResNet. This object is not saved.
Applying the state from the best checkpoint...
Unable to resume the state from the best checkpoint, using the latest state.
Finished, total runtime is 812.21 s
{ 'best_config': { 'batch_size': 16,
                   'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
),
                   'custom_optimizer': <__main__.Adam object at 0x7f81c04b1e80>,
                   'dist_ip_addrs': None,
                   'early_stop_baseline': -inf,
                   'early_stop_max_value': inf,
                   'early_stop_patience': 10,
                   'epochs': 1,
                   'final_fit': False,
                   'gpus': [0],
                   'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/62d5d7b7',
                   'lr': 0.01,
                   'model': 'resnet50',
                   'ngpus_per_trial': 1,
                   'nthreads_per_trial': 128,
                   'num_trials': 1,
                   'num_workers': 8,
                   'problem_type': 'multiclass',
                   'scheduler': 'local',
                   'search_strategy': 'random',
                   'searcher': 'random',
                   'seed': 554,
                   'time_limits': 7200,
                   'wall_clock_tick': 1634869672.0826032},
  'total_time': 798.5396816730499,
  'train_acc': 0.22883333333333333,
  'valid_acc': 0.3263333333333333}
print(classifier.fit_summary())
{'train_acc': 0.22883333333333333, 'valid_acc': 0.3263333333333333, 'total_time': 798.5396816730499, 'best_config': {'model': 'resnet50', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/62d5d7b7', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
), 'custom_optimizer': <__main__.Adam object at 0x7f81c04b1e80>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 554, 'final_fit': False, 'wall_clock_tick': 1634869672.0826032, 'problem_type': 'multiclass'}, 'fit_history': {'train_acc': 0.22883333333333333, 'valid_acc': 0.3263333333333333, 'total_time': 798.5396816730499, 'best_config': {'model': 'resnet50', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/62d5d7b7', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet(
  (features): HybridSequential(
    (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
    (2): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (3): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (1): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (2): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (3): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (4): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (5): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (6): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
      (7): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (4): HybridSequential(
      (0): CIFARBasicBlockV1(
        (body): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
          (2): Activation(relu)
          (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
        (downsample): HybridSequential(
          (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None)
        )
      )
    )
    (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
  )
  (output): Dense(64 -> 10, linear)
), 'custom_optimizer': <__main__.Adam object at 0x7f81c04b1e80>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 554, 'final_fit': False, 'wall_clock_tick': 1634869672.0826032, 'problem_type': 'multiclass'}}}