.. _sec_customobj: Searchable Objects ================== When defining custom Python objects such as network architectures, or specialized optimizers, it may be hard to decide what values to set for all of their attributes. AutoGluon provides an API that allows you to instead specify a search space of possible values to consider for such attributes, within which the optimal value will be automatically searched for at runtime. This tutorial demonstrates how easy this is to do, without having to modify your existing code at all! Example for Constructing a Network ---------------------------------- This tutorial covers an example of selecting a neural network's architecture as a hyperparameter optimization (HPO) task. If you are interested in efficient neural architecture search (NAS), please refer to this other tutorial instead: ``sec_proxyless``\ \_ . CIFAR ResNet in GluonCV ~~~~~~~~~~~~~~~~~~~~~~~ GluonCV provides `CIFARResNet `__, which allow user to specify how many layers at each stage. For example, we can construct a CIFAR ResNet with only 1 layer per stage: .. code:: python import pickle from gluoncv.model_zoo.cifarresnet import CIFARResNetV1, CIFARBasicBlockV1 layers = [1, 1, 1] channels = [16, 16, 32, 64] net = CIFARResNetV1(CIFARBasicBlockV1, layers, channels) .. parsed-literal:: :class: output /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/venv/lib/python3.9/site-packages/gluoncv/__init__.py:40: UserWarning: Both `mxnet==1.7.0` and `torch==1.7.1+cu101` are installed. You might encounter increased GPU memory footprint if both framework are used at the same time. warnings.warn(f'Both `mxnet=={mx.__version__}` and `torch=={torch.__version__}` are installed. ' We can visualize the network: .. code:: python import autogluon.core as ag from autogluon.vision.utils import plot_network plot_network(net, (1, 3, 32, 32)) .. figure:: output_object_d3e86d_3_0.svg Searchable Network Architecture Using AutoGluon Object ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :func:`autogluon.obj` enables customized search space to any user defined class. It can also be used within ``autogluon.Categorical()`` if you have multiple networks to choose from. .. code:: python @ag.obj( nstage1=ag.space.Int(2, 4), nstage2=ag.space.Int(2, 4), ) class MyCifarResNet(CIFARResNetV1): def __init__(self, nstage1, nstage2): nstage3 = 9 - nstage1 - nstage2 layers = [nstage1, nstage2, nstage3] channels = [16, 16, 32, 64] super().__init__(CIFARBasicBlockV1, layers=layers, channels=channels) Create one network instance and print the configuration space: .. code:: python mynet=MyCifarResNet() print(mynet.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: nstage1, Type: UniformInteger, Range: [2, 4], Default: 3 nstage2, Type: UniformInteger, Range: [2, 4], Default: 3 We can also overwrite existing search spaces: .. code:: python mynet1 = MyCifarResNet(nstage1=1, nstage2=ag.space.Int(5, 10)) print(mynet1.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: nstage2, Type: UniformInteger, Range: [5, 10], Default: 8 Decorate Existing Class ~~~~~~~~~~~~~~~~~~~~~~~ We can also use :func:`autogluon.obj` to easily decorate any existing classes. For example, if we want to search learning rate and weight decay for Adam optimizer, we only need to add a decorator: .. code:: python from mxnet import optimizer as optim @ag.obj() class Adam(optim.Adam): pass Then we can create an instance: .. code:: python myoptim = Adam(learning_rate=ag.Real(1e-2, 1e-1, log=True), wd=ag.Real(1e-5, 1e-3, log=True)) print(myoptim.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: learning_rate, Type: UniformFloat, Range: [0.01, 0.1], Default: 0.0316227766, on log-scale wd, Type: UniformFloat, Range: [1e-05, 0.001], Default: 0.0001, on log-scale Launch Experiments Using AutoGluon Object ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ AutoGluon Object is compatible with Fit API in AutoGluon tasks, and also works with user-defined training scripts using :func:`autogluon.autogluon_register_args`. We can start fitting: .. code:: python from autogluon.vision import ImagePredictor classifier = ImagePredictor().fit('cifar10', hyperparameters={'net': mynet, 'optimizer': myoptim, 'epochs': 1}, ngpus_per_trial=1) .. parsed-literal:: :class: output `time_limit=auto` set to `time_limit=7200`. Starting fit without HPO modified configs( != ): { root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto root.train.early_stop_baseline 0.0 != -inf root.train.lr 0.1 != 0.01 root.train.epochs 10 != 1 root.train.early_stop_max_value 1.0 != inf root.train.num_training_samples 1281167 != -1 root.train.data_dir ~/.mxnet/datasets/imagenet != auto root.train.num_workers 4 != 8 root.train.rec_val ~/.mxnet/datasets/imagenet/rec/val.rec != auto root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto root.train.early_stop_patience -1 != 10 root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto root.train.batch_size 128 != 16 root.img_cls.model resnet50_v1 != resnet50 root.valid.num_workers 4 != 8 root.valid.batch_size 128 != 16 } Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/aae1b29f/.trial_0/config.yaml Start training from [Epoch 0] Epoch[0] Batch [49] Speed: 72.343340 samples/sec accuracy=0.136250 lr=0.010000 Epoch[0] Batch [99] Speed: 72.583042 samples/sec accuracy=0.153750 lr=0.010000 Epoch[0] Batch [149] Speed: 71.757374 samples/sec accuracy=0.155833 lr=0.010000 Epoch[0] Batch [199] Speed: 71.314618 samples/sec accuracy=0.163125 lr=0.010000 Epoch[0] Batch [249] Speed: 70.797352 samples/sec accuracy=0.168750 lr=0.010000 Epoch[0] Batch [299] Speed: 70.067545 samples/sec accuracy=0.174167 lr=0.010000 Epoch[0] Batch [349] Speed: 69.548124 samples/sec accuracy=0.175000 lr=0.010000 Epoch[0] Batch [399] Speed: 68.948529 samples/sec accuracy=0.177656 lr=0.010000 Epoch[0] Batch [449] Speed: 68.328321 samples/sec accuracy=0.178750 lr=0.010000 Epoch[0] Batch [499] Speed: 67.768025 samples/sec accuracy=0.182500 lr=0.010000 Epoch[0] Batch [549] Speed: 67.078169 samples/sec accuracy=0.182045 lr=0.010000 Epoch[0] Batch [599] Speed: 66.649762 samples/sec accuracy=0.184896 lr=0.010000 Epoch[0] Batch [649] Speed: 66.290015 samples/sec accuracy=0.186538 lr=0.010000 Epoch[0] Batch [699] Speed: 65.763010 samples/sec accuracy=0.188750 lr=0.010000 Epoch[0] Batch [749] Speed: 65.398779 samples/sec accuracy=0.189333 lr=0.010000 Epoch[0] Batch [799] Speed: 65.314920 samples/sec accuracy=0.190938 lr=0.010000 Epoch[0] Batch [849] Speed: 65.441313 samples/sec accuracy=0.193235 lr=0.010000 Epoch[0] Batch [899] Speed: 65.567551 samples/sec accuracy=0.195764 lr=0.010000 Epoch[0] Batch [949] Speed: 65.563890 samples/sec accuracy=0.196513 lr=0.010000 Epoch[0] Batch [999] Speed: 65.563365 samples/sec accuracy=0.198500 lr=0.010000 Epoch[0] Batch [1049] Speed: 65.620749 samples/sec accuracy=0.199881 lr=0.010000 Epoch[0] Batch [1099] Speed: 65.543888 samples/sec accuracy=0.199602 lr=0.010000 Epoch[0] Batch [1149] Speed: 65.712121 samples/sec accuracy=0.200163 lr=0.010000 Epoch[0] Batch [1199] Speed: 65.729935 samples/sec accuracy=0.200521 lr=0.010000 Epoch[0] Batch [1249] Speed: 65.744988 samples/sec accuracy=0.200700 lr=0.010000 Epoch[0] Batch [1299] Speed: 65.744506 samples/sec accuracy=0.202692 lr=0.010000 Epoch[0] Batch [1349] Speed: 65.742627 samples/sec accuracy=0.202593 lr=0.010000 Epoch[0] Batch [1399] Speed: 65.693858 samples/sec accuracy=0.202366 lr=0.010000 Epoch[0] Batch [1449] Speed: 65.783839 samples/sec accuracy=0.203276 lr=0.010000 Epoch[0] Batch [1499] Speed: 65.719681 samples/sec accuracy=0.203750 lr=0.010000 Epoch[0] Batch [1549] Speed: 65.854035 samples/sec accuracy=0.204194 lr=0.010000 Epoch[0] Batch [1599] Speed: 65.792340 samples/sec accuracy=0.204453 lr=0.010000 Epoch[0] Batch [1649] Speed: 65.813707 samples/sec accuracy=0.205114 lr=0.010000 Epoch[0] Batch [1699] Speed: 65.879841 samples/sec accuracy=0.206618 lr=0.010000 Epoch[0] Batch [1749] Speed: 65.780570 samples/sec accuracy=0.207357 lr=0.010000 Epoch[0] Batch [1799] Speed: 65.813223 samples/sec accuracy=0.207361 lr=0.010000 Epoch[0] Batch [1849] Speed: 65.756948 samples/sec accuracy=0.208345 lr=0.010000 Epoch[0] Batch [1899] Speed: 65.778110 samples/sec accuracy=0.208816 lr=0.010000 Epoch[0] Batch [1949] Speed: 65.713213 samples/sec accuracy=0.209071 lr=0.010000 Epoch[0] Batch [1999] Speed: 65.808846 samples/sec accuracy=0.209937 lr=0.010000 Epoch[0] Batch [2049] Speed: 65.797956 samples/sec accuracy=0.210030 lr=0.010000 Epoch[0] Batch [2099] Speed: 65.797731 samples/sec accuracy=0.210774 lr=0.010000 Epoch[0] Batch [2149] Speed: 65.771468 samples/sec accuracy=0.211570 lr=0.010000 Epoch[0] Batch [2199] Speed: 65.716778 samples/sec accuracy=0.212074 lr=0.010000 Epoch[0] Batch [2249] Speed: 65.738751 samples/sec accuracy=0.213028 lr=0.010000 Epoch[0] Batch [2299] Speed: 65.812407 samples/sec accuracy=0.213207 lr=0.010000 Epoch[0] Batch [2349] Speed: 65.782234 samples/sec accuracy=0.213856 lr=0.010000 Epoch[0] Batch [2399] Speed: 65.887428 samples/sec accuracy=0.214141 lr=0.010000 Epoch[0] Batch [2449] Speed: 66.006821 samples/sec accuracy=0.214872 lr=0.010000 Epoch[0] Batch [2499] Speed: 66.030213 samples/sec accuracy=0.215650 lr=0.010000 Epoch[0] Batch [2549] Speed: 66.008420 samples/sec accuracy=0.216078 lr=0.010000 Epoch[0] Batch [2599] Speed: 65.924558 samples/sec accuracy=0.216130 lr=0.010000 Epoch[0] Batch [2649] Speed: 66.070871 samples/sec accuracy=0.216604 lr=0.010000 Epoch[0] Batch [2699] Speed: 65.887324 samples/sec accuracy=0.217199 lr=0.010000 Epoch[0] Batch [2749] Speed: 66.110434 samples/sec accuracy=0.217705 lr=0.010000 Epoch[0] Batch [2799] Speed: 66.064644 samples/sec accuracy=0.218214 lr=0.010000 Epoch[0] Batch [2849] Speed: 66.064488 samples/sec accuracy=0.218443 lr=0.010000 Epoch[0] Batch [2899] Speed: 66.056154 samples/sec accuracy=0.219030 lr=0.010000 Epoch[0] Batch [2949] Speed: 66.036293 samples/sec accuracy=0.219407 lr=0.010000 Epoch[0] Batch [2999] Speed: 65.925377 samples/sec accuracy=0.219833 lr=0.010000 Epoch[0] Batch [3049] Speed: 66.003197 samples/sec accuracy=0.221270 lr=0.010000 Epoch[0] Batch [3099] Speed: 65.904175 samples/sec accuracy=0.221613 lr=0.010000 Epoch[0] Batch [3149] Speed: 66.047819 samples/sec accuracy=0.221746 lr=0.010000 Epoch[0] Batch [3199] Speed: 65.970173 samples/sec accuracy=0.222207 lr=0.010000 Epoch[0] Batch [3249] Speed: 66.010120 samples/sec accuracy=0.222885 lr=0.010000 Epoch[0] Batch [3299] Speed: 65.984186 samples/sec accuracy=0.223106 lr=0.010000 Epoch[0] Batch [3349] Speed: 65.932756 samples/sec accuracy=0.223451 lr=0.010000 [Epoch 0] training: accuracy=0.223630 [Epoch 0] speed: 66 samples/sec time cost: 812.301834 [Epoch 0] validation: top1=0.281667 top5=0.827000 [Epoch 0] Current best top-1: 0.281667 vs previous -inf, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/aae1b29f/.trial_0/best_checkpoint.pkl Unable to pickle object due to the reason: Can't pickle : it's not the same object as __main__.MyCifarResNet. This object is not saved. Applying the state from the best checkpoint... Unable to resume the state from the best checkpoint, using the latest state. Finished, total runtime is 836.86 s { 'best_config': { 'batch_size': 16, 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7f51a7468d30>, 'dist_ip_addrs': None, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'early_stop_patience': 10, 'epochs': 1, 'final_fit': False, 'gpus': [0], 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/aae1b29f', 'lr': 0.01, 'model': 'resnet50', 'ngpus_per_trial': 1, 'nthreads_per_trial': 128, 'num_trials': 1, 'num_workers': 8, 'problem_type': 'multiclass', 'scheduler': 'local', 'search_strategy': 'random', 'searcher': 'random', 'seed': 411, 'time_limits': 7200, 'wall_clock_tick': 1637619624.8002355}, 'total_time': 823.2866365909576, 'train_acc': 0.22362962962962962, 'valid_acc': 0.2816666666666667} .. code:: python print(classifier.fit_summary()) .. parsed-literal:: :class: output {'train_acc': 0.22362962962962962, 'valid_acc': 0.2816666666666667, 'total_time': 823.2866365909576, 'best_config': {'model': 'resnet50', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/aae1b29f', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7f51a7468d30>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 411, 'final_fit': False, 'wall_clock_tick': 1637619624.8002355, 'problem_type': 'multiclass'}, 'fit_history': {'train_acc': 0.22362962962962962, 'valid_acc': 0.2816666666666667, 'total_time': 823.2866365909576, 'best_config': {'model': 'resnet50', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/aae1b29f', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7f51a7468d30>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 411, 'final_fit': False, 'wall_clock_tick': 1637619624.8002355, 'problem_type': 'multiclass'}}}