.. _sec_customobj: Searchable Objects ================== When defining custom Python objects such as network architectures, or specialized optimizers, it may be hard to decide what values to set for all of their attributes. AutoGluon provides an API that allows you to instead specify a search space of possible values to consider for such attributes, within which the optimal value will be automatically searched for at runtime. This tutorial demonstrates how easy this is to do, without having to modify your existing code at all! Example for Constructing a Network ---------------------------------- This tutorial covers an example of selecting a neural network's architecture as a hyperparameter optimization (HPO) task. If you are interested in efficient neural architecture search (NAS), please refer to this other tutorial instead: ``sec_proxyless``\ \_ . CIFAR ResNet in GluonCV ~~~~~~~~~~~~~~~~~~~~~~~ GluonCV provides `CIFARResNet `__, which allow user to specify how many layers at each stage. For example, we can construct a CIFAR ResNet with only 1 layer per stage: .. code:: python import pickle from gluoncv.model_zoo.cifarresnet import CIFARResNetV1, CIFARBasicBlockV1 layers = [1, 1, 1] channels = [16, 16, 32, 64] net = CIFARResNetV1(CIFARBasicBlockV1, layers, channels) .. parsed-literal:: :class: output /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/venv/lib/python3.7/site-packages/gluoncv/__init__.py:40: UserWarning: Both `mxnet==1.7.0` and `torch==1.7.1+cu101` are installed. You might encounter increased GPU memory footprint if both framework are used at the same time. warnings.warn(f'Both `mxnet=={mx.__version__}` and `torch=={torch.__version__}` are installed. ' We can visualize the network: .. code:: python import autogluon.core as ag from autogluon.vision.utils import plot_network plot_network(net, (1, 3, 32, 32)) .. figure:: output_object_d3e86d_3_0.svg Searchable Network Architecture Using AutoGluon Object ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :func:`autogluon.obj` enables customized search space to any user defined class. It can also be used within ``autogluon.Categorical()`` if you have multiple networks to choose from. .. code:: python @ag.obj( nstage1=ag.space.Int(2, 4), nstage2=ag.space.Int(2, 4), ) class MyCifarResNet(CIFARResNetV1): def __init__(self, nstage1, nstage2): nstage3 = 9 - nstage1 - nstage2 layers = [nstage1, nstage2, nstage3] channels = [16, 16, 32, 64] super().__init__(CIFARBasicBlockV1, layers=layers, channels=channels) Create one network instance and print the configuration space: .. code:: python mynet=MyCifarResNet() print(mynet.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: nstage1, Type: UniformInteger, Range: [2, 4], Default: 3 nstage2, Type: UniformInteger, Range: [2, 4], Default: 3 We can also overwrite existing search spaces: .. code:: python mynet1 = MyCifarResNet(nstage1=1, nstage2=ag.space.Int(5, 10)) print(mynet1.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: nstage2, Type: UniformInteger, Range: [5, 10], Default: 8 Decorate Existing Class ~~~~~~~~~~~~~~~~~~~~~~~ We can also use :func:`autogluon.obj` to easily decorate any existing classes. For example, if we want to search learning rate and weight decay for Adam optimizer, we only need to add a decorator: .. code:: python from mxnet import optimizer as optim @ag.obj() class Adam(optim.Adam): pass Then we can create an instance: .. code:: python myoptim = Adam(learning_rate=ag.Real(1e-2, 1e-1, log=True), wd=ag.Real(1e-5, 1e-3, log=True)) print(myoptim.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: learning_rate, Type: UniformFloat, Range: [0.01, 0.1], Default: 0.0316227766, on log-scale wd, Type: UniformFloat, Range: [1e-05, 0.001], Default: 0.0001, on log-scale Launch Experiments Using AutoGluon Object ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ AutoGluon Object is compatible with Fit API in AutoGluon tasks, and also works with user-defined training scripts using :func:`autogluon.autogluon_register_args`. We can start fitting: .. code:: python from autogluon.vision import ImagePredictor classifier = ImagePredictor().fit('cifar10', hyperparameters={'net': mynet, 'optimizer': myoptim, 'epochs': 1}, ngpus_per_trial=1) .. parsed-literal:: :class: output `time_limit=auto` set to `time_limit=7200`. Starting fit without HPO modified configs( != ): { root.valid.batch_size 128 != 16 root.valid.num_workers 4 != 8 root.train.num_workers 4 != 8 root.train.rec_val ~/.mxnet/datasets/imagenet/rec/val.rec != auto root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto root.train.epochs 10 != 1 root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto root.train.early_stop_max_value 1.0 != inf root.train.data_dir ~/.mxnet/datasets/imagenet != auto root.train.num_training_samples 1281167 != -1 root.train.early_stop_patience -1 != 10 root.train.early_stop_baseline 0.0 != -inf root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto root.train.lr 0.1 != 0.01 root.train.batch_size 128 != 16 root.img_cls.model resnet50_v1 != resnet50 } Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/91aec8aa/.trial_0/config.yaml Start training from [Epoch 0] Epoch[0] Batch [49] Speed: 72.278254 samples/sec accuracy=0.158750 lr=0.010000 Epoch[0] Batch [99] Speed: 72.968321 samples/sec accuracy=0.165000 lr=0.010000 Epoch[0] Batch [149] Speed: 72.555819 samples/sec accuracy=0.166250 lr=0.010000 Epoch[0] Batch [199] Speed: 72.057637 samples/sec accuracy=0.174375 lr=0.010000 Epoch[0] Batch [249] Speed: 71.869353 samples/sec accuracy=0.174000 lr=0.010000 Epoch[0] Batch [299] Speed: 71.952340 samples/sec accuracy=0.177917 lr=0.010000 Epoch[0] Batch [349] Speed: 71.749925 samples/sec accuracy=0.177679 lr=0.010000 Epoch[0] Batch [399] Speed: 71.367287 samples/sec accuracy=0.181719 lr=0.010000 Epoch[0] Batch [449] Speed: 71.074033 samples/sec accuracy=0.183750 lr=0.010000 Epoch[0] Batch [499] Speed: 70.870735 samples/sec accuracy=0.188625 lr=0.010000 Epoch[0] Batch [549] Speed: 70.467299 samples/sec accuracy=0.188523 lr=0.010000 Epoch[0] Batch [599] Speed: 70.131650 samples/sec accuracy=0.190833 lr=0.010000 Epoch[0] Batch [649] Speed: 69.837431 samples/sec accuracy=0.194038 lr=0.010000 Epoch[0] Batch [699] Speed: 69.441294 samples/sec accuracy=0.196339 lr=0.010000 Epoch[0] Batch [749] Speed: 68.977313 samples/sec accuracy=0.198417 lr=0.010000 Epoch[0] Batch [799] Speed: 68.515449 samples/sec accuracy=0.199922 lr=0.010000 Epoch[0] Batch [849] Speed: 67.906139 samples/sec accuracy=0.202500 lr=0.010000 Epoch[0] Batch [899] Speed: 68.611221 samples/sec accuracy=0.205556 lr=0.010000 Epoch[0] Batch [949] Speed: 69.916964 samples/sec accuracy=0.206776 lr=0.010000 Epoch[0] Batch [999] Speed: 70.623258 samples/sec accuracy=0.207625 lr=0.010000 Epoch[0] Batch [1049] Speed: 71.090078 samples/sec accuracy=0.208750 lr=0.010000 Epoch[0] Batch [1099] Speed: 71.241425 samples/sec accuracy=0.208125 lr=0.010000 Epoch[0] Batch [1149] Speed: 71.418239 samples/sec accuracy=0.208315 lr=0.010000 Epoch[0] Batch [1199] Speed: 71.626897 samples/sec accuracy=0.208646 lr=0.010000 Epoch[0] Batch [1249] Speed: 71.638809 samples/sec accuracy=0.209750 lr=0.010000 Epoch[0] Batch [1299] Speed: 71.489135 samples/sec accuracy=0.212212 lr=0.010000 Epoch[0] Batch [1349] Speed: 71.319494 samples/sec accuracy=0.213380 lr=0.010000 Epoch[0] Batch [1399] Speed: 71.283605 samples/sec accuracy=0.214062 lr=0.010000 Epoch[0] Batch [1449] Speed: 71.316281 samples/sec accuracy=0.215216 lr=0.010000 Epoch[0] Batch [1499] Speed: 71.190837 samples/sec accuracy=0.216292 lr=0.010000 Epoch[0] Batch [1549] Speed: 71.041426 samples/sec accuracy=0.216653 lr=0.010000 Epoch[0] Batch [1599] Speed: 71.034183 samples/sec accuracy=0.217461 lr=0.010000 Epoch[0] Batch [1649] Speed: 70.890309 samples/sec accuracy=0.218636 lr=0.010000 Epoch[0] Batch [1699] Speed: 70.646984 samples/sec accuracy=0.219338 lr=0.010000 Epoch[0] Batch [1749] Speed: 70.504651 samples/sec accuracy=0.219607 lr=0.010000 Epoch[0] Batch [1799] Speed: 70.275851 samples/sec accuracy=0.219861 lr=0.010000 Epoch[0] Batch [1849] Speed: 70.005143 samples/sec accuracy=0.219899 lr=0.010000 Epoch[0] Batch [1899] Speed: 69.774077 samples/sec accuracy=0.220296 lr=0.010000 Epoch[0] Batch [1949] Speed: 69.440847 samples/sec accuracy=0.220481 lr=0.010000 Epoch[0] Batch [1999] Speed: 69.015333 samples/sec accuracy=0.221281 lr=0.010000 Epoch[0] Batch [2049] Speed: 68.637815 samples/sec accuracy=0.221280 lr=0.010000 Epoch[0] Batch [2099] Speed: 68.121316 samples/sec accuracy=0.221786 lr=0.010000 Epoch[0] Batch [2149] Speed: 67.598312 samples/sec accuracy=0.222209 lr=0.010000 Epoch[0] Batch [2199] Speed: 67.642253 samples/sec accuracy=0.222983 lr=0.010000 Epoch[0] Batch [2249] Speed: 69.143878 samples/sec accuracy=0.223417 lr=0.010000 Epoch[0] Batch [2299] Speed: 70.120760 samples/sec accuracy=0.223804 lr=0.010000 Epoch[0] Batch [2349] Speed: 70.710918 samples/sec accuracy=0.224894 lr=0.010000 Epoch[0] Batch [2399] Speed: 71.067930 samples/sec accuracy=0.225443 lr=0.010000 Epoch[0] Batch [2449] Speed: 71.199916 samples/sec accuracy=0.226046 lr=0.010000 Epoch[0] Batch [2499] Speed: 71.239341 samples/sec accuracy=0.226750 lr=0.010000 Epoch[0] Batch [2549] Speed: 71.379220 samples/sec accuracy=0.227451 lr=0.010000 Epoch[0] Batch [2599] Speed: 71.322384 samples/sec accuracy=0.227813 lr=0.010000 Epoch[0] Batch [2649] Speed: 71.328304 samples/sec accuracy=0.228443 lr=0.010000 Epoch[0] Batch [2699] Speed: 71.256280 samples/sec accuracy=0.229005 lr=0.010000 Epoch[0] Batch [2749] Speed: 71.249807 samples/sec accuracy=0.229886 lr=0.010000 Epoch[0] Batch [2799] Speed: 71.161437 samples/sec accuracy=0.230446 lr=0.010000 Epoch[0] Batch [2849] Speed: 71.072413 samples/sec accuracy=0.231513 lr=0.010000 Epoch[0] Batch [2899] Speed: 70.911057 samples/sec accuracy=0.232134 lr=0.010000 Epoch[0] Batch [2949] Speed: 70.870319 samples/sec accuracy=0.232627 lr=0.010000 Epoch[0] Batch [2999] Speed: 70.678320 samples/sec accuracy=0.232937 lr=0.010000 Epoch[0] Batch [3049] Speed: 70.412623 samples/sec accuracy=0.233893 lr=0.010000 Epoch[0] Batch [3099] Speed: 70.230713 samples/sec accuracy=0.234415 lr=0.010000 Epoch[0] Batch [3149] Speed: 69.988621 samples/sec accuracy=0.234544 lr=0.010000 Epoch[0] Batch [3199] Speed: 69.720767 samples/sec accuracy=0.235039 lr=0.010000 Epoch[0] Batch [3249] Speed: 69.518797 samples/sec accuracy=0.235635 lr=0.010000 Epoch[0] Batch [3299] Speed: 69.145559 samples/sec accuracy=0.236383 lr=0.010000 Epoch[0] Batch [3349] Speed: 68.718622 samples/sec accuracy=0.236866 lr=0.010000 [Epoch 0] training: accuracy=0.237130 [Epoch 0] speed: 70 samples/sec time cost: 766.344004 [Epoch 0] validation: top1=0.318000 top5=0.863500 [Epoch 0] Current best top-1: 0.318000 vs previous -inf, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/91aec8aa/.trial_0/best_checkpoint.pkl Unable to pickle object due to the reason: Can't pickle : it's not the same object as __main__.MyCifarResNet. This object is not saved. Applying the state from the best checkpoint... Unable to resume the state from the best checkpoint, using the latest state. Finished, total runtime is 791.73 s { 'best_config': { 'batch_size': 16, 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7fe1dc7574d0>, 'dist_ip_addrs': None, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'early_stop_patience': 10, 'epochs': 1, 'final_fit': False, 'gpus': [0], 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/91aec8aa', 'lr': 0.01, 'model': 'resnet50', 'ngpus_per_trial': 1, 'nthreads_per_trial': 128, 'num_trials': 1, 'num_workers': 8, 'problem_type': 'multiclass', 'scheduler': 'local', 'search_strategy': 'random', 'searcher': 'random', 'seed': 211, 'time_limits': 7200, 'wall_clock_tick': 1630109480.8089828}, 'total_time': 777.5042235851288, 'train_acc': 0.23712962962962963, 'valid_acc': 0.318} .. code:: python print(classifier.fit_summary()) .. parsed-literal:: :class: output {'train_acc': 0.23712962962962963, 'valid_acc': 0.318, 'total_time': 777.5042235851288, 'best_config': {'model': 'resnet50', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/91aec8aa', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7fe1dc7574d0>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 211, 'final_fit': False, 'wall_clock_tick': 1630109480.8089828, 'problem_type': 'multiclass'}, 'fit_history': {'train_acc': 0.23712962962962963, 'valid_acc': 0.318, 'total_time': 777.5042235851288, 'best_config': {'model': 'resnet50', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/91aec8aa', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7fe1dc7574d0>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 211, 'final_fit': False, 'wall_clock_tick': 1630109480.8089828, 'problem_type': 'multiclass'}}}