Getting started with Advanced HPO Algorithms

Loading libraries

# Basic utils for folder manipulations etc
import time
import multiprocessing # to count the number of CPUs available

# External tools to load and process data
import numpy as np
import pandas as pd

# MXNet (NeuralNets)
import mxnet as mx
from mxnet import gluon, autograd
from mxnet.gluon import nn

# AutoGluon and HPO tools
import autogluon as ag
from autogluon.utils import load_and_split_openml_data

Check the version of MxNet, you should be fine with version >= 1.5

mx.__version__
/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
  and should_run_async(code)
'1.7.0'

You can also check the version of AutoGluon and the specific commit and check that it matches what you want.

ag.__version__
'0.0.14b20201027'

Hyperparameter Optimization of a 2-layer MLP

Setting up the context

Here we declare a few “environment variables” setting the context for what we’re doing

OPENML_TASK_ID = 6                # describes the problem we will tackle
RATIO_TRAIN_VALID = 0.33          # split of the training data used for validation
RESOURCE_ATTR_NAME = 'epoch'      # how do we measure resources   (will become clearer further)
REWARD_ATTR_NAME = 'objective'    # how do we measure performance (will become clearer further)

NUM_CPUS = multiprocessing.cpu_count()

Preparing the data

We will use a multi-way classification task from OpenML. Data preparation includes:

  • Missing values are imputed, using the ‘mean’ strategy of sklearn.impute.SimpleImputer

  • Split training set into training and validation

  • Standardize inputs to mean 0, variance 1

X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data(
    OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False)
n_classes
Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff...
100%|██████████| 704/704 [00:00<00:00, 57150.41KB/s]
Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3...
100%|██████████| 2521/2521 [00:00<00:00, 51041.66KB/s]
Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml...
3KB [00:00, 2490.68KB/s]
Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml...
8KB [00:00, 4545.44KB/s]
Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml...
15KB [00:00, 9384.63KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff...
2998KB [00:00, 13106.61KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3...
881KB [00:00, 34260.96KB/s]
Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml...
3KB [00:00, 2543.03KB/s]
pickle load data letter
26

The problem has 26 classes.

Declaring a model specifying a hyperparameter space with AutoGluon

Two layer MLP where we optimize over:

  • the number of units on the first layer

  • the number of units on the second layer

  • the dropout rate after each layer

  • the learning rate

  • the scaling

  • the @ag.args decorator allows us to specify the space we will optimize over, this matches the ConfigSpace syntax

The body of the function run_mlp_openml is pretty simple:

  • it reads the hyperparameters given via the decorator

  • it defines a 2 layer MLP with dropout

  • it declares a trainer with the ‘adam’ loss function and a provided learning rate

  • it trains the NN with a number of epochs (most of that is boilerplate code from mxnet)

  • the reporter at the end is used to keep track of training history in the hyperparameter optimization

Note: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment

@ag.args(n_units_1=ag.space.Int(lower=16, upper=128),
         n_units_2=ag.space.Int(lower=16, upper=128),
         dropout_1=ag.space.Real(lower=0, upper=.75),
         dropout_2=ag.space.Real(lower=0, upper=.75),
         learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True),
         batch_size=ag.space.Int(lower=8, upper=128),
         scale_1=ag.space.Real(lower=0.001, upper=10, log=True),
         scale_2=ag.space.Real(lower=0.001, upper=10, log=True),
         epochs=9)
def run_mlp_openml(args, reporter, **kwargs):
    # Time stamp for elapsed_time
    ts_start = time.time()
    # Unwrap hyperparameters
    n_units_1 = args.n_units_1
    n_units_2 = args.n_units_2
    dropout_1 = args.dropout_1
    dropout_2 = args.dropout_2
    scale_1 = args.scale_1
    scale_2 = args.scale_2
    batch_size = args.batch_size
    learning_rate = args.learning_rate

    ctx = mx.cpu()
    net = nn.Sequential()
    with net.name_scope():
        # Layer 1
        net.add(nn.Dense(n_units_1, activation='relu',
                         weight_initializer=mx.initializer.Uniform(scale=scale_1)))
        # Dropout
        net.add(gluon.nn.Dropout(dropout_1))
        # Layer 2
        net.add(nn.Dense(n_units_2, activation='relu',
                         weight_initializer=mx.initializer.Uniform(scale=scale_2)))
        # Dropout
        net.add(gluon.nn.Dropout(dropout_2))
        # Output
        net.add(nn.Dense(n_classes))
    net.initialize(ctx=ctx)

    trainer = gluon.Trainer(net.collect_params(), 'adam',
                            {'learning_rate': learning_rate})

    for epoch in range(args.epochs):
        ts_epoch = time.time()

        train_iter = mx.io.NDArrayIter(
                        data={'data': X_train},
                        label={'label': y_train},
                        batch_size=batch_size,
                        shuffle=True)
        valid_iter = mx.io.NDArrayIter(
                        data={'data': X_valid},
                        label={'label': y_valid},
                        batch_size=batch_size,
                        shuffle=False)

        metric = mx.metric.Accuracy()
        loss = gluon.loss.SoftmaxCrossEntropyLoss()

        for batch in train_iter:
            data = batch.data[0].as_in_context(ctx)
            label = batch.label[0].as_in_context(ctx)
            with autograd.record():
                output = net(data)
                L = loss(output, label)
            L.backward()
            trainer.step(data.shape[0])
            metric.update([label], [output])

        name, train_acc = metric.get()

        metric = mx.metric.Accuracy()
        for batch in valid_iter:
            data = batch.data[0].as_in_context(ctx)
            label = batch.label[0].as_in_context(ctx)
            output = net(data)
            metric.update([label], [output])

        name, val_acc = metric.get()

        print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % (
            epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc))

        ts_now = time.time()
        eval_time = ts_now - ts_epoch
        elapsed_time = ts_now - ts_start

        # The resource reported back (as 'epoch') is the number of epochs
        # done, starting at 1
        reporter(
            epoch=epoch + 1,
            objective=float(val_acc),
            eval_time=eval_time,
            time_step=ts_now,
            elapsed_time=elapsed_time)
/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
  and should_run_async(code)

Note: The annotation epochs=9 specifies the maximum number of epochs for training. It becomes available as args.epochs. Importantly, it is also processed by HyperbandScheduler below in order to set its max_t attribute.

Recommendation: Whenever writing training code to be passed as train_fn to a scheduler, if this training code reports a resource (or time) attribute, the corresponding maximum resource value should be included in train_fn.args:

  • If the resource attribute (time_attr of scheduler) in train_fn is epoch, make sure to include epochs=XYZ in the annotation. This allows the scheduler to read max_t from train_fn.args.epochs. This case corresponds to our example here.

  • If the resource attribute is something else than epoch, you can also include the annotation max_t=XYZ, which allows the scheduler to read max_t from train_fn.args.max_t.

Annotating the training function by the correct value for max_t simplifies scheduler creation (since max_t does not have to be passed), and avoids inconsistencies between train_fn and the scheduler.

Running the Hyperparameter Optimization

You can use the following schedulers:

  • FIFO (fifo)

  • Hyperband (either the stopping (hbs) or promotion (hbp) variant)

And the following searchers:

  • Random search (random)

  • Gaussian process based Bayesian optimization (bayesopt)

  • SkOpt Bayesian optimization (skopt; only with FIFO scheduler)

Note that the method known as (asynchronous) Hyperband is using random search. Combining Hyperband scheduling with the bayesopt searcher uses a novel method called asynchronous BOHB.

Pick the combination you’re interested in (doing the full experiment takes around 120 seconds, see the time_out parameter), running everything with multiple runs can take a fair bit of time. In real life, you will want to choose a larger time_out in order to obtain good performance.

SCHEDULER = "hbs"
SEARCHER = "bayesopt"
def compute_error(df):
    return 1.0 - df["objective"]

def compute_runtime(df, start_timestamp):
        return df["time_step"] - start_timestamp

def process_training_history(task_dicts, start_timestamp,
                             runtime_fn=compute_runtime,
                             error_fn=compute_error):
    task_dfs = []
    for task_id in task_dicts:
        task_df = pd.DataFrame(task_dicts[task_id])
        task_df = task_df.assign(task_id=task_id,
                                 runtime=runtime_fn(task_df, start_timestamp),
                                 error=error_fn(task_df),
                                 target_epoch=task_df["epoch"].iloc[-1])
        task_dfs.append(task_df)

    result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True)
    # re-order by runtime
    result = result.sort_values(by="runtime")
    # calculate incumbent best -- the cumulative minimum of the error.
    result = result.assign(best=result["error"].cummin())
    return result

resources = dict(num_cpus=NUM_CPUS, num_gpus=0)
search_options = {
    'num_init_random': 2,
    'debug_log': True}
if SCHEDULER == 'fifo':
    myscheduler = ag.scheduler.FIFOScheduler(
        run_mlp_openml,
        resource=resources,
        searcher=SEARCHER,
        search_options=search_options,
        time_out=120,
        time_attr=RESOURCE_ATTR_NAME,
        reward_attr=REWARD_ATTR_NAME)

else:
    # This setup uses rung levels at 1, 3, 9 epochs. We just use a single
    # bracket, so this is in fact successive halving (Hyperband would use
    # more than 1 bracket).
    # Also note that since we do not use the max_t argument of
    # HyperbandScheduler, this value is obtained from train_fn.args.epochs.
    sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion'
    myscheduler = ag.scheduler.HyperbandScheduler(
        run_mlp_openml,
        resource=resources,
        searcher=SEARCHER,
        search_options=search_options,
        time_out=120,
        time_attr=RESOURCE_ATTR_NAME,
        reward_attr=REWARD_ATTR_NAME,
        type=sch_type,
        grace_period=1,
        reduction_factor=3,
        brackets=1)

# run tasks
myscheduler.run()
myscheduler.join_jobs()

results_df = process_training_history(
                myscheduler.training_history.copy(),
                start_timestamp=myscheduler._start_time)
max_t = 9, as inferred from train_fn.args
scheduler_options: Key 'resume': Imputing default value False
scheduler_options: Key 'keep_size_ratios': Imputing default value False
scheduler_options: Key 'maxt_pending': Imputing default value False
scheduler_options: Key 'searcher_data': Imputing default value rungs
scheduler_options: Key 'do_snapshots': Imputing default value False

scheduler_options: Key 'visualizer': Imputing default value none
scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60
scheduler_options: Key 'delay_get_config': Imputing default value True

search_options: Key 'random_seed': Imputing default value 7058
search_options: Key 'opt_skip_init_length': Imputing default value 150
search_options: Key 'opt_skip_period': Imputing default value 1
search_options: Key 'profiler': Imputing default value False
search_options: Key 'opt_maxiter': Imputing default value 50
search_options: Key 'opt_nstarts': Imputing default value 2
search_options: Key 'opt_warmstart': Imputing default value False
search_options: Key 'opt_verbose': Imputing default value False
search_options: Key 'opt_debug_writer': Imputing default value False
search_options: Key 'num_fantasy_samples': Imputing default value 20
search_options: Key 'num_init_candidates': Imputing default value 250
search_options: Key 'initial_scoring': Imputing default value thompson_indep
search_options: Key 'first_is_default': Imputing default value True
search_options: Key 'opt_skip_num_max_resource': Imputing default value False
search_options: Key 'gp_resource_kernel': Imputing default value matern52
search_options: Key 'resource_acq': Imputing default value bohb

[GPMultiFidelitySearcher.__init__]
- acquisition_class = <class 'autogluon.searcher.bayesopt.models.nphead_acqfunc.EIAcquisitionFunction'>
- local_minimizer_class = <class 'autogluon.searcher.bayesopt.tuning_algorithms.bo_algorithm_components.LBFGSOptimizeAcquisition'>
- num_initial_candidates = 250
- num_initial_random_choices = 2
- initial_scoring = thompson_indep
- first_is_default = True
Starting Experiments
Num of Finished Tasks is 0
Time out (secs) is 120
Starting get_config[random] for config_id 0
Start with default config:
{'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1}
[0: random]
batch_size: 68
dropout_1: 0.375
dropout_2: 0.375
learning_rate: 0.001
n_units_1: 72
n_units_2: 72
scale_1: 0.1
scale_2: 0.1
/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/distributed/worker.py:3382: UserWarning: Large object of size 1.30 MB detected in task graph:
  (<function run_mlp_openml at 0x7f2386a5fb90>, {'ar ... sReporter}, [])
Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and
keep data on workers

    future = client.submit(func, big_data)    # bad

    big_future = client.scatter(big_data)     # good
    future = client.submit(func, big_future)  # good
  % (format_bytes(len(b)), s)
Epoch 1 ; Time: 0.486747 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250
Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875
config_id 0: Reaches 1, continues to 3
Epoch 2 ; Time: 0.920213 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247
Epoch 3 ; Time: 1.383752 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686
Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464
config_id 0: Reaches 3, continues to 9
Epoch 4 ; Time: 1.819390 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063
Epoch 5 ; Time: 2.266399 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939
Epoch 6 ; Time: 2.734506 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321
Epoch 7 ; Time: 3.159239 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501
Epoch 8 ; Time: 3.584904 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202
Epoch 9 ; Time: 4.010921 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043
config_id 0: Terminating evaluation at 9
Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695
Starting get_config[random] for config_id 1
[1: random]
batch_size: 12
dropout_1: 0.3619427579190121
dropout_2: 0.7008556940852432
learning_rate: 0.0006534843560615438
n_units_1: 17
n_units_2: 57
scale_1: 4.706210274490814
scale_2: 0.14101744075889586
Epoch 1 ; Time: 2.440908 ; Training: accuracy=0.121227 ; Validation: accuracy=0.414478
config_id 1: Terminating evaluation at 1
Update for config_id 1:1: reward = 0.4144781144781145, crit_val = 0.5855218855218856
Starting get_config[BO] for config_id 2
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3983858189740543
- self.std = 0.1375410133514106
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 4
Current best is [0.46875]
[2: BO] (15 evaluations)
batch_size: 128
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 1.0
n_units_1: 128
n_units_2: 16
scale_1: 0.0010000000000000002
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 47
dropout_1: 0.04435796153318
dropout_2: 0.007812119878330748
learning_rate: 0.002236512923826564
n_units_1: 53
n_units_2: 50
scale_1: 0.018695005465870176
scale_2: 0.001233683341855758
Top score values: [0.14496524 0.24058498 0.24844708 0.24848887 0.26669571]
Labeled: 0:1, 0:3, 0:9, 1:1. Pending:
Targets: [ 0.5115869  -0.67668287 -1.19548777  1.36058374]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.9278942362552179, 'kernel_inv_bw1': 1.0004995788011803, 'kernel_inv_bw2': 0.9796615029762179, 'kernel_inv_bw3': 1.0015600012174053, 'kernel_inv_bw4': 0.8780658298992018, 'kernel_inv_bw5': 1.0273639466830176, 'kernel_inv_bw6': 1.0050984544236925, 'kernel_inv_bw7': 1.0022879688034811, 'kernel_inv_bw8': 100.00000000000004, 'kernel_covariance_scale': 0.705889844789842, 'mean_mean_value': -0.17864666284973565}
Epoch 1 ; Time: 0.299392 ; Training: accuracy=0.036513 ; Validation: accuracy=0.037733
config_id 2: Terminating evaluation at 1
Update for config_id 2:1: reward = 0.037732712765957445, crit_val = 0.9622672872340425
Starting get_config[BO] for config_id 3
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5111621126260519
- self.std = 0.2569202092528199
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 5
Current best is [0.46875]
[3: BO] (1 evaluations)
batch_size: 46
dropout_1: 0.42695183226405875
dropout_2: 0.022833753493764902
learning_rate: 1.710004895394534e-05
n_units_1: 93
n_units_2: 128
scale_1: 0.027714348983233907
scale_2: 0.0010089600551032943
Started BO from (top scorer):
batch_size: 46
dropout_1: 0.42695183226405875
dropout_2: 0.022833753493764902
learning_rate: 1.710004895394534e-05
n_units_1: 93
n_units_2: 128
scale_1: 0.027714348983233907
scale_2: 0.0010089600551032943
Top score values: [-0.04826039 -0.01248857  0.01462949  0.04008222  0.07387793]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending:
Targets: [-0.16507893 -0.80121351 -1.07895324  0.2894275   1.75581818]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.7044274833313655, 'kernel_inv_bw1': 0.4336078534639402, 'kernel_inv_bw2': 0.6308971073833392, 'kernel_inv_bw3': 1.109899251121662, 'kernel_inv_bw4': 7.643179765072924, 'kernel_inv_bw5': 0.37062151700188917, 'kernel_inv_bw6': 100.00000000000004, 'kernel_inv_bw7': 0.20299548392421968, 'kernel_inv_bw8': 1.5554819021123705, 'kernel_covariance_scale': 0.8063802816624212, 'mean_mean_value': 0.303660891397239}
Epoch 1 ; Time: 0.775516 ; Training: accuracy=0.293106 ; Validation: accuracy=0.389632
config_id 3: Terminating evaluation at 1
Update for config_id 3:1: reward = 0.3896321070234114, crit_val = 0.6103678929765886
Starting get_config[BO] for config_id 4
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5276964093511415
- self.std = 0.23743120360024392
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 6
Current best is [0.46875]
[4: BO] (88 evaluations)
batch_size: 128
dropout_1: 0.39942908126701493
dropout_2: 0.0
learning_rate: 0.0022116630251412697
n_units_1: 128
n_units_2: 100
scale_1: 0.0010000000000000002
scale_2: 0.052690027692572375
Started BO from (top scorer):
batch_size: 38
dropout_1: 0.009530064696840201
dropout_2: 0.3094404663366098
learning_rate: 3.1584484383421487e-06
n_units_1: 16
n_units_2: 70
scale_1: 0.09619627668811008
scale_2: 0.32195755009882676
Top score values: [0.0656661  0.09464238 0.11711916 0.11773622 0.16386244]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending:
Targets: [-0.24826732 -0.93661758 -1.23715496  0.24354624  1.8303023   0.34819132]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.26770433903981694, 'kernel_inv_bw1': 9.008462609616624, 'kernel_inv_bw2': 0.27484411839257467, 'kernel_inv_bw3': 1.2371853972057303, 'kernel_inv_bw4': 0.26141064709947986, 'kernel_inv_bw5': 1.014167928090242, 'kernel_inv_bw6': 0.3191357183696273, 'kernel_inv_bw7': 9.319953352593487, 'kernel_inv_bw8': 1.5418386647973328, 'kernel_covariance_scale': 0.8852584286657372, 'mean_mean_value': 0.34448915870871255}
Epoch 1 ; Time: 0.322027 ; Training: accuracy=0.370230 ; Validation: accuracy=0.614694
Update for config_id 4:1: reward = 0.6146941489361702, crit_val = 0.38530585106382975
config_id 4: Reaches 1, continues to 3
Epoch 2 ; Time: 0.558452 ; Training: accuracy=0.626563 ; Validation: accuracy=0.702460
Epoch 3 ; Time: 0.791223 ; Training: accuracy=0.691201 ; Validation: accuracy=0.767287
Update for config_id 4:3: reward = 0.7672872340425532, crit_val = 0.23271276595744683
config_id 4: Reaches 3, continues to 9
Epoch 4 ; Time: 1.019551 ; Training: accuracy=0.730839 ; Validation: accuracy=0.791888
Epoch 5 ; Time: 1.270133 ; Training: accuracy=0.757977 ; Validation: accuracy=0.808178
Epoch 6 ; Time: 1.501433 ; Training: accuracy=0.778783 ; Validation: accuracy=0.833610
Epoch 7 ; Time: 1.725927 ; Training: accuracy=0.793914 ; Validation: accuracy=0.847407
Epoch 8 ; Time: 1.951463 ; Training: accuracy=0.810033 ; Validation: accuracy=0.862699
Epoch 9 ; Time: 2.176672 ; Training: accuracy=0.821793 ; Validation: accuracy=0.872008
config_id 4: Terminating evaluation at 9
Update for config_id 4:9: reward = 0.8720079787234043, crit_val = 0.1279920212765957
Starting get_config[BO] for config_id 5
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.43468767715608003
- self.std = 0.24208351614056364
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 9
Current best is [0.38530585]
[5: BO] (24 evaluations)
batch_size: 128
dropout_1: 0.4494668530275533
dropout_2: 0.5660951649140447
learning_rate: 1.0
n_units_1: 128
n_units_2: 76
scale_1: 0.0013457507502543137
scale_2: 0.002436972541321956
Started BO from (top scorer):
batch_size: 103
dropout_1: 0.5756338329443644
dropout_2: 0.5660949820168737
learning_rate: 0.6204655178098845
n_units_1: 114
n_units_2: 76
scale_1: 0.0013457541111761644
scale_2: 0.0022790372086615323
Top score values: [-0.11081183  0.09391514  0.11175324  0.14753379  0.18762432]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9. Pending:
Targets: [ 0.14070484 -0.53441683 -0.82917855  0.62306683  2.17932893  0.72570086
 -0.20398674 -0.83431914 -1.2669002 ]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.4557972912475852, 'kernel_inv_bw1': 3.7514858616114557, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 1.338831201356535, 'kernel_inv_bw4': 0.03567763552406428, 'kernel_inv_bw5': 0.004564602359102062, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.011084852077624103, 'kernel_inv_bw8': 1.1457244980868717, 'kernel_covariance_scale': 1.138206063971665, 'mean_mean_value': 0.7269209083511647}
Epoch 1 ; Time: 0.290375 ; Training: accuracy=0.039638 ; Validation: accuracy=0.037566
config_id 5: Terminating evaluation at 1
Update for config_id 5:1: reward = 0.03756648936170213, crit_val = 0.9624335106382979
Starting get_config[BO] for config_id 6
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4874622605043018
- self.std = 0.27894514825765715
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 10
Current best is [0.38530585]
[6: BO] (18 evaluations)
batch_size: 73
dropout_1: 0.6415441394989139
dropout_2: 0.576949866176584
learning_rate: 0.008350048931638588
n_units_1: 63
n_units_2: 75
scale_1: 3.3378723585220875
scale_2: 9.155286357407212
Started BO from (top scorer):
batch_size: 12
dropout_1: 0.6415643976704684
dropout_2: 0.5777698271418924
learning_rate: 0.0738083253883884
n_units_1: 63
n_units_2: 75
scale_1: 3.469695096164748
scale_2: 9.155955641147331
Top score values: [0.23580231 0.24592884 0.30603732 0.32449038 0.33338612]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1. Pending:
Targets: [-0.06708222 -0.65298891 -0.90879889  0.3515373   1.70214478  0.4406086
 -0.36622401 -0.91326017 -1.28867715  1.70274068]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.4469623643229872, 'kernel_inv_bw1': 0.00010020827693492103, 'kernel_inv_bw2': 0.0005666174769466442, 'kernel_inv_bw3': 3.680085817082452, 'kernel_inv_bw4': 0.00034420299746394623, 'kernel_inv_bw5': 0.00028859512699021175, 'kernel_inv_bw6': 0.005353495891411828, 'kernel_inv_bw7': 0.0001367486776381668, 'kernel_inv_bw8': 1.1422612480374623, 'kernel_covariance_scale': 0.882616090617811, 'mean_mean_value': 0.46655516540495395}
Epoch 1 ; Time: 0.466035 ; Training: accuracy=0.090857 ; Validation: accuracy=0.316906
config_id 6: Terminating evaluation at 1
Update for config_id 6:1: reward = 0.31690611426662213, crit_val = 0.6830938857333779
Starting get_config[BO] for config_id 7
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5052469537069451
- self.std = 0.27184494941037113
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 11
Current best is [0.38530585]
[7: BO] (2 evaluations)
batch_size: 38
dropout_1: 0.01428523178697837
dropout_2: 0.320010360931437
learning_rate: 1.0000000000000004e-06
n_units_1: 103
n_units_2: 110
scale_1: 0.1729494836765736
scale_2: 1.3251555487835274
Started BO from (top scorer):
batch_size: 38
dropout_1: 0.0142852321792937
dropout_2: 0.32001035746238826
learning_rate: 6.073612217036369e-06
n_units_1: 103
n_units_2: 110
scale_1: 0.17294927471988455
scale_2: 1.325155545206798
Top score values: [0.2609173  0.31760945 0.31867298 0.33532668 0.3531575 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1. Pending:
Targets: [-0.13425651 -0.73546624 -0.9979576   0.29529676  1.68118015  0.38669447
 -0.44121144 -1.00253541 -1.38775774  1.68179162  0.65422195]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0061766551001926585, 'kernel_inv_bw1': 0.00016481337620937624, 'kernel_inv_bw2': 0.0006016214023291494, 'kernel_inv_bw3': 8.219180666926277, 'kernel_inv_bw4': 0.006813346793894043, 'kernel_inv_bw5': 0.000341856188600768, 'kernel_inv_bw6': 0.004630052137556817, 'kernel_inv_bw7': 0.00010430707112655825, 'kernel_inv_bw8': 1.1778023413966061, 'kernel_covariance_scale': 0.8224249977911093, 'mean_mean_value': 0.5441760364967412}
Epoch 1 ; Time: 0.831820 ; Training: accuracy=0.064465 ; Validation: accuracy=0.069896
config_id 7: Terminating evaluation at 1
Update for config_id 7:1: reward = 0.06989607777405296, crit_val = 0.9301039222259471
Starting get_config[BO] for config_id 8
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5406517010835286
- self.std = 0.2855342845405141
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 12
Current best is [0.38530585]
[8: BO] (5 evaluations)
batch_size: 115
dropout_1: 0.24187944810886564
dropout_2: 0.29346258186805196
learning_rate: 6.751481632121916e-05
n_units_1: 18
n_units_2: 24
scale_1: 0.1411654746911316
scale_2: 0.08896336352586504
Started BO from (top scorer):
batch_size: 115
dropout_1: 0.24187944834262717
dropout_2: 0.2934625871421564
learning_rate: 0.00011907707756307102
n_units_1: 18
n_units_2: 24
scale_1: 0.14116563031490334
scale_2: 0.08896336349981186
Top score values: [0.30833232 0.3272424  0.34884514 0.36395197 0.36966334]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1. Pending:
Targets: [-0.2518146  -0.8242006  -1.07410738  0.15714465  1.47658481  0.24416049
 -0.54405323 -1.07846571 -1.44521937  1.47716695  0.49886193  1.36394206]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.000907949230138543, 'kernel_inv_bw1': 0.00019848100171743864, 'kernel_inv_bw2': 0.0004935556436282336, 'kernel_inv_bw3': 8.225855470755034, 'kernel_inv_bw4': 0.002235578895046009, 'kernel_inv_bw5': 0.00060980732315924, 'kernel_inv_bw6': 0.002180083609459959, 'kernel_inv_bw7': 0.00010328699027201036, 'kernel_inv_bw8': 1.1303894549573745, 'kernel_covariance_scale': 0.7690471563932195, 'mean_mean_value': 0.5960626251912128}
Epoch 1 ; Time: 0.322462 ; Training: accuracy=0.044141 ; Validation: accuracy=0.069565
config_id 8: Terminating evaluation at 1
Update for config_id 8:1: reward = 0.06956521739130435, crit_val = 0.9304347826086956
Starting get_config[BO] for config_id 9
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.570635015047003
- self.std = 0.29333648228468523
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 13
Current best is [0.38530585]
[9: BO] (7 evaluations)
batch_size: 117
dropout_1: 0.31690126549933817
dropout_2: 0.23555191634866235
learning_rate: 0.001757353043631389
n_units_1: 44
n_units_2: 73
scale_1: 1.8397367144274908
scale_2: 0.010548059673332347
Started BO from (top scorer):
batch_size: 117
dropout_1: 0.31690126898227333
dropout_2: 0.23555191623431485
learning_rate: 0.0018560533023470626
n_units_1: 44
n_units_2: 73
scale_1: 1.8397366918432363
scale_2: 0.01054814057574058
Top score values: [0.36933265 0.3694303  0.3727333  0.37760432 0.37779149]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1. Pending:
Targets: [-0.34733155 -0.90449317 -1.14775289  0.05075015  1.33509569  0.13545154
 -0.63179719 -1.1519953  -1.50899401  1.33566235  0.3833784   1.22544903
  1.22657695]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0001461217271029719, 'kernel_inv_bw1': 0.00035216819863875484, 'kernel_inv_bw2': 0.00010049584476263256, 'kernel_inv_bw3': 8.306817442621444, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.0001431142771725629, 'kernel_inv_bw6': 0.00010000090099432115, 'kernel_inv_bw7': 0.0044658766082197, 'kernel_inv_bw8': 1.0875606987022017, 'kernel_covariance_scale': 0.7892306882657111, 'mean_mean_value': 0.5684728018701748}
Epoch 1 ; Time: 0.332736 ; Training: accuracy=0.357084 ; Validation: accuracy=0.631138
Update for config_id 9:1: reward = 0.6311379252555723, crit_val = 0.36886207474442767
config_id 9: Reaches 1, continues to 3
Epoch 2 ; Time: 0.602031 ; Training: accuracy=0.508629 ; Validation: accuracy=0.680409
Epoch 3 ; Time: 0.864318 ; Training: accuracy=0.555720 ; Validation: accuracy=0.710407
config_id 9: Terminating evaluation at 3
Update for config_id 9:3: reward = 0.7104072398190046, crit_val = 0.28959276018099545
Starting get_config[BO] for config_id 10
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5384473353690975
- self.std = 0.2855121902268272
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 15
Current best is [0.36886208]
[10: BO] (11 evaluations)
batch_size: 20
dropout_1: 0.295623405043272
dropout_2: 0.32083397832786376
learning_rate: 0.0043078719806179
n_units_1: 110
n_units_2: 56
scale_1: 1.8620051590717166
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 20
dropout_1: 0.2956229459366064
dropout_2: 0.32083398214435827
learning_rate: 0.0005836358528158626
n_units_1: 110
n_units_2: 56
scale_1: 1.8620053228479063
scale_2: 0.002773646147822261
Top score values: [0.22729669 0.26204432 0.27243974 0.27259687 0.31154225]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3. Pending:
Targets: [-0.24411334 -0.81654364 -1.06646976  0.16487755  1.48441981  0.25190013
 -0.53637459 -1.07082843 -1.43761047  1.48500201  0.50662128  1.37176835
  1.37292718 -0.59396855 -0.87160753]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000001592422845, 'kernel_inv_bw1': 0.002647264694538466, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.825342705816994, 'kernel_inv_bw4': 0.0016805124935032668, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00010000000267193107, 'kernel_inv_bw7': 2.078001098193213, 'kernel_inv_bw8': 1.111294642672605, 'kernel_covariance_scale': 0.7131454962634407, 'mean_mean_value': 0.615868796574879}
Epoch 1 ; Time: 1.622923 ; Training: accuracy=0.509784 ; Validation: accuracy=0.731650
Update for config_id 10:1: reward = 0.7316498316498317, crit_val = 0.26835016835016834
config_id 10: Reaches 1, continues to 3
Epoch 2 ; Time: 3.278953 ; Training: accuracy=0.625788 ; Validation: accuracy=0.798485
Epoch 3 ; Time: 4.785232 ; Training: accuracy=0.656799 ; Validation: accuracy=0.810101
Update for config_id 10:3: reward = 0.8101010101010101, crit_val = 0.1898989898989899
config_id 10: Reaches 3, continues to 9
Epoch 4 ; Time: 6.300443 ; Training: accuracy=0.679353 ; Validation: accuracy=0.812121
Epoch 5 ; Time: 7.785443 ; Training: accuracy=0.691874 ; Validation: accuracy=0.803367
Epoch 6 ; Time: 9.316005 ; Training: accuracy=0.703897 ; Validation: accuracy=0.825926
Epoch 7 ; Time: 10.978170 ; Training: accuracy=0.718740 ; Validation: accuracy=0.837037
Epoch 8 ; Time: 12.569396 ; Training: accuracy=0.718823 ; Validation: accuracy=0.842761
Epoch 9 ; Time: 14.177347 ; Training: accuracy=0.733582 ; Validation: accuracy=0.844781
config_id 10: Terminating evaluation at 9
Update for config_id 10:9: reward = 0.8447811447811447, crit_val = 0.15521885521885526
Starting get_config[BO] for config_id 11
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4827876691113599
- self.std = 0.28947238269911535
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 18
Current best is [0.26835017]
[11: BO] (15 evaluations)
batch_size: 116
dropout_1: 0.6963488927154586
dropout_2: 0.5109253028686711
learning_rate: 0.008166010866125228
n_units_1: 16
n_units_2: 63
scale_1: 0.05786531400463329
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 116
dropout_1: 0.695457505335067
dropout_2: 0.5109252161008294
learning_rate: 0.03449832398876095
n_units_1: 71
n_units_2: 63
scale_1: 0.05786671441184137
scale_2: 0.0011630243727054094
Top score values: [0.17851842 0.2113652  0.22020969 0.23529457 0.23548254]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9. Pending:
Targets: [-0.04849398 -0.61309302 -0.85959996  0.35490162  1.65639158  0.44073366
 -0.33675689 -0.863899   -1.2256632   1.65696581  0.69197004  1.54528128
  1.54642425 -0.39356291 -0.6674036  -0.74078742 -1.0118018  -1.13160644]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.007623330561608118, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 3.889470558609023, 'kernel_inv_bw4': 0.41178743060982825, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.0003738855720075773, 'kernel_inv_bw7': 0.8864354946634273, 'kernel_inv_bw8': 1.0716508686536432, 'kernel_covariance_scale': 0.9509275207021439, 'mean_mean_value': 0.6619089106763194}
Epoch 1 ; Time: 0.386148 ; Training: accuracy=0.123011 ; Validation: accuracy=0.375663
config_id 11: Terminating evaluation at 1
Update for config_id 11:1: reward = 0.3756631299734748, crit_val = 0.6243368700265253
Starting get_config[BO] for config_id 12
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4902376270542633
- self.std = 0.2835190819629447
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 19
Current best is [0.26835017]
[12: BO] (15 evaluations)
batch_size: 112
dropout_1: 0.7432191952093735
dropout_2: 0.012650458902374555
learning_rate: 2.6896464714959505e-05
n_units_1: 128
n_units_2: 84
scale_1: 0.001010230236569222
scale_2: 0.03746371741472742
Started BO from (top scorer):
batch_size: 112
dropout_1: 0.7099736817858229
dropout_2: 0.012669738594579677
learning_rate: 0.00015540974327731252
n_units_1: 24
n_units_2: 84
scale_1: 0.0010102333144228614
scale_2: 0.07309960274761212
Top score values: [0.17172392 0.18330908 0.20783873 0.23382025 0.2386564 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9, 11:1. Pending:
Targets: [-0.075789   -0.65224342 -0.90392649  0.33607706  1.66489556  0.4237114
 -0.37010481 -0.9083158  -1.27767628  1.66548184  0.68022321  1.55145217
  1.55261915 -0.42810364 -0.7076944  -0.78261914 -1.05932424 -1.18164453
  0.47298137]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0021290482003229246, 'kernel_inv_bw1': 3.3437633303320276, 'kernel_inv_bw2': 0.0008780812931283295, 'kernel_inv_bw3': 4.711914530156885, 'kernel_inv_bw4': 0.3405253744426396, 'kernel_inv_bw5': 0.007521251637694033, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.03632877220280582, 'kernel_inv_bw8': 1.0738772719586922, 'kernel_covariance_scale': 0.798083650178426, 'mean_mean_value': 0.767529988077872}
Epoch 1 ; Time: 0.364694 ; Training: accuracy=0.134507 ; Validation: accuracy=0.254134
config_id 12: Terminating evaluation at 1
Update for config_id 12:1: reward = 0.2541335978835979, crit_val = 0.7458664021164021
Starting get_config[BO] for config_id 13
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5030190658073702
- self.std = 0.2819004340199647
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 20
Current best is [0.26835017]
[13: BO] (29 evaluations)
batch_size: 20
dropout_1: 0.29707543173731416
dropout_2: 0.44903406035061033
learning_rate: 0.009128453517686305
n_units_1: 16
n_units_2: 57
scale_1: 0.9551109652013693
scale_2: 10.0
Started BO from (top scorer):
batch_size: 20
dropout_1: 0.21690067329658475
dropout_2: 0.44904599729788264
learning_rate: 0.004202839773499723
n_units_1: 59
n_units_2: 57
scale_1: 0.9554397704530159
scale_2: 0.5637429930115935
Top score values: [0.17354037 0.19240304 0.22751181 0.25383203 0.25416248]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9, 11:1, 12:1. Pending:
Targets: [-0.12156443 -0.70132881 -0.95445701  0.29266652  1.62911499  0.38080405
 -0.41757018 -0.95887153 -1.33035285  1.62970464  0.63878873  1.51502022
  1.51619389 -0.47590204 -0.75709818 -0.83245313 -1.11074705 -1.23376969
  0.43035693  0.86146493]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00017916047497614195, 'kernel_inv_bw1': 3.5694337893103407, 'kernel_inv_bw2': 0.00033292873515583375, 'kernel_inv_bw3': 4.877173642973564, 'kernel_inv_bw4': 0.3535915667753214, 'kernel_inv_bw5': 0.0013011635988467787, 'kernel_inv_bw6': 0.0004686064235519834, 'kernel_inv_bw7': 0.055341861820093464, 'kernel_inv_bw8': 1.0769154946518156, 'kernel_covariance_scale': 0.7392458458758574, 'mean_mean_value': 0.7909489816473}
Epoch 1 ; Time: 1.670342 ; Training: accuracy=0.230348 ; Validation: accuracy=0.451515
Update for config_id 13:1: reward = 0.45151515151515154, crit_val = 0.5484848484848485
config_id 13: Reaches 1, continues to 3
Epoch 2 ; Time: 3.289912 ; Training: accuracy=0.315257 ; Validation: accuracy=0.488384
Epoch 3 ; Time: 4.773031 ; Training: accuracy=0.349668 ; Validation: accuracy=0.511953
config_id 13: Terminating evaluation at 3
Update for config_id 13:3: reward = 0.5119528619528619, crit_val = 0.48804713804713806
Starting get_config[BO] for config_id 14
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5044051501217905
- self.std = 0.2689716209239652
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 22
Current best is [0.26835017]
[14: BO] (18 evaluations)
batch_size: 65
dropout_1: 0.0
dropout_2: 0.38119864599587466
learning_rate: 0.0029592758005345028
n_units_1: 128
n_units_2: 69
scale_1: 0.6099056924156824
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 65
dropout_1: 0.013295147184842981
dropout_2: 0.3811986268366724
learning_rate: 0.0033963935098966715
n_units_1: 17
n_units_2: 69
scale_1: 0.609905634090425
scale_2: 0.05055686634320212
Top score values: [0.21592655 0.29315608 0.30704734 0.31535854 0.33610127]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9, 11:1, 12:1, 13:1, 13:3. Pending:
Targets: [-0.13256101 -0.74019325 -1.00548872  0.30158102  1.70226932  0.3939551
 -0.44279504 -1.01011543 -1.39945295  1.70288731  0.66434048  1.58269029
  1.58392038 -0.50393077 -0.79864333 -0.8776204  -1.16929124 -1.29822728
  0.44588987  0.89772018  0.16388234 -0.06081687]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.9091876382445208, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.378404460638416, 'kernel_inv_bw4': 0.4499972065223455, 'kernel_inv_bw5': 0.00010537071575831221, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.8499783102221391, 'kernel_inv_bw8': 1.0525964467979392, 'kernel_covariance_scale': 0.9037893854953893, 'mean_mean_value': 0.7466917471842721}
Epoch 1 ; Time: 0.526799 ; Training: accuracy=0.522911 ; Validation: accuracy=0.734448
Update for config_id 14:1: reward = 0.7344481605351171, crit_val = 0.2655518394648829
config_id 14: Reaches 1, continues to 3
Epoch 2 ; Time: 0.979934 ; Training: accuracy=0.711993 ; Validation: accuracy=0.807023
Epoch 3 ; Time: 1.423608 ; Training: accuracy=0.754425 ; Validation: accuracy=0.829599
Update for config_id 14:3: reward = 0.8295986622073579, crit_val = 0.1704013377926421
config_id 14: Reaches 3, continues to 9
Epoch 4 ; Time: 1.869881 ; Training: accuracy=0.784202 ; Validation: accuracy=0.858027
Epoch 5 ; Time: 2.346884 ; Training: accuracy=0.804053 ; Validation: accuracy=0.869732
Epoch 6 ; Time: 2.839576 ; Training: accuracy=0.818528 ; Validation: accuracy=0.872910
Epoch 7 ; Time: 3.313582 ; Training: accuracy=0.827130 ; Validation: accuracy=0.886455
Epoch 8 ; Time: 3.782759 ; Training: accuracy=0.841853 ; Validation: accuracy=0.894314
Epoch 9 ; Time: 4.283800 ; Training: accuracy=0.842928 ; Validation: accuracy=0.898328
config_id 14: Terminating evaluation at 9
Update for config_id 14:9: reward = 0.8983277591973244, crit_val = 0.10167224080267556
Starting get_config[BO] for config_id 15
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4653815488295836
- self.std = 0.27454240717526435
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 25
Current best is [0.26555184]
[15: BO] (17 evaluations)
batch_size: 8
dropout_1: 0.17552079386093805
dropout_2: 0.26843500669914777
learning_rate: 0.002917151177111673
n_units_1: 99
n_units_2: 112
scale_1: 0.24841212241690597
scale_2: 0.039100417280603116
Started BO from (top scorer):
batch_size: 8
dropout_1: 0.17552297508250822
dropout_2: 0.2684350192517277
learning_rate: 0.0023652786617869956
n_units_1: 58
n_units_2: 112
scale_1: 0.24841209747555762
scale_2: 0.039100945687575016
Top score values: [0.16873854 0.18244659 0.19398429 0.20906855 0.2092251 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9. Pending:
Targets: [ 0.01226933 -0.58303334 -0.84294566  0.43760211  1.80986881  0.52810182
 -0.29166969 -0.84747848 -1.22891589  1.81047426  0.79300076  1.69271618
  1.69392131 -0.3515649  -0.6402974  -0.71767193 -1.00342443 -1.12974421
  0.57898276  1.02164491  0.3026975   0.0825577  -0.72786464 -1.07444316
 -1.32478371]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.36284319027722034, 'kernel_inv_bw1': 0.0008338880917810281, 'kernel_inv_bw2': 0.00010006397121532856, 'kernel_inv_bw3': 10.607319913452702, 'kernel_inv_bw4': 0.7614918914038229, 'kernel_inv_bw5': 0.00010061494903382198, 'kernel_inv_bw6': 0.00011078368201324454, 'kernel_inv_bw7': 0.0004933036322447669, 'kernel_inv_bw8': 1.0723232530793676, 'kernel_covariance_scale': 0.8390145301459427, 'mean_mean_value': 0.8001095706507626}
Epoch 1 ; Time: 3.792020 ; Training: accuracy=0.632460 ; Validation: accuracy=0.794415
Update for config_id 15:1: reward = 0.794414535666218, crit_val = 0.205585464333782
config_id 15: Reaches 1, continues to 3
Epoch 2 ; Time: 7.530799 ; Training: accuracy=0.764174 ; Validation: accuracy=0.853970
Epoch 3 ; Time: 11.560687 ; Training: accuracy=0.806946 ; Validation: accuracy=0.883580
Update for config_id 15:3: reward = 0.8835800807537012, crit_val = 0.1164199192462988
config_id 15: Reaches 3, continues to 9
Epoch 4 ; Time: 15.305123 ; Training: accuracy=0.822779 ; Validation: accuracy=0.891992
Epoch 5 ; Time: 19.044413 ; Training: accuracy=0.836041 ; Validation: accuracy=0.881393
Epoch 6 ; Time: 22.953450 ; Training: accuracy=0.843501 ; Validation: accuracy=0.906797
Epoch 7 ; Time: 26.752977 ; Training: accuracy=0.853863 ; Validation: accuracy=0.910834
Epoch 8 ; Time: 30.548261 ; Training: accuracy=0.859085 ; Validation: accuracy=0.909825
Epoch 9 ; Time: 34.393370 ; Training: accuracy=0.862566 ; Validation: accuracy=0.915713
config_id 15: Terminating evaluation at 9
Update for config_id 15:9: reward = 0.9157133243606999, crit_val = 0.08428667563930015
Starting get_config[BO] for config_id 16
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.43002967071282033
- self.std = 0.27927499966326685
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 28
Current best is [0.20558547]
[16: BO] (22 evaluations)
batch_size: 8
dropout_1: 0.4926500768949307
dropout_2: 0.022876403562322414
learning_rate: 0.0032025353360561116
n_units_1: 30
n_units_2: 78
scale_1: 0.0011797356756528227
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 36
dropout_1: 0.49265016285483343
dropout_2: 0.02287645320573864
learning_rate: 0.0028182780048393113
n_units_1: 30
n_units_2: 78
scale_1: 0.007107555640624739
scale_2: 0.003961365035264215
Top score values: [0.13136521 0.19833481 0.21751083 0.25087781 0.25100138]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9. Pending:
Targets: [ 0.13864588 -0.44656879 -0.70207663  0.55677098  1.90578325  0.64573708
 -0.16014258 -0.70653265 -1.08150622  1.90637845  0.90614704  1.79061589
  1.7918006  -0.21902281 -0.50286245 -0.5789258  -0.85983594 -0.9840151
  0.69575579  1.13091659  0.42415246  0.20774315 -0.58894578 -0.92965118
 -1.17574946 -0.80366738 -1.12294245 -1.23800195]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.4507817105292052, 'kernel_inv_bw1': 0.00010772479136131647, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.6539016638089095, 'kernel_inv_bw4': 0.0015622863141175674, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.43866329895157713, 'kernel_inv_bw7': 0.16606051109863562, 'kernel_inv_bw8': 0.9922613303342391, 'kernel_covariance_scale': 0.8878139393998372, 'mean_mean_value': 0.9871061530361999}
Epoch 1 ; Time: 3.708866 ; Training: accuracy=0.370855 ; Validation: accuracy=0.621299
Update for config_id 16:1: reward = 0.6212987886944819, crit_val = 0.37870121130551815
config_id 16: Reaches 1, continues to 3
Epoch 2 ; Time: 7.410609 ; Training: accuracy=0.518319 ; Validation: accuracy=0.690276
Epoch 3 ; Time: 11.131569 ; Training: accuracy=0.561754 ; Validation: accuracy=0.722577
config_id 16: Terminating evaluation at 3
Update for config_id 16:3: reward = 0.7225773889636609, crit_val = 0.27742261103633914
Starting get_config[BO] for config_id 17
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.42323182007669424
- self.std = 0.27131678375678564
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 30
Current best is [0.11641992]
[17: BO] (47 evaluations)
batch_size: 8
dropout_1: 0.1648077691378252
dropout_2: 0.7397040372988888
learning_rate: 0.0033375547923769583
n_units_1: 128
n_units_2: 83
scale_1: 0.04213633054276213
scale_2: 0.00997862760923571
Started BO from (top scorer):
batch_size: 125
dropout_1: 0.17407151057614528
dropout_2: 0.7396531753211542
learning_rate: 0.004727392946779631
n_units_1: 16
n_units_2: 83
scale_1: 0.7314773339706804
scale_2: 0.009977500935167959
Top score values: [0.13138863 0.14610964 0.15976456 0.16293169 0.17515208]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3. Pending:
Targets: [ 0.16776765 -0.43461244 -0.69761479  0.59815712  1.98673838  0.68973276
 -0.13978482 -0.70220151 -1.08817374  1.98735103  0.95778102  1.86819295
  1.86941241 -0.20039212 -0.49255729 -0.57085172 -0.86000146 -0.98782302
  0.74121861  1.18914347  0.46164866  0.23889166 -0.5811656  -0.93186451
 -1.1851813  -0.80218537 -1.13082536 -1.24925978 -0.16412773 -0.53741316]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.1662544162256941, 'kernel_inv_bw1': 0.6981079593675076, 'kernel_inv_bw2': 0.0009268439641641938, 'kernel_inv_bw3': 8.910385622177154, 'kernel_inv_bw4': 0.19964924494306252, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.6853629883339367, 'kernel_inv_bw7': 0.0030785674069184056, 'kernel_inv_bw8': 0.9486385749525671, 'kernel_covariance_scale': 0.753260879078906, 'mean_mean_value': 1.1545656021148591}
Epoch 1 ; Time: 4.091071 ; Training: accuracy=0.285809 ; Validation: accuracy=0.645525
Update for config_id 17:1: reward = 0.6455248990578735, crit_val = 0.35447510094212653
config_id 17: Reaches 1, continues to 3
Epoch 2 ; Time: 8.357033 ; Training: accuracy=0.404592 ; Validation: accuracy=0.736709
Epoch 3 ; Time: 12.600374 ; Training: accuracy=0.442722 ; Validation: accuracy=0.746131
config_id 17: Terminating evaluation at 3
Update for config_id 17:3: reward = 0.7461305518169583, crit_val = 0.2538694481830417
Starting get_config[BO] for config_id 18
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4157905984820624
- self.std = 0.26457649632012925
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 32
Current best is [0.11641992]
[18: BO] (33 evaluations)
batch_size: 8
dropout_1: 0.01286681640433462
dropout_2: 0.0
learning_rate: 0.0032240808434012545
n_units_1: 128
n_units_2: 126
scale_1: 0.32095395030988577
scale_2: 0.04355736556114528
Started BO from (top scorer):
batch_size: 100
dropout_1: 0.04577874257939943
dropout_2: 0.02299654986240901
learning_rate: 0.0014648030242655203
n_units_1: 80
n_units_2: 126
scale_1: 0.014022515662553641
scale_2: 0.043557323303113495
Top score values: [0.14629997 0.15171403 0.1680729  0.18017993 0.19063563]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 17:1, 17:3. Pending:
Targets: [ 0.20016669 -0.41755949 -0.68726203  0.64152065  2.06547708  0.73542925
 -0.11522092 -0.69196559 -1.08777076  2.06610534  1.01030625  1.94391161
  1.94516214 -0.17737223 -0.47698053 -0.55726957 -0.85378562 -0.98486353
  0.78822675  1.24756283  0.50153453  0.27310264 -0.5678462  -0.92747944
 -1.18724967 -0.79449663 -1.13150897 -1.25296059 -0.14018398 -0.52297914
 -0.2317496  -0.61200126]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.21934502678618983, 'kernel_inv_bw1': 0.4097228157244553, 'kernel_inv_bw2': 0.46682982981681437, 'kernel_inv_bw3': 8.13367879311205, 'kernel_inv_bw4': 0.22909121316543313, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.6748938129646664, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9150837312096575, 'kernel_covariance_scale': 0.8389424096690691, 'mean_mean_value': 1.1941598139605007}
Epoch 1 ; Time: 4.305675 ; Training: accuracy=0.731598 ; Validation: accuracy=0.838829
Update for config_id 18:1: reward = 0.8388290713324361, crit_val = 0.16117092866756388
config_id 18: Reaches 1, continues to 3
Epoch 2 ; Time: 8.289160 ; Training: accuracy=0.868783 ; Validation: accuracy=0.896534
Epoch 3 ; Time: 12.288969 ; Training: accuracy=0.899287 ; Validation: accuracy=0.905283
Update for config_id 18:3: reward = 0.9052826379542396, crit_val = 0.09471736204576042
config_id 18: Reaches 3, continues to 9
Epoch 4 ; Time: 16.652077 ; Training: accuracy=0.918684 ; Validation: accuracy=0.909657
Epoch 5 ; Time: 20.656675 ; Training: accuracy=0.928714 ; Validation: accuracy=0.919751
Epoch 6 ; Time: 24.799152 ; Training: accuracy=0.936174 ; Validation: accuracy=0.923116
Epoch 7 ; Time: 28.818707 ; Training: accuracy=0.945209 ; Validation: accuracy=0.924293
Epoch 8 ; Time: 33.152905 ; Training: accuracy=0.947530 ; Validation: accuracy=0.927490
Epoch 9 ; Time: 37.230219 ; Training: accuracy=0.954244 ; Validation: accuracy=0.936575
config_id 18: Terminating evaluation at 9
Update for config_id 18:9: reward = 0.9365746971736204, crit_val = 0.06342530282637959
Starting get_config[BO] for config_id 19
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3892746498561629
- self.std = 0.2676615813379797
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 35
Current best is [0.09471735]
[19: BO] (14 evaluations)
batch_size: 8
dropout_1: 0.6295893687035936
dropout_2: 0.0
learning_rate: 0.06521492367050961
n_units_1: 120
n_units_2: 48
scale_1: 0.0010000000000000002
scale_2: 0.4477349279506013
Started BO from (top scorer):
batch_size: 11
dropout_1: 0.654268531300859
dropout_2: 0.017232746721372982
learning_rate: 0.10194094521859276
n_units_1: 120
n_units_2: 48
scale_1: 0.07002951486623496
scale_2: 0.44773544751826444
Top score values: [0.07600676 0.10839947 0.12889904 0.1321471  0.14164108]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 4:9, 5:1, 6:1, 7:1, 8:1, 9:1, 9:3, 10:1, 10:3, 10:9, 11:1, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 17:1, 17:3, 18:1, 18:3, 18:9. Pending:
Targets: [ 0.29692476 -0.31368147 -0.5802754   0.73319165  2.14073546  0.82601785
 -0.01482767 -0.58492475 -0.97616784  2.14135648  1.09772659  2.02057116
  2.02180728 -0.07626263 -0.37241762 -0.45178124 -0.74487963 -0.87444673
  0.8782068   1.33224854  0.59481902  0.36902004 -0.46223597 -0.81772405
 -1.07450015 -0.68627401 -1.01940192 -1.13945368 -0.03950301 -0.41788604
 -0.13001324 -0.5058821  -0.85220942 -1.100484   -1.21739304]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.4424760575681563, 'kernel_inv_bw1': 0.11589630556012034, 'kernel_inv_bw2': 0.30569943473400973, 'kernel_inv_bw3': 7.841854343934342, 'kernel_inv_bw4': 0.008247855723296114, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.7758458300727824, 'kernel_inv_bw7': 0.00014618812473428605, 'kernel_inv_bw8': 0.906714419671346, 'kernel_covariance_scale': 0.8636918247366842, 'mean_mean_value': 1.2460826912812344}
Epoch 1 ; Time: 4.281150 ; Training: accuracy=0.040451 ; Validation: accuracy=0.038526
config_id 19: Terminating evaluation at 1
Update for config_id 19:1: reward = 0.038526244952893675, crit_val = 0.9614737550471063

Analysing the results

The training history is stored in the results_df, the main fields are the runtime and 'best' (the objective).

Note: You will get slightly different curves for different pairs of scheduler/searcher, the time_out here is a bit too short to really see the difference in a significant way (it would be better to set it to >1000s). Generally speaking though, hyperband stopping / promotion + model will tend to significantly outperform other combinations given enough time.

results_df.head()
/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
  and should_run_async(code)
bracket elapsed_time epoch error eval_time objective runtime searcher_data_size searcher_params_kernel_covariance_scale searcher_params_kernel_inv_bw0 ... searcher_params_kernel_inv_bw7 searcher_params_kernel_inv_bw8 searcher_params_mean_mean_value searcher_params_noise_variance target_epoch task_id time_since_start time_step time_this_iter best
0 0 0.489492 1 0.468750 0.484402 0.531250 0.581579 NaN 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.584056 1.603831e+09 0.520296 0.468750
1 0 0.922045 2 0.344753 0.425398 0.655247 1.014132 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.015150 1.603831e+09 0.432537 0.344753
2 0 1.385753 3 0.305314 0.461647 0.694686 1.477840 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.480152 1.603831e+09 0.463708 0.305314
3 0 1.821344 4 0.288937 0.429753 0.711063 1.913430 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.914552 1.603831e+09 0.435593 0.288937
4 0 2.268243 5 0.273061 0.444681 0.726939 2.360330 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 2.361275 1.603831e+09 0.446897 0.273061

5 rows × 26 columns

import matplotlib.pyplot as plt

plt.figure(figsize=(12, 8))

runtime = results_df['runtime'].values
objective = results_df['best'].values

plt.plot(runtime, objective, lw=2)
plt.xticks(fontsize=12)
plt.xlim(0, 120)
plt.ylim(0, 0.5)
plt.yticks(fontsize=12)
plt.xlabel("Runtime [s]", fontsize=14)
plt.ylabel("Objective", fontsize=14)
/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
  and should_run_async(code)
Text(0, 0.5, 'Objective')
../../_images/output_mlp_cb387f_18_2.png

Diving Deeper

Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at MNIST Training in PyTorch). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful.

Logging the Search Progress

First, it is a good idea in general to switch on debug_log, which outputs useful information about the search progress. This is already done in the example above.

The outputs show which configurations are chosen, stopped, or promoted. For BO and BOHB, a range of information is displayed for every get_config decision. This log output is very useful in order to figure out what is going on during the search.

Configuring HyperbandScheduler

The most important knobs to turn with HyperbandScheduler are max_t, grace_period, reduction_factor, brackets, and type. The first three determine the rung levels at which stopping or promotion decisions are being made.

  • The maximum resource level max_t (usually, resource equates to epochs, so max_t is the maximum number of training epochs) is typically hardcoded in train_fn passed to the scheduler (this is run_mlp_openml in the example above). As already noted above, the value is best fixed in the ag.args decorator as epochs=XYZ, it can then be accessed as args.epochs in the train_fn code. If this is done, you do not have to pass max_t when creating the scheduler.

  • grace_period and reduction_factor determine the rung levels, which are grace_period, grace_period * reduction_factor, grace_period * (reduction_factor ** 2), etc. All rung levels must be less or equal than max_t. It is recommended to make max_t equal to the largest rung level. For example, if grace_period = 1, reduction_factor = 3, it is in general recommended to use max_t = 9, max_t = 27, or max_t = 81. Choosing a max_t value “off the grid” works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you set max_t = 10, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch.

  • With reduction_factor, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used.

  • Finally, grace_period should be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. While grace_period = 1 should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung.

  • brackets sets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). For brackets = 1, you are running successive halving (single bracket). Higher brackets have larger effective grace_period values (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (brackets = 1) in a comparison.

  • Finally, with type (values stopping, promotion) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the default stopping is simpler and seems to perform well, but promotion is more careful promoting configurations to higher resource levels, which can work better in some cases.

Asynchronous BOHB

Finally, here are some ideas for tuning asynchronous BOHB, apart from tuning its HyperbandScheduling component. You need to pass these options in search_options.

  • We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by gp_resource_kernel, values are matern52, matern52-res-warp, exp-decay-sum, exp-decay-combined, exp-decay-delta1. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found here.

  • Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper train_fn and find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the options opt_skip_period and opt_skip_num_max_resource. The basic idea is as follows. By far the most expensive part of a get_config call (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for most get_config calls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers – we would love to learn about your use case.