Getting started with Advanced HPO Algorithms

Loading libraries

# Basic utils for folder manipulations etc
import time
import multiprocessing # to count the number of CPUs available

# External tools to load and process data
import numpy as np
import pandas as pd

# MXNet (NeuralNets)
import mxnet as mx
from mxnet import gluon, autograd
from mxnet.gluon import nn

# AutoGluon and HPO tools
import autogluon as ag
from autogluon.utils import load_and_split_openml_data

Check the version of MxNet, you should be fine with version >= 1.5

mx.__version__

/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
  and should_run_async(code)

'1.7.0'

You can also check the version of AutoGluon and the specific commit and check that it matches what you want.

ag.__version__

'0.0.14b20201027'

Hyperparameter Optimization of a 2-layer MLP

Setting up the context

Here we declare a few “environment variables” setting the context for what we’re doing

OPENML_TASK_ID = 6                # describes the problem we will tackle
RATIO_TRAIN_VALID = 0.33          # split of the training data used for validation
RESOURCE_ATTR_NAME = 'epoch'      # how do we measure resources   (will become clearer further)
REWARD_ATTR_NAME = 'objective'    # how do we measure performance (will become clearer further)

NUM_CPUS = multiprocessing.cpu_count()

Preparing the data

We will use a multi-way classification task from OpenML. Data preparation includes:

• Missing values are imputed, using the ‘mean’ strategy of sklearn.impute.SimpleImputer

• Split training set into training and validation

• Standardize inputs to mean 0, variance 1

X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data(
    OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False)
n_classes

Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff...
100%|██████████| 704/704 [00:00<00:00, 54179.63KB/s]
Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3...
100%|██████████| 2521/2521 [00:00<00:00, 45588.69KB/s]
Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml...
3KB [00:00, 3857.42KB/s]
Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml...
8KB [00:00, 10381.94KB/s]
Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml...
15KB [00:00, 14169.95KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff...
2998KB [00:00, 51813.32KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3...
881KB [00:00, 56292.09KB/s]
Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml...
3KB [00:00, 4228.13KB/s]
pickle load data letter

26

The problem has 26 classes.

Declaring a model specifying a hyperparameter space with AutoGluon

Two layer MLP where we optimize over:

• the number of units on the first layer

• the number of units on the second layer

• the dropout rate after each layer

• the learning rate

• the scaling

• the @ag.args decorator allows us to specify the space we will optimize over, this matches the ConfigSpace syntax

The body of the function run_mlp_openml is pretty simple:

• it reads the hyperparameters given via the decorator

• it defines a 2 layer MLP with dropout

• it declares a trainer with the ‘adam’ loss function and a provided learning rate

• it trains the NN with a number of epochs (most of that is boilerplate code from mxnet)

• the reporter at the end is used to keep track of training history in the hyperparameter optimization

Note: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment

@ag.args(n_units_1=ag.space.Int(lower=16, upper=128),
         n_units_2=ag.space.Int(lower=16, upper=128),
         dropout_1=ag.space.Real(lower=0, upper=.75),
         dropout_2=ag.space.Real(lower=0, upper=.75),
         learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True),
         batch_size=ag.space.Int(lower=8, upper=128),
         scale_1=ag.space.Real(lower=0.001, upper=10, log=True),
         scale_2=ag.space.Real(lower=0.001, upper=10, log=True),
         epochs=9)
def run_mlp_openml(args, reporter, **kwargs):
    # Time stamp for elapsed_time
    ts_start = time.time()
    # Unwrap hyperparameters
    n_units_1 = args.n_units_1
    n_units_2 = args.n_units_2
    dropout_1 = args.dropout_1
    dropout_2 = args.dropout_2
    scale_1 = args.scale_1
    scale_2 = args.scale_2
    batch_size = args.batch_size
    learning_rate = args.learning_rate

    ctx = mx.cpu()
    net = nn.Sequential()
    with net.name_scope():
        # Layer 1
        net.add(nn.Dense(n_units_1, activation='relu',
                         weight_initializer=mx.initializer.Uniform(scale=scale_1)))
        # Dropout
        net.add(gluon.nn.Dropout(dropout_1))
        # Layer 2
        net.add(nn.Dense(n_units_2, activation='relu',
                         weight_initializer=mx.initializer.Uniform(scale=scale_2)))
        # Dropout
        net.add(gluon.nn.Dropout(dropout_2))
        # Output
        net.add(nn.Dense(n_classes))
    net.initialize(ctx=ctx)

    trainer = gluon.Trainer(net.collect_params(), 'adam',
                            {'learning_rate': learning_rate})

    for epoch in range(args.epochs):
        ts_epoch = time.time()

        train_iter = mx.io.NDArrayIter(
                        data={'data': X_train},
                        label={'label': y_train},
                        batch_size=batch_size,
                        shuffle=True)
        valid_iter = mx.io.NDArrayIter(
                        data={'data': X_valid},
                        label={'label': y_valid},
                        batch_size=batch_size,
                        shuffle=False)

        metric = mx.metric.Accuracy()
        loss = gluon.loss.SoftmaxCrossEntropyLoss()

        for batch in train_iter:
            data = batch.data[0].as_in_context(ctx)
            label = batch.label[0].as_in_context(ctx)
            with autograd.record():
                output = net(data)
                L = loss(output, label)
            L.backward()
            trainer.step(data.shape[0])
            metric.update([label], [output])

        name, train_acc = metric.get()

        metric = mx.metric.Accuracy()
        for batch in valid_iter:
            data = batch.data[0].as_in_context(ctx)
            label = batch.label[0].as_in_context(ctx)
            output = net(data)
            metric.update([label], [output])

        name, val_acc = metric.get()

        print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % (
            epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc))

        ts_now = time.time()
        eval_time = ts_now - ts_epoch
        elapsed_time = ts_now - ts_start

        # The resource reported back (as 'epoch') is the number of epochs
        # done, starting at 1
        reporter(
            epoch=epoch + 1,
            objective=float(val_acc),
            eval_time=eval_time,
            time_step=ts_now,
            elapsed_time=elapsed_time)

/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
  and should_run_async(code)

Note: The annotation epochs=9 specifies the maximum number of epochs for training. It becomes available as args.epochs. Importantly, it is also processed by HyperbandScheduler below in order to set its max_t attribute.

Recommendation: Whenever writing training code to be passed as train_fn to a scheduler, if this training code reports a resource (or time) attribute, the corresponding maximum resource value should be included in train_fn.args:

• If the resource attribute (time_attr of scheduler) in train_fn is epoch, make sure to include epochs=XYZ in the annotation. This allows the scheduler to read max_t from train_fn.args.epochs. This case corresponds to our example here.

• If the resource attribute is something else than epoch, you can also include the annotation max_t=XYZ, which allows the scheduler to read max_t from train_fn.args.max_t.

Annotating the training function by the correct value for max_t simplifies scheduler creation (since max_t does not have to be passed), and avoids inconsistencies between train_fn and the scheduler.

Running the Hyperparameter Optimization

You can use the following schedulers:

• FIFO (fifo)

• Hyperband (either the stopping (hbs) or promotion (hbp) variant)

And the following searchers:

• Random search (random)

• Gaussian process based Bayesian optimization (bayesopt)

• SkOpt Bayesian optimization (skopt; only with FIFO scheduler)

Note that the method known as (asynchronous) Hyperband is using random search. Combining Hyperband scheduling with the bayesopt searcher uses a novel method called asynchronous BOHB.

Pick the combination you’re interested in (doing the full experiment takes around 120 seconds, see the time_out parameter), running everything with multiple runs can take a fair bit of time. In real life, you will want to choose a larger time_out in order to obtain good performance.

SCHEDULER = "hbs"
SEARCHER = "bayesopt"

def compute_error(df):
    return 1.0 - df["objective"]

def compute_runtime(df, start_timestamp):
        return df["time_step"] - start_timestamp

def process_training_history(task_dicts, start_timestamp,
                             runtime_fn=compute_runtime,
                             error_fn=compute_error):
    task_dfs = []
    for task_id in task_dicts:
        task_df = pd.DataFrame(task_dicts[task_id])
        task_df = task_df.assign(task_id=task_id,
                                 runtime=runtime_fn(task_df, start_timestamp),
                                 error=error_fn(task_df),
                                 target_epoch=task_df["epoch"].iloc[-1])
        task_dfs.append(task_df)

    result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True)
    # re-order by runtime
    result = result.sort_values(by="runtime")
    # calculate incumbent best -- the cumulative minimum of the error.
    result = result.assign(best=result["error"].cummin())
    return result

resources = dict(num_cpus=NUM_CPUS, num_gpus=0)

search_options = {
    'num_init_random': 2,
    'debug_log': True}
if SCHEDULER == 'fifo':
    myscheduler = ag.scheduler.FIFOScheduler(
        run_mlp_openml,
        resource=resources,
        searcher=SEARCHER,
        search_options=search_options,
        time_out=120,
        time_attr=RESOURCE_ATTR_NAME,
        reward_attr=REWARD_ATTR_NAME)

else:
    # This setup uses rung levels at 1, 3, 9 epochs. We just use a single
    # bracket, so this is in fact successive halving (Hyperband would use
    # more than 1 bracket).
    # Also note that since we do not use the max_t argument of
    # HyperbandScheduler, this value is obtained from train_fn.args.epochs.
    sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion'
    myscheduler = ag.scheduler.HyperbandScheduler(
        run_mlp_openml,
        resource=resources,
        searcher=SEARCHER,
        search_options=search_options,
        time_out=120,
        time_attr=RESOURCE_ATTR_NAME,
        reward_attr=REWARD_ATTR_NAME,
        type=sch_type,
        grace_period=1,
        reduction_factor=3,
        brackets=1)

# run tasks
myscheduler.run()
myscheduler.join_jobs()

results_df = process_training_history(
                myscheduler.training_history.copy(),
                start_timestamp=myscheduler._start_time)

max_t = 9, as inferred from train_fn.args
scheduler_options: Key 'resume': Imputing default value False
scheduler_options: Key 'keep_size_ratios': Imputing default value False
scheduler_options: Key 'maxt_pending': Imputing default value False
scheduler_options: Key 'searcher_data': Imputing default value rungs
scheduler_options: Key 'do_snapshots': Imputing default value False

scheduler_options: Key 'visualizer': Imputing default value none
scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60
scheduler_options: Key 'delay_get_config': Imputing default value True

search_options: Key 'random_seed': Imputing default value 1822
search_options: Key 'opt_skip_init_length': Imputing default value 150
search_options: Key 'opt_skip_period': Imputing default value 1
search_options: Key 'profiler': Imputing default value False
search_options: Key 'opt_maxiter': Imputing default value 50
search_options: Key 'opt_nstarts': Imputing default value 2
search_options: Key 'opt_warmstart': Imputing default value False
search_options: Key 'opt_verbose': Imputing default value False
search_options: Key 'opt_debug_writer': Imputing default value False
search_options: Key 'num_fantasy_samples': Imputing default value 20
search_options: Key 'num_init_candidates': Imputing default value 250
search_options: Key 'initial_scoring': Imputing default value thompson_indep
search_options: Key 'first_is_default': Imputing default value True
search_options: Key 'opt_skip_num_max_resource': Imputing default value False
search_options: Key 'gp_resource_kernel': Imputing default value matern52
search_options: Key 'resource_acq': Imputing default value bohb

[GPMultiFidelitySearcher.__init__]
- acquisition_class = <class 'autogluon.searcher.bayesopt.models.nphead_acqfunc.EIAcquisitionFunction'>
- local_minimizer_class = <class 'autogluon.searcher.bayesopt.tuning_algorithms.bo_algorithm_components.LBFGSOptimizeAcquisition'>
- num_initial_candidates = 250
- num_initial_random_choices = 2
- initial_scoring = thompson_indep
- first_is_default = True
Starting Experiments
Num of Finished Tasks is 0
Time out (secs) is 120
Starting get_config[random] for config_id 0
Start with default config:
{'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1}
[0: random]
batch_size: 68
dropout_1: 0.375
dropout_2: 0.375
learning_rate: 0.001
n_units_1: 72
n_units_2: 72
scale_1: 0.1
scale_2: 0.1
/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/distributed/worker.py:3382: UserWarning: Large object of size 1.30 MB detected in task graph:
  (<function run_mlp_openml at 0x7fd6a3ad0f80>, {'ar ... sReporter}, [])
Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and
keep data on workers

    future = client.submit(func, big_data)    # bad

    big_future = client.scatter(big_data)     # good
    future = client.submit(func, big_future)  # good
  % (format_bytes(len(b)), s)

Epoch 1 ; Time: 0.481761 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250

Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875
config_id 0: Reaches 1, continues to 3

Epoch 2 ; Time: 0.923535 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247
Epoch 3 ; Time: 1.345758 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686

Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464
config_id 0: Reaches 3, continues to 9

Epoch 4 ; Time: 1.771330 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063
Epoch 5 ; Time: 2.190708 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939
Epoch 6 ; Time: 2.614471 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321
Epoch 7 ; Time: 3.063879 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501
Epoch 8 ; Time: 3.488436 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202
Epoch 9 ; Time: 3.925083 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043

config_id 0: Terminating evaluation at 9
Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695
Starting get_config[random] for config_id 1
[1: random]
batch_size: 81
dropout_1: 0.20829386966719157
dropout_2: 0.22302955829164872
learning_rate: 3.4371538766396122e-06
n_units_1: 100
n_units_2: 123
scale_1: 0.0029958953582211044
scale_2: 0.1287354014226165

Epoch 1 ; Time: 0.417130 ; Training: accuracy=0.055266 ; Validation: accuracy=0.090757

config_id 1: Terminating evaluation at 1
Update for config_id 1:1: reward = 0.09075742409075742, crit_val = 0.9092425759092426
Starting get_config[BO] for config_id 2
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.47931599157089355
- self.std = 0.2624052537450702
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 4
Current best is [0.46875]
[2: BO] (1 evaluations)
batch_size: 123
dropout_1: 0.4640710798470552
dropout_2: 0.09873615428539195
learning_rate: 0.0005378457242779595
n_units_1: 32
n_units_2: 120
scale_1: 0.045864005010116914
scale_2: 2.1665539764398583
Started BO from (top scorer):
batch_size: 123
dropout_1: 0.4640710798470552
dropout_2: 0.09873615428539195
learning_rate: 0.0005378457242779595
n_units_1: 32
n_units_2: 120
scale_1: 0.045864005010116914
scale_2: 2.1665539764398583
Top score values: [-0.25960743  0.01441937  0.02376365  0.07418799  0.09351457]
Labeled: 0:1, 0:3, 0:9, 1:1. Pending:
Targets: [-0.04026593 -0.66310342 -0.93503758  1.63840692]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 1.3033469106956084, 'kernel_inv_bw1': 3.4007518670314734, 'kernel_inv_bw2': 2.7166619261817617, 'kernel_inv_bw3': 84.00968747668335, 'kernel_inv_bw4': 4.7662174436449565, 'kernel_inv_bw5': 82.50051779133331, 'kernel_inv_bw6': 67.22964070460776, 'kernel_inv_bw7': 1.0170591364397439, 'kernel_inv_bw8': 1.5777241662196506, 'kernel_covariance_scale': 0.8127451069073666, 'mean_mean_value': 0.3271186419562175}

Epoch 1 ; Time: 0.307112 ; Training: accuracy=0.162766 ; Validation: accuracy=0.369670

config_id 2: Terminating evaluation at 1
Update for config_id 2:1: reward = 0.36966981914717106, crit_val = 0.630330180852829
Starting get_config[BO] for config_id 3
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5095188294272807
- self.std = 0.2423511077192109
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 5
Current best is [0.46875]
[3: BO] (1 evaluations)
batch_size: 75
dropout_1: 0.5302630865931415
dropout_2: 0.12336644847504774
learning_rate: 0.7028645562543027
n_units_1: 73
n_units_2: 126
scale_1: 0.10722353807156274
scale_2: 0.0016236501394851991
Started BO from (top scorer):
batch_size: 75
dropout_1: 0.5302630865931415
dropout_2: 0.12336644847504774
learning_rate: 0.7028645562543027
n_units_1: 73
n_units_2: 126
scale_1: 0.10722353807156274
scale_2: 0.0016236501394851991
Top score values: [0.00719892 0.0303531  0.07280485 0.11276195 0.1538033 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending:
Targets: [-0.16822217 -0.84259841 -1.13703466  1.64935803  0.49849721]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00048025899994700926, 'kernel_inv_bw1': 1.1131207570561128, 'kernel_inv_bw2': 1.0824826486741035, 'kernel_inv_bw3': 84.3637388830916, 'kernel_inv_bw4': 0.0006325818414342524, 'kernel_inv_bw5': 100.00000000000004, 'kernel_inv_bw6': 39.43198460539477, 'kernel_inv_bw7': 0.013328152652716078, 'kernel_inv_bw8': 1.6151629858567016, 'kernel_covariance_scale': 0.8034397943414244, 'mean_mean_value': 0.3077191486825238}

Epoch 1 ; Time: 0.460667 ; Training: accuracy=0.041159 ; Validation: accuracy=0.034333

config_id 3: Terminating evaluation at 1
Update for config_id 3:1: reward = 0.034333333333333334, crit_val = 0.9656666666666667
Starting get_config[BO] for config_id 4
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5855434689671783
- self.std = 0.279004979519119
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 6
Current best is [0.46875]
[4: BO] (21 evaluations)
batch_size: 128
dropout_1: 0.39366683240859135
dropout_2: 0.6114522662562138
learning_rate: 1.8354238197457777e-06
n_units_1: 50
n_units_2: 71
scale_1: 0.04226495155290692
scale_2: 0.0013550158837172756
Started BO from (top scorer):
batch_size: 16
dropout_1: 0.39366684595540025
dropout_2: 0.611452209042092
learning_rate: 1.8354821086802627e-06
n_units_1: 50
n_units_2: 35
scale_1: 0.0422649570093964
scale_2: 0.0013547855784457326
Top score values: [0.00599251 0.07967276 0.11748946 0.15673679 0.16525097]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending:
Targets: [-0.41860711 -1.00438816 -1.26014328  1.16019115  0.16052298  1.36242442]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 2.0053376121249324, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0010704263242157716, 'kernel_inv_bw3': 0.0008335469721583438, 'kernel_inv_bw4': 0.0037120842504946355, 'kernel_inv_bw5': 4.686944148532531, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0022817208637103594, 'kernel_inv_bw8': 1.5124799741334436, 'kernel_covariance_scale': 0.7561001736211863, 'mean_mean_value': -0.0006020526887166488}

Epoch 1 ; Time: 0.298823 ; Training: accuracy=0.052385 ; Validation: accuracy=0.130818

config_id 4: Terminating evaluation at 1
Update for config_id 4:1: reward = 0.13081781914893617, crit_val = 0.8691821808510638
Starting get_config[BO] for config_id 5
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6260632849505906
- self.std = 0.2767207468205929
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 7
Current best is [0.46875]
[5: BO] (19 evaluations)
batch_size: 8
dropout_1: 0.26555362657067627
dropout_2: 0.7173603551260431
learning_rate: 0.0022299009480851073
n_units_1: 127
n_units_2: 16
scale_1: 0.37877273777029896
scale_2: 0.36495062851271215
Started BO from (top scorer):
batch_size: 29
dropout_1: 0.4799799052588535
dropout_2: 0.5647643800414299
learning_rate: 0.004077220423509741
n_units_1: 111
n_units_2: 89
scale_1: 0.15393299323611206
scale_2: 0.3845928002559493
Top score values: [0.06202583 0.13401968 0.16643212 0.21561172 0.26673195]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1. Pending:
Targets: [-0.56849111 -1.15910758 -1.41697386  1.02333957  0.0154195   1.22724221
  0.87857126]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.3335819072441991, 'kernel_inv_bw1': 0.2750752295317643, 'kernel_inv_bw2': 0.24890601177148425, 'kernel_inv_bw3': 11.183471528369342, 'kernel_inv_bw4': 0.2744568212856452, 'kernel_inv_bw5': 0.2952497007090965, 'kernel_inv_bw6': 0.3312379223384557, 'kernel_inv_bw7': 0.027407776944603238, 'kernel_inv_bw8': 1.0971022798993004, 'kernel_covariance_scale': 0.732949008356558, 'mean_mean_value': 0.15940542653147902}

Epoch 1 ; Time: 3.883657 ; Training: accuracy=0.166446 ; Validation: accuracy=0.514300

Update for config_id 5:1: reward = 0.514300134589502, crit_val = 0.48569986541049803
config_id 5: Reaches 1, continues to 3

Epoch 2 ; Time: 7.507255 ; Training: accuracy=0.223972 ; Validation: accuracy=0.605316
Epoch 3 ; Time: 10.984456 ; Training: accuracy=0.252487 ; Validation: accuracy=0.644179

config_id 5: Terminating evaluation at 3
Update for config_id 5:3: reward = 0.644179004037685, crit_val = 0.355820995962315
Starting get_config[BO] for config_id 6
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5804404284474385
- self.std = 0.26034600503297584
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 9
Current best is [0.46875]
[6: BO] (3 evaluations)
batch_size: 104
dropout_1: 0.06753273656053299
dropout_2: 0.5787103086119473
learning_rate: 0.0028205883583779776
n_units_1: 46
n_units_2: 84
scale_1: 0.0010000000000000002
scale_2: 10.0
Started BO from (top scorer):
batch_size: 104
dropout_1: 0.06753790263018208
dropout_2: 0.5785861703504277
learning_rate: 0.0028205883438605244
n_units_1: 46
n_units_2: 84
scale_1: 0.004630219176146915
scale_2: 9.435508332838014
Top score values: [0.09120854 0.21865244 0.22627411 0.28955933 0.30119195]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3. Pending:
Targets: [-0.42900765 -1.05677157 -1.33085664  1.26294293  0.19162865  1.47967025
  1.10906926 -0.3639025  -0.86277273]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.038848425403682475, 'kernel_inv_bw1': 0.013826037746051094, 'kernel_inv_bw2': 0.04933080557942479, 'kernel_inv_bw3': 0.00010000000000000009, 'kernel_inv_bw4': 0.04813505724162551, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 3.8015174579942212, 'kernel_inv_bw7': 3.983663262254571, 'kernel_inv_bw8': 1.3227034312264876, 'kernel_covariance_scale': 0.8009725817505569, 'mean_mean_value': 0.3852913191841205}

Epoch 1 ; Time: 0.434894 ; Training: accuracy=0.427387 ; Validation: accuracy=0.733753

Update for config_id 6:1: reward = 0.7337533156498673, crit_val = 0.26624668435013266
config_id 6: Reaches 1, continues to 3

Epoch 2 ; Time: 0.739148 ; Training: accuracy=0.601459 ; Validation: accuracy=0.772712
Epoch 3 ; Time: 1.023004 ; Training: accuracy=0.640252 ; Validation: accuracy=0.786638

Update for config_id 6:3: reward = 0.7866379310344828, crit_val = 0.21336206896551724
config_id 6: Reaches 3, continues to 9

Epoch 4 ; Time: 1.314426 ; Training: accuracy=0.653846 ; Validation: accuracy=0.777023
Epoch 5 ; Time: 1.598032 ; Training: accuracy=0.671751 ; Validation: accuracy=0.806034
Epoch 6 ; Time: 1.881462 ; Training: accuracy=0.674072 ; Validation: accuracy=0.797745
Epoch 7 ; Time: 2.165841 ; Training: accuracy=0.684682 ; Validation: accuracy=0.804708
Epoch 8 ; Time: 2.445447 ; Training: accuracy=0.689821 ; Validation: accuracy=0.808853
Epoch 9 ; Time: 2.727328 ; Training: accuracy=0.697281 ; Validation: accuracy=0.816645

config_id 6: Terminating evaluation at 9
Update for config_id 6:9: reward = 0.8166445623342176, crit_val = 0.18335543766578244
Starting get_config[BO] for config_id 7
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.49057733725069824
- self.std = 0.27450813586157435
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 12
Current best is [0.26624669]
[7: BO] (11 evaluations)
batch_size: 39
dropout_1: 0.027346760299878224
dropout_2: 0.5143530092361976
learning_rate: 1.0
n_units_1: 109
n_units_2: 62
scale_1: 2.2834582168904713
scale_2: 10.0
Started BO from (top scorer):
batch_size: 39
dropout_1: 0.027346252810436844
dropout_2: 0.5143570802897102
learning_rate: 3.053475061997591e-05
n_units_1: 109
n_units_2: 64
scale_1: 2.2783933896225537
scale_2: 7.120331459024733
Top score values: [0.13905466 0.14949949 0.15337874 0.18225681 0.22215788]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9. Pending:
Targets: [-0.07951435 -0.67489135 -0.93483611  1.52514692  0.50910274  1.73069307
  1.37921174 -0.01776804 -0.49090108 -0.81720949 -1.00986176 -1.11917229]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0006731845953042116, 'kernel_inv_bw1': 0.0004435239703577121, 'kernel_inv_bw2': 0.0010950254706804787, 'kernel_inv_bw3': 0.5006292391496392, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 1.6812251804964524, 'kernel_inv_bw6': 0.006431440498054594, 'kernel_inv_bw7': 1.2196658205787714, 'kernel_inv_bw8': 1.0870399802800832, 'kernel_covariance_scale': 1.0663920505796691, 'mean_mean_value': 0.43197612961607723}

Epoch 1 ; Time: 0.823566 ; Training: accuracy=0.037634 ; Validation: accuracy=0.038043

config_id 7: Terminating evaluation at 1
Update for config_id 7:1: reward = 0.03804256745433216, crit_val = 0.9619574325456678
Starting get_config[BO] for config_id 8
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5268373445810804
- self.std = 0.2921226755451922
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 13
Current best is [0.26624669]
[8: BO] (14 evaluations)
batch_size: 128
dropout_1: 0.08296817905926457
dropout_2: 0.19054579790821474
learning_rate: 0.005552797405875484
n_units_1: 72
n_units_2: 76
scale_1: 0.0015941704119388613
scale_2: 2.8428356369373566
Started BO from (top scorer):
batch_size: 85
dropout_1: 0.08296818203857925
dropout_2: 0.19054613346766927
learning_rate: 0.014308176452172097
n_units_1: 72
n_units_2: 76
scale_1: 0.0015951451801685214
scale_2: 2.8428356956092933
Top score values: [0.06523108 0.18698123 0.18731839 0.18876787 0.21413681]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1. Pending:
Targets: [-0.19884572 -0.75832242 -1.00259292  1.30905699  0.35427868  1.50220903
  1.17192147 -0.14082262 -0.58542648 -0.89205899 -1.07309463 -1.17581392
  1.48951151]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.6475724989455702, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0005582372769792866, 'kernel_inv_bw3': 6.972856257358789, 'kernel_inv_bw4': 0.008774491917625664, 'kernel_inv_bw5': 0.0003003923479295185, 'kernel_inv_bw6': 0.006691532727480968, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1916430438869066, 'kernel_covariance_scale': 0.6989353646776911, 'mean_mean_value': 0.5762116803881481}

Epoch 1 ; Time: 0.307458 ; Training: accuracy=0.570888 ; Validation: accuracy=0.795545

Update for config_id 8:1: reward = 0.7955452127659575, crit_val = 0.20445478723404253
config_id 8: Reaches 1, continues to 3

Epoch 2 ; Time: 0.549475 ; Training: accuracy=0.742105 ; Validation: accuracy=0.842919
Epoch 3 ; Time: 0.784090 ; Training: accuracy=0.780181 ; Validation: accuracy=0.860040

Update for config_id 8:3: reward = 0.8600398936170213, crit_val = 0.13996010638297873
config_id 8: Reaches 3, continues to 9

Epoch 4 ; Time: 1.029320 ; Training: accuracy=0.804194 ; Validation: accuracy=0.859043
Epoch 5 ; Time: 1.345355 ; Training: accuracy=0.810444 ; Validation: accuracy=0.880652
Epoch 6 ; Time: 1.581157 ; Training: accuracy=0.825247 ; Validation: accuracy=0.873172
Epoch 7 ; Time: 1.813629 ; Training: accuracy=0.835280 ; Validation: accuracy=0.894116
Epoch 8 ; Time: 2.050075 ; Training: accuracy=0.843257 ; Validation: accuracy=0.893949
Epoch 9 ; Time: 2.286526 ; Training: accuracy=0.843339 ; Validation: accuracy=0.898936

config_id 8: Terminating evaluation at 9
Update for config_id 8:9: reward = 0.898936170212766, crit_val = 0.10106382978723405
Starting get_config[BO] for config_id 9
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4558977626848939
- self.std = 0.30246201791748595
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 16
Current best is [0.20445479]
[9: BO] (15 evaluations)
batch_size: 95
dropout_1: 0.4262497145288378
dropout_2: 0.24000456457264768
learning_rate: 0.007365011875480271
n_units_1: 100
n_units_2: 16
scale_1: 0.0010000000000000002
scale_2: 0.0011590104176322858
Started BO from (top scorer):
batch_size: 95
dropout_1: 0.4256455858790655
dropout_2: 0.24000416180847647
learning_rate: 0.007915982639017364
n_units_1: 100
n_units_2: 61
scale_1: 0.0036892633125513137
scale_2: 0.0011591512852441098
Top score values: [0.19647575 0.2062344  0.20680512 0.20890162 0.21861883]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9. Pending:
Targets: [ 0.04249207 -0.49785951 -0.73377988  1.49884874  0.5767085   1.68539808
  1.36640105  0.09853172 -0.33087383 -0.62702444 -0.80187157 -0.9010795
  1.67313461 -0.83132083 -1.04455316 -1.17315204]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.006775924221977387, 'kernel_inv_bw1': 0.015172921861280587, 'kernel_inv_bw2': 0.0013188699217722508, 'kernel_inv_bw3': 4.415642720480877, 'kernel_inv_bw4': 0.0009644522562784017, 'kernel_inv_bw5': 0.6640762984435752, 'kernel_inv_bw6': 1.2045893849895135, 'kernel_inv_bw7': 0.0012584335539753077, 'kernel_inv_bw8': 0.9331349919847574, 'kernel_covariance_scale': 0.7694699862952703, 'mean_mean_value': 0.7784628343310049}

Epoch 1 ; Time: 0.363448 ; Training: accuracy=0.218400 ; Validation: accuracy=0.468839

config_id 9: Terminating evaluation at 1
Update for config_id 9:1: reward = 0.46883876357560567, crit_val = 0.5311612364243943
Starting get_config[BO] for config_id 10
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.46032502584604096
- self.std = 0.29396515757905645
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 17
Current best is [0.20445479]
[10: BO] (15 evaluations)
batch_size: 31
dropout_1: 0.0
dropout_2: 0.0222361448993122
learning_rate: 0.010132292274826368
n_units_1: 44
n_units_2: 107
scale_1: 1.5325745511866318
scale_2: 0.41088043849938755
Started BO from (top scorer):
batch_size: 31
dropout_1: 0.37741926641904855
dropout_2: 0.022238614298746323
learning_rate: 0.0279472326159558
n_units_1: 44
n_units_2: 123
scale_1: 1.5324224383172704
scale_2: 0.4108804457105683
Top score values: [0.14409397 0.17617909 0.19075953 0.20373432 0.21153185]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1. Pending:
Targets: [ 0.02865977 -0.5273103  -0.77004979  1.52711142  0.57831736  1.71905285
  1.39083543  0.08631921 -0.35549801 -0.66020866 -0.84010962 -0.94218509
  1.70643491 -0.87041009 -1.08980575 -1.22212169  0.24096805]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0020638596342242126, 'kernel_inv_bw1': 1.1254919271494774, 'kernel_inv_bw2': 0.0010445566099235965, 'kernel_inv_bw3': 4.826738255199483, 'kernel_inv_bw4': 0.006503434857039732, 'kernel_inv_bw5': 0.915326168235606, 'kernel_inv_bw6': 0.004022186724770975, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9927458895104945, 'kernel_covariance_scale': 0.7803189333803716, 'mean_mean_value': 0.8409017073194343}

Epoch 1 ; Time: 1.004785 ; Training: accuracy=0.721836 ; Validation: accuracy=0.834005

Update for config_id 10:1: reward = 0.834005376344086, crit_val = 0.165994623655914
config_id 10: Reaches 1, continues to 3

Epoch 2 ; Time: 1.903632 ; Training: accuracy=0.846402 ; Validation: accuracy=0.871136
Epoch 3 ; Time: 2.803965 ; Training: accuracy=0.887179 ; Validation: accuracy=0.876680

Update for config_id 10:3: reward = 0.8766801075268817, crit_val = 0.12331989247311825
config_id 10: Reaches 3, continues to 9

Epoch 4 ; Time: 3.779331 ; Training: accuracy=0.894127 ; Validation: accuracy=0.898522
Epoch 5 ; Time: 4.691466 ; Training: accuracy=0.906782 ; Validation: accuracy=0.899194
Epoch 6 ; Time: 5.745457 ; Training: accuracy=0.916791 ; Validation: accuracy=0.895665
Epoch 7 ; Time: 6.643373 ; Training: accuracy=0.919603 ; Validation: accuracy=0.904066
Epoch 8 ; Time: 7.577041 ; Training: accuracy=0.922415 ; Validation: accuracy=0.911290
Epoch 9 ; Time: 8.509091 ; Training: accuracy=0.926303 ; Validation: accuracy=0.911962

config_id 10: Terminating evaluation at 9
Update for config_id 10:9: reward = 0.9119623655913979, crit_val = 0.08803763440860213
Starting get_config[BO] for config_id 11
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4101438794960165
- self.std = 0.2964373065434322
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 20
Current best is [0.16599463]
[11: BO] (35 evaluations)
batch_size: 100
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.004820472426324419
n_units_1: 107
n_units_2: 128
scale_1: 0.06680137561468431
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 100
dropout_1: 0.00911833153273836
dropout_2: 0.23742831006045706
learning_rate: 0.007257355221166904
n_units_1: 96
n_units_2: 81
scale_1: 0.06682046297392799
scale_2: 0.0024843701783189196
Top score values: [0.13889474 0.14858531 0.15217332 0.21747817 0.22456437]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9. Pending:
Targets: [ 0.19770157 -0.35363197 -0.59434712  1.68365683  0.74277527  1.87399755
  1.54851731  0.25488015 -0.18325252 -0.48542202 -0.66382269 -0.7650469
  1.86148484 -0.69387047 -0.91143647 -1.04264896  0.40823929 -0.82361177
 -0.96757048 -1.08659146]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00015375309178579185, 'kernel_inv_bw1': 1.032070936012607, 'kernel_inv_bw2': 0.2807788155180474, 'kernel_inv_bw3': 3.8647366048770095, 'kernel_inv_bw4': 0.0033982880287331653, 'kernel_inv_bw5': 0.5295507523300419, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.01296890476903277, 'kernel_inv_bw8': 0.9070917227422385, 'kernel_covariance_scale': 0.8877448885848056, 'mean_mean_value': 0.969264783177449}

Epoch 1 ; Time: 0.339826 ; Training: accuracy=0.562645 ; Validation: accuracy=0.764667

Update for config_id 11:1: reward = 0.7646666666666667, crit_val = 0.23533333333333328
config_id 11: Reaches 1, continues to 3

Epoch 2 ; Time: 0.624418 ; Training: accuracy=0.804215 ; Validation: accuracy=0.819667
Epoch 3 ; Time: 0.898253 ; Training: accuracy=0.861405 ; Validation: accuracy=0.874667

Update for config_id 11:3: reward = 0.8746666666666667, crit_val = 0.1253333333333333
config_id 11: Reaches 3, continues to 9

Epoch 4 ; Time: 1.185551 ; Training: accuracy=0.888926 ; Validation: accuracy=0.891000
Epoch 5 ; Time: 1.476594 ; Training: accuracy=0.911901 ; Validation: accuracy=0.915500
Epoch 6 ; Time: 1.762834 ; Training: accuracy=0.927686 ; Validation: accuracy=0.915667
Epoch 7 ; Time: 2.077373 ; Training: accuracy=0.935372 ; Validation: accuracy=0.929833
Epoch 8 ; Time: 2.379926 ; Training: accuracy=0.945455 ; Validation: accuracy=0.929500
Epoch 9 ; Time: 2.769369 ; Training: accuracy=0.946529 ; Validation: accuracy=0.935333

config_id 11: Terminating evaluation at 9
Update for config_id 11:9: reward = 0.9353333333333333, crit_val = 0.06466666666666665
Starting get_config[BO] for config_id 12
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.375139605358855
- self.std = 0.29194623770057787
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 23
Current best is [0.16599462]
[12: BO] (23 evaluations)
batch_size: 43
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.005048365786215596
n_units_1: 50
n_units_2: 99
scale_1: 0.08427330560427197
scale_2: 0.019575394403800444
Started BO from (top scorer):
batch_size: 43
dropout_1: 0.10030417821472981
dropout_2: 0.2123012790467915
learning_rate: 0.005482631492530546
n_units_1: 16
n_units_2: 99
scale_1: 0.0842733102219232
scale_2: 0.01957864750341544
Top score values: [0.22747641 0.24507579 0.2511109  0.26006843 0.26608125]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9. Pending:
Targets: [ 0.32064258 -0.23917224 -0.48359036  1.8294566   0.87410126  2.02272537
  1.6922382   0.37870075 -0.0661718  -0.37298964 -0.55413468 -0.65691604
  2.01002017 -0.58464469 -0.80555756 -0.93878852  0.53441905 -0.71638184
 -0.86255509 -0.98340699 -0.47887677 -0.85565847 -1.06345929]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0010985734536101213, 'kernel_inv_bw1': 0.8649270179329417, 'kernel_inv_bw2': 0.2636393579306121, 'kernel_inv_bw3': 3.8035263685806937, 'kernel_inv_bw4': 0.669976463457446, 'kernel_inv_bw5': 0.00011549998074446066, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0016618407331475172, 'kernel_inv_bw8': 0.9154142608685621, 'kernel_covariance_scale': 0.9179515989083743, 'mean_mean_value': 1.04777913925491}

Epoch 1 ; Time: 0.684860 ; Training: accuracy=0.589506 ; Validation: accuracy=0.767275

Update for config_id 12:1: reward = 0.7672745524510624, crit_val = 0.23272544754893765
config_id 12: Reaches 1, continues to 3

Epoch 2 ; Time: 1.311083 ; Training: accuracy=0.803526 ; Validation: accuracy=0.833361
Epoch 3 ; Time: 1.938117 ; Training: accuracy=0.857072 ; Validation: accuracy=0.867325

Update for config_id 12:3: reward = 0.8673247448552786, crit_val = 0.13267525514472145
config_id 12: Reaches 3, continues to 9

Epoch 4 ; Time: 2.593575 ; Training: accuracy=0.878259 ; Validation: accuracy=0.887569
Epoch 5 ; Time: 3.265805 ; Training: accuracy=0.906232 ; Validation: accuracy=0.892421
Epoch 6 ; Time: 3.906039 ; Training: accuracy=0.918977 ; Validation: accuracy=0.905304
Epoch 7 ; Time: 4.549798 ; Training: accuracy=0.924108 ; Validation: accuracy=0.913502
Epoch 8 ; Time: 5.284969 ; Training: accuracy=0.935529 ; Validation: accuracy=0.898109
Epoch 9 ; Time: 5.914810 ; Training: accuracy=0.940081 ; Validation: accuracy=0.907311

config_id 12: Terminating evaluation at 9
Update for config_id 12:9: reward = 0.9073113602141543, crit_val = 0.09268863978584574
Starting get_config[BO] for config_id 13
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3494730871435835
- self.std = 0.2843393969162383
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 26
Current best is [0.16599463]
[13: BO] (19 evaluations)
batch_size: 49
dropout_1: 0.0
dropout_2: 0.5645088370464826
learning_rate: 0.008117520067527348
n_units_1: 83
n_units_2: 128
scale_1: 0.011738370954664094
scale_2: 10.0
Started BO from (top scorer):
batch_size: 49
dropout_1: 0.18507369082965996
dropout_2: 0.5644247025103575
learning_rate: 0.021536534166200078
n_units_1: 83
n_units_2: 32
scale_1: 0.011738400285525419
scale_2: 5.12869757717152
Top score values: [0.14710688 0.16359598 0.16610807 0.17599915 0.17805276]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9. Pending:
Targets: [ 0.41948782 -0.15530354 -0.40626051  1.96866665  0.987753    2.16710588
  1.82777729  0.4790992   0.02232511 -0.29270092 -0.47869208 -0.58422312
  2.15406079 -0.51001831 -0.73684119 -0.87363644  0.63898338 -0.64527978
 -0.79536356 -0.91944857 -0.40142082 -0.78828244 -1.00164249 -0.41059256
 -0.76246146 -0.90309134]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010452280079272239, 'kernel_inv_bw1': 0.3851257033268748, 'kernel_inv_bw2': 0.004672299443157123, 'kernel_inv_bw3': 4.850239092869646, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.47895621368559343, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.41223347675408245, 'kernel_inv_bw8': 0.939813631659752, 'kernel_covariance_scale': 1.0011470092026824, 'mean_mean_value': 1.0431072156529337}

Epoch 1 ; Time: 0.685233 ; Training: accuracy=0.508717 ; Validation: accuracy=0.706591

config_id 13: Terminating evaluation at 1
Update for config_id 13:1: reward = 0.7065908330545333, crit_val = 0.2934091669454667
Starting get_config[BO] for config_id 14
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3473966456547643
- self.std = 0.2792249828512775
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 27
Current best is [0.16599463]
[14: BO] (30 evaluations)
batch_size: 53
dropout_1: 0.0
dropout_2: 0.11705125052606932
learning_rate: 0.01360292210731249
n_units_1: 108
n_units_2: 128
scale_1: 0.0010033865698770206
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 45
dropout_1: 0.07271341792828104
dropout_2: 0.3777104736924482
learning_rate: 0.004073244311091114
n_units_1: 106
n_units_2: 46
scale_1: 0.0010035328114221392
scale_2: 0.0015301629446470427
Top score values: [0.14323158 0.17123515 0.18547989 0.18802694 0.19629106]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1. Pending:
Targets: [ 0.4346078  -0.15071171 -0.40626532  2.01216211  1.01328159  2.21423604
  1.86869216  0.49531105  0.03017047 -0.29062572 -0.48002358 -0.58748757
  2.20095201 -0.5119236  -0.74290107 -0.88220192  0.65812374 -0.64966258
 -0.80249536 -0.92885317 -0.40133698 -0.79528454 -1.01255259 -0.41067671
 -0.76899061 -0.91219633 -0.19334759]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.004395445140104485, 'kernel_inv_bw1': 1.222434567552199, 'kernel_inv_bw2': 0.7771588193504158, 'kernel_inv_bw3': 4.605173301537861, 'kernel_inv_bw4': 0.003034444769035247, 'kernel_inv_bw5': 0.18456224385325032, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.09757269492936227, 'kernel_inv_bw8': 1.0344175929580213, 'kernel_covariance_scale': 0.8950917105716353, 'mean_mean_value': 1.1691876572286595}

Epoch 1 ; Time: 0.589378 ; Training: accuracy=0.639110 ; Validation: accuracy=0.778761

Update for config_id 14:1: reward = 0.7787610619469026, crit_val = 0.22123893805309736
config_id 14: Reaches 1, continues to 3

Epoch 2 ; Time: 1.126719 ; Training: accuracy=0.823155 ; Validation: accuracy=0.843046
Epoch 3 ; Time: 1.674030 ; Training: accuracy=0.858160 ; Validation: accuracy=0.887126

Update for config_id 14:3: reward = 0.8871263983970613, crit_val = 0.11287360160293869
config_id 14: Reaches 3, continues to 9

Epoch 4 ; Time: 2.356794 ; Training: accuracy=0.869497 ; Validation: accuracy=0.878611
Epoch 5 ; Time: 3.028255 ; Training: accuracy=0.885137 ; Validation: accuracy=0.905994
Epoch 6 ; Time: 3.693399 ; Training: accuracy=0.894820 ; Validation: accuracy=0.913508
Epoch 7 ; Time: 4.384751 ; Training: accuracy=0.903178 ; Validation: accuracy=0.906495
Epoch 8 ; Time: 5.078993 ; Training: accuracy=0.908474 ; Validation: accuracy=0.918350
Epoch 9 ; Time: 5.772203 ; Training: accuracy=0.903840 ; Validation: accuracy=0.886959

config_id 14: Terminating evaluation at 9
Update for config_id 14:9: reward = 0.886959425613625, crit_val = 0.11304057438637505
Starting get_config[BO] for config_id 15
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3275620848907016
- self.std = 0.27197642408867573
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 30
Current best is [0.11287361]
[15: BO] (33 evaluations)
batch_size: 128
dropout_1: 0.0
dropout_2: 0.05978471049967446
learning_rate: 0.008771087280096419
n_units_1: 81
n_units_2: 128
scale_1: 1.6051836086730857
scale_2: 0.013635796499749771
Started BO from (top scorer):
batch_size: 10
dropout_1: 0.04231197881231458
dropout_2: 0.26711820263125935
learning_rate: 0.0010374236949651373
n_units_1: 81
n_units_2: 22
scale_1: 1.6051803295987732
scale_2: 0.26385189842384354
Top score values: [0.11687344 0.14021704 0.14263611 0.14613091 0.19120987]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9. Pending:
Targets: [ 0.51911821 -0.08180089 -0.34416537  2.13871659  1.11321449  2.34617608
  1.99142296  0.5814393   0.10390206 -0.22544381 -0.41988939 -0.53021745
  2.33253801 -0.45263959 -0.68977294 -0.83278636  0.74859118 -0.59404951
 -0.7509555  -0.88068093 -0.33910568 -0.74355251 -0.96661106 -0.34869433
 -0.71655781 -0.86358016 -0.12557308 -0.39092781 -0.78936431 -0.78875039]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.072806109016839, 'kernel_inv_bw1': 0.9779305280174871, 'kernel_inv_bw2': 0.6350438861248543, 'kernel_inv_bw3': 4.38767732128345, 'kernel_inv_bw4': 0.00015799760540255293, 'kernel_inv_bw5': 0.34358579783209253, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.3837206410237312, 'kernel_inv_bw8': 1.0972874897515217, 'kernel_covariance_scale': 0.8787819972958487, 'mean_mean_value': 1.2176464810673702}

Epoch 1 ; Time: 0.281261 ; Training: accuracy=0.667105 ; Validation: accuracy=0.812001

Update for config_id 15:1: reward = 0.812001329787234, crit_val = 0.18799867021276595
config_id 15: Reaches 1, continues to 3

Epoch 2 ; Time: 0.541857 ; Training: accuracy=0.835938 ; Validation: accuracy=0.872340
Epoch 3 ; Time: 0.765531 ; Training: accuracy=0.884622 ; Validation: accuracy=0.896110

Update for config_id 15:3: reward = 0.8961103723404256, crit_val = 0.10388962765957444
config_id 15: Reaches 3, continues to 9

Epoch 4 ; Time: 0.992504 ; Training: accuracy=0.908553 ; Validation: accuracy=0.912566
Epoch 5 ; Time: 1.228107 ; Training: accuracy=0.917188 ; Validation: accuracy=0.913730
Epoch 6 ; Time: 1.464472 ; Training: accuracy=0.935197 ; Validation: accuracy=0.929854
Epoch 7 ; Time: 1.687305 ; Training: accuracy=0.935937 ; Validation: accuracy=0.926695
Epoch 8 ; Time: 1.922657 ; Training: accuracy=0.941776 ; Validation: accuracy=0.938331
Epoch 9 ; Time: 2.179316 ; Training: accuracy=0.949013 ; Validation: accuracy=0.922872

config_id 15: Terminating evaluation at 9
Update for config_id 15:9: reward = 0.9228723404255319, crit_val = 0.0771276595744681
Starting get_config[BO] for config_id 16
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.30896601527781375
- self.std = 0.26628465735221635
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 33
Current best is [0.06466667]
[16: BO] (32 evaluations)
batch_size: 84
dropout_1: 0.1872000861932956
dropout_2: 0.0
learning_rate: 0.006197816480758794
n_units_1: 83
n_units_2: 102
scale_1: 1.5185688244258915
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 84
dropout_1: 0.45161421544435854
dropout_2: 0.5638747171714953
learning_rate: 0.021592731858966305
n_units_1: 83
n_units_2: 102
scale_1: 1.518495173672961
scale_2: 0.05388875466468348
Top score values: [-0.05190702 -0.01884247  0.02839783  0.06007975  0.07019629]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9. Pending:
Targets: [ 0.60004953 -0.01371406 -0.28168651  2.25426642  1.20684447  2.4661603
  2.10382442  0.66370272  0.17595824 -0.16042731 -0.35902912 -0.47171541
  2.45223072 -0.39247935 -0.63468136 -0.78075165  0.83442743 -0.53691186
 -0.69717168 -0.82966996 -0.27651868 -0.68961045 -0.91743682 -0.28631228
 -0.66203874 -0.81220367 -0.05842187 -0.32944849 -0.73640147 -0.73577443
 -0.45427831 -0.77013971 -0.87064106]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0011225369972828222, 'kernel_inv_bw1': 1.1410997493950654, 'kernel_inv_bw2': 0.7917189890607564, 'kernel_inv_bw3': 4.34788067006306, 'kernel_inv_bw4': 0.0038802911728676643, 'kernel_inv_bw5': 0.001087465565066201, 'kernel_inv_bw6': 0.000328139144163153, 'kernel_inv_bw7': 0.381417927287251, 'kernel_inv_bw8': 1.0745248268360206, 'kernel_covariance_scale': 0.9388174021125767, 'mean_mean_value': 1.2434091179730569}

Epoch 1 ; Time: 0.404633 ; Training: accuracy=0.615575 ; Validation: accuracy=0.787223

Update for config_id 16:1: reward = 0.7872233400402414, crit_val = 0.21277665995975859
config_id 16: Reaches 1, continues to 3

Epoch 2 ; Time: 0.763939 ; Training: accuracy=0.771577 ; Validation: accuracy=0.853789
Epoch 3 ; Time: 1.106980 ; Training: accuracy=0.824487 ; Validation: accuracy=0.881791

Update for config_id 16:3: reward = 0.8817907444668008, crit_val = 0.11820925553319916
config_id 16: Reaches 3, continues to 9

Epoch 4 ; Time: 1.454785 ; Training: accuracy=0.843998 ; Validation: accuracy=0.892857
Epoch 5 ; Time: 1.792809 ; Training: accuracy=0.866319 ; Validation: accuracy=0.896546
Epoch 6 ; Time: 2.307078 ; Training: accuracy=0.877976 ; Validation: accuracy=0.924212
Epoch 7 ; Time: 2.651515 ; Training: accuracy=0.889964 ; Validation: accuracy=0.913984
Epoch 8 ; Time: 2.987180 ; Training: accuracy=0.891782 ; Validation: accuracy=0.918846
Epoch 9 ; Time: 3.323330 ; Training: accuracy=0.899636 ; Validation: accuracy=0.933434

config_id 16: Terminating evaluation at 9
Update for config_id 16:9: reward = 0.9334339369550637, crit_val = 0.06656606304493629
Starting get_config[BO] for config_id 17
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.2942619578529374
- self.std = 0.26015822396056204
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 36
Current best is [0.06466667]
[17: BO] (17 evaluations)
batch_size: 19
dropout_1: 0.44699880885461124
dropout_2: 0.35902378542272395
learning_rate: 0.00994454798419592
n_units_1: 41
n_units_2: 41
scale_1: 0.20722456251139096
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 19
dropout_1: 0.446998616449144
dropout_2: 0.44450165886895776
learning_rate: 0.007050608524261196
n_units_1: 41
n_units_2: 40
scale_1: 0.20722353706958116
scale_2: 0.001180574218724413
Top score values: [-0.04060306 -0.01667316  0.02036014  0.02545359  0.04380131]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9. Pending:
Targets: [ 0.6706997   0.04248266 -0.23180024  2.36387153  1.29178397  2.58075527
  2.20988679  0.73585184  0.23662153 -0.10768552 -0.31096418 -0.42630411
  2.56649767 -0.34520212 -0.59310772 -0.7426178   0.91059692 -0.49303586
 -0.65706962 -0.79268808 -0.22651071 -0.64933033 -0.88252175 -0.23653494
 -0.62110934 -0.77481048 -0.00327797 -0.28068696 -0.69722323 -0.69658141
 -0.40845638 -0.73175596 -0.834624   -0.31321438 -0.67671396 -0.87522082]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010024528982465707, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.6296139381452603, 'kernel_inv_bw3': 4.578274729809311, 'kernel_inv_bw4': 0.00018607512015933497, 'kernel_inv_bw5': 0.8959493144763312, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.14840236754369968, 'kernel_inv_bw8': 1.4027812901311545, 'kernel_covariance_scale': 0.8876423821230857, 'mean_mean_value': 1.3226521421150887}

Epoch 1 ; Time: 1.561811 ; Training: accuracy=0.382677 ; Validation: accuracy=0.670254

config_id 17: Terminating evaluation at 1
Update for config_id 17:1: reward = 0.6702539095342189, crit_val = 0.32974609046578107
Starting get_config[BO] for config_id 18
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.29522098846409534
- self.std = 0.256682996577975
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 37
Current best is [0.06466667]
[18: BO] (23 evaluations)
batch_size: 64
dropout_1: 0.013136665274062341
dropout_2: 0.75
learning_rate: 0.00010583972178928185
n_units_1: 128
n_units_2: 16
scale_1: 0.5429125928862846
scale_2: 0.011730911155531775
Started BO from (top scorer):
batch_size: 64
dropout_1: 0.013136987059864602
dropout_2: 0.5034888824382017
learning_rate: 4.629414809281751e-06
n_units_1: 16
n_units_2: 73
scale_1: 0.5429104968156312
scale_2: 0.01173088986204388
Top score values: [-0.02428519  0.05932551  0.07152494  0.07359064  0.08147375]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1. Pending:
Targets: [ 0.67604405  0.03932159 -0.23867482  2.3921397   1.30553717  2.61195984
  2.23607017  0.74207828  0.2360889  -0.11287972 -0.31891057 -0.43581208
  2.5975092  -0.35361205 -0.60487404 -0.75640834  0.91918924 -0.50344731
 -0.66970192 -0.80715652 -0.23331368 -0.66185785 -0.89820644 -0.24347363
 -0.63325478 -0.78903687 -0.0070586  -0.28822342 -0.71039917 -0.70974866
 -0.41772272 -0.74539944 -0.84966021 -0.32119123 -0.68961223 -0.89080667
  0.13450483]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.6191424312423364, 'kernel_inv_bw3': 4.422173979447018, 'kernel_inv_bw4': 0.1853160297787087, 'kernel_inv_bw5': 0.9528081584043367, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00012639453445066284, 'kernel_inv_bw8': 1.5240871677818384, 'kernel_covariance_scale': 0.8876537823197109, 'mean_mean_value': 1.3439923159251053}

Epoch 1 ; Time: 0.559545 ; Training: accuracy=0.061756 ; Validation: accuracy=0.174059

config_id 18: Terminating evaluation at 1
Update for config_id 18:1: reward = 0.17405913978494625, crit_val = 0.8259408602150538
Starting get_config[BO] for config_id 19
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.30918730087859425
- self.std = 0.2671506242706291
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 38
Current best is [0.06466667]
[19: BO] (41 evaluations)
batch_size: 82
dropout_1: 0.5033788670142001
dropout_2: 0.0
learning_rate: 0.0068366382394597375
n_units_1: 128
n_units_2: 112
scale_1: 0.0023098041291240022
scale_2: 10.0
Started BO from (top scorer):
batch_size: 82
dropout_1: 0.5033734160492536
dropout_2: 0.12329436421898304
learning_rate: 0.014691508241772754
n_units_1: 74
n_units_2: 52
scale_1: 0.0023096446056230517
scale_2: 0.0028549744663600007
Top score values: [0.03802058 0.05108491 0.05179572 0.06859465 0.08715316]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1. Pending:
Targets: [ 0.59727616 -0.01449793 -0.28160174  2.24613091  1.20210417  2.45733794
  2.09617657  0.66072301  0.17455956 -0.1607356  -0.35869365 -0.47101467
  2.44345351 -0.39203544 -0.63345236 -0.77904917  0.83089432 -0.53599979
 -0.69574012 -0.82780891 -0.27645066 -0.6882034  -0.91529127 -0.28621252
 -0.66072107 -0.81039923 -0.05906082 -0.3292089  -0.73484275 -0.73421774
 -0.45363409 -0.76847162 -0.8686472  -0.36088495 -0.71487029 -0.90818144
  0.0769558   1.93431537]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.5985100096701567, 'kernel_inv_bw3': 4.267699620387008, 'kernel_inv_bw4': 0.15621448972299348, 'kernel_inv_bw5': 0.9103553579184126, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0732959268315928, 'kernel_inv_bw8': 1.4976041672770317, 'kernel_covariance_scale': 0.8556419538080655, 'mean_mean_value': 1.3261775399137425}

Epoch 1 ; Time: 0.408917 ; Training: accuracy=0.496786 ; Validation: accuracy=0.691113

config_id 19: Terminating evaluation at 1
Update for config_id 19:1: reward = 0.6911125960574674, crit_val = 0.30888740394253256
Starting get_config[BO] for config_id 20
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3091796112135671
- self.std = 0.2637033792780162
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 39
Current best is [0.06466667]
[20: BO] (25 evaluations)
batch_size: 128
dropout_1: 0.41108457637056517
dropout_2: 0.2626242802150693
learning_rate: 0.0058569004185053146
n_units_1: 128
n_units_2: 102
scale_1: 0.9534302215684295
scale_2: 0.018070562137925787
Started BO from (top scorer):
batch_size: 128
dropout_1: 0.41106820571967884
dropout_2: 0.26262421544361153
learning_rate: 0.025796940714592858
n_units_1: 95
n_units_2: 47
scale_1: 0.9533993859134777
scale_2: 0.03822432087523404
Top score values: [-0.01791735  0.01786102  0.04308918  0.04802902  0.05093939]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1. Pending:
Targets: [ 6.05113174e-01 -1.46582880e-02 -2.85253804e-01  2.27552247e+00
  1.21784776e+00  2.48949049e+00  2.12360786e+00  6.69389428e-01
  1.76870637e-01 -1.62807648e-01 -3.63353486e-01 -4.77142818e-01
  2.47542456e+00 -3.97131141e-01 -6.41703968e-01 -7.89204075e-01
  8.41785289e-01 -5.42977447e-01 -7.04805980e-01 -8.38601225e-01
 -2.80035387e-01 -6.97170732e-01 -9.27227194e-01 -2.89924854e-01
 -6.69329140e-01 -8.20963963e-01 -5.98037246e-02 -3.33483300e-01
 -7.44419773e-01 -7.43786588e-01 -4.59535033e-01 -7.78488255e-01
 -8.79973371e-01 -3.65573439e-01 -7.24186228e-01 -9.20024418e-01
  7.79909583e-02  1.95963074e+00 -1.10809073e-03]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.001165694398243441, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.2523015365445955, 'kernel_inv_bw4': 0.40855856284176645, 'kernel_inv_bw5': 0.7504960382174999, 'kernel_inv_bw6': 0.0002734363676833348, 'kernel_inv_bw7': 0.5131858700249358, 'kernel_inv_bw8': 1.00815607605841, 'kernel_covariance_scale': 0.9259442809609314, 'mean_mean_value': 1.38937512547733}

Epoch 1 ; Time: 0.298842 ; Training: accuracy=0.539145 ; Validation: accuracy=0.750665

config_id 20: Terminating evaluation at 1
Update for config_id 20:1: reward = 0.7506648936170213, crit_val = 0.24933510638297873
Starting get_config[BO] for config_id 21
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.30768349859280236
- self.std = 0.2605537971371087
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 40
Current best is [0.06466667]
[21: BO] (24 evaluations)
batch_size: 48
dropout_1: 0.18570593072122676
dropout_2: 0.5713371911053257
learning_rate: 0.013632647785826724
n_units_1: 16
n_units_2: 32
scale_1: 10.0
scale_2: 10.0
Started BO from (top scorer):
batch_size: 48
dropout_1: 0.18570598047780576
dropout_2: 0.5713282997895286
learning_rate: 0.033333077372950354
n_units_1: 47
n_units_2: 44
scale_1: 0.09619764668914697
scale_2: 0.008348100279749712
Top score values: [-0.03398156  0.045603    0.04790041  0.05168735  0.05303241]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1. Pending:
Targets: [ 0.61816985 -0.00909343 -0.28295991  2.3087711   1.23831119  2.52532558
  2.15502015  0.68322308  0.1847507  -0.15903362 -0.36200367 -0.47716849
  2.51108961 -0.39618963 -0.64371886 -0.79300195  0.85770286 -0.54379893
 -0.70758365 -0.84299621 -0.27767841 -0.6998561  -0.9326935  -0.28768743
 -0.67167796 -0.82514575 -0.05478459 -0.33177241 -0.74767629 -0.74703545
 -0.45934786 -0.7821566  -0.88486847 -0.36425045 -0.72719816 -0.92540365
  0.08467576  1.98906087  0.00462056 -0.2239399 ]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.000688950031210672, 'kernel_inv_bw3': 3.8843756119878727, 'kernel_inv_bw4': 0.3354257626321199, 'kernel_inv_bw5': 0.7226708893548897, 'kernel_inv_bw6': 0.08692415231211728, 'kernel_inv_bw7': 0.474513251952304, 'kernel_inv_bw8': 1.0015103321713872, 'kernel_covariance_scale': 1.0393298947058967, 'mean_mean_value': 1.3624631143322377}

Epoch 1 ; Time: 0.848740 ; Training: accuracy=0.100612 ; Validation: accuracy=0.204637

config_id 21: Terminating evaluation at 1
Update for config_id 21:1: reward = 0.20463709677419356, crit_val = 0.7953629032258065
Starting get_config[BO] for config_id 22
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3195781182179976
- self.std = 0.2681263666763133
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 41
Current best is [0.06466667]
[22: BO] (27 evaluations)
batch_size: 90
dropout_1: 0.0
dropout_2: 0.6879712739929725
learning_rate: 0.004995912830531246
n_units_1: 16
n_units_2: 58
scale_1: 0.03908079586733884
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 90
dropout_1: 0.025893045336624287
dropout_2: 0.6879725645841153
learning_rate: 0.008779959281369732
n_units_1: 73
n_units_2: 43
scale_1: 0.03907780447755258
scale_2: 0.012925835170614252
Top score values: [-0.08560423 -0.01200155  0.01235796  0.03412326  0.05451195]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1. Pending:
Targets: [ 0.55634917 -0.0531986  -0.3193304   2.19920355  1.15897614  2.40964198
  2.04979491  0.61956513  0.13517088 -0.1989041  -0.39614175 -0.50805403
  2.39580807 -0.42936222 -0.66990059 -0.81496755  0.78911716 -0.57280265
 -0.73196168 -0.86354985 -0.31419806 -0.72445238 -0.95071385 -0.32392439
 -0.69707006 -0.84620353 -0.09759932 -0.3667643  -0.77092201 -0.77029927
 -0.4907367  -0.8044285  -0.90423953 -0.39832509 -0.75102223 -0.94362989
  0.03792231  1.88852274 -0.03987192 -0.26197726  1.77447966]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.180300429625024, 'kernel_inv_bw2': 0.00012375616416109952, 'kernel_inv_bw3': 5.830164570865315, 'kernel_inv_bw4': 0.04290500448812368, 'kernel_inv_bw5': 1.5281996769225803, 'kernel_inv_bw6': 0.00012536666893201966, 'kernel_inv_bw7': 0.5410548461976437, 'kernel_inv_bw8': 0.9484448071973319, 'kernel_covariance_scale': 0.9935261698170694, 'mean_mean_value': 1.3437429881542777}

Epoch 1 ; Time: 0.404031 ; Training: accuracy=0.192123 ; Validation: accuracy=0.465320

config_id 22: Terminating evaluation at 1

Update for config_id 22:1: reward = 0.4653198653198653, crit_val = 0.5346801346801346
Starting get_config[BO] for config_id 23
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3246995948004294
- self.std = 0.26693716606537654
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 42
Current best is [0.06466667]
[23: BO] (28 evaluations)
batch_size: 109
dropout_1: 0.11852060048440546
dropout_2: 0.0
learning_rate: 0.006138751368351793
n_units_1: 128
n_units_2: 102
scale_1: 0.02891738158418114
scale_2: 0.18188937111131168
Started BO from (top scorer):
batch_size: 109
dropout_1: 0.04031775066009935
dropout_2: 0.7182959892329118
learning_rate: 0.0045752284522110245
n_units_1: 103
n_units_2: 65
scale_1: 0.02891742255788364
scale_2: 0.9099861166037259
Top score values: [-0.02438883 -0.00867721  0.00772424  0.00978592  0.02696178]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1. Pending:
Targets: [ 0.53964162 -0.07262167 -0.33993908  2.18981489  1.14495329  2.40119082
  2.03974064  0.60313921  0.11658699 -0.21897629 -0.41709263 -0.52950348
  2.38729528 -0.45046109 -0.69207106 -0.83778429  0.77344659 -0.59454056
 -0.75440863 -0.88658303 -0.33478388 -0.74686588 -0.97413534 -0.34455355
 -0.71936157 -0.86915943 -0.1172202  -0.38758431 -0.79354253 -0.79291701
 -0.512109   -0.82719829 -0.92745397 -0.41928569 -0.7735541  -0.96701983
  0.01890518  1.87775001 -0.05923563 -0.28233044  1.76319887  0.78662909]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.9812140563421585, 'kernel_inv_bw2': 0.33496536307372765, 'kernel_inv_bw3': 5.3016968001601725, 'kernel_inv_bw4': 0.3085800529760349, 'kernel_inv_bw5': 1.289053703882947, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.5175407006989317, 'kernel_inv_bw8': 0.9188839771275492, 'kernel_covariance_scale': 0.9524801588092421, 'mean_mean_value': 1.4147913481699443}

Epoch 1 ; Time: 0.470745 ; Training: accuracy=0.588313 ; Validation: accuracy=0.784654

Update for config_id 23:1: reward = 0.7846538782318598, crit_val = 0.21534612176814016
config_id 23: Reaches 1, continues to 3

Epoch 2 ; Time: 0.747943 ; Training: accuracy=0.791305 ; Validation: accuracy=0.856214
Epoch 3 ; Time: 1.036803 ; Training: accuracy=0.851393 ; Validation: accuracy=0.870225

config_id 23: Terminating evaluation at 3
Update for config_id 23:3: reward = 0.8702251876563804, crit_val = 0.12977481234361965
Starting get_config[BO] for config_id 24
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3177841799029499
- self.std = 0.26287650785603406
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 44
Current best is [0.06466667]
[24: BO] (26 evaluations)
batch_size: 128
dropout_1: 0.21841606653758272
dropout_2: 0.20374184939425172
learning_rate: 0.00508606125379937
n_units_1: 65
n_units_2: 111
scale_1: 0.34275490027583067
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 128
dropout_1: 0.573300893527115
dropout_2: 0.3422561033358465
learning_rate: 0.015760765651058696
n_units_1: 65
n_units_2: 88
scale_1: 0.3427537059973482
scale_2: 1.0430141723650779
Top score values: [-0.058857   -0.03617593 -0.00602565  0.00015744  0.00246559]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3. Pending:
Targets: [ 0.57428418 -0.04743676 -0.31888342  2.24994771  1.18894611  2.46458876
  2.09755526  0.63876261  0.14469462 -0.19605212 -0.39722877 -0.51137602
  2.45047858 -0.43111267 -0.67645479 -0.82441886  0.81170074 -0.57741773
 -0.73975529 -0.87397138 -0.31364859 -0.73209603 -0.96287612 -0.32356917
 -0.70416686 -0.85627865 -0.0927242  -0.36726462 -0.77949369 -0.77885851
 -0.49371285 -0.81366933 -0.91547367 -0.3994557  -0.7591965  -0.95565069
  0.04550392  1.93306235 -0.03384394 -0.2603849   1.81674174  0.82508687
 -0.3896813  -0.71520034]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.4773107164096427, 'kernel_inv_bw2': 1.4679705090422104, 'kernel_inv_bw3': 6.238172565895472, 'kernel_inv_bw4': 0.0013688879382363422, 'kernel_inv_bw5': 1.2841568273470492, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.07584607661842979, 'kernel_inv_bw8': 1.0557207868122573, 'kernel_covariance_scale': 0.9777257034912031, 'mean_mean_value': 1.3924430037176574}

Epoch 1 ; Time: 0.290785 ; Training: accuracy=0.493339 ; Validation: accuracy=0.717254

config_id 24: Terminating evaluation at 1
Update for config_id 24:1: reward = 0.7172539893617021, crit_val = 0.28274601063829785
Starting get_config[BO] for config_id 25
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.31700555391929097
- self.std = 0.25999055369635105
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 45
Current best is [0.06466667]
[25: BO] (28 evaluations)
batch_size: 67
dropout_1: 0.4415543835380574
dropout_2: 0.17713599260891252
learning_rate: 0.007534325733484996
n_units_1: 47
n_units_2: 128
scale_1: 0.18011623235166907
scale_2: 0.011939981901592116
Started BO from (top scorer):
batch_size: 67
dropout_1: 0.27446038977900605
dropout_2: 0.3751223727356925
learning_rate: 0.03731393813805193
n_units_1: 47
n_units_2: 66
scale_1: 0.025936041515244566
scale_2: 0.05929575041254891
Top score values: [-0.01769174 -0.00732726  0.0144894   0.019887    0.02668765]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1. Pending:
Targets: [ 0.58365369 -0.04496849 -0.31942828  2.27791746  1.2051385   2.49494108
  2.12383342  0.64884785  0.14929559 -0.19523351 -0.39864327 -0.51405759
  2.48067428 -0.43290329 -0.68096877 -0.83057527  0.82370563 -0.58083237
 -0.74497192 -0.88067784 -0.31413534 -0.73722763 -0.97056944 -0.32416603
 -0.70898845 -0.86278871 -0.09075863 -0.36834652 -0.78515142 -0.78450919
 -0.49619835 -0.81970642 -0.92264081 -0.40089493 -0.76462893 -0.96326381
  0.04900384  1.95751461 -0.03122479 -0.26028041  1.83990281  0.83724034
 -0.39101202 -0.7201444  -0.13177226]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.6355233403702053, 'kernel_inv_bw2': 0.9029958244613535, 'kernel_inv_bw3': 4.901554552077261, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.2950753092823263, 'kernel_inv_bw6': 0.9148325451466071, 'kernel_inv_bw7': 0.37603134858345694, 'kernel_inv_bw8': 0.8614864415955265, 'kernel_covariance_scale': 1.1426790498461497, 'mean_mean_value': 1.2717253602063514}

Epoch 1 ; Time: 0.496053 ; Training: accuracy=0.463516 ; Validation: accuracy=0.705517

config_id 25: Terminating evaluation at 1
Update for config_id 25:1: reward = 0.7055173570350495, crit_val = 0.2944826429649505
Starting get_config[BO] for config_id 26
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.31651592542028356
- self.std = 0.2571700171079883
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 46
Current best is [0.06466667]
[26: BO] (25 evaluations)
batch_size: 115
dropout_1: 0.431217753143572
dropout_2: 0.220775368953466
learning_rate: 0.005866651154538663
n_units_1: 111
n_units_2: 107
scale_1: 2.1462066529246764
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 115
dropout_1: 0.3500250446723522
dropout_2: 0.3577246428515128
learning_rate: 0.007746856505449004
n_units_1: 111
n_units_2: 111
scale_1: 2.1364316530563725
scale_2: 0.12210755388377395
Top score values: [-0.16187454 -0.03440598  0.02215888  0.02287187  0.02482086]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1. Pending:
Targets: [ 0.59195888 -0.04355778 -0.32102773  2.30480465  1.22025988  2.52420849
  2.14903067  0.65786806  0.15283691 -0.19547085 -0.40111152 -0.51779165
  2.50978522 -0.43574729 -0.68653345 -0.83778077  0.83464361 -0.5852988
 -0.75123856 -0.88843285 -0.31567674 -0.74340934 -0.97931035 -0.32581744
 -0.71486043 -0.87034752 -0.08985013 -0.37048249 -0.79185873 -0.79120946
 -0.49973654 -0.82679272 -0.93085605 -0.40338787 -0.77111116 -0.97192459
  0.05144521  1.98088774 -0.02966334 -0.26123115  1.86198602  0.84832677
 -0.39339657 -0.72613874 -0.13131358 -0.08567594]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.4016969832248494, 'kernel_inv_bw2': 1.5548431716715594, 'kernel_inv_bw3': 6.410101329245941, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 1.2789051987867037, 'kernel_inv_bw6': 0.002500601231354757, 'kernel_inv_bw7': 0.07994470027723147, 'kernel_inv_bw8': 1.050533756162631, 'kernel_covariance_scale': 1.013963918545403, 'mean_mean_value': 1.3947599890293165}

Epoch 1 ; Time: 0.316076 ; Training: accuracy=0.518758 ; Validation: accuracy=0.727090

config_id 26: Terminating evaluation at 1
Update for config_id 26:1: reward = 0.7270903010033445, crit_val = 0.2729096989966555
Starting get_config[BO] for config_id 27
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.315588133368717
- self.std = 0.25449726243737253
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 47
Current best is [0.06466667]
[27: BO] (32 evaluations)
batch_size: 21
dropout_1: 0.44977523402211345
dropout_2: 0.22775354317878593
learning_rate: 0.005830667219304713
n_units_1: 56
n_units_2: 107
scale_1: 0.14328846989149557
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 21
dropout_1: 0.6307901388503625
dropout_2: 0.048396805897776285
learning_rate: 0.005206382249900967
n_units_1: 56
n_units_2: 68
scale_1: 0.14327058821032795
scale_2: 0.2616426747102914
Top score values: [-0.01671427 -0.01380325  0.00721527  0.00772511  0.01064867]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1. Pending:
Targets: [ 0.60182127 -0.04036964 -0.3207536   2.33265551  1.23672076  2.55436356
  2.17524559  0.66842264  0.1580876  -0.19387811 -0.40167844 -0.51958396
  2.53978881 -0.43667796 -0.6900979  -0.84293364  0.8470547  -0.58780007
 -0.75548255 -0.89411767 -0.31534642 -0.74757111 -0.98594957 -0.32559362
 -0.71872238 -0.8758424  -0.08714815 -0.37072774 -0.79652932 -0.79587323
 -0.50133924 -0.83183019 -0.9369864  -0.4039787  -0.77556385 -0.97848624
  0.05563108  2.0053368  -0.02632928 -0.26032904  1.88518637  0.86088156
 -0.39388247 -0.73011913 -0.12904706 -0.08293013 -0.16769703]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.3954479158901061, 'kernel_inv_bw2': 1.4648046751511983, 'kernel_inv_bw3': 6.3635395955492475, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 1.2760616829331175, 'kernel_inv_bw6': 0.000459823768002302, 'kernel_inv_bw7': 0.07631597784979986, 'kernel_inv_bw8': 1.0572767610160405, 'kernel_covariance_scale': 1.0481146015880773, 'mean_mean_value': 1.407447219082324}

Epoch 1 ; Time: 1.420753 ; Training: accuracy=0.491925 ; Validation: accuracy=0.739021

config_id 27: Terminating evaluation at 1
Update for config_id 27:1: reward = 0.7390206966178697, crit_val = 0.26097930338213027
Starting get_config[BO] for config_id 28
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3144504494106631
- self.std = 0.2519530485512195
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 48
Current best is [0.06466667]
[28: BO] (52 evaluations)
batch_size: 8
dropout_1: 0.42184871402947194
dropout_2: 0.22973772747988677
learning_rate: 0.005612837601248531
n_units_1: 16
n_units_2: 107
scale_1: 2.569994101154867
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 86
dropout_1: 0.012485018503003986
dropout_2: 0.23159412369703133
learning_rate: 0.0030719495858766528
n_units_1: 83
n_units_2: 38
scale_1: 2.427669128386413
scale_2: 0.12413975027733427
Top score values: [0.01561047 0.04301952 0.06196485 0.06372489 0.06632395]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1. Pending:
Targets: [ 0.61241391 -0.03626183 -0.3194771   2.36072606  1.25372459  2.5846729
  2.20172661  0.67968781  0.16419943 -0.19132043 -0.40121912 -0.52031524
  2.56995098 -0.43657206 -0.69255103 -0.8469301   0.86012369 -0.5892202
 -0.75859593 -0.89863098 -0.31401532 -0.7506046  -0.9913902  -0.324366
 -0.72146456 -0.88017117 -0.08351271 -0.36995588 -0.80005719 -0.79939448
 -0.50188628 -0.83571452 -0.9419326  -0.4035426  -0.77888001 -0.98385151
  0.0607083   2.03010209 -0.02207969 -0.25844237  1.90873838  0.87409018
 -0.39334443 -0.73297639 -0.12583471 -0.07925209 -0.16487497 -0.21222663]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.043354515241428186, 'kernel_inv_bw1': 0.48722866739628845, 'kernel_inv_bw2': 1.344945514678171, 'kernel_inv_bw3': 6.392254368452178, 'kernel_inv_bw4': 0.008478845030866105, 'kernel_inv_bw5': 1.3001252151102531, 'kernel_inv_bw6': 0.0023953033039684018, 'kernel_inv_bw7': 0.06077238974733418, 'kernel_inv_bw8': 1.0780826260251941, 'kernel_covariance_scale': 1.065768961640694, 'mean_mean_value': 1.4203155605053284}

Epoch 1 ; Time: 3.696474 ; Training: accuracy=0.291777 ; Validation: accuracy=0.577894

config_id 28: Terminating evaluation at 1
Update for config_id 28:1: reward = 0.5778936742934051, crit_val = 0.4221063257065949
Starting get_config[BO] for config_id 29
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.31664750811058007
- self.std = 0.24983298575475252
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 49
Current best is [0.06466667]
[29: BO] (25 evaluations)
batch_size: 56
dropout_1: 0.0
dropout_2: 0.24848470693697405
learning_rate: 0.006213675660362365
n_units_1: 118
n_units_2: 101
scale_1: 0.06714003227160924
scale_2: 0.011263999369006229
Started BO from (top scorer):
batch_size: 56
dropout_1: 0.7361319246310831
dropout_2: 0.2156009675340169
learning_rate: 0.003663253609023281
n_units_1: 77
n_units_2: 107
scale_1: 0.5478226343206746
scale_2: 0.011264025318744222
Top score values: [-0.03000711  0.0269265   0.04830531  0.05073006  0.05651974]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1. Pending:
Targets: [ 0.60881669 -0.04536365 -0.33098227  2.37196488  1.25556948  2.59781212
  2.21161618  0.67666148  0.1567987  -0.20173807 -0.41341794 -0.53352471
  2.58296527 -0.44907089 -0.70722207 -0.86291119  0.85862853 -0.60301439
 -0.77382742 -0.9150508  -0.32547413 -0.76576828 -1.00859717 -0.33591265
 -0.73638096 -0.89643434 -0.0930155  -0.3818894  -0.81564052 -0.81497218
 -0.51493936 -0.85160044 -0.95871987 -0.41576114 -0.79428364 -1.0009945
  0.05242936  2.03853527 -0.03106117 -0.2694296   1.91614167  0.87271353
 -0.40547643 -0.74799048 -0.13569664 -0.08871873 -0.17506819 -0.22282168
  0.42211727]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.22349405064456837, 'kernel_inv_bw2': 0.9635925241399346, 'kernel_inv_bw3': 5.176157668350697, 'kernel_inv_bw4': 0.5131296608518512, 'kernel_inv_bw5': 0.8035989804015595, 'kernel_inv_bw6': 0.7874489280538858, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9134532801855584, 'kernel_covariance_scale': 1.0785513392934298, 'mean_mean_value': 1.563999257733009}

Epoch 1 ; Time: 0.562677 ; Training: accuracy=0.597801 ; Validation: accuracy=0.773031

Update for config_id 29:1: reward = 0.7730307076101469, crit_val = 0.22696929238985308
config_id 29: Reaches 1, continues to 3

Epoch 2 ; Time: 1.068709 ; Training: accuracy=0.787946 ; Validation: accuracy=0.851636
Epoch 3 ; Time: 1.575621 ; Training: accuracy=0.835317 ; Validation: accuracy=0.875668

config_id 29: Terminating evaluation at 3
Update for config_id 29:3: reward = 0.8756675567423231, crit_val = 0.1243324432576769
Starting get_config[BO] for config_id 30
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.31111822809933243
- self.std = 0.2466194005265664
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 51
Current best is [0.06466667]
[30: BO] (29 evaluations)
batch_size: 70
dropout_1: 0.31958342053905936
dropout_2: 0.24399002542126408
learning_rate: 0.006599322198136962
n_units_1: 105
n_units_2: 105
scale_1: 0.06022014151194356
scale_2: 0.0032339328704914226
Started BO from (top scorer):
batch_size: 70
dropout_1: 0.551843930100634
dropout_2: 0.3970268561101558
learning_rate: 0.006599975278097348
n_units_1: 110
n_units_2: 60
scale_1: 0.03873172490128809
scale_2: 0.0032339282381031492
Top score values: [-0.0579298   0.0043006   0.04447643  0.04534082  0.0519806 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3. Pending:
Targets: [ 0.6391702  -0.02353447 -0.31287485  2.42529317  1.29435053  2.65408333
  2.26285504  0.70789904  0.18126217 -0.18194653 -0.39638471 -0.51805653
  2.63904301 -0.43250223 -0.69401726 -0.85173509  0.89223722 -0.5884517
 -0.76149052 -0.90455412 -0.30729494 -0.75332636 -0.99931944 -0.31786948
 -0.72355611 -0.88569507 -0.07180725 -0.36444534 -0.80384846 -0.80317142
 -0.499229   -0.84027696 -0.94879222 -0.39875844 -0.78221329 -0.99161771
  0.07553283  2.08751879 -0.00904562 -0.25052012  1.96353034  0.90650576
 -0.38833971 -0.73531691 -0.11504455 -0.06745449 -0.15492913 -0.20330487
  0.45003798 -0.34120972 -0.7573848 ]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.4772729027548042, 'kernel_inv_bw2': 0.7926641719348785, 'kernel_inv_bw3': 5.040873509127367, 'kernel_inv_bw4': 0.4864132209497254, 'kernel_inv_bw5': 0.7698955526075097, 'kernel_inv_bw6': 0.8901911588711701, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9006913454112195, 'kernel_covariance_scale': 1.0845781061187012, 'mean_mean_value': 1.5750446495137802}

Epoch 1 ; Time: 0.509809 ; Training: accuracy=0.540462 ; Validation: accuracy=0.745378

Update for config_id 30:1: reward = 0.7453781512605042, crit_val = 0.2546218487394958
config_id 30: Reaches 1, continues to 3

Epoch 2 ; Time: 0.934910 ; Training: accuracy=0.706936 ; Validation: accuracy=0.803025
Epoch 3 ; Time: 1.358011 ; Training: accuracy=0.752601 ; Validation: accuracy=0.842185

config_id 30: Terminating evaluation at 3
Update for config_id 30:3: reward = 0.8421848739495799, crit_val = 0.15781512605042014
Starting get_config[BO] for config_id 31
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3071597473180353
- self.std = 0.24292792468190896
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 53
Current best is [0.06466667]
[31: BO] (28 evaluations)
batch_size: 43
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.003057861367333125
n_units_1: 128
n_units_2: 92
scale_1: 1.2045672735960884
scale_2: 0.03066580760656
Started BO from (top scorer):
batch_size: 43
dropout_1: 0.6582795958638858
dropout_2: 0.4639146886680176
learning_rate: 0.00823844766070984
n_units_1: 121
n_units_2: 74
scale_1: 1.2042649225423718
scale_2: 0.020779856609750333
Top score values: [0.0052399  0.01694925 0.02123839 0.03249024 0.07142258]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3. Pending:
Targets: [ 0.66517776 -0.00759722 -0.30133435  2.47844223  1.33031406  2.71070903
  2.31353573  0.73495099  0.20031147 -0.16841647 -0.3861132  -0.50963392
  2.69544016 -0.42277956 -0.68826851 -0.84838298  0.92209033 -0.58109879
 -0.75676707 -0.90200463 -0.29566965 -0.74847885 -0.99820999 -0.30640487
 -0.71825622 -0.88285901 -0.05660354 -0.35368848 -0.79976868 -0.79908135
 -0.49052029 -0.83675074 -0.94691497 -0.38852301 -0.77780474 -0.99039122
  0.09297549  2.13553511  0.00711181 -0.23803209  2.00966256  0.93657568
 -0.37794595 -0.73019574 -0.10049786 -0.05218463 -0.14098852 -0.19009936
  0.47317153 -0.33009978 -0.75259896 -0.21626949 -0.61476926]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0008014496833669769, 'kernel_inv_bw1': 0.17868168288748057, 'kernel_inv_bw2': 0.22283389263016143, 'kernel_inv_bw3': 5.488584767342549, 'kernel_inv_bw4': 1.1209009450195109, 'kernel_inv_bw5': 1.0100879793676378, 'kernel_inv_bw6': 0.0005979962870949027, 'kernel_inv_bw7': 0.5120192852396444, 'kernel_inv_bw8': 0.8515727779691762, 'kernel_covariance_scale': 1.1708921719564838, 'mean_mean_value': 1.6315836590273203}

Epoch 1 ; Time: 0.687358 ; Training: accuracy=0.691716 ; Validation: accuracy=0.800402

Update for config_id 31:1: reward = 0.8004015392337293, crit_val = 0.19959846076627075
config_id 31: Reaches 1, continues to 3

Epoch 2 ; Time: 1.427055 ; Training: accuracy=0.845568 ; Validation: accuracy=0.866656
Epoch 3 ; Time: 2.101790 ; Training: accuracy=0.885376 ; Validation: accuracy=0.892923

Update for config_id 31:3: reward = 0.8929228710055211, crit_val = 0.10707712899447885
config_id 31: Reaches 3, continues to 9

Epoch 4 ; Time: 2.788547 ; Training: accuracy=0.911694 ; Validation: accuracy=0.914673
Epoch 5 ; Time: 3.472511 ; Training: accuracy=0.929405 ; Validation: accuracy=0.905638
Epoch 6 ; Time: 4.160065 ; Training: accuracy=0.940329 ; Validation: accuracy=0.917183
Epoch 7 ; Time: 4.986891 ; Training: accuracy=0.947695 ; Validation: accuracy=0.931906
Epoch 8 ; Time: 5.803370 ; Training: accuracy=0.951833 ; Validation: accuracy=0.928392
Epoch 9 ; Time: 6.543173 ; Training: accuracy=0.959199 ; Validation: accuracy=0.934081

config_id 31: Terminating evaluation at 9
Update for config_id 31:9: reward = 0.934080642462774, crit_val = 0.065919357537226
Starting get_config[BO] for config_id 32
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.29735824205631867
- self.std = 0.24024393396396798
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 56
Current best is [0.06466667]
[32: BO] (20 evaluations)
batch_size: 67
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.005029844658978558
n_units_1: 97
n_units_2: 101
scale_1: 0.4282244661770721
scale_2: 0.04365398118444292
Started BO from (top scorer):
batch_size: 67
dropout_1: 0.1734863815084967
dropout_2: 0.05448423762046489
learning_rate: 0.0027946254009417865
n_units_1: 98
n_units_2: 113
scale_1: 0.4282230841629656
scale_2: 0.677369051872654
Top score values: [0.02265813 0.04217471 0.04999923 0.05146959 0.08312105]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9. Pending:
Targets: [ 0.71340722  0.03311605 -0.2639027   2.54692938  1.38597439  2.78179105
  2.38018055  0.78395995  0.24334747 -0.12949987 -0.34962869 -0.47452938
  2.7663516  -0.38670469 -0.65515967 -0.81706293  0.97319    -0.54679265
 -0.72442349 -0.87128363 -0.25817471 -0.71604267 -0.96856379 -0.26902987
 -0.68548239 -0.85192412 -0.01643777 -0.31684173 -0.76790551 -0.7672105
 -0.45520222 -0.80530073 -0.91669571 -0.35206542 -0.74569619 -0.96065767
  0.13481235  2.20019132  0.0479894  -0.19989323  2.07291253  0.98783719
 -0.3413702  -0.6975553  -0.06082248 -0.0119695  -0.1017655  -0.151425
  0.51925591 -0.2929895  -0.72020881 -0.1778875  -0.58083929 -0.40691883
 -0.79203296 -0.96334955]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00020134389738157523, 'kernel_inv_bw1': 0.18494793920169467, 'kernel_inv_bw2': 0.20357012643357442, 'kernel_inv_bw3': 5.170417209501338, 'kernel_inv_bw4': 1.1096493201061086, 'kernel_inv_bw5': 0.9933926176588416, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.5008653892049818, 'kernel_inv_bw8': 0.8322989515866606, 'kernel_covariance_scale': 1.2250883952275229, 'mean_mean_value': 1.6802901279825868}

Epoch 1 ; Time: 0.465850 ; Training: accuracy=0.657960 ; Validation: accuracy=0.791548

Update for config_id 32:1: reward = 0.7915478785846051, crit_val = 0.20845212141539493
config_id 32: Reaches 1, continues to 3

Epoch 2 ; Time: 0.884421 ; Training: accuracy=0.842206 ; Validation: accuracy=0.863827
Epoch 3 ; Time: 1.309256 ; Training: accuracy=0.888308 ; Validation: accuracy=0.887640

Update for config_id 32:3: reward = 0.8876404494382022, crit_val = 0.1123595505617978
config_id 32: Reaches 3, continues to 9

Epoch 4 ; Time: 1.732339 ; Training: accuracy=0.912106 ; Validation: accuracy=0.905752
Epoch 5 ; Time: 2.167060 ; Training: accuracy=0.925539 ; Validation: accuracy=0.912125
Epoch 6 ; Time: 2.606297 ; Training: accuracy=0.936982 ; Validation: accuracy=0.915982
Epoch 7 ; Time: 3.026454 ; Training: accuracy=0.943864 ; Validation: accuracy=0.928056
Epoch 8 ; Time: 3.443968 ; Training: accuracy=0.953317 ; Validation: accuracy=0.931913
Epoch 9 ; Time: 3.860855 ; Training: accuracy=0.957629 ; Validation: accuracy=0.931410

config_id 32: Terminating evaluation at 9
Update for config_id 32:9: reward = 0.9314103639107831, crit_val = 0.06858963608921687
Starting get_config[BO] for config_id 33
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.28883835361390264
- self.std = 0.23729911914726853
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 59
Current best is [0.06466667]
[33: BO] (23 evaluations)
batch_size: 49
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.006599144075261778
n_units_1: 128
n_units_2: 126
scale_1: 10.0
scale_2: 0.6131334928571974
Started BO from (top scorer):
batch_size: 49
dropout_1: 0.31412038526172237
dropout_2: 0.25623758001373487
learning_rate: 0.004673782052180023
n_units_1: 70
n_units_2: 64
scale_1: 0.4245429161746598
scale_2: 0.6131350287995055
Top score values: [0.02428879 0.02514878 0.04077546 0.04530218 0.05353108]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9. Pending:
Targets: [ 0.75816399  0.06943059 -0.23127408  2.61443963  1.43907752  2.85221587
  2.4456215   0.82959226  0.28227093 -0.09520334 -0.31806391 -0.44451457
  2.83658482 -0.3556     -0.62738643 -0.79129887  1.0211706  -0.51767461
 -0.6975098  -0.84619243 -0.22547501 -0.68902498 -0.94467981 -0.23646487
 -0.65808545 -0.82659268  0.01926182 -0.28487007 -0.74153142 -0.74082778
 -0.42494757 -0.7793907  -0.89216806 -0.32053087 -0.71904649 -0.93667558
  0.17238891  2.26339865  0.08448852 -0.16647026  2.13454037  1.03599955
 -0.30970293 -0.67030818 -0.02567369  0.02378555 -0.06712479 -0.11740056
  0.56160332 -0.26072183 -0.69324282 -0.14419145 -0.55214376 -0.376065
 -0.76595828 -0.93940086 -0.33875487 -0.74369767 -0.92814806]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.15376987171783166, 'kernel_inv_bw2': 0.8003658086258035, 'kernel_inv_bw3': 4.661541391068402, 'kernel_inv_bw4': 0.820536287484395, 'kernel_inv_bw5': 0.7192071222695686, 'kernel_inv_bw6': 0.5184585130388424, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.8193951701553809, 'kernel_covariance_scale': 1.463581095872539, 'mean_mean_value': 1.5846775355833405}

Epoch 1 ; Time: 0.629033 ; Training: accuracy=0.670412 ; Validation: accuracy=0.773670

Update for config_id 33:1: reward = 0.7736701237872198, crit_val = 0.22632987621278022
config_id 33: Reaches 1, continues to 3

Epoch 2 ; Time: 1.192476 ; Training: accuracy=0.804759 ; Validation: accuracy=0.792071
Epoch 3 ; Time: 1.743613 ; Training: accuracy=0.829464 ; Validation: accuracy=0.811308

config_id 33: Terminating evaluation at 3
Update for config_id 33:3: reward = 0.8113081298093008, crit_val = 0.18869187019069922
Starting get_config[BO] for config_id 34
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.28617187884629075
- self.std = 0.23385032194414795
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 61
Current best is [0.06466667]
[34: BO] (22 evaluations)
batch_size: 31
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.010848410822500352
n_units_1: 70
n_units_2: 109
scale_1: 0.05201511999198041
scale_2: 0.0015266972045049883
Started BO from (top scorer):
batch_size: 31
dropout_1: 0.2175351876089488
dropout_2: 0.02805791405897856
learning_rate: 0.008679743599523712
n_units_1: 66
n_units_2: 110
scale_1: 0.010279419696210736
scale_2: 0.00152670465144634
Top score values: [0.00367144 0.00807341 0.02277784 0.02796257 0.03939434]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3. Pending:
Targets: [ 0.78074779  0.08185703 -0.22328239  2.66439957  1.47170335  2.9056825
  2.49309172  0.85322947  0.29783631 -0.08520491 -0.31135219 -0.43966773
  2.88982093 -0.34944186 -0.62523657 -0.79156636  1.04763318 -0.51390673
 -0.69639411 -0.8472695  -0.2173978  -0.68778415 -0.94720935 -0.22854974
 -0.65638834 -0.82738068  0.03094838 -0.27766881 -0.74106495 -0.74035093
 -0.41981216 -0.77948258 -0.89392316 -0.31385554 -0.71824842 -0.93908708
  0.18633377  2.30818148  0.09713703 -0.15752286  2.17742281  1.06268084
 -0.3028679  -0.66879132 -0.01464983  0.03553882 -0.05671226 -0.10772949
  0.58128826 -0.25316444 -0.6920642  -0.13491549 -0.54888423 -0.37020868
 -0.76585206 -0.94185255 -0.3323483  -0.74326316 -0.9304338  -0.25589874
 -0.41684787]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.18190102112267384, 'kernel_inv_bw2': 0.7123611616932807, 'kernel_inv_bw3': 5.197488621083075, 'kernel_inv_bw4': 1.0067438436023535, 'kernel_inv_bw5': 0.7926447175972984, 'kernel_inv_bw6': 0.5969046798403365, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.8499041380800628, 'kernel_covariance_scale': 1.2637911058686522, 'mean_mean_value': 1.685471460467687}

Epoch 1 ; Time: 0.923618 ; Training: accuracy=0.660711 ; Validation: accuracy=0.799899

Update for config_id 34:1: reward = 0.7998991935483871, crit_val = 0.20010080645161288
config_id 34: Reaches 1, continues to 3

Epoch 2 ; Time: 1.836628 ; Training: accuracy=0.835649 ; Validation: accuracy=0.856183
Epoch 3 ; Time: 2.797171 ; Training: accuracy=0.871216 ; Validation: accuracy=0.873656

config_id 34: Terminating evaluation at 3
Update for config_id 34:3: reward = 0.8736559139784946, crit_val = 0.12634408602150538
Starting get_config[BO] for config_id 35
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.2822687222555056
- self.std = 0.23120930284746374
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 63
Current best is [0.06466667]
[35: BO] (26 evaluations)
batch_size: 121
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.004709749912666916
n_units_1: 128
n_units_2: 84
scale_1: 1.785630978661036
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 121
dropout_1: 0.08199638119198988
dropout_2: 0.12202712352247957
learning_rate: 0.0030296314321197958
n_units_1: 60
n_units_2: 56
scale_1: 1.7851756051748051
scale_2: 0.0018481959606542833
Top score values: [0.00223451 0.02319435 0.0250267  0.05159533 0.05400897]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3. Pending:
Targets: [ 8.06547468e-01  9.96735364e-02 -2.08951380e-01  2.71171551e+00
  1.50539556e+00  2.95575453e+00  2.53845088e+00  8.79857085e-01
  3.18119871e-01 -6.92966836e-02 -2.98027166e-01 -4.27808412e-01
  2.93971178e+00 -3.36551921e-01 -6.15496929e-01 -7.83726650e-01
  1.07648140e+00 -5.02895416e-01 -6.87467277e-01 -8.40066059e-01
 -2.02999569e-01 -6.78758973e-01 -9.41147492e-01 -2.14278898e-01
 -6.47004533e-01 -8.19950063e-01  4.81833756e-02 -2.63959034e-01
 -7.32648378e-01 -7.31926206e-01 -4.07726034e-01 -7.71504833e-01
 -8.87252633e-01 -3.00559110e-01 -7.09571218e-01 -9.32932441e-01
  2.05343676e-01  2.35142847e+00  1.15128074e-01 -1.42440704e-01
  2.21917620e+00  1.09170094e+00 -2.89445968e-01 -6.59549196e-01
  2.06431306e-03  5.28262512e-02 -4.04785757e-02 -9.20785566e-02
  6.04809589e-01 -2.39174762e-01 -6.83087908e-01 -1.19575091e-01
 -5.38272443e-01 -3.57555948e-01 -7.57718617e-01 -9.35729497e-01
 -3.19263109e-01 -7.34871692e-01 -9.24180314e-01 -2.41940291e-01
 -4.04727885e-01 -3.55383260e-01 -6.74387381e-01]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00025014083267642167, 'kernel_inv_bw1': 0.28919596584551394, 'kernel_inv_bw2': 0.22420097977804268, 'kernel_inv_bw3': 4.672336346114276, 'kernel_inv_bw4': 1.037841560138486, 'kernel_inv_bw5': 1.1382833259248413, 'kernel_inv_bw6': 0.0006661490330757411, 'kernel_inv_bw7': 0.5873591706441644, 'kernel_inv_bw8': 0.833929744581794, 'kernel_covariance_scale': 1.2661342913985998, 'mean_mean_value': 1.7163904325693817}

Epoch 1 ; Time: 0.297275 ; Training: accuracy=0.665041 ; Validation: accuracy=0.805455

Update for config_id 35:1: reward = 0.8054545454545454, crit_val = 0.19454545454545458
config_id 35: Reaches 1, continues to 3

Epoch 2 ; Time: 0.548198 ; Training: accuracy=0.833884 ; Validation: accuracy=0.842645
Epoch 3 ; Time: 0.778134 ; Training: accuracy=0.877686 ; Validation: accuracy=0.880661

Update for config_id 35:3: reward = 0.8806611570247934, crit_val = 0.11933884297520658
config_id 35: Reaches 3, continues to 9

Epoch 4 ; Time: 1.011846 ; Training: accuracy=0.905372 ; Validation: accuracy=0.908595
Epoch 5 ; Time: 1.251584 ; Training: accuracy=0.929669 ; Validation: accuracy=0.915207
Epoch 6 ; Time: 1.610493 ; Training: accuracy=0.935289 ; Validation: accuracy=0.917686
Epoch 7 ; Time: 1.844067 ; Training: accuracy=0.940331 ; Validation: accuracy=0.915372
Epoch 8 ; Time: 2.110958 ; Training: accuracy=0.951240 ; Validation: accuracy=0.914876
Epoch 9 ; Time: 2.357718 ; Training: accuracy=0.955620 ; Validation: accuracy=0.928595

config_id 35: Terminating evaluation at 9
Update for config_id 35:9: reward = 0.9285950413223141, crit_val = 0.07140495867768593
Starting get_config[BO] for config_id 36
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.2752760417923516
- self.std = 0.2284106990916211
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 66
Current best is [0.06466667]
[36: BO] (42 evaluations)
batch_size: 13
dropout_1: 0.0
dropout_2: 0.3229218378252589
learning_rate: 0.003350210782489933
n_units_1: 128
n_units_2: 110
scale_1: 0.017430610159085962
scale_2: 0.003138239662520903
Started BO from (top scorer):
batch_size: 13
dropout_1: 0.26884076770212595
dropout_2: 0.3645837923519833
learning_rate: 0.2588421122908576
n_units_1: 61
n_units_2: 42
scale_1: 0.0011346200476860303
scale_2: 0.003138245794989121
Top score values: [-0.0624444   0.00170651  0.0057119   0.04304142  0.08263659]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9. Pending:
Targets: [ 0.8470442   0.13150929 -0.18089705  2.77555533  1.55445494  3.02258444
  2.60016777  0.92125204  0.35263214 -0.03953124 -0.27106424 -0.40243563
  3.00634512 -0.31006102 -0.5924238  -0.76271476  1.1202855  -0.47844264
 -0.66527597 -0.81974447 -0.17487232 -0.65646097 -0.9220644  -0.18628985
 -0.62431746 -0.799382    0.07938825 -0.23657869 -0.71101065 -0.71027963
 -0.3821072  -0.7503432  -0.8675092  -0.27362721 -0.68765074 -0.9137487
  0.23847416  2.41085387  0.14715319 -0.11357145  2.27698117  1.13569151
 -0.2623779  -0.63701582  0.03270411  0.08408801 -0.01036003 -0.06259224
  0.64283453 -0.21149075 -0.66084294 -0.09042568 -0.51425312 -0.3313224
 -0.73638807 -0.91658002 -0.29256038 -0.71326121 -0.90488934 -0.21429016
 -0.37907231 -0.32912309 -0.65203581 -0.35344486 -0.68270532 -0.89256363]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0001765975485634436, 'kernel_inv_bw1': 0.20265830659692838, 'kernel_inv_bw2': 0.7421344493969121, 'kernel_inv_bw3': 4.575089827805363, 'kernel_inv_bw4': 0.8444089666067263, 'kernel_inv_bw5': 0.7323448181989877, 'kernel_inv_bw6': 0.9186947228696695, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.8135774028321318, 'kernel_covariance_scale': 1.3221065106332446, 'mean_mean_value': 1.6866603747506494}

Epoch 1 ; Time: 2.381794 ; Training: accuracy=0.572696 ; Validation: accuracy=0.767548

config_id 36: Terminating evaluation at 1
Update for config_id 36:1: reward = 0.7675475509173539, crit_val = 0.23245244908264606
Starting get_config[BO] for config_id 37
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.27463688369220673
- self.std = 0.22675919311532314
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 67
Current best is [0.06466667]
[37: BO] (38 evaluations)
batch_size: 84
dropout_1: 0.0
dropout_2: 0.2922265871470264
learning_rate: 0.003341868460106545
n_units_1: 128
n_units_2: 125
scale_1: 0.0029292853735109286
scale_2: 0.06986498582894866
Started BO from (top scorer):
batch_size: 84
dropout_1: 0.6165336826147805
dropout_2: 0.716672390336065
learning_rate: 0.04661339810236861
n_units_1: 43
n_units_2: 121
scale_1: 0.0032860110061669885
scale_2: 0.06986493574617235
Top score values: [0.01804793 0.04567762 0.05125348 0.05544129 0.07355351]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9, 36:1. Pending:
Targets: [ 0.85603196  0.13528575 -0.17939588  2.7985886   1.56859483  3.04741684
  2.62192368  0.93078026  0.35801906 -0.03700048 -0.27021976 -0.40254794
  3.03105925 -0.30950056 -0.59391981 -0.76545101  1.1312633  -0.47910851
 -0.66730257 -0.82289607 -0.17332726 -0.65842336 -0.92596121 -0.18482795
 -0.62604575 -0.8023853   0.0827851  -0.23548305 -0.71337034 -0.71263399
 -0.38207145 -0.75298934 -0.87100867 -0.27280139 -0.68984029 -0.91758494
  0.24302965  2.43123099  0.15104358 -0.11157994  2.29638328  1.14678151
 -0.26147016 -0.6388366   0.03576096  0.08751909 -0.00761682 -0.06022944
  0.65033501 -0.21021239 -0.66283725 -0.08826559 -0.51517981 -0.33091678
 -0.73893258 -0.92043689 -0.29187245 -0.71563728 -0.90866105 -0.21303219
 -0.37901446 -0.32870146 -0.65396598 -0.35320036 -0.68485885 -0.89624558
 -0.18603186]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.003285371084878471, 'kernel_inv_bw1': 0.3304262058754804, 'kernel_inv_bw2': 0.6352604558931586, 'kernel_inv_bw3': 4.346194226510676, 'kernel_inv_bw4': 0.7173119046553594, 'kernel_inv_bw5': 0.7149273369845481, 'kernel_inv_bw6': 1.01614715613028, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.8127427752719871, 'kernel_covariance_scale': 1.3645023654628092, 'mean_mean_value': 1.6754358600797579}

Epoch 1 ; Time: 0.397591 ; Training: accuracy=0.499835 ; Validation: accuracy=0.712106

config_id 37: Terminating evaluation at 1
Update for config_id 37:1: reward = 0.7121059691482227, crit_val = 0.28789403085177734
Starting get_config[BO] for config_id 38
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.27483184173867103
- self.std = 0.22509132750228003
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 68
Current best is [0.06466667]
[38: BO] (22 evaluations)
batch_size: 44
dropout_1: 0.75
dropout_2: 0.0
learning_rate: 0.0063692003585150226
n_units_1: 62
n_units_2: 89
scale_1: 4.453876363616511
scale_2: 0.08121415983173202
Started BO from (top scorer):
batch_size: 44
dropout_1: 0.6643415461271273
dropout_2: 0.1987793444951125
learning_rate: 0.016764470238010157
n_units_1: 77
n_units_2: 124
scale_1: 4.45385772810035
scale_2: 0.033619294881536775
Top score values: [0.03750278 0.06083585 0.06409336 0.08235989 0.09293021]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9, 36:1, 37:1. Pending:
Targets: [ 0.86150879  0.13542205 -0.18159128  2.81845925  1.57935156  3.06913124
  2.64048529  0.93681096  0.35980575 -0.03814077 -0.27308814 -0.40639684
  3.05265244 -0.31266    -0.59918673 -0.77198893  1.13877952 -0.48352471
 -0.67311322 -0.82985964 -0.1754777  -0.66416823 -0.93368846 -0.1870636
 -0.63155071 -0.80919689  0.08253239 -0.23809404 -0.71952235 -0.71878055
 -0.38576862 -0.75943492 -0.87832874 -0.27568891 -0.69581795 -0.92525012
  0.2439643   2.44837962  0.15129664 -0.11327285  2.31253273  1.15441273
 -0.26427371 -0.64443633  0.03515981  0.08730146 -0.00853939 -0.06154186
  0.65428769 -0.21263613 -0.66861482 -0.08978575 -0.51986328 -0.33423492
 -0.74527399 -0.9281232  -0.29490128 -0.72180609 -0.91626012 -0.21547683
 -0.38268898 -0.33200317 -0.65967782 -0.35668361 -0.6907996  -0.90375265
 -0.18827643  0.05803062]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.3137479251964829, 'kernel_inv_bw2': 0.21388040891089755, 'kernel_inv_bw3': 4.64126465396062, 'kernel_inv_bw4': 1.0288777129326367, 'kernel_inv_bw5': 1.1441479346655605, 'kernel_inv_bw6': 0.00011695284618913166, 'kernel_inv_bw7': 0.5685924461821816, 'kernel_inv_bw8': 0.8436950628722043, 'kernel_covariance_scale': 1.3190266323792321, 'mean_mean_value': 1.762606161449074}

Epoch 1 ; Time: 0.701976 ; Training: accuracy=0.237438 ; Validation: accuracy=0.469024

config_id 38: Terminating evaluation at 1
Update for config_id 38:1: reward = 0.469023569023569, crit_val = 0.530976430976431
Starting get_config[BO] for config_id 39
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.27854408216240667
- self.std = 0.225541357402729
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 69
Current best is [0.06466667]
[39: BO] (28 evaluations)
batch_size: 8
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.005187968909871167
n_units_1: 63
n_units_2: 117
scale_1: 0.0035297615259256587
scale_2: 0.002209076216210477
Started BO from (top scorer):
batch_size: 100
dropout_1: 0.044803819744552636
dropout_2: 0.050523243428423986
learning_rate: 0.0036037353512263152
n_units_1: 107
n_units_2: 61
scale_1: 0.019097693338704407
scale_2: 0.0022102786701961525
Top score values: [0.02971497 0.06644624 0.07417161 0.08208605 0.1004852 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9, 36:1, 37:1, 38:1. Pending:
Targets: [ 0.84333055  0.11869259 -0.19768819  2.79637624  1.55974098  3.04654806
  2.6187574   0.91848247  0.34262857 -0.05452392 -0.28900249 -0.42204519
  3.03010214 -0.32849538 -0.61445039 -0.7869078   1.12004804 -0.49901916
 -0.68822938 -0.84466304 -0.19158681 -0.67930224 -0.94828469 -0.20314959
 -0.6467498  -0.82404152  0.06590847 -0.25407821 -0.73454591 -0.73380559
 -0.40145813 -0.77437884 -0.89303543 -0.29159806 -0.71088881 -0.93986319
  0.22701827  2.42703504  0.13453551 -0.12950607  2.29145921  1.13565004
 -0.28020564 -0.65960971  0.01863041  0.07066802 -0.0249816  -0.07787831
  0.63652292 -0.2286711  -0.68373996 -0.10606584 -0.53528522 -0.35002725
 -0.76024617 -0.94273054 -0.3107721  -0.73682509 -0.93089112 -0.23150613
 -0.39838464 -0.34779996 -0.67482079 -0.37243115 -0.70588047 -0.91840861
 -0.20436001  0.04145558  1.11922865]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.10767130316015067, 'kernel_inv_bw1': 0.5735987472433268, 'kernel_inv_bw2': 0.4501678262011913, 'kernel_inv_bw3': 3.847041571312879, 'kernel_inv_bw4': 0.43384795376286595, 'kernel_inv_bw5': 0.726479391236762, 'kernel_inv_bw6': 1.5954055912198324, 'kernel_inv_bw7': 0.0014678419708413885, 'kernel_inv_bw8': 0.7949160188252204, 'kernel_covariance_scale': 1.367033050795075, 'mean_mean_value': 1.8026277861637732}

Epoch 1 ; Time: 3.842823 ; Training: accuracy=0.657410 ; Validation: accuracy=0.807369

Update for config_id 39:1: reward = 0.8073687752355316, crit_val = 0.19263122476446837
config_id 39: Reaches 1, continues to 3

Epoch 2 ; Time: 7.860336 ; Training: accuracy=0.830902 ; Validation: accuracy=0.867429
Epoch 3 ; Time: 11.884933 ; Training: accuracy=0.874917 ; Validation: accuracy=0.875168

config_id 39: Terminating evaluation at 3
Update for config_id 39:3: reward = 0.8751682368775235, crit_val = 0.12483176312247646
Starting get_config[BO] for config_id 40
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.27516907967736626
- self.std = 0.22329649743884794
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 71
Current best is [0.06466667]
[40: BO] (26 evaluations)
batch_size: 8
dropout_1: 0.19742704332913752
dropout_2: 0.3252681176512454
learning_rate: 0.005219528352860058
n_units_1: 128
n_units_2: 108
scale_1: 0.7796167062520984
scale_2: 0.29756654929393117
Started BO from (top scorer):
batch_size: 31
dropout_1: 0.4878886531255304
dropout_2: 0.18114926469664466
learning_rate: 0.0016224230284530277
n_units_1: 92
n_units_2: 116
scale_1: 0.060583648432504245
scale_2: 0.29827246813684094
Top score values: [-0.02123424  0.02655392  0.03792998  0.04227423  0.0445185 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9, 36:1, 37:1, 38:1, 39:1, 39:3. Pending:
Targets: [ 0.86692323  0.13500029 -0.18456116  2.83960341  1.59053593  3.09229027
  2.66019892  0.94283067  0.36118756 -0.03995761 -0.27679346 -0.41117368
  3.07567902 -0.31668339 -0.60551318 -0.77970435  1.14642262 -0.48892149
 -0.6800339  -0.83804022 -0.17839844 -0.671017   -0.9427036  -0.19007746
 -0.6381373  -0.81721138  0.08168551 -0.24151808 -0.72681605 -0.72606829
 -0.39037965 -0.76704943 -0.88689891 -0.27941513 -0.70292112 -0.93419744
  0.24441499  2.46654913  0.15100248 -0.11569359  2.32961031  1.16218149
 -0.26790818 -0.6511265   0.03393215  0.08649291 -0.0101183  -0.06354679
  0.6580365  -0.21585555 -0.67549934 -0.0920177  -0.52555215 -0.33843173
 -0.75277469 -0.93709362 -0.29878193 -0.72911815 -0.92513517 -0.21871908
 -0.38727526 -0.33618205 -0.6664905  -0.36106086 -0.69786243 -0.91252717
 -0.19130005  0.05698679  1.145595   -0.36963345 -0.67326321]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.10341476367436149, 'kernel_inv_bw1': 0.5983479644078403, 'kernel_inv_bw2': 0.4179785036924045, 'kernel_inv_bw3': 3.91803420329974, 'kernel_inv_bw4': 0.4149104350412135, 'kernel_inv_bw5': 0.7649108916443379, 'kernel_inv_bw6': 1.610976592213731, 'kernel_inv_bw7': 0.002604672903894858, 'kernel_inv_bw8': 0.800340076458261, 'kernel_covariance_scale': 1.363105011961438, 'mean_mean_value': 1.8215236543541617}

Epoch 1 ; Time: 3.738302 ; Training: accuracy=0.602371 ; Validation: accuracy=0.794583

Update for config_id 40:1: reward = 0.7945827725437415, crit_val = 0.20541722745625846
config_id 40: Reaches 1, continues to 3

Epoch 2 ; Time: 7.337619 ; Training: accuracy=0.734251 ; Validation: accuracy=0.816117
Epoch 3 ; Time: 10.956102 ; Training: accuracy=0.763843 ; Validation: accuracy=0.851279

config_id 40: Terminating evaluation at 3
Update for config_id 40:3: reward = 0.851278600269179, crit_val = 0.148721399730821

Analysing the results

The training history is stored in the results_df, the main fields are the runtime and 'best' (the objective).

Note: You will get slightly different curves for different pairs of scheduler/searcher, the time_out here is a bit too short to really see the difference in a significant way (it would be better to set it to >1000s). Generally speaking though, hyperband stopping / promotion + model will tend to significantly outperform other combinations given enough time.

results_df.head()

/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
  and should_run_async(code)

	bracket	elapsed_time	epoch	error	eval_time	objective	runtime	searcher_data_size	searcher_params_kernel_covariance_scale	searcher_params_kernel_inv_bw0	...	searcher_params_kernel_inv_bw7	searcher_params_kernel_inv_bw8	searcher_params_mean_mean_value	searcher_params_noise_variance	target_epoch	task_id	time_since_start	time_step	time_this_iter	best
0	0	0.484257	1	0.468750	0.479386	0.531250	0.571462	NaN	1.0	1.0	...	1.0	1.0	0.0	0.001	9	0	0.573196	1.603831e+09	0.513793	0.468750
1	0	0.925274	2	0.344753	0.434060	0.655247	1.012479	1.0	1.0	1.0	...	1.0	1.0	0.0	0.001	9	0	1.017026	1.603831e+09	0.441000	0.344753
2	0	1.347351	3	0.305314	0.416302	0.694686	1.434556	1.0	1.0	1.0	...	1.0	1.0	0.0	0.001	9	0	1.435394	1.603831e+09	0.422076	0.305314
3	0	1.773053	4	0.288937	0.421576	0.711063	1.860258	2.0	1.0	1.0	...	1.0	1.0	0.0	0.001	9	0	1.861271	1.603831e+09	0.425702	0.288937
4	0	2.192330	5	0.273061	0.417183	0.726939	2.279535	2.0	1.0	1.0	...	1.0	1.0	0.0	0.001	9	0	2.280613	1.603832e+09	0.419278	0.273061

5 rows × 26 columns

import matplotlib.pyplot as plt

plt.figure(figsize=(12, 8))

runtime = results_df['runtime'].values
objective = results_df['best'].values

plt.plot(runtime, objective, lw=2)
plt.xticks(fontsize=12)
plt.xlim(0, 120)
plt.ylim(0, 0.5)
plt.yticks(fontsize=12)
plt.xlabel("Runtime [s]", fontsize=14)
plt.ylabel("Objective", fontsize=14)

/var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
  and should_run_async(code)

Text(0, 0.5, 'Objective')

Diving Deeper

Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at MNIST Training in PyTorch). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful.

Logging the Search Progress

First, it is a good idea in general to switch on debug_log, which outputs useful information about the search progress. This is already done in the example above.

The outputs show which configurations are chosen, stopped, or promoted. For BO and BOHB, a range of information is displayed for every get_config decision. This log output is very useful in order to figure out what is going on during the search.

Configuring HyperbandScheduler

The most important knobs to turn with HyperbandScheduler are max_t, grace_period, reduction_factor, brackets, and type. The first three determine the rung levels at which stopping or promotion decisions are being made.

• The maximum resource level max_t (usually, resource equates to epochs, so max_t is the maximum number of training epochs) is typically hardcoded in train_fn passed to the scheduler (this is run_mlp_openml in the example above). As already noted above, the value is best fixed in the ag.args decorator as epochs=XYZ, it can then be accessed as args.epochs in the train_fn code. If this is done, you do not have to pass max_t when creating the scheduler.

• grace_period and reduction_factor determine the rung levels, which are grace_period, grace_period * reduction_factor, grace_period * (reduction_factor ** 2), etc. All rung levels must be less or equal than max_t. It is recommended to make max_t equal to the largest rung level. For example, if grace_period = 1, reduction_factor = 3, it is in general recommended to use max_t = 9, max_t = 27, or max_t = 81. Choosing a max_t value “off the grid” works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you set max_t = 10, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch.

• With reduction_factor, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used.

• Finally, grace_period should be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. While grace_period = 1 should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung.

• brackets sets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). For brackets = 1, you are running successive halving (single bracket). Higher brackets have larger effective grace_period values (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (brackets = 1) in a comparison.

• Finally, with type (values stopping, promotion) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the default stopping is simpler and seems to perform well, but promotion is more careful promoting configurations to higher resource levels, which can work better in some cases.

Asynchronous BOHB

Finally, here are some ideas for tuning asynchronous BOHB, apart from tuning its HyperbandScheduling component. You need to pass these options in search_options.

• We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by gp_resource_kernel, values are matern52, matern52-res-warp, exp-decay-sum, exp-decay-combined, exp-decay-delta1. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found here.

• Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper train_fn and find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the options opt_skip_period and opt_skip_num_max_resource. The basic idea is as follows. By far the most expensive part of a get_config call (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for most get_config calls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers – we would love to learn about your use case.