.. _sec_custom_advancedhpo: Getting started with Advanced HPO Algorithms ============================================ Loading libraries ----------------- .. code:: python # Basic utils for folder manipulations etc import time import multiprocessing # to count the number of CPUs available # External tools to load and process data import numpy as np import pandas as pd # MXNet (NeuralNets) import mxnet as mx from mxnet import gluon, autograd from mxnet.gluon import nn # AutoGluon and HPO tools import autogluon as ag from autogluon.utils import load_and_split_openml_data Check the version of MxNet, you should be fine with version >= 1.5 .. code:: python mx.__version__ .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) .. parsed-literal:: :class: output '1.7.0' You can also check the version of AutoGluon and the specific commit and check that it matches what you want. .. code:: python ag.__version__ .. parsed-literal:: :class: output '0.0.14b20201027' Hyperparameter Optimization of a 2-layer MLP -------------------------------------------- Setting up the context ~~~~~~~~~~~~~~~~~~~~~~ Here we declare a few "environment variables" setting the context for what we're doing .. code:: python OPENML_TASK_ID = 6 # describes the problem we will tackle RATIO_TRAIN_VALID = 0.33 # split of the training data used for validation RESOURCE_ATTR_NAME = 'epoch' # how do we measure resources (will become clearer further) REWARD_ATTR_NAME = 'objective' # how do we measure performance (will become clearer further) NUM_CPUS = multiprocessing.cpu_count() Preparing the data ~~~~~~~~~~~~~~~~~~ We will use a multi-way classification task from OpenML. Data preparation includes: - Missing values are imputed, using the 'mean' strategy of ``sklearn.impute.SimpleImputer`` - Split training set into training and validation - Standardize inputs to mean 0, variance 1 .. code:: python X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data( OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False) n_classes .. parsed-literal:: :class: output Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff... 100%|██████████| 704/704 [00:00<00:00, 54179.63KB/s] Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3... 100%|██████████| 2521/2521 [00:00<00:00, 45588.69KB/s] Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml... 3KB [00:00, 3857.42KB/s] Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml... 8KB [00:00, 10381.94KB/s] Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml... 15KB [00:00, 14169.95KB/s] Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff... 2998KB [00:00, 51813.32KB/s] Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3... 881KB [00:00, 56292.09KB/s] Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml... 3KB [00:00, 4228.13KB/s] pickle load data letter .. parsed-literal:: :class: output 26 The problem has 26 classes. Declaring a model specifying a hyperparameter space with AutoGluon ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Two layer MLP where we optimize over: - the number of units on the first layer - the number of units on the second layer - the dropout rate after each layer - the learning rate - the scaling - the ``@ag.args`` decorator allows us to specify the space we will optimize over, this matches the `ConfigSpace `__ syntax The body of the function ``run_mlp_openml`` is pretty simple: - it reads the hyperparameters given via the decorator - it defines a 2 layer MLP with dropout - it declares a trainer with the 'adam' loss function and a provided learning rate - it trains the NN with a number of epochs (most of that is boilerplate code from ``mxnet``) - the ``reporter`` at the end is used to keep track of training history in the hyperparameter optimization **Note**: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment .. code:: python @ag.args(n_units_1=ag.space.Int(lower=16, upper=128), n_units_2=ag.space.Int(lower=16, upper=128), dropout_1=ag.space.Real(lower=0, upper=.75), dropout_2=ag.space.Real(lower=0, upper=.75), learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True), batch_size=ag.space.Int(lower=8, upper=128), scale_1=ag.space.Real(lower=0.001, upper=10, log=True), scale_2=ag.space.Real(lower=0.001, upper=10, log=True), epochs=9) def run_mlp_openml(args, reporter, **kwargs): # Time stamp for elapsed_time ts_start = time.time() # Unwrap hyperparameters n_units_1 = args.n_units_1 n_units_2 = args.n_units_2 dropout_1 = args.dropout_1 dropout_2 = args.dropout_2 scale_1 = args.scale_1 scale_2 = args.scale_2 batch_size = args.batch_size learning_rate = args.learning_rate ctx = mx.cpu() net = nn.Sequential() with net.name_scope(): # Layer 1 net.add(nn.Dense(n_units_1, activation='relu', weight_initializer=mx.initializer.Uniform(scale=scale_1))) # Dropout net.add(gluon.nn.Dropout(dropout_1)) # Layer 2 net.add(nn.Dense(n_units_2, activation='relu', weight_initializer=mx.initializer.Uniform(scale=scale_2))) # Dropout net.add(gluon.nn.Dropout(dropout_2)) # Output net.add(nn.Dense(n_classes)) net.initialize(ctx=ctx) trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate': learning_rate}) for epoch in range(args.epochs): ts_epoch = time.time() train_iter = mx.io.NDArrayIter( data={'data': X_train}, label={'label': y_train}, batch_size=batch_size, shuffle=True) valid_iter = mx.io.NDArrayIter( data={'data': X_valid}, label={'label': y_valid}, batch_size=batch_size, shuffle=False) metric = mx.metric.Accuracy() loss = gluon.loss.SoftmaxCrossEntropyLoss() for batch in train_iter: data = batch.data[0].as_in_context(ctx) label = batch.label[0].as_in_context(ctx) with autograd.record(): output = net(data) L = loss(output, label) L.backward() trainer.step(data.shape[0]) metric.update([label], [output]) name, train_acc = metric.get() metric = mx.metric.Accuracy() for batch in valid_iter: data = batch.data[0].as_in_context(ctx) label = batch.label[0].as_in_context(ctx) output = net(data) metric.update([label], [output]) name, val_acc = metric.get() print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % ( epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc)) ts_now = time.time() eval_time = ts_now - ts_epoch elapsed_time = ts_now - ts_start # The resource reported back (as 'epoch') is the number of epochs # done, starting at 1 reporter( epoch=epoch + 1, objective=float(val_acc), eval_time=eval_time, time_step=ts_now, elapsed_time=elapsed_time) .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) **Note**: The annotation ``epochs=9`` specifies the maximum number of epochs for training. It becomes available as ``args.epochs``. Importantly, it is also processed by ``HyperbandScheduler`` below in order to set its ``max_t`` attribute. **Recommendation**: Whenever writing training code to be passed as ``train_fn`` to a scheduler, if this training code reports a resource (or time) attribute, the corresponding maximum resource value should be included in ``train_fn.args``: - If the resource attribute (``time_attr`` of scheduler) in ``train_fn`` is ``epoch``, make sure to include ``epochs=XYZ`` in the annotation. This allows the scheduler to read ``max_t`` from ``train_fn.args.epochs``. This case corresponds to our example here. - If the resource attribute is something else than ``epoch``, you can also include the annotation ``max_t=XYZ``, which allows the scheduler to read ``max_t`` from ``train_fn.args.max_t``. Annotating the training function by the correct value for ``max_t`` simplifies scheduler creation (since ``max_t`` does not have to be passed), and avoids inconsistencies between ``train_fn`` and the scheduler. Running the Hyperparameter Optimization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can use the following schedulers: - FIFO (``fifo``) - Hyperband (either the stopping (``hbs``) or promotion (``hbp``) variant) And the following searchers: - Random search (``random``) - Gaussian process based Bayesian optimization (``bayesopt``) - SkOpt Bayesian optimization (``skopt``; only with FIFO scheduler) Note that the method known as (asynchronous) Hyperband is using random search. Combining Hyperband scheduling with the ``bayesopt`` searcher uses a novel method called asynchronous BOHB. Pick the combination you're interested in (doing the full experiment takes around 120 seconds, see the ``time_out`` parameter), running everything with multiple runs can take a fair bit of time. In real life, you will want to choose a larger ``time_out`` in order to obtain good performance. .. code:: python SCHEDULER = "hbs" SEARCHER = "bayesopt" .. code:: python def compute_error(df): return 1.0 - df["objective"] def compute_runtime(df, start_timestamp): return df["time_step"] - start_timestamp def process_training_history(task_dicts, start_timestamp, runtime_fn=compute_runtime, error_fn=compute_error): task_dfs = [] for task_id in task_dicts: task_df = pd.DataFrame(task_dicts[task_id]) task_df = task_df.assign(task_id=task_id, runtime=runtime_fn(task_df, start_timestamp), error=error_fn(task_df), target_epoch=task_df["epoch"].iloc[-1]) task_dfs.append(task_df) result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True) # re-order by runtime result = result.sort_values(by="runtime") # calculate incumbent best -- the cumulative minimum of the error. result = result.assign(best=result["error"].cummin()) return result resources = dict(num_cpus=NUM_CPUS, num_gpus=0) .. code:: python search_options = { 'num_init_random': 2, 'debug_log': True} if SCHEDULER == 'fifo': myscheduler = ag.scheduler.FIFOScheduler( run_mlp_openml, resource=resources, searcher=SEARCHER, search_options=search_options, time_out=120, time_attr=RESOURCE_ATTR_NAME, reward_attr=REWARD_ATTR_NAME) else: # This setup uses rung levels at 1, 3, 9 epochs. We just use a single # bracket, so this is in fact successive halving (Hyperband would use # more than 1 bracket). # Also note that since we do not use the max_t argument of # HyperbandScheduler, this value is obtained from train_fn.args.epochs. sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion' myscheduler = ag.scheduler.HyperbandScheduler( run_mlp_openml, resource=resources, searcher=SEARCHER, search_options=search_options, time_out=120, time_attr=RESOURCE_ATTR_NAME, reward_attr=REWARD_ATTR_NAME, type=sch_type, grace_period=1, reduction_factor=3, brackets=1) # run tasks myscheduler.run() myscheduler.join_jobs() results_df = process_training_history( myscheduler.training_history.copy(), start_timestamp=myscheduler._start_time) .. parsed-literal:: :class: output max_t = 9, as inferred from train_fn.args scheduler_options: Key 'resume': Imputing default value False scheduler_options: Key 'keep_size_ratios': Imputing default value False scheduler_options: Key 'maxt_pending': Imputing default value False scheduler_options: Key 'searcher_data': Imputing default value rungs scheduler_options: Key 'do_snapshots': Imputing default value False scheduler_options: Key 'visualizer': Imputing default value none scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60 scheduler_options: Key 'delay_get_config': Imputing default value True search_options: Key 'random_seed': Imputing default value 1822 search_options: Key 'opt_skip_init_length': Imputing default value 150 search_options: Key 'opt_skip_period': Imputing default value 1 search_options: Key 'profiler': Imputing default value False search_options: Key 'opt_maxiter': Imputing default value 50 search_options: Key 'opt_nstarts': Imputing default value 2 search_options: Key 'opt_warmstart': Imputing default value False search_options: Key 'opt_verbose': Imputing default value False search_options: Key 'opt_debug_writer': Imputing default value False search_options: Key 'num_fantasy_samples': Imputing default value 20 search_options: Key 'num_init_candidates': Imputing default value 250 search_options: Key 'initial_scoring': Imputing default value thompson_indep search_options: Key 'first_is_default': Imputing default value True search_options: Key 'opt_skip_num_max_resource': Imputing default value False search_options: Key 'gp_resource_kernel': Imputing default value matern52 search_options: Key 'resource_acq': Imputing default value bohb [GPMultiFidelitySearcher.__init__] - acquisition_class = - local_minimizer_class = - num_initial_candidates = 250 - num_initial_random_choices = 2 - initial_scoring = thompson_indep - first_is_default = True Starting Experiments Num of Finished Tasks is 0 Time out (secs) is 120 Starting get_config[random] for config_id 0 Start with default config: {'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1} [0: random] batch_size: 68 dropout_1: 0.375 dropout_2: 0.375 learning_rate: 0.001 n_units_1: 72 n_units_2: 72 scale_1: 0.1 scale_2: 0.1 /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/distributed/worker.py:3382: UserWarning: Large object of size 1.30 MB detected in task graph: (, {'ar ... sReporter}, []) Consider scattering large objects ahead of time with client.scatter to reduce scheduler burden and keep data on workers future = client.submit(func, big_data) # bad big_future = client.scatter(big_data) # good future = client.submit(func, big_future) # good % (format_bytes(len(b)), s) .. parsed-literal:: :class: output Epoch 1 ; Time: 0.481761 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250 .. parsed-literal:: :class: output Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875 config_id 0: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.923535 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247 Epoch 3 ; Time: 1.345758 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686 .. parsed-literal:: :class: output Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464 config_id 0: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.771330 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063 Epoch 5 ; Time: 2.190708 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939 Epoch 6 ; Time: 2.614471 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321 Epoch 7 ; Time: 3.063879 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501 Epoch 8 ; Time: 3.488436 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202 Epoch 9 ; Time: 3.925083 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043 .. parsed-literal:: :class: output config_id 0: Terminating evaluation at 9 Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695 Starting get_config[random] for config_id 1 [1: random] batch_size: 81 dropout_1: 0.20829386966719157 dropout_2: 0.22302955829164872 learning_rate: 3.4371538766396122e-06 n_units_1: 100 n_units_2: 123 scale_1: 0.0029958953582211044 scale_2: 0.1287354014226165 .. parsed-literal:: :class: output Epoch 1 ; Time: 0.417130 ; Training: accuracy=0.055266 ; Validation: accuracy=0.090757 .. parsed-literal:: :class: output config_id 1: Terminating evaluation at 1 Update for config_id 1:1: reward = 0.09075742409075742, crit_val = 0.9092425759092426 Starting get_config[BO] for config_id 2 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.47931599157089355 - self.std = 0.2624052537450702 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 4 Current best is [0.46875] [2: BO] (1 evaluations) batch_size: 123 dropout_1: 0.4640710798470552 dropout_2: 0.09873615428539195 learning_rate: 0.0005378457242779595 n_units_1: 32 n_units_2: 120 scale_1: 0.045864005010116914 scale_2: 2.1665539764398583 Started BO from (top scorer): batch_size: 123 dropout_1: 0.4640710798470552 dropout_2: 0.09873615428539195 learning_rate: 0.0005378457242779595 n_units_1: 32 n_units_2: 120 scale_1: 0.045864005010116914 scale_2: 2.1665539764398583 Top score values: [-0.25960743 0.01441937 0.02376365 0.07418799 0.09351457] Labeled: 0:1, 0:3, 0:9, 1:1. Pending: Targets: [-0.04026593 -0.66310342 -0.93503758 1.63840692] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 1.3033469106956084, 'kernel_inv_bw1': 3.4007518670314734, 'kernel_inv_bw2': 2.7166619261817617, 'kernel_inv_bw3': 84.00968747668335, 'kernel_inv_bw4': 4.7662174436449565, 'kernel_inv_bw5': 82.50051779133331, 'kernel_inv_bw6': 67.22964070460776, 'kernel_inv_bw7': 1.0170591364397439, 'kernel_inv_bw8': 1.5777241662196506, 'kernel_covariance_scale': 0.8127451069073666, 'mean_mean_value': 0.3271186419562175} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.307112 ; Training: accuracy=0.162766 ; Validation: accuracy=0.369670 .. parsed-literal:: :class: output config_id 2: Terminating evaluation at 1 Update for config_id 2:1: reward = 0.36966981914717106, crit_val = 0.630330180852829 Starting get_config[BO] for config_id 3 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5095188294272807 - self.std = 0.2423511077192109 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 5 Current best is [0.46875] [3: BO] (1 evaluations) batch_size: 75 dropout_1: 0.5302630865931415 dropout_2: 0.12336644847504774 learning_rate: 0.7028645562543027 n_units_1: 73 n_units_2: 126 scale_1: 0.10722353807156274 scale_2: 0.0016236501394851991 Started BO from (top scorer): batch_size: 75 dropout_1: 0.5302630865931415 dropout_2: 0.12336644847504774 learning_rate: 0.7028645562543027 n_units_1: 73 n_units_2: 126 scale_1: 0.10722353807156274 scale_2: 0.0016236501394851991 Top score values: [0.00719892 0.0303531 0.07280485 0.11276195 0.1538033 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending: Targets: [-0.16822217 -0.84259841 -1.13703466 1.64935803 0.49849721] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00048025899994700926, 'kernel_inv_bw1': 1.1131207570561128, 'kernel_inv_bw2': 1.0824826486741035, 'kernel_inv_bw3': 84.3637388830916, 'kernel_inv_bw4': 0.0006325818414342524, 'kernel_inv_bw5': 100.00000000000004, 'kernel_inv_bw6': 39.43198460539477, 'kernel_inv_bw7': 0.013328152652716078, 'kernel_inv_bw8': 1.6151629858567016, 'kernel_covariance_scale': 0.8034397943414244, 'mean_mean_value': 0.3077191486825238} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.460667 ; Training: accuracy=0.041159 ; Validation: accuracy=0.034333 .. parsed-literal:: :class: output config_id 3: Terminating evaluation at 1 Update for config_id 3:1: reward = 0.034333333333333334, crit_val = 0.9656666666666667 Starting get_config[BO] for config_id 4 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5855434689671783 - self.std = 0.279004979519119 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 6 Current best is [0.46875] [4: BO] (21 evaluations) batch_size: 128 dropout_1: 0.39366683240859135 dropout_2: 0.6114522662562138 learning_rate: 1.8354238197457777e-06 n_units_1: 50 n_units_2: 71 scale_1: 0.04226495155290692 scale_2: 0.0013550158837172756 Started BO from (top scorer): batch_size: 16 dropout_1: 0.39366684595540025 dropout_2: 0.611452209042092 learning_rate: 1.8354821086802627e-06 n_units_1: 50 n_units_2: 35 scale_1: 0.0422649570093964 scale_2: 0.0013547855784457326 Top score values: [0.00599251 0.07967276 0.11748946 0.15673679 0.16525097] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending: Targets: [-0.41860711 -1.00438816 -1.26014328 1.16019115 0.16052298 1.36242442] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 2.0053376121249324, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0010704263242157716, 'kernel_inv_bw3': 0.0008335469721583438, 'kernel_inv_bw4': 0.0037120842504946355, 'kernel_inv_bw5': 4.686944148532531, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0022817208637103594, 'kernel_inv_bw8': 1.5124799741334436, 'kernel_covariance_scale': 0.7561001736211863, 'mean_mean_value': -0.0006020526887166488} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.298823 ; Training: accuracy=0.052385 ; Validation: accuracy=0.130818 .. parsed-literal:: :class: output config_id 4: Terminating evaluation at 1 Update for config_id 4:1: reward = 0.13081781914893617, crit_val = 0.8691821808510638 Starting get_config[BO] for config_id 5 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6260632849505906 - self.std = 0.2767207468205929 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 7 Current best is [0.46875] [5: BO] (19 evaluations) batch_size: 8 dropout_1: 0.26555362657067627 dropout_2: 0.7173603551260431 learning_rate: 0.0022299009480851073 n_units_1: 127 n_units_2: 16 scale_1: 0.37877273777029896 scale_2: 0.36495062851271215 Started BO from (top scorer): batch_size: 29 dropout_1: 0.4799799052588535 dropout_2: 0.5647643800414299 learning_rate: 0.004077220423509741 n_units_1: 111 n_units_2: 89 scale_1: 0.15393299323611206 scale_2: 0.3845928002559493 Top score values: [0.06202583 0.13401968 0.16643212 0.21561172 0.26673195] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1. Pending: Targets: [-0.56849111 -1.15910758 -1.41697386 1.02333957 0.0154195 1.22724221 0.87857126] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.3335819072441991, 'kernel_inv_bw1': 0.2750752295317643, 'kernel_inv_bw2': 0.24890601177148425, 'kernel_inv_bw3': 11.183471528369342, 'kernel_inv_bw4': 0.2744568212856452, 'kernel_inv_bw5': 0.2952497007090965, 'kernel_inv_bw6': 0.3312379223384557, 'kernel_inv_bw7': 0.027407776944603238, 'kernel_inv_bw8': 1.0971022798993004, 'kernel_covariance_scale': 0.732949008356558, 'mean_mean_value': 0.15940542653147902} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.883657 ; Training: accuracy=0.166446 ; Validation: accuracy=0.514300 .. parsed-literal:: :class: output Update for config_id 5:1: reward = 0.514300134589502, crit_val = 0.48569986541049803 config_id 5: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 7.507255 ; Training: accuracy=0.223972 ; Validation: accuracy=0.605316 Epoch 3 ; Time: 10.984456 ; Training: accuracy=0.252487 ; Validation: accuracy=0.644179 .. parsed-literal:: :class: output config_id 5: Terminating evaluation at 3 Update for config_id 5:3: reward = 0.644179004037685, crit_val = 0.355820995962315 Starting get_config[BO] for config_id 6 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5804404284474385 - self.std = 0.26034600503297584 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 9 Current best is [0.46875] [6: BO] (3 evaluations) batch_size: 104 dropout_1: 0.06753273656053299 dropout_2: 0.5787103086119473 learning_rate: 0.0028205883583779776 n_units_1: 46 n_units_2: 84 scale_1: 0.0010000000000000002 scale_2: 10.0 Started BO from (top scorer): batch_size: 104 dropout_1: 0.06753790263018208 dropout_2: 0.5785861703504277 learning_rate: 0.0028205883438605244 n_units_1: 46 n_units_2: 84 scale_1: 0.004630219176146915 scale_2: 9.435508332838014 Top score values: [0.09120854 0.21865244 0.22627411 0.28955933 0.30119195] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3. Pending: Targets: [-0.42900765 -1.05677157 -1.33085664 1.26294293 0.19162865 1.47967025 1.10906926 -0.3639025 -0.86277273] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.038848425403682475, 'kernel_inv_bw1': 0.013826037746051094, 'kernel_inv_bw2': 0.04933080557942479, 'kernel_inv_bw3': 0.00010000000000000009, 'kernel_inv_bw4': 0.04813505724162551, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 3.8015174579942212, 'kernel_inv_bw7': 3.983663262254571, 'kernel_inv_bw8': 1.3227034312264876, 'kernel_covariance_scale': 0.8009725817505569, 'mean_mean_value': 0.3852913191841205} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.434894 ; Training: accuracy=0.427387 ; Validation: accuracy=0.733753 .. parsed-literal:: :class: output Update for config_id 6:1: reward = 0.7337533156498673, crit_val = 0.26624668435013266 config_id 6: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.739148 ; Training: accuracy=0.601459 ; Validation: accuracy=0.772712 Epoch 3 ; Time: 1.023004 ; Training: accuracy=0.640252 ; Validation: accuracy=0.786638 .. parsed-literal:: :class: output Update for config_id 6:3: reward = 0.7866379310344828, crit_val = 0.21336206896551724 config_id 6: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.314426 ; Training: accuracy=0.653846 ; Validation: accuracy=0.777023 Epoch 5 ; Time: 1.598032 ; Training: accuracy=0.671751 ; Validation: accuracy=0.806034 Epoch 6 ; Time: 1.881462 ; Training: accuracy=0.674072 ; Validation: accuracy=0.797745 Epoch 7 ; Time: 2.165841 ; Training: accuracy=0.684682 ; Validation: accuracy=0.804708 Epoch 8 ; Time: 2.445447 ; Training: accuracy=0.689821 ; Validation: accuracy=0.808853 Epoch 9 ; Time: 2.727328 ; Training: accuracy=0.697281 ; Validation: accuracy=0.816645 .. parsed-literal:: :class: output config_id 6: Terminating evaluation at 9 Update for config_id 6:9: reward = 0.8166445623342176, crit_val = 0.18335543766578244 Starting get_config[BO] for config_id 7 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.49057733725069824 - self.std = 0.27450813586157435 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 12 Current best is [0.26624669] [7: BO] (11 evaluations) batch_size: 39 dropout_1: 0.027346760299878224 dropout_2: 0.5143530092361976 learning_rate: 1.0 n_units_1: 109 n_units_2: 62 scale_1: 2.2834582168904713 scale_2: 10.0 Started BO from (top scorer): batch_size: 39 dropout_1: 0.027346252810436844 dropout_2: 0.5143570802897102 learning_rate: 3.053475061997591e-05 n_units_1: 109 n_units_2: 64 scale_1: 2.2783933896225537 scale_2: 7.120331459024733 Top score values: [0.13905466 0.14949949 0.15337874 0.18225681 0.22215788] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9. Pending: Targets: [-0.07951435 -0.67489135 -0.93483611 1.52514692 0.50910274 1.73069307 1.37921174 -0.01776804 -0.49090108 -0.81720949 -1.00986176 -1.11917229] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0006731845953042116, 'kernel_inv_bw1': 0.0004435239703577121, 'kernel_inv_bw2': 0.0010950254706804787, 'kernel_inv_bw3': 0.5006292391496392, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 1.6812251804964524, 'kernel_inv_bw6': 0.006431440498054594, 'kernel_inv_bw7': 1.2196658205787714, 'kernel_inv_bw8': 1.0870399802800832, 'kernel_covariance_scale': 1.0663920505796691, 'mean_mean_value': 0.43197612961607723} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.823566 ; Training: accuracy=0.037634 ; Validation: accuracy=0.038043 .. parsed-literal:: :class: output config_id 7: Terminating evaluation at 1 Update for config_id 7:1: reward = 0.03804256745433216, crit_val = 0.9619574325456678 Starting get_config[BO] for config_id 8 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5268373445810804 - self.std = 0.2921226755451922 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 13 Current best is [0.26624669] [8: BO] (14 evaluations) batch_size: 128 dropout_1: 0.08296817905926457 dropout_2: 0.19054579790821474 learning_rate: 0.005552797405875484 n_units_1: 72 n_units_2: 76 scale_1: 0.0015941704119388613 scale_2: 2.8428356369373566 Started BO from (top scorer): batch_size: 85 dropout_1: 0.08296818203857925 dropout_2: 0.19054613346766927 learning_rate: 0.014308176452172097 n_units_1: 72 n_units_2: 76 scale_1: 0.0015951451801685214 scale_2: 2.8428356956092933 Top score values: [0.06523108 0.18698123 0.18731839 0.18876787 0.21413681] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1. Pending: Targets: [-0.19884572 -0.75832242 -1.00259292 1.30905699 0.35427868 1.50220903 1.17192147 -0.14082262 -0.58542648 -0.89205899 -1.07309463 -1.17581392 1.48951151] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.6475724989455702, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0005582372769792866, 'kernel_inv_bw3': 6.972856257358789, 'kernel_inv_bw4': 0.008774491917625664, 'kernel_inv_bw5': 0.0003003923479295185, 'kernel_inv_bw6': 0.006691532727480968, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1916430438869066, 'kernel_covariance_scale': 0.6989353646776911, 'mean_mean_value': 0.5762116803881481} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.307458 ; Training: accuracy=0.570888 ; Validation: accuracy=0.795545 .. parsed-literal:: :class: output Update for config_id 8:1: reward = 0.7955452127659575, crit_val = 0.20445478723404253 config_id 8: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.549475 ; Training: accuracy=0.742105 ; Validation: accuracy=0.842919 Epoch 3 ; Time: 0.784090 ; Training: accuracy=0.780181 ; Validation: accuracy=0.860040 .. parsed-literal:: :class: output Update for config_id 8:3: reward = 0.8600398936170213, crit_val = 0.13996010638297873 config_id 8: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.029320 ; Training: accuracy=0.804194 ; Validation: accuracy=0.859043 Epoch 5 ; Time: 1.345355 ; Training: accuracy=0.810444 ; Validation: accuracy=0.880652 Epoch 6 ; Time: 1.581157 ; Training: accuracy=0.825247 ; Validation: accuracy=0.873172 Epoch 7 ; Time: 1.813629 ; Training: accuracy=0.835280 ; Validation: accuracy=0.894116 Epoch 8 ; Time: 2.050075 ; Training: accuracy=0.843257 ; Validation: accuracy=0.893949 Epoch 9 ; Time: 2.286526 ; Training: accuracy=0.843339 ; Validation: accuracy=0.898936 .. parsed-literal:: :class: output config_id 8: Terminating evaluation at 9 Update for config_id 8:9: reward = 0.898936170212766, crit_val = 0.10106382978723405 Starting get_config[BO] for config_id 9 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4558977626848939 - self.std = 0.30246201791748595 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 16 Current best is [0.20445479] [9: BO] (15 evaluations) batch_size: 95 dropout_1: 0.4262497145288378 dropout_2: 0.24000456457264768 learning_rate: 0.007365011875480271 n_units_1: 100 n_units_2: 16 scale_1: 0.0010000000000000002 scale_2: 0.0011590104176322858 Started BO from (top scorer): batch_size: 95 dropout_1: 0.4256455858790655 dropout_2: 0.24000416180847647 learning_rate: 0.007915982639017364 n_units_1: 100 n_units_2: 61 scale_1: 0.0036892633125513137 scale_2: 0.0011591512852441098 Top score values: [0.19647575 0.2062344 0.20680512 0.20890162 0.21861883] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9. Pending: Targets: [ 0.04249207 -0.49785951 -0.73377988 1.49884874 0.5767085 1.68539808 1.36640105 0.09853172 -0.33087383 -0.62702444 -0.80187157 -0.9010795 1.67313461 -0.83132083 -1.04455316 -1.17315204] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.006775924221977387, 'kernel_inv_bw1': 0.015172921861280587, 'kernel_inv_bw2': 0.0013188699217722508, 'kernel_inv_bw3': 4.415642720480877, 'kernel_inv_bw4': 0.0009644522562784017, 'kernel_inv_bw5': 0.6640762984435752, 'kernel_inv_bw6': 1.2045893849895135, 'kernel_inv_bw7': 0.0012584335539753077, 'kernel_inv_bw8': 0.9331349919847574, 'kernel_covariance_scale': 0.7694699862952703, 'mean_mean_value': 0.7784628343310049} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.363448 ; Training: accuracy=0.218400 ; Validation: accuracy=0.468839 .. parsed-literal:: :class: output config_id 9: Terminating evaluation at 1 Update for config_id 9:1: reward = 0.46883876357560567, crit_val = 0.5311612364243943 Starting get_config[BO] for config_id 10 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.46032502584604096 - self.std = 0.29396515757905645 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 17 Current best is [0.20445479] [10: BO] (15 evaluations) batch_size: 31 dropout_1: 0.0 dropout_2: 0.0222361448993122 learning_rate: 0.010132292274826368 n_units_1: 44 n_units_2: 107 scale_1: 1.5325745511866318 scale_2: 0.41088043849938755 Started BO from (top scorer): batch_size: 31 dropout_1: 0.37741926641904855 dropout_2: 0.022238614298746323 learning_rate: 0.0279472326159558 n_units_1: 44 n_units_2: 123 scale_1: 1.5324224383172704 scale_2: 0.4108804457105683 Top score values: [0.14409397 0.17617909 0.19075953 0.20373432 0.21153185] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1. Pending: Targets: [ 0.02865977 -0.5273103 -0.77004979 1.52711142 0.57831736 1.71905285 1.39083543 0.08631921 -0.35549801 -0.66020866 -0.84010962 -0.94218509 1.70643491 -0.87041009 -1.08980575 -1.22212169 0.24096805] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0020638596342242126, 'kernel_inv_bw1': 1.1254919271494774, 'kernel_inv_bw2': 0.0010445566099235965, 'kernel_inv_bw3': 4.826738255199483, 'kernel_inv_bw4': 0.006503434857039732, 'kernel_inv_bw5': 0.915326168235606, 'kernel_inv_bw6': 0.004022186724770975, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9927458895104945, 'kernel_covariance_scale': 0.7803189333803716, 'mean_mean_value': 0.8409017073194343} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.004785 ; Training: accuracy=0.721836 ; Validation: accuracy=0.834005 .. parsed-literal:: :class: output Update for config_id 10:1: reward = 0.834005376344086, crit_val = 0.165994623655914 config_id 10: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.903632 ; Training: accuracy=0.846402 ; Validation: accuracy=0.871136 Epoch 3 ; Time: 2.803965 ; Training: accuracy=0.887179 ; Validation: accuracy=0.876680 .. parsed-literal:: :class: output Update for config_id 10:3: reward = 0.8766801075268817, crit_val = 0.12331989247311825 config_id 10: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 3.779331 ; Training: accuracy=0.894127 ; Validation: accuracy=0.898522 Epoch 5 ; Time: 4.691466 ; Training: accuracy=0.906782 ; Validation: accuracy=0.899194 Epoch 6 ; Time: 5.745457 ; Training: accuracy=0.916791 ; Validation: accuracy=0.895665 Epoch 7 ; Time: 6.643373 ; Training: accuracy=0.919603 ; Validation: accuracy=0.904066 Epoch 8 ; Time: 7.577041 ; Training: accuracy=0.922415 ; Validation: accuracy=0.911290 Epoch 9 ; Time: 8.509091 ; Training: accuracy=0.926303 ; Validation: accuracy=0.911962 .. parsed-literal:: :class: output config_id 10: Terminating evaluation at 9 Update for config_id 10:9: reward = 0.9119623655913979, crit_val = 0.08803763440860213 Starting get_config[BO] for config_id 11 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4101438794960165 - self.std = 0.2964373065434322 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 20 Current best is [0.16599463] [11: BO] (35 evaluations) batch_size: 100 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.004820472426324419 n_units_1: 107 n_units_2: 128 scale_1: 0.06680137561468431 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 100 dropout_1: 0.00911833153273836 dropout_2: 0.23742831006045706 learning_rate: 0.007257355221166904 n_units_1: 96 n_units_2: 81 scale_1: 0.06682046297392799 scale_2: 0.0024843701783189196 Top score values: [0.13889474 0.14858531 0.15217332 0.21747817 0.22456437] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9. Pending: Targets: [ 0.19770157 -0.35363197 -0.59434712 1.68365683 0.74277527 1.87399755 1.54851731 0.25488015 -0.18325252 -0.48542202 -0.66382269 -0.7650469 1.86148484 -0.69387047 -0.91143647 -1.04264896 0.40823929 -0.82361177 -0.96757048 -1.08659146] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00015375309178579185, 'kernel_inv_bw1': 1.032070936012607, 'kernel_inv_bw2': 0.2807788155180474, 'kernel_inv_bw3': 3.8647366048770095, 'kernel_inv_bw4': 0.0033982880287331653, 'kernel_inv_bw5': 0.5295507523300419, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.01296890476903277, 'kernel_inv_bw8': 0.9070917227422385, 'kernel_covariance_scale': 0.8877448885848056, 'mean_mean_value': 0.969264783177449} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.339826 ; Training: accuracy=0.562645 ; Validation: accuracy=0.764667 .. parsed-literal:: :class: output Update for config_id 11:1: reward = 0.7646666666666667, crit_val = 0.23533333333333328 config_id 11: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.624418 ; Training: accuracy=0.804215 ; Validation: accuracy=0.819667 Epoch 3 ; Time: 0.898253 ; Training: accuracy=0.861405 ; Validation: accuracy=0.874667 .. parsed-literal:: :class: output Update for config_id 11:3: reward = 0.8746666666666667, crit_val = 0.1253333333333333 config_id 11: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.185551 ; Training: accuracy=0.888926 ; Validation: accuracy=0.891000 Epoch 5 ; Time: 1.476594 ; Training: accuracy=0.911901 ; Validation: accuracy=0.915500 Epoch 6 ; Time: 1.762834 ; Training: accuracy=0.927686 ; Validation: accuracy=0.915667 Epoch 7 ; Time: 2.077373 ; Training: accuracy=0.935372 ; Validation: accuracy=0.929833 Epoch 8 ; Time: 2.379926 ; Training: accuracy=0.945455 ; Validation: accuracy=0.929500 Epoch 9 ; Time: 2.769369 ; Training: accuracy=0.946529 ; Validation: accuracy=0.935333 .. parsed-literal:: :class: output config_id 11: Terminating evaluation at 9 Update for config_id 11:9: reward = 0.9353333333333333, crit_val = 0.06466666666666665 Starting get_config[BO] for config_id 12 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.375139605358855 - self.std = 0.29194623770057787 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 23 Current best is [0.16599462] [12: BO] (23 evaluations) batch_size: 43 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.005048365786215596 n_units_1: 50 n_units_2: 99 scale_1: 0.08427330560427197 scale_2: 0.019575394403800444 Started BO from (top scorer): batch_size: 43 dropout_1: 0.10030417821472981 dropout_2: 0.2123012790467915 learning_rate: 0.005482631492530546 n_units_1: 16 n_units_2: 99 scale_1: 0.0842733102219232 scale_2: 0.01957864750341544 Top score values: [0.22747641 0.24507579 0.2511109 0.26006843 0.26608125] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9. Pending: Targets: [ 0.32064258 -0.23917224 -0.48359036 1.8294566 0.87410126 2.02272537 1.6922382 0.37870075 -0.0661718 -0.37298964 -0.55413468 -0.65691604 2.01002017 -0.58464469 -0.80555756 -0.93878852 0.53441905 -0.71638184 -0.86255509 -0.98340699 -0.47887677 -0.85565847 -1.06345929] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0010985734536101213, 'kernel_inv_bw1': 0.8649270179329417, 'kernel_inv_bw2': 0.2636393579306121, 'kernel_inv_bw3': 3.8035263685806937, 'kernel_inv_bw4': 0.669976463457446, 'kernel_inv_bw5': 0.00011549998074446066, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0016618407331475172, 'kernel_inv_bw8': 0.9154142608685621, 'kernel_covariance_scale': 0.9179515989083743, 'mean_mean_value': 1.04777913925491} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.684860 ; Training: accuracy=0.589506 ; Validation: accuracy=0.767275 .. parsed-literal:: :class: output Update for config_id 12:1: reward = 0.7672745524510624, crit_val = 0.23272544754893765 config_id 12: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.311083 ; Training: accuracy=0.803526 ; Validation: accuracy=0.833361 Epoch 3 ; Time: 1.938117 ; Training: accuracy=0.857072 ; Validation: accuracy=0.867325 .. parsed-literal:: :class: output Update for config_id 12:3: reward = 0.8673247448552786, crit_val = 0.13267525514472145 config_id 12: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 2.593575 ; Training: accuracy=0.878259 ; Validation: accuracy=0.887569 Epoch 5 ; Time: 3.265805 ; Training: accuracy=0.906232 ; Validation: accuracy=0.892421 Epoch 6 ; Time: 3.906039 ; Training: accuracy=0.918977 ; Validation: accuracy=0.905304 Epoch 7 ; Time: 4.549798 ; Training: accuracy=0.924108 ; Validation: accuracy=0.913502 Epoch 8 ; Time: 5.284969 ; Training: accuracy=0.935529 ; Validation: accuracy=0.898109 Epoch 9 ; Time: 5.914810 ; Training: accuracy=0.940081 ; Validation: accuracy=0.907311 .. parsed-literal:: :class: output config_id 12: Terminating evaluation at 9 Update for config_id 12:9: reward = 0.9073113602141543, crit_val = 0.09268863978584574 Starting get_config[BO] for config_id 13 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3494730871435835 - self.std = 0.2843393969162383 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 26 Current best is [0.16599463] [13: BO] (19 evaluations) batch_size: 49 dropout_1: 0.0 dropout_2: 0.5645088370464826 learning_rate: 0.008117520067527348 n_units_1: 83 n_units_2: 128 scale_1: 0.011738370954664094 scale_2: 10.0 Started BO from (top scorer): batch_size: 49 dropout_1: 0.18507369082965996 dropout_2: 0.5644247025103575 learning_rate: 0.021536534166200078 n_units_1: 83 n_units_2: 32 scale_1: 0.011738400285525419 scale_2: 5.12869757717152 Top score values: [0.14710688 0.16359598 0.16610807 0.17599915 0.17805276] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9. Pending: Targets: [ 0.41948782 -0.15530354 -0.40626051 1.96866665 0.987753 2.16710588 1.82777729 0.4790992 0.02232511 -0.29270092 -0.47869208 -0.58422312 2.15406079 -0.51001831 -0.73684119 -0.87363644 0.63898338 -0.64527978 -0.79536356 -0.91944857 -0.40142082 -0.78828244 -1.00164249 -0.41059256 -0.76246146 -0.90309134] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010452280079272239, 'kernel_inv_bw1': 0.3851257033268748, 'kernel_inv_bw2': 0.004672299443157123, 'kernel_inv_bw3': 4.850239092869646, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.47895621368559343, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.41223347675408245, 'kernel_inv_bw8': 0.939813631659752, 'kernel_covariance_scale': 1.0011470092026824, 'mean_mean_value': 1.0431072156529337} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.685233 ; Training: accuracy=0.508717 ; Validation: accuracy=0.706591 .. parsed-literal:: :class: output config_id 13: Terminating evaluation at 1 Update for config_id 13:1: reward = 0.7065908330545333, crit_val = 0.2934091669454667 Starting get_config[BO] for config_id 14 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3473966456547643 - self.std = 0.2792249828512775 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 27 Current best is [0.16599463] [14: BO] (30 evaluations) batch_size: 53 dropout_1: 0.0 dropout_2: 0.11705125052606932 learning_rate: 0.01360292210731249 n_units_1: 108 n_units_2: 128 scale_1: 0.0010033865698770206 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 45 dropout_1: 0.07271341792828104 dropout_2: 0.3777104736924482 learning_rate: 0.004073244311091114 n_units_1: 106 n_units_2: 46 scale_1: 0.0010035328114221392 scale_2: 0.0015301629446470427 Top score values: [0.14323158 0.17123515 0.18547989 0.18802694 0.19629106] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1. Pending: Targets: [ 0.4346078 -0.15071171 -0.40626532 2.01216211 1.01328159 2.21423604 1.86869216 0.49531105 0.03017047 -0.29062572 -0.48002358 -0.58748757 2.20095201 -0.5119236 -0.74290107 -0.88220192 0.65812374 -0.64966258 -0.80249536 -0.92885317 -0.40133698 -0.79528454 -1.01255259 -0.41067671 -0.76899061 -0.91219633 -0.19334759] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.004395445140104485, 'kernel_inv_bw1': 1.222434567552199, 'kernel_inv_bw2': 0.7771588193504158, 'kernel_inv_bw3': 4.605173301537861, 'kernel_inv_bw4': 0.003034444769035247, 'kernel_inv_bw5': 0.18456224385325032, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.09757269492936227, 'kernel_inv_bw8': 1.0344175929580213, 'kernel_covariance_scale': 0.8950917105716353, 'mean_mean_value': 1.1691876572286595} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.589378 ; Training: accuracy=0.639110 ; Validation: accuracy=0.778761 .. parsed-literal:: :class: output Update for config_id 14:1: reward = 0.7787610619469026, crit_val = 0.22123893805309736 config_id 14: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.126719 ; Training: accuracy=0.823155 ; Validation: accuracy=0.843046 Epoch 3 ; Time: 1.674030 ; Training: accuracy=0.858160 ; Validation: accuracy=0.887126 .. parsed-literal:: :class: output Update for config_id 14:3: reward = 0.8871263983970613, crit_val = 0.11287360160293869 config_id 14: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 2.356794 ; Training: accuracy=0.869497 ; Validation: accuracy=0.878611 Epoch 5 ; Time: 3.028255 ; Training: accuracy=0.885137 ; Validation: accuracy=0.905994 Epoch 6 ; Time: 3.693399 ; Training: accuracy=0.894820 ; Validation: accuracy=0.913508 Epoch 7 ; Time: 4.384751 ; Training: accuracy=0.903178 ; Validation: accuracy=0.906495 Epoch 8 ; Time: 5.078993 ; Training: accuracy=0.908474 ; Validation: accuracy=0.918350 Epoch 9 ; Time: 5.772203 ; Training: accuracy=0.903840 ; Validation: accuracy=0.886959 .. parsed-literal:: :class: output config_id 14: Terminating evaluation at 9 Update for config_id 14:9: reward = 0.886959425613625, crit_val = 0.11304057438637505 Starting get_config[BO] for config_id 15 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3275620848907016 - self.std = 0.27197642408867573 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 30 Current best is [0.11287361] [15: BO] (33 evaluations) batch_size: 128 dropout_1: 0.0 dropout_2: 0.05978471049967446 learning_rate: 0.008771087280096419 n_units_1: 81 n_units_2: 128 scale_1: 1.6051836086730857 scale_2: 0.013635796499749771 Started BO from (top scorer): batch_size: 10 dropout_1: 0.04231197881231458 dropout_2: 0.26711820263125935 learning_rate: 0.0010374236949651373 n_units_1: 81 n_units_2: 22 scale_1: 1.6051803295987732 scale_2: 0.26385189842384354 Top score values: [0.11687344 0.14021704 0.14263611 0.14613091 0.19120987] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9. Pending: Targets: [ 0.51911821 -0.08180089 -0.34416537 2.13871659 1.11321449 2.34617608 1.99142296 0.5814393 0.10390206 -0.22544381 -0.41988939 -0.53021745 2.33253801 -0.45263959 -0.68977294 -0.83278636 0.74859118 -0.59404951 -0.7509555 -0.88068093 -0.33910568 -0.74355251 -0.96661106 -0.34869433 -0.71655781 -0.86358016 -0.12557308 -0.39092781 -0.78936431 -0.78875039] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.072806109016839, 'kernel_inv_bw1': 0.9779305280174871, 'kernel_inv_bw2': 0.6350438861248543, 'kernel_inv_bw3': 4.38767732128345, 'kernel_inv_bw4': 0.00015799760540255293, 'kernel_inv_bw5': 0.34358579783209253, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.3837206410237312, 'kernel_inv_bw8': 1.0972874897515217, 'kernel_covariance_scale': 0.8787819972958487, 'mean_mean_value': 1.2176464810673702} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.281261 ; Training: accuracy=0.667105 ; Validation: accuracy=0.812001 .. parsed-literal:: :class: output Update for config_id 15:1: reward = 0.812001329787234, crit_val = 0.18799867021276595 config_id 15: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.541857 ; Training: accuracy=0.835938 ; Validation: accuracy=0.872340 Epoch 3 ; Time: 0.765531 ; Training: accuracy=0.884622 ; Validation: accuracy=0.896110 .. parsed-literal:: :class: output Update for config_id 15:3: reward = 0.8961103723404256, crit_val = 0.10388962765957444 config_id 15: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 0.992504 ; Training: accuracy=0.908553 ; Validation: accuracy=0.912566 Epoch 5 ; Time: 1.228107 ; Training: accuracy=0.917188 ; Validation: accuracy=0.913730 Epoch 6 ; Time: 1.464472 ; Training: accuracy=0.935197 ; Validation: accuracy=0.929854 Epoch 7 ; Time: 1.687305 ; Training: accuracy=0.935937 ; Validation: accuracy=0.926695 Epoch 8 ; Time: 1.922657 ; Training: accuracy=0.941776 ; Validation: accuracy=0.938331 Epoch 9 ; Time: 2.179316 ; Training: accuracy=0.949013 ; Validation: accuracy=0.922872 .. parsed-literal:: :class: output config_id 15: Terminating evaluation at 9 Update for config_id 15:9: reward = 0.9228723404255319, crit_val = 0.0771276595744681 Starting get_config[BO] for config_id 16 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.30896601527781375 - self.std = 0.26628465735221635 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 33 Current best is [0.06466667] [16: BO] (32 evaluations) batch_size: 84 dropout_1: 0.1872000861932956 dropout_2: 0.0 learning_rate: 0.006197816480758794 n_units_1: 83 n_units_2: 102 scale_1: 1.5185688244258915 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 84 dropout_1: 0.45161421544435854 dropout_2: 0.5638747171714953 learning_rate: 0.021592731858966305 n_units_1: 83 n_units_2: 102 scale_1: 1.518495173672961 scale_2: 0.05388875466468348 Top score values: [-0.05190702 -0.01884247 0.02839783 0.06007975 0.07019629] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9. Pending: Targets: [ 0.60004953 -0.01371406 -0.28168651 2.25426642 1.20684447 2.4661603 2.10382442 0.66370272 0.17595824 -0.16042731 -0.35902912 -0.47171541 2.45223072 -0.39247935 -0.63468136 -0.78075165 0.83442743 -0.53691186 -0.69717168 -0.82966996 -0.27651868 -0.68961045 -0.91743682 -0.28631228 -0.66203874 -0.81220367 -0.05842187 -0.32944849 -0.73640147 -0.73577443 -0.45427831 -0.77013971 -0.87064106] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0011225369972828222, 'kernel_inv_bw1': 1.1410997493950654, 'kernel_inv_bw2': 0.7917189890607564, 'kernel_inv_bw3': 4.34788067006306, 'kernel_inv_bw4': 0.0038802911728676643, 'kernel_inv_bw5': 0.001087465565066201, 'kernel_inv_bw6': 0.000328139144163153, 'kernel_inv_bw7': 0.381417927287251, 'kernel_inv_bw8': 1.0745248268360206, 'kernel_covariance_scale': 0.9388174021125767, 'mean_mean_value': 1.2434091179730569} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.404633 ; Training: accuracy=0.615575 ; Validation: accuracy=0.787223 .. parsed-literal:: :class: output Update for config_id 16:1: reward = 0.7872233400402414, crit_val = 0.21277665995975859 config_id 16: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.763939 ; Training: accuracy=0.771577 ; Validation: accuracy=0.853789 Epoch 3 ; Time: 1.106980 ; Training: accuracy=0.824487 ; Validation: accuracy=0.881791 .. parsed-literal:: :class: output Update for config_id 16:3: reward = 0.8817907444668008, crit_val = 0.11820925553319916 config_id 16: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.454785 ; Training: accuracy=0.843998 ; Validation: accuracy=0.892857 Epoch 5 ; Time: 1.792809 ; Training: accuracy=0.866319 ; Validation: accuracy=0.896546 Epoch 6 ; Time: 2.307078 ; Training: accuracy=0.877976 ; Validation: accuracy=0.924212 Epoch 7 ; Time: 2.651515 ; Training: accuracy=0.889964 ; Validation: accuracy=0.913984 Epoch 8 ; Time: 2.987180 ; Training: accuracy=0.891782 ; Validation: accuracy=0.918846 Epoch 9 ; Time: 3.323330 ; Training: accuracy=0.899636 ; Validation: accuracy=0.933434 .. parsed-literal:: :class: output config_id 16: Terminating evaluation at 9 Update for config_id 16:9: reward = 0.9334339369550637, crit_val = 0.06656606304493629 Starting get_config[BO] for config_id 17 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.2942619578529374 - self.std = 0.26015822396056204 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 36 Current best is [0.06466667] [17: BO] (17 evaluations) batch_size: 19 dropout_1: 0.44699880885461124 dropout_2: 0.35902378542272395 learning_rate: 0.00994454798419592 n_units_1: 41 n_units_2: 41 scale_1: 0.20722456251139096 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 19 dropout_1: 0.446998616449144 dropout_2: 0.44450165886895776 learning_rate: 0.007050608524261196 n_units_1: 41 n_units_2: 40 scale_1: 0.20722353706958116 scale_2: 0.001180574218724413 Top score values: [-0.04060306 -0.01667316 0.02036014 0.02545359 0.04380131] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9. Pending: Targets: [ 0.6706997 0.04248266 -0.23180024 2.36387153 1.29178397 2.58075527 2.20988679 0.73585184 0.23662153 -0.10768552 -0.31096418 -0.42630411 2.56649767 -0.34520212 -0.59310772 -0.7426178 0.91059692 -0.49303586 -0.65706962 -0.79268808 -0.22651071 -0.64933033 -0.88252175 -0.23653494 -0.62110934 -0.77481048 -0.00327797 -0.28068696 -0.69722323 -0.69658141 -0.40845638 -0.73175596 -0.834624 -0.31321438 -0.67671396 -0.87522082] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010024528982465707, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.6296139381452603, 'kernel_inv_bw3': 4.578274729809311, 'kernel_inv_bw4': 0.00018607512015933497, 'kernel_inv_bw5': 0.8959493144763312, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.14840236754369968, 'kernel_inv_bw8': 1.4027812901311545, 'kernel_covariance_scale': 0.8876423821230857, 'mean_mean_value': 1.3226521421150887} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.561811 ; Training: accuracy=0.382677 ; Validation: accuracy=0.670254 .. parsed-literal:: :class: output config_id 17: Terminating evaluation at 1 Update for config_id 17:1: reward = 0.6702539095342189, crit_val = 0.32974609046578107 Starting get_config[BO] for config_id 18 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.29522098846409534 - self.std = 0.256682996577975 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 37 Current best is [0.06466667] [18: BO] (23 evaluations) batch_size: 64 dropout_1: 0.013136665274062341 dropout_2: 0.75 learning_rate: 0.00010583972178928185 n_units_1: 128 n_units_2: 16 scale_1: 0.5429125928862846 scale_2: 0.011730911155531775 Started BO from (top scorer): batch_size: 64 dropout_1: 0.013136987059864602 dropout_2: 0.5034888824382017 learning_rate: 4.629414809281751e-06 n_units_1: 16 n_units_2: 73 scale_1: 0.5429104968156312 scale_2: 0.01173088986204388 Top score values: [-0.02428519 0.05932551 0.07152494 0.07359064 0.08147375] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1. Pending: Targets: [ 0.67604405 0.03932159 -0.23867482 2.3921397 1.30553717 2.61195984 2.23607017 0.74207828 0.2360889 -0.11287972 -0.31891057 -0.43581208 2.5975092 -0.35361205 -0.60487404 -0.75640834 0.91918924 -0.50344731 -0.66970192 -0.80715652 -0.23331368 -0.66185785 -0.89820644 -0.24347363 -0.63325478 -0.78903687 -0.0070586 -0.28822342 -0.71039917 -0.70974866 -0.41772272 -0.74539944 -0.84966021 -0.32119123 -0.68961223 -0.89080667 0.13450483] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.6191424312423364, 'kernel_inv_bw3': 4.422173979447018, 'kernel_inv_bw4': 0.1853160297787087, 'kernel_inv_bw5': 0.9528081584043367, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00012639453445066284, 'kernel_inv_bw8': 1.5240871677818384, 'kernel_covariance_scale': 0.8876537823197109, 'mean_mean_value': 1.3439923159251053} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.559545 ; Training: accuracy=0.061756 ; Validation: accuracy=0.174059 .. parsed-literal:: :class: output config_id 18: Terminating evaluation at 1 Update for config_id 18:1: reward = 0.17405913978494625, crit_val = 0.8259408602150538 Starting get_config[BO] for config_id 19 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.30918730087859425 - self.std = 0.2671506242706291 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 38 Current best is [0.06466667] [19: BO] (41 evaluations) batch_size: 82 dropout_1: 0.5033788670142001 dropout_2: 0.0 learning_rate: 0.0068366382394597375 n_units_1: 128 n_units_2: 112 scale_1: 0.0023098041291240022 scale_2: 10.0 Started BO from (top scorer): batch_size: 82 dropout_1: 0.5033734160492536 dropout_2: 0.12329436421898304 learning_rate: 0.014691508241772754 n_units_1: 74 n_units_2: 52 scale_1: 0.0023096446056230517 scale_2: 0.0028549744663600007 Top score values: [0.03802058 0.05108491 0.05179572 0.06859465 0.08715316] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1. Pending: Targets: [ 0.59727616 -0.01449793 -0.28160174 2.24613091 1.20210417 2.45733794 2.09617657 0.66072301 0.17455956 -0.1607356 -0.35869365 -0.47101467 2.44345351 -0.39203544 -0.63345236 -0.77904917 0.83089432 -0.53599979 -0.69574012 -0.82780891 -0.27645066 -0.6882034 -0.91529127 -0.28621252 -0.66072107 -0.81039923 -0.05906082 -0.3292089 -0.73484275 -0.73421774 -0.45363409 -0.76847162 -0.8686472 -0.36088495 -0.71487029 -0.90818144 0.0769558 1.93431537] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.5985100096701567, 'kernel_inv_bw3': 4.267699620387008, 'kernel_inv_bw4': 0.15621448972299348, 'kernel_inv_bw5': 0.9103553579184126, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0732959268315928, 'kernel_inv_bw8': 1.4976041672770317, 'kernel_covariance_scale': 0.8556419538080655, 'mean_mean_value': 1.3261775399137425} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.408917 ; Training: accuracy=0.496786 ; Validation: accuracy=0.691113 .. parsed-literal:: :class: output config_id 19: Terminating evaluation at 1 Update for config_id 19:1: reward = 0.6911125960574674, crit_val = 0.30888740394253256 Starting get_config[BO] for config_id 20 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3091796112135671 - self.std = 0.2637033792780162 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 39 Current best is [0.06466667] [20: BO] (25 evaluations) batch_size: 128 dropout_1: 0.41108457637056517 dropout_2: 0.2626242802150693 learning_rate: 0.0058569004185053146 n_units_1: 128 n_units_2: 102 scale_1: 0.9534302215684295 scale_2: 0.018070562137925787 Started BO from (top scorer): batch_size: 128 dropout_1: 0.41106820571967884 dropout_2: 0.26262421544361153 learning_rate: 0.025796940714592858 n_units_1: 95 n_units_2: 47 scale_1: 0.9533993859134777 scale_2: 0.03822432087523404 Top score values: [-0.01791735 0.01786102 0.04308918 0.04802902 0.05093939] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1. Pending: Targets: [ 6.05113174e-01 -1.46582880e-02 -2.85253804e-01 2.27552247e+00 1.21784776e+00 2.48949049e+00 2.12360786e+00 6.69389428e-01 1.76870637e-01 -1.62807648e-01 -3.63353486e-01 -4.77142818e-01 2.47542456e+00 -3.97131141e-01 -6.41703968e-01 -7.89204075e-01 8.41785289e-01 -5.42977447e-01 -7.04805980e-01 -8.38601225e-01 -2.80035387e-01 -6.97170732e-01 -9.27227194e-01 -2.89924854e-01 -6.69329140e-01 -8.20963963e-01 -5.98037246e-02 -3.33483300e-01 -7.44419773e-01 -7.43786588e-01 -4.59535033e-01 -7.78488255e-01 -8.79973371e-01 -3.65573439e-01 -7.24186228e-01 -9.20024418e-01 7.79909583e-02 1.95963074e+00 -1.10809073e-03] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.001165694398243441, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.2523015365445955, 'kernel_inv_bw4': 0.40855856284176645, 'kernel_inv_bw5': 0.7504960382174999, 'kernel_inv_bw6': 0.0002734363676833348, 'kernel_inv_bw7': 0.5131858700249358, 'kernel_inv_bw8': 1.00815607605841, 'kernel_covariance_scale': 0.9259442809609314, 'mean_mean_value': 1.38937512547733} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.298842 ; Training: accuracy=0.539145 ; Validation: accuracy=0.750665 .. parsed-literal:: :class: output config_id 20: Terminating evaluation at 1 Update for config_id 20:1: reward = 0.7506648936170213, crit_val = 0.24933510638297873 Starting get_config[BO] for config_id 21 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.30768349859280236 - self.std = 0.2605537971371087 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 40 Current best is [0.06466667] [21: BO] (24 evaluations) batch_size: 48 dropout_1: 0.18570593072122676 dropout_2: 0.5713371911053257 learning_rate: 0.013632647785826724 n_units_1: 16 n_units_2: 32 scale_1: 10.0 scale_2: 10.0 Started BO from (top scorer): batch_size: 48 dropout_1: 0.18570598047780576 dropout_2: 0.5713282997895286 learning_rate: 0.033333077372950354 n_units_1: 47 n_units_2: 44 scale_1: 0.09619764668914697 scale_2: 0.008348100279749712 Top score values: [-0.03398156 0.045603 0.04790041 0.05168735 0.05303241] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1. Pending: Targets: [ 0.61816985 -0.00909343 -0.28295991 2.3087711 1.23831119 2.52532558 2.15502015 0.68322308 0.1847507 -0.15903362 -0.36200367 -0.47716849 2.51108961 -0.39618963 -0.64371886 -0.79300195 0.85770286 -0.54379893 -0.70758365 -0.84299621 -0.27767841 -0.6998561 -0.9326935 -0.28768743 -0.67167796 -0.82514575 -0.05478459 -0.33177241 -0.74767629 -0.74703545 -0.45934786 -0.7821566 -0.88486847 -0.36425045 -0.72719816 -0.92540365 0.08467576 1.98906087 0.00462056 -0.2239399 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.000688950031210672, 'kernel_inv_bw3': 3.8843756119878727, 'kernel_inv_bw4': 0.3354257626321199, 'kernel_inv_bw5': 0.7226708893548897, 'kernel_inv_bw6': 0.08692415231211728, 'kernel_inv_bw7': 0.474513251952304, 'kernel_inv_bw8': 1.0015103321713872, 'kernel_covariance_scale': 1.0393298947058967, 'mean_mean_value': 1.3624631143322377} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.848740 ; Training: accuracy=0.100612 ; Validation: accuracy=0.204637 .. parsed-literal:: :class: output config_id 21: Terminating evaluation at 1 Update for config_id 21:1: reward = 0.20463709677419356, crit_val = 0.7953629032258065 Starting get_config[BO] for config_id 22 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3195781182179976 - self.std = 0.2681263666763133 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 41 Current best is [0.06466667] [22: BO] (27 evaluations) batch_size: 90 dropout_1: 0.0 dropout_2: 0.6879712739929725 learning_rate: 0.004995912830531246 n_units_1: 16 n_units_2: 58 scale_1: 0.03908079586733884 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 90 dropout_1: 0.025893045336624287 dropout_2: 0.6879725645841153 learning_rate: 0.008779959281369732 n_units_1: 73 n_units_2: 43 scale_1: 0.03907780447755258 scale_2: 0.012925835170614252 Top score values: [-0.08560423 -0.01200155 0.01235796 0.03412326 0.05451195] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1. Pending: Targets: [ 0.55634917 -0.0531986 -0.3193304 2.19920355 1.15897614 2.40964198 2.04979491 0.61956513 0.13517088 -0.1989041 -0.39614175 -0.50805403 2.39580807 -0.42936222 -0.66990059 -0.81496755 0.78911716 -0.57280265 -0.73196168 -0.86354985 -0.31419806 -0.72445238 -0.95071385 -0.32392439 -0.69707006 -0.84620353 -0.09759932 -0.3667643 -0.77092201 -0.77029927 -0.4907367 -0.8044285 -0.90423953 -0.39832509 -0.75102223 -0.94362989 0.03792231 1.88852274 -0.03987192 -0.26197726 1.77447966] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.180300429625024, 'kernel_inv_bw2': 0.00012375616416109952, 'kernel_inv_bw3': 5.830164570865315, 'kernel_inv_bw4': 0.04290500448812368, 'kernel_inv_bw5': 1.5281996769225803, 'kernel_inv_bw6': 0.00012536666893201966, 'kernel_inv_bw7': 0.5410548461976437, 'kernel_inv_bw8': 0.9484448071973319, 'kernel_covariance_scale': 0.9935261698170694, 'mean_mean_value': 1.3437429881542777} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.404031 ; Training: accuracy=0.192123 ; Validation: accuracy=0.465320 .. parsed-literal:: :class: output config_id 22: Terminating evaluation at 1 .. parsed-literal:: :class: output .. parsed-literal:: :class: output Update for config_id 22:1: reward = 0.4653198653198653, crit_val = 0.5346801346801346 Starting get_config[BO] for config_id 23 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3246995948004294 - self.std = 0.26693716606537654 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 42 Current best is [0.06466667] [23: BO] (28 evaluations) batch_size: 109 dropout_1: 0.11852060048440546 dropout_2: 0.0 learning_rate: 0.006138751368351793 n_units_1: 128 n_units_2: 102 scale_1: 0.02891738158418114 scale_2: 0.18188937111131168 Started BO from (top scorer): batch_size: 109 dropout_1: 0.04031775066009935 dropout_2: 0.7182959892329118 learning_rate: 0.0045752284522110245 n_units_1: 103 n_units_2: 65 scale_1: 0.02891742255788364 scale_2: 0.9099861166037259 Top score values: [-0.02438883 -0.00867721 0.00772424 0.00978592 0.02696178] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1. Pending: Targets: [ 0.53964162 -0.07262167 -0.33993908 2.18981489 1.14495329 2.40119082 2.03974064 0.60313921 0.11658699 -0.21897629 -0.41709263 -0.52950348 2.38729528 -0.45046109 -0.69207106 -0.83778429 0.77344659 -0.59454056 -0.75440863 -0.88658303 -0.33478388 -0.74686588 -0.97413534 -0.34455355 -0.71936157 -0.86915943 -0.1172202 -0.38758431 -0.79354253 -0.79291701 -0.512109 -0.82719829 -0.92745397 -0.41928569 -0.7735541 -0.96701983 0.01890518 1.87775001 -0.05923563 -0.28233044 1.76319887 0.78662909] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.9812140563421585, 'kernel_inv_bw2': 0.33496536307372765, 'kernel_inv_bw3': 5.3016968001601725, 'kernel_inv_bw4': 0.3085800529760349, 'kernel_inv_bw5': 1.289053703882947, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.5175407006989317, 'kernel_inv_bw8': 0.9188839771275492, 'kernel_covariance_scale': 0.9524801588092421, 'mean_mean_value': 1.4147913481699443} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.470745 ; Training: accuracy=0.588313 ; Validation: accuracy=0.784654 .. parsed-literal:: :class: output Update for config_id 23:1: reward = 0.7846538782318598, crit_val = 0.21534612176814016 config_id 23: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.747943 ; Training: accuracy=0.791305 ; Validation: accuracy=0.856214 Epoch 3 ; Time: 1.036803 ; Training: accuracy=0.851393 ; Validation: accuracy=0.870225 .. parsed-literal:: :class: output config_id 23: Terminating evaluation at 3 Update for config_id 23:3: reward = 0.8702251876563804, crit_val = 0.12977481234361965 Starting get_config[BO] for config_id 24 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3177841799029499 - self.std = 0.26287650785603406 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 44 Current best is [0.06466667] [24: BO] (26 evaluations) batch_size: 128 dropout_1: 0.21841606653758272 dropout_2: 0.20374184939425172 learning_rate: 0.00508606125379937 n_units_1: 65 n_units_2: 111 scale_1: 0.34275490027583067 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 128 dropout_1: 0.573300893527115 dropout_2: 0.3422561033358465 learning_rate: 0.015760765651058696 n_units_1: 65 n_units_2: 88 scale_1: 0.3427537059973482 scale_2: 1.0430141723650779 Top score values: [-0.058857 -0.03617593 -0.00602565 0.00015744 0.00246559] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3. Pending: Targets: [ 0.57428418 -0.04743676 -0.31888342 2.24994771 1.18894611 2.46458876 2.09755526 0.63876261 0.14469462 -0.19605212 -0.39722877 -0.51137602 2.45047858 -0.43111267 -0.67645479 -0.82441886 0.81170074 -0.57741773 -0.73975529 -0.87397138 -0.31364859 -0.73209603 -0.96287612 -0.32356917 -0.70416686 -0.85627865 -0.0927242 -0.36726462 -0.77949369 -0.77885851 -0.49371285 -0.81366933 -0.91547367 -0.3994557 -0.7591965 -0.95565069 0.04550392 1.93306235 -0.03384394 -0.2603849 1.81674174 0.82508687 -0.3896813 -0.71520034] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.4773107164096427, 'kernel_inv_bw2': 1.4679705090422104, 'kernel_inv_bw3': 6.238172565895472, 'kernel_inv_bw4': 0.0013688879382363422, 'kernel_inv_bw5': 1.2841568273470492, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.07584607661842979, 'kernel_inv_bw8': 1.0557207868122573, 'kernel_covariance_scale': 0.9777257034912031, 'mean_mean_value': 1.3924430037176574} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.290785 ; Training: accuracy=0.493339 ; Validation: accuracy=0.717254 .. parsed-literal:: :class: output config_id 24: Terminating evaluation at 1 Update for config_id 24:1: reward = 0.7172539893617021, crit_val = 0.28274601063829785 Starting get_config[BO] for config_id 25 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.31700555391929097 - self.std = 0.25999055369635105 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 45 Current best is [0.06466667] [25: BO] (28 evaluations) batch_size: 67 dropout_1: 0.4415543835380574 dropout_2: 0.17713599260891252 learning_rate: 0.007534325733484996 n_units_1: 47 n_units_2: 128 scale_1: 0.18011623235166907 scale_2: 0.011939981901592116 Started BO from (top scorer): batch_size: 67 dropout_1: 0.27446038977900605 dropout_2: 0.3751223727356925 learning_rate: 0.03731393813805193 n_units_1: 47 n_units_2: 66 scale_1: 0.025936041515244566 scale_2: 0.05929575041254891 Top score values: [-0.01769174 -0.00732726 0.0144894 0.019887 0.02668765] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1. Pending: Targets: [ 0.58365369 -0.04496849 -0.31942828 2.27791746 1.2051385 2.49494108 2.12383342 0.64884785 0.14929559 -0.19523351 -0.39864327 -0.51405759 2.48067428 -0.43290329 -0.68096877 -0.83057527 0.82370563 -0.58083237 -0.74497192 -0.88067784 -0.31413534 -0.73722763 -0.97056944 -0.32416603 -0.70898845 -0.86278871 -0.09075863 -0.36834652 -0.78515142 -0.78450919 -0.49619835 -0.81970642 -0.92264081 -0.40089493 -0.76462893 -0.96326381 0.04900384 1.95751461 -0.03122479 -0.26028041 1.83990281 0.83724034 -0.39101202 -0.7201444 -0.13177226] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.6355233403702053, 'kernel_inv_bw2': 0.9029958244613535, 'kernel_inv_bw3': 4.901554552077261, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.2950753092823263, 'kernel_inv_bw6': 0.9148325451466071, 'kernel_inv_bw7': 0.37603134858345694, 'kernel_inv_bw8': 0.8614864415955265, 'kernel_covariance_scale': 1.1426790498461497, 'mean_mean_value': 1.2717253602063514} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.496053 ; Training: accuracy=0.463516 ; Validation: accuracy=0.705517 .. parsed-literal:: :class: output config_id 25: Terminating evaluation at 1 Update for config_id 25:1: reward = 0.7055173570350495, crit_val = 0.2944826429649505 Starting get_config[BO] for config_id 26 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.31651592542028356 - self.std = 0.2571700171079883 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 46 Current best is [0.06466667] [26: BO] (25 evaluations) batch_size: 115 dropout_1: 0.431217753143572 dropout_2: 0.220775368953466 learning_rate: 0.005866651154538663 n_units_1: 111 n_units_2: 107 scale_1: 2.1462066529246764 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 115 dropout_1: 0.3500250446723522 dropout_2: 0.3577246428515128 learning_rate: 0.007746856505449004 n_units_1: 111 n_units_2: 111 scale_1: 2.1364316530563725 scale_2: 0.12210755388377395 Top score values: [-0.16187454 -0.03440598 0.02215888 0.02287187 0.02482086] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1. Pending: Targets: [ 0.59195888 -0.04355778 -0.32102773 2.30480465 1.22025988 2.52420849 2.14903067 0.65786806 0.15283691 -0.19547085 -0.40111152 -0.51779165 2.50978522 -0.43574729 -0.68653345 -0.83778077 0.83464361 -0.5852988 -0.75123856 -0.88843285 -0.31567674 -0.74340934 -0.97931035 -0.32581744 -0.71486043 -0.87034752 -0.08985013 -0.37048249 -0.79185873 -0.79120946 -0.49973654 -0.82679272 -0.93085605 -0.40338787 -0.77111116 -0.97192459 0.05144521 1.98088774 -0.02966334 -0.26123115 1.86198602 0.84832677 -0.39339657 -0.72613874 -0.13131358 -0.08567594] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.4016969832248494, 'kernel_inv_bw2': 1.5548431716715594, 'kernel_inv_bw3': 6.410101329245941, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 1.2789051987867037, 'kernel_inv_bw6': 0.002500601231354757, 'kernel_inv_bw7': 0.07994470027723147, 'kernel_inv_bw8': 1.050533756162631, 'kernel_covariance_scale': 1.013963918545403, 'mean_mean_value': 1.3947599890293165} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.316076 ; Training: accuracy=0.518758 ; Validation: accuracy=0.727090 .. parsed-literal:: :class: output config_id 26: Terminating evaluation at 1 Update for config_id 26:1: reward = 0.7270903010033445, crit_val = 0.2729096989966555 Starting get_config[BO] for config_id 27 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.315588133368717 - self.std = 0.25449726243737253 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 47 Current best is [0.06466667] [27: BO] (32 evaluations) batch_size: 21 dropout_1: 0.44977523402211345 dropout_2: 0.22775354317878593 learning_rate: 0.005830667219304713 n_units_1: 56 n_units_2: 107 scale_1: 0.14328846989149557 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 21 dropout_1: 0.6307901388503625 dropout_2: 0.048396805897776285 learning_rate: 0.005206382249900967 n_units_1: 56 n_units_2: 68 scale_1: 0.14327058821032795 scale_2: 0.2616426747102914 Top score values: [-0.01671427 -0.01380325 0.00721527 0.00772511 0.01064867] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1. Pending: Targets: [ 0.60182127 -0.04036964 -0.3207536 2.33265551 1.23672076 2.55436356 2.17524559 0.66842264 0.1580876 -0.19387811 -0.40167844 -0.51958396 2.53978881 -0.43667796 -0.6900979 -0.84293364 0.8470547 -0.58780007 -0.75548255 -0.89411767 -0.31534642 -0.74757111 -0.98594957 -0.32559362 -0.71872238 -0.8758424 -0.08714815 -0.37072774 -0.79652932 -0.79587323 -0.50133924 -0.83183019 -0.9369864 -0.4039787 -0.77556385 -0.97848624 0.05563108 2.0053368 -0.02632928 -0.26032904 1.88518637 0.86088156 -0.39388247 -0.73011913 -0.12904706 -0.08293013 -0.16769703] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.3954479158901061, 'kernel_inv_bw2': 1.4648046751511983, 'kernel_inv_bw3': 6.3635395955492475, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 1.2760616829331175, 'kernel_inv_bw6': 0.000459823768002302, 'kernel_inv_bw7': 0.07631597784979986, 'kernel_inv_bw8': 1.0572767610160405, 'kernel_covariance_scale': 1.0481146015880773, 'mean_mean_value': 1.407447219082324} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.420753 ; Training: accuracy=0.491925 ; Validation: accuracy=0.739021 .. parsed-literal:: :class: output config_id 27: Terminating evaluation at 1 Update for config_id 27:1: reward = 0.7390206966178697, crit_val = 0.26097930338213027 Starting get_config[BO] for config_id 28 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3144504494106631 - self.std = 0.2519530485512195 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 48 Current best is [0.06466667] [28: BO] (52 evaluations) batch_size: 8 dropout_1: 0.42184871402947194 dropout_2: 0.22973772747988677 learning_rate: 0.005612837601248531 n_units_1: 16 n_units_2: 107 scale_1: 2.569994101154867 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 86 dropout_1: 0.012485018503003986 dropout_2: 0.23159412369703133 learning_rate: 0.0030719495858766528 n_units_1: 83 n_units_2: 38 scale_1: 2.427669128386413 scale_2: 0.12413975027733427 Top score values: [0.01561047 0.04301952 0.06196485 0.06372489 0.06632395] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1. Pending: Targets: [ 0.61241391 -0.03626183 -0.3194771 2.36072606 1.25372459 2.5846729 2.20172661 0.67968781 0.16419943 -0.19132043 -0.40121912 -0.52031524 2.56995098 -0.43657206 -0.69255103 -0.8469301 0.86012369 -0.5892202 -0.75859593 -0.89863098 -0.31401532 -0.7506046 -0.9913902 -0.324366 -0.72146456 -0.88017117 -0.08351271 -0.36995588 -0.80005719 -0.79939448 -0.50188628 -0.83571452 -0.9419326 -0.4035426 -0.77888001 -0.98385151 0.0607083 2.03010209 -0.02207969 -0.25844237 1.90873838 0.87409018 -0.39334443 -0.73297639 -0.12583471 -0.07925209 -0.16487497 -0.21222663] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.043354515241428186, 'kernel_inv_bw1': 0.48722866739628845, 'kernel_inv_bw2': 1.344945514678171, 'kernel_inv_bw3': 6.392254368452178, 'kernel_inv_bw4': 0.008478845030866105, 'kernel_inv_bw5': 1.3001252151102531, 'kernel_inv_bw6': 0.0023953033039684018, 'kernel_inv_bw7': 0.06077238974733418, 'kernel_inv_bw8': 1.0780826260251941, 'kernel_covariance_scale': 1.065768961640694, 'mean_mean_value': 1.4203155605053284} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.696474 ; Training: accuracy=0.291777 ; Validation: accuracy=0.577894 .. parsed-literal:: :class: output config_id 28: Terminating evaluation at 1 Update for config_id 28:1: reward = 0.5778936742934051, crit_val = 0.4221063257065949 Starting get_config[BO] for config_id 29 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.31664750811058007 - self.std = 0.24983298575475252 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 49 Current best is [0.06466667] [29: BO] (25 evaluations) batch_size: 56 dropout_1: 0.0 dropout_2: 0.24848470693697405 learning_rate: 0.006213675660362365 n_units_1: 118 n_units_2: 101 scale_1: 0.06714003227160924 scale_2: 0.011263999369006229 Started BO from (top scorer): batch_size: 56 dropout_1: 0.7361319246310831 dropout_2: 0.2156009675340169 learning_rate: 0.003663253609023281 n_units_1: 77 n_units_2: 107 scale_1: 0.5478226343206746 scale_2: 0.011264025318744222 Top score values: [-0.03000711 0.0269265 0.04830531 0.05073006 0.05651974] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1. Pending: Targets: [ 0.60881669 -0.04536365 -0.33098227 2.37196488 1.25556948 2.59781212 2.21161618 0.67666148 0.1567987 -0.20173807 -0.41341794 -0.53352471 2.58296527 -0.44907089 -0.70722207 -0.86291119 0.85862853 -0.60301439 -0.77382742 -0.9150508 -0.32547413 -0.76576828 -1.00859717 -0.33591265 -0.73638096 -0.89643434 -0.0930155 -0.3818894 -0.81564052 -0.81497218 -0.51493936 -0.85160044 -0.95871987 -0.41576114 -0.79428364 -1.0009945 0.05242936 2.03853527 -0.03106117 -0.2694296 1.91614167 0.87271353 -0.40547643 -0.74799048 -0.13569664 -0.08871873 -0.17506819 -0.22282168 0.42211727] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.22349405064456837, 'kernel_inv_bw2': 0.9635925241399346, 'kernel_inv_bw3': 5.176157668350697, 'kernel_inv_bw4': 0.5131296608518512, 'kernel_inv_bw5': 0.8035989804015595, 'kernel_inv_bw6': 0.7874489280538858, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9134532801855584, 'kernel_covariance_scale': 1.0785513392934298, 'mean_mean_value': 1.563999257733009} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.562677 ; Training: accuracy=0.597801 ; Validation: accuracy=0.773031 .. parsed-literal:: :class: output Update for config_id 29:1: reward = 0.7730307076101469, crit_val = 0.22696929238985308 config_id 29: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.068709 ; Training: accuracy=0.787946 ; Validation: accuracy=0.851636 Epoch 3 ; Time: 1.575621 ; Training: accuracy=0.835317 ; Validation: accuracy=0.875668 .. parsed-literal:: :class: output config_id 29: Terminating evaluation at 3 Update for config_id 29:3: reward = 0.8756675567423231, crit_val = 0.1243324432576769 Starting get_config[BO] for config_id 30 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.31111822809933243 - self.std = 0.2466194005265664 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 51 Current best is [0.06466667] [30: BO] (29 evaluations) batch_size: 70 dropout_1: 0.31958342053905936 dropout_2: 0.24399002542126408 learning_rate: 0.006599322198136962 n_units_1: 105 n_units_2: 105 scale_1: 0.06022014151194356 scale_2: 0.0032339328704914226 Started BO from (top scorer): batch_size: 70 dropout_1: 0.551843930100634 dropout_2: 0.3970268561101558 learning_rate: 0.006599975278097348 n_units_1: 110 n_units_2: 60 scale_1: 0.03873172490128809 scale_2: 0.0032339282381031492 Top score values: [-0.0579298 0.0043006 0.04447643 0.04534082 0.0519806 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3. Pending: Targets: [ 0.6391702 -0.02353447 -0.31287485 2.42529317 1.29435053 2.65408333 2.26285504 0.70789904 0.18126217 -0.18194653 -0.39638471 -0.51805653 2.63904301 -0.43250223 -0.69401726 -0.85173509 0.89223722 -0.5884517 -0.76149052 -0.90455412 -0.30729494 -0.75332636 -0.99931944 -0.31786948 -0.72355611 -0.88569507 -0.07180725 -0.36444534 -0.80384846 -0.80317142 -0.499229 -0.84027696 -0.94879222 -0.39875844 -0.78221329 -0.99161771 0.07553283 2.08751879 -0.00904562 -0.25052012 1.96353034 0.90650576 -0.38833971 -0.73531691 -0.11504455 -0.06745449 -0.15492913 -0.20330487 0.45003798 -0.34120972 -0.7573848 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.4772729027548042, 'kernel_inv_bw2': 0.7926641719348785, 'kernel_inv_bw3': 5.040873509127367, 'kernel_inv_bw4': 0.4864132209497254, 'kernel_inv_bw5': 0.7698955526075097, 'kernel_inv_bw6': 0.8901911588711701, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9006913454112195, 'kernel_covariance_scale': 1.0845781061187012, 'mean_mean_value': 1.5750446495137802} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.509809 ; Training: accuracy=0.540462 ; Validation: accuracy=0.745378 .. parsed-literal:: :class: output Update for config_id 30:1: reward = 0.7453781512605042, crit_val = 0.2546218487394958 config_id 30: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.934910 ; Training: accuracy=0.706936 ; Validation: accuracy=0.803025 Epoch 3 ; Time: 1.358011 ; Training: accuracy=0.752601 ; Validation: accuracy=0.842185 .. parsed-literal:: :class: output config_id 30: Terminating evaluation at 3 Update for config_id 30:3: reward = 0.8421848739495799, crit_val = 0.15781512605042014 Starting get_config[BO] for config_id 31 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3071597473180353 - self.std = 0.24292792468190896 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 53 Current best is [0.06466667] [31: BO] (28 evaluations) batch_size: 43 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.003057861367333125 n_units_1: 128 n_units_2: 92 scale_1: 1.2045672735960884 scale_2: 0.03066580760656 Started BO from (top scorer): batch_size: 43 dropout_1: 0.6582795958638858 dropout_2: 0.4639146886680176 learning_rate: 0.00823844766070984 n_units_1: 121 n_units_2: 74 scale_1: 1.2042649225423718 scale_2: 0.020779856609750333 Top score values: [0.0052399 0.01694925 0.02123839 0.03249024 0.07142258] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3. Pending: Targets: [ 0.66517776 -0.00759722 -0.30133435 2.47844223 1.33031406 2.71070903 2.31353573 0.73495099 0.20031147 -0.16841647 -0.3861132 -0.50963392 2.69544016 -0.42277956 -0.68826851 -0.84838298 0.92209033 -0.58109879 -0.75676707 -0.90200463 -0.29566965 -0.74847885 -0.99820999 -0.30640487 -0.71825622 -0.88285901 -0.05660354 -0.35368848 -0.79976868 -0.79908135 -0.49052029 -0.83675074 -0.94691497 -0.38852301 -0.77780474 -0.99039122 0.09297549 2.13553511 0.00711181 -0.23803209 2.00966256 0.93657568 -0.37794595 -0.73019574 -0.10049786 -0.05218463 -0.14098852 -0.19009936 0.47317153 -0.33009978 -0.75259896 -0.21626949 -0.61476926] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0008014496833669769, 'kernel_inv_bw1': 0.17868168288748057, 'kernel_inv_bw2': 0.22283389263016143, 'kernel_inv_bw3': 5.488584767342549, 'kernel_inv_bw4': 1.1209009450195109, 'kernel_inv_bw5': 1.0100879793676378, 'kernel_inv_bw6': 0.0005979962870949027, 'kernel_inv_bw7': 0.5120192852396444, 'kernel_inv_bw8': 0.8515727779691762, 'kernel_covariance_scale': 1.1708921719564838, 'mean_mean_value': 1.6315836590273203} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.687358 ; Training: accuracy=0.691716 ; Validation: accuracy=0.800402 .. parsed-literal:: :class: output Update for config_id 31:1: reward = 0.8004015392337293, crit_val = 0.19959846076627075 config_id 31: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.427055 ; Training: accuracy=0.845568 ; Validation: accuracy=0.866656 Epoch 3 ; Time: 2.101790 ; Training: accuracy=0.885376 ; Validation: accuracy=0.892923 .. parsed-literal:: :class: output Update for config_id 31:3: reward = 0.8929228710055211, crit_val = 0.10707712899447885 config_id 31: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 2.788547 ; Training: accuracy=0.911694 ; Validation: accuracy=0.914673 Epoch 5 ; Time: 3.472511 ; Training: accuracy=0.929405 ; Validation: accuracy=0.905638 Epoch 6 ; Time: 4.160065 ; Training: accuracy=0.940329 ; Validation: accuracy=0.917183 Epoch 7 ; Time: 4.986891 ; Training: accuracy=0.947695 ; Validation: accuracy=0.931906 Epoch 8 ; Time: 5.803370 ; Training: accuracy=0.951833 ; Validation: accuracy=0.928392 Epoch 9 ; Time: 6.543173 ; Training: accuracy=0.959199 ; Validation: accuracy=0.934081 .. parsed-literal:: :class: output config_id 31: Terminating evaluation at 9 Update for config_id 31:9: reward = 0.934080642462774, crit_val = 0.065919357537226 Starting get_config[BO] for config_id 32 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.29735824205631867 - self.std = 0.24024393396396798 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 56 Current best is [0.06466667] [32: BO] (20 evaluations) batch_size: 67 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.005029844658978558 n_units_1: 97 n_units_2: 101 scale_1: 0.4282244661770721 scale_2: 0.04365398118444292 Started BO from (top scorer): batch_size: 67 dropout_1: 0.1734863815084967 dropout_2: 0.05448423762046489 learning_rate: 0.0027946254009417865 n_units_1: 98 n_units_2: 113 scale_1: 0.4282230841629656 scale_2: 0.677369051872654 Top score values: [0.02265813 0.04217471 0.04999923 0.05146959 0.08312105] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9. Pending: Targets: [ 0.71340722 0.03311605 -0.2639027 2.54692938 1.38597439 2.78179105 2.38018055 0.78395995 0.24334747 -0.12949987 -0.34962869 -0.47452938 2.7663516 -0.38670469 -0.65515967 -0.81706293 0.97319 -0.54679265 -0.72442349 -0.87128363 -0.25817471 -0.71604267 -0.96856379 -0.26902987 -0.68548239 -0.85192412 -0.01643777 -0.31684173 -0.76790551 -0.7672105 -0.45520222 -0.80530073 -0.91669571 -0.35206542 -0.74569619 -0.96065767 0.13481235 2.20019132 0.0479894 -0.19989323 2.07291253 0.98783719 -0.3413702 -0.6975553 -0.06082248 -0.0119695 -0.1017655 -0.151425 0.51925591 -0.2929895 -0.72020881 -0.1778875 -0.58083929 -0.40691883 -0.79203296 -0.96334955] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00020134389738157523, 'kernel_inv_bw1': 0.18494793920169467, 'kernel_inv_bw2': 0.20357012643357442, 'kernel_inv_bw3': 5.170417209501338, 'kernel_inv_bw4': 1.1096493201061086, 'kernel_inv_bw5': 0.9933926176588416, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.5008653892049818, 'kernel_inv_bw8': 0.8322989515866606, 'kernel_covariance_scale': 1.2250883952275229, 'mean_mean_value': 1.6802901279825868} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.465850 ; Training: accuracy=0.657960 ; Validation: accuracy=0.791548 .. parsed-literal:: :class: output Update for config_id 32:1: reward = 0.7915478785846051, crit_val = 0.20845212141539493 config_id 32: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.884421 ; Training: accuracy=0.842206 ; Validation: accuracy=0.863827 Epoch 3 ; Time: 1.309256 ; Training: accuracy=0.888308 ; Validation: accuracy=0.887640 .. parsed-literal:: :class: output Update for config_id 32:3: reward = 0.8876404494382022, crit_val = 0.1123595505617978 config_id 32: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.732339 ; Training: accuracy=0.912106 ; Validation: accuracy=0.905752 Epoch 5 ; Time: 2.167060 ; Training: accuracy=0.925539 ; Validation: accuracy=0.912125 Epoch 6 ; Time: 2.606297 ; Training: accuracy=0.936982 ; Validation: accuracy=0.915982 Epoch 7 ; Time: 3.026454 ; Training: accuracy=0.943864 ; Validation: accuracy=0.928056 Epoch 8 ; Time: 3.443968 ; Training: accuracy=0.953317 ; Validation: accuracy=0.931913 Epoch 9 ; Time: 3.860855 ; Training: accuracy=0.957629 ; Validation: accuracy=0.931410 .. parsed-literal:: :class: output config_id 32: Terminating evaluation at 9 Update for config_id 32:9: reward = 0.9314103639107831, crit_val = 0.06858963608921687 Starting get_config[BO] for config_id 33 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.28883835361390264 - self.std = 0.23729911914726853 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 59 Current best is [0.06466667] [33: BO] (23 evaluations) batch_size: 49 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.006599144075261778 n_units_1: 128 n_units_2: 126 scale_1: 10.0 scale_2: 0.6131334928571974 Started BO from (top scorer): batch_size: 49 dropout_1: 0.31412038526172237 dropout_2: 0.25623758001373487 learning_rate: 0.004673782052180023 n_units_1: 70 n_units_2: 64 scale_1: 0.4245429161746598 scale_2: 0.6131350287995055 Top score values: [0.02428879 0.02514878 0.04077546 0.04530218 0.05353108] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9. Pending: Targets: [ 0.75816399 0.06943059 -0.23127408 2.61443963 1.43907752 2.85221587 2.4456215 0.82959226 0.28227093 -0.09520334 -0.31806391 -0.44451457 2.83658482 -0.3556 -0.62738643 -0.79129887 1.0211706 -0.51767461 -0.6975098 -0.84619243 -0.22547501 -0.68902498 -0.94467981 -0.23646487 -0.65808545 -0.82659268 0.01926182 -0.28487007 -0.74153142 -0.74082778 -0.42494757 -0.7793907 -0.89216806 -0.32053087 -0.71904649 -0.93667558 0.17238891 2.26339865 0.08448852 -0.16647026 2.13454037 1.03599955 -0.30970293 -0.67030818 -0.02567369 0.02378555 -0.06712479 -0.11740056 0.56160332 -0.26072183 -0.69324282 -0.14419145 -0.55214376 -0.376065 -0.76595828 -0.93940086 -0.33875487 -0.74369767 -0.92814806] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.15376987171783166, 'kernel_inv_bw2': 0.8003658086258035, 'kernel_inv_bw3': 4.661541391068402, 'kernel_inv_bw4': 0.820536287484395, 'kernel_inv_bw5': 0.7192071222695686, 'kernel_inv_bw6': 0.5184585130388424, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.8193951701553809, 'kernel_covariance_scale': 1.463581095872539, 'mean_mean_value': 1.5846775355833405} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.629033 ; Training: accuracy=0.670412 ; Validation: accuracy=0.773670 .. parsed-literal:: :class: output Update for config_id 33:1: reward = 0.7736701237872198, crit_val = 0.22632987621278022 config_id 33: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.192476 ; Training: accuracy=0.804759 ; Validation: accuracy=0.792071 Epoch 3 ; Time: 1.743613 ; Training: accuracy=0.829464 ; Validation: accuracy=0.811308 .. parsed-literal:: :class: output config_id 33: Terminating evaluation at 3 Update for config_id 33:3: reward = 0.8113081298093008, crit_val = 0.18869187019069922 Starting get_config[BO] for config_id 34 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.28617187884629075 - self.std = 0.23385032194414795 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 61 Current best is [0.06466667] [34: BO] (22 evaluations) batch_size: 31 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.010848410822500352 n_units_1: 70 n_units_2: 109 scale_1: 0.05201511999198041 scale_2: 0.0015266972045049883 Started BO from (top scorer): batch_size: 31 dropout_1: 0.2175351876089488 dropout_2: 0.02805791405897856 learning_rate: 0.008679743599523712 n_units_1: 66 n_units_2: 110 scale_1: 0.010279419696210736 scale_2: 0.00152670465144634 Top score values: [0.00367144 0.00807341 0.02277784 0.02796257 0.03939434] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3. Pending: Targets: [ 0.78074779 0.08185703 -0.22328239 2.66439957 1.47170335 2.9056825 2.49309172 0.85322947 0.29783631 -0.08520491 -0.31135219 -0.43966773 2.88982093 -0.34944186 -0.62523657 -0.79156636 1.04763318 -0.51390673 -0.69639411 -0.8472695 -0.2173978 -0.68778415 -0.94720935 -0.22854974 -0.65638834 -0.82738068 0.03094838 -0.27766881 -0.74106495 -0.74035093 -0.41981216 -0.77948258 -0.89392316 -0.31385554 -0.71824842 -0.93908708 0.18633377 2.30818148 0.09713703 -0.15752286 2.17742281 1.06268084 -0.3028679 -0.66879132 -0.01464983 0.03553882 -0.05671226 -0.10772949 0.58128826 -0.25316444 -0.6920642 -0.13491549 -0.54888423 -0.37020868 -0.76585206 -0.94185255 -0.3323483 -0.74326316 -0.9304338 -0.25589874 -0.41684787] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.18190102112267384, 'kernel_inv_bw2': 0.7123611616932807, 'kernel_inv_bw3': 5.197488621083075, 'kernel_inv_bw4': 1.0067438436023535, 'kernel_inv_bw5': 0.7926447175972984, 'kernel_inv_bw6': 0.5969046798403365, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.8499041380800628, 'kernel_covariance_scale': 1.2637911058686522, 'mean_mean_value': 1.685471460467687} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.923618 ; Training: accuracy=0.660711 ; Validation: accuracy=0.799899 .. parsed-literal:: :class: output Update for config_id 34:1: reward = 0.7998991935483871, crit_val = 0.20010080645161288 config_id 34: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.836628 ; Training: accuracy=0.835649 ; Validation: accuracy=0.856183 Epoch 3 ; Time: 2.797171 ; Training: accuracy=0.871216 ; Validation: accuracy=0.873656 .. parsed-literal:: :class: output config_id 34: Terminating evaluation at 3 Update for config_id 34:3: reward = 0.8736559139784946, crit_val = 0.12634408602150538 Starting get_config[BO] for config_id 35 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.2822687222555056 - self.std = 0.23120930284746374 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 63 Current best is [0.06466667] [35: BO] (26 evaluations) batch_size: 121 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.004709749912666916 n_units_1: 128 n_units_2: 84 scale_1: 1.785630978661036 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 121 dropout_1: 0.08199638119198988 dropout_2: 0.12202712352247957 learning_rate: 0.0030296314321197958 n_units_1: 60 n_units_2: 56 scale_1: 1.7851756051748051 scale_2: 0.0018481959606542833 Top score values: [0.00223451 0.02319435 0.0250267 0.05159533 0.05400897] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3. Pending: Targets: [ 8.06547468e-01 9.96735364e-02 -2.08951380e-01 2.71171551e+00 1.50539556e+00 2.95575453e+00 2.53845088e+00 8.79857085e-01 3.18119871e-01 -6.92966836e-02 -2.98027166e-01 -4.27808412e-01 2.93971178e+00 -3.36551921e-01 -6.15496929e-01 -7.83726650e-01 1.07648140e+00 -5.02895416e-01 -6.87467277e-01 -8.40066059e-01 -2.02999569e-01 -6.78758973e-01 -9.41147492e-01 -2.14278898e-01 -6.47004533e-01 -8.19950063e-01 4.81833756e-02 -2.63959034e-01 -7.32648378e-01 -7.31926206e-01 -4.07726034e-01 -7.71504833e-01 -8.87252633e-01 -3.00559110e-01 -7.09571218e-01 -9.32932441e-01 2.05343676e-01 2.35142847e+00 1.15128074e-01 -1.42440704e-01 2.21917620e+00 1.09170094e+00 -2.89445968e-01 -6.59549196e-01 2.06431306e-03 5.28262512e-02 -4.04785757e-02 -9.20785566e-02 6.04809589e-01 -2.39174762e-01 -6.83087908e-01 -1.19575091e-01 -5.38272443e-01 -3.57555948e-01 -7.57718617e-01 -9.35729497e-01 -3.19263109e-01 -7.34871692e-01 -9.24180314e-01 -2.41940291e-01 -4.04727885e-01 -3.55383260e-01 -6.74387381e-01] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00025014083267642167, 'kernel_inv_bw1': 0.28919596584551394, 'kernel_inv_bw2': 0.22420097977804268, 'kernel_inv_bw3': 4.672336346114276, 'kernel_inv_bw4': 1.037841560138486, 'kernel_inv_bw5': 1.1382833259248413, 'kernel_inv_bw6': 0.0006661490330757411, 'kernel_inv_bw7': 0.5873591706441644, 'kernel_inv_bw8': 0.833929744581794, 'kernel_covariance_scale': 1.2661342913985998, 'mean_mean_value': 1.7163904325693817} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.297275 ; Training: accuracy=0.665041 ; Validation: accuracy=0.805455 .. parsed-literal:: :class: output Update for config_id 35:1: reward = 0.8054545454545454, crit_val = 0.19454545454545458 config_id 35: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.548198 ; Training: accuracy=0.833884 ; Validation: accuracy=0.842645 Epoch 3 ; Time: 0.778134 ; Training: accuracy=0.877686 ; Validation: accuracy=0.880661 .. parsed-literal:: :class: output Update for config_id 35:3: reward = 0.8806611570247934, crit_val = 0.11933884297520658 config_id 35: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.011846 ; Training: accuracy=0.905372 ; Validation: accuracy=0.908595 Epoch 5 ; Time: 1.251584 ; Training: accuracy=0.929669 ; Validation: accuracy=0.915207 Epoch 6 ; Time: 1.610493 ; Training: accuracy=0.935289 ; Validation: accuracy=0.917686 Epoch 7 ; Time: 1.844067 ; Training: accuracy=0.940331 ; Validation: accuracy=0.915372 Epoch 8 ; Time: 2.110958 ; Training: accuracy=0.951240 ; Validation: accuracy=0.914876 Epoch 9 ; Time: 2.357718 ; Training: accuracy=0.955620 ; Validation: accuracy=0.928595 .. parsed-literal:: :class: output config_id 35: Terminating evaluation at 9 Update for config_id 35:9: reward = 0.9285950413223141, crit_val = 0.07140495867768593 Starting get_config[BO] for config_id 36 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.2752760417923516 - self.std = 0.2284106990916211 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 66 Current best is [0.06466667] [36: BO] (42 evaluations) batch_size: 13 dropout_1: 0.0 dropout_2: 0.3229218378252589 learning_rate: 0.003350210782489933 n_units_1: 128 n_units_2: 110 scale_1: 0.017430610159085962 scale_2: 0.003138239662520903 Started BO from (top scorer): batch_size: 13 dropout_1: 0.26884076770212595 dropout_2: 0.3645837923519833 learning_rate: 0.2588421122908576 n_units_1: 61 n_units_2: 42 scale_1: 0.0011346200476860303 scale_2: 0.003138245794989121 Top score values: [-0.0624444 0.00170651 0.0057119 0.04304142 0.08263659] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9. Pending: Targets: [ 0.8470442 0.13150929 -0.18089705 2.77555533 1.55445494 3.02258444 2.60016777 0.92125204 0.35263214 -0.03953124 -0.27106424 -0.40243563 3.00634512 -0.31006102 -0.5924238 -0.76271476 1.1202855 -0.47844264 -0.66527597 -0.81974447 -0.17487232 -0.65646097 -0.9220644 -0.18628985 -0.62431746 -0.799382 0.07938825 -0.23657869 -0.71101065 -0.71027963 -0.3821072 -0.7503432 -0.8675092 -0.27362721 -0.68765074 -0.9137487 0.23847416 2.41085387 0.14715319 -0.11357145 2.27698117 1.13569151 -0.2623779 -0.63701582 0.03270411 0.08408801 -0.01036003 -0.06259224 0.64283453 -0.21149075 -0.66084294 -0.09042568 -0.51425312 -0.3313224 -0.73638807 -0.91658002 -0.29256038 -0.71326121 -0.90488934 -0.21429016 -0.37907231 -0.32912309 -0.65203581 -0.35344486 -0.68270532 -0.89256363] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0001765975485634436, 'kernel_inv_bw1': 0.20265830659692838, 'kernel_inv_bw2': 0.7421344493969121, 'kernel_inv_bw3': 4.575089827805363, 'kernel_inv_bw4': 0.8444089666067263, 'kernel_inv_bw5': 0.7323448181989877, 'kernel_inv_bw6': 0.9186947228696695, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.8135774028321318, 'kernel_covariance_scale': 1.3221065106332446, 'mean_mean_value': 1.6866603747506494} .. parsed-literal:: :class: output Epoch 1 ; Time: 2.381794 ; Training: accuracy=0.572696 ; Validation: accuracy=0.767548 .. parsed-literal:: :class: output config_id 36: Terminating evaluation at 1 Update for config_id 36:1: reward = 0.7675475509173539, crit_val = 0.23245244908264606 Starting get_config[BO] for config_id 37 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.27463688369220673 - self.std = 0.22675919311532314 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 67 Current best is [0.06466667] [37: BO] (38 evaluations) batch_size: 84 dropout_1: 0.0 dropout_2: 0.2922265871470264 learning_rate: 0.003341868460106545 n_units_1: 128 n_units_2: 125 scale_1: 0.0029292853735109286 scale_2: 0.06986498582894866 Started BO from (top scorer): batch_size: 84 dropout_1: 0.6165336826147805 dropout_2: 0.716672390336065 learning_rate: 0.04661339810236861 n_units_1: 43 n_units_2: 121 scale_1: 0.0032860110061669885 scale_2: 0.06986493574617235 Top score values: [0.01804793 0.04567762 0.05125348 0.05544129 0.07355351] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9, 36:1. Pending: Targets: [ 0.85603196 0.13528575 -0.17939588 2.7985886 1.56859483 3.04741684 2.62192368 0.93078026 0.35801906 -0.03700048 -0.27021976 -0.40254794 3.03105925 -0.30950056 -0.59391981 -0.76545101 1.1312633 -0.47910851 -0.66730257 -0.82289607 -0.17332726 -0.65842336 -0.92596121 -0.18482795 -0.62604575 -0.8023853 0.0827851 -0.23548305 -0.71337034 -0.71263399 -0.38207145 -0.75298934 -0.87100867 -0.27280139 -0.68984029 -0.91758494 0.24302965 2.43123099 0.15104358 -0.11157994 2.29638328 1.14678151 -0.26147016 -0.6388366 0.03576096 0.08751909 -0.00761682 -0.06022944 0.65033501 -0.21021239 -0.66283725 -0.08826559 -0.51517981 -0.33091678 -0.73893258 -0.92043689 -0.29187245 -0.71563728 -0.90866105 -0.21303219 -0.37901446 -0.32870146 -0.65396598 -0.35320036 -0.68485885 -0.89624558 -0.18603186] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.003285371084878471, 'kernel_inv_bw1': 0.3304262058754804, 'kernel_inv_bw2': 0.6352604558931586, 'kernel_inv_bw3': 4.346194226510676, 'kernel_inv_bw4': 0.7173119046553594, 'kernel_inv_bw5': 0.7149273369845481, 'kernel_inv_bw6': 1.01614715613028, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.8127427752719871, 'kernel_covariance_scale': 1.3645023654628092, 'mean_mean_value': 1.6754358600797579} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.397591 ; Training: accuracy=0.499835 ; Validation: accuracy=0.712106 .. parsed-literal:: :class: output config_id 37: Terminating evaluation at 1 Update for config_id 37:1: reward = 0.7121059691482227, crit_val = 0.28789403085177734 Starting get_config[BO] for config_id 38 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.27483184173867103 - self.std = 0.22509132750228003 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 68 Current best is [0.06466667] [38: BO] (22 evaluations) batch_size: 44 dropout_1: 0.75 dropout_2: 0.0 learning_rate: 0.0063692003585150226 n_units_1: 62 n_units_2: 89 scale_1: 4.453876363616511 scale_2: 0.08121415983173202 Started BO from (top scorer): batch_size: 44 dropout_1: 0.6643415461271273 dropout_2: 0.1987793444951125 learning_rate: 0.016764470238010157 n_units_1: 77 n_units_2: 124 scale_1: 4.45385772810035 scale_2: 0.033619294881536775 Top score values: [0.03750278 0.06083585 0.06409336 0.08235989 0.09293021] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9, 36:1, 37:1. Pending: Targets: [ 0.86150879 0.13542205 -0.18159128 2.81845925 1.57935156 3.06913124 2.64048529 0.93681096 0.35980575 -0.03814077 -0.27308814 -0.40639684 3.05265244 -0.31266 -0.59918673 -0.77198893 1.13877952 -0.48352471 -0.67311322 -0.82985964 -0.1754777 -0.66416823 -0.93368846 -0.1870636 -0.63155071 -0.80919689 0.08253239 -0.23809404 -0.71952235 -0.71878055 -0.38576862 -0.75943492 -0.87832874 -0.27568891 -0.69581795 -0.92525012 0.2439643 2.44837962 0.15129664 -0.11327285 2.31253273 1.15441273 -0.26427371 -0.64443633 0.03515981 0.08730146 -0.00853939 -0.06154186 0.65428769 -0.21263613 -0.66861482 -0.08978575 -0.51986328 -0.33423492 -0.74527399 -0.9281232 -0.29490128 -0.72180609 -0.91626012 -0.21547683 -0.38268898 -0.33200317 -0.65967782 -0.35668361 -0.6907996 -0.90375265 -0.18827643 0.05803062] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.3137479251964829, 'kernel_inv_bw2': 0.21388040891089755, 'kernel_inv_bw3': 4.64126465396062, 'kernel_inv_bw4': 1.0288777129326367, 'kernel_inv_bw5': 1.1441479346655605, 'kernel_inv_bw6': 0.00011695284618913166, 'kernel_inv_bw7': 0.5685924461821816, 'kernel_inv_bw8': 0.8436950628722043, 'kernel_covariance_scale': 1.3190266323792321, 'mean_mean_value': 1.762606161449074} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.701976 ; Training: accuracy=0.237438 ; Validation: accuracy=0.469024 .. parsed-literal:: :class: output config_id 38: Terminating evaluation at 1 Update for config_id 38:1: reward = 0.469023569023569, crit_val = 0.530976430976431 Starting get_config[BO] for config_id 39 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.27854408216240667 - self.std = 0.225541357402729 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 69 Current best is [0.06466667] [39: BO] (28 evaluations) batch_size: 8 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.005187968909871167 n_units_1: 63 n_units_2: 117 scale_1: 0.0035297615259256587 scale_2: 0.002209076216210477 Started BO from (top scorer): batch_size: 100 dropout_1: 0.044803819744552636 dropout_2: 0.050523243428423986 learning_rate: 0.0036037353512263152 n_units_1: 107 n_units_2: 61 scale_1: 0.019097693338704407 scale_2: 0.0022102786701961525 Top score values: [0.02971497 0.06644624 0.07417161 0.08208605 0.1004852 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9, 36:1, 37:1, 38:1. Pending: Targets: [ 0.84333055 0.11869259 -0.19768819 2.79637624 1.55974098 3.04654806 2.6187574 0.91848247 0.34262857 -0.05452392 -0.28900249 -0.42204519 3.03010214 -0.32849538 -0.61445039 -0.7869078 1.12004804 -0.49901916 -0.68822938 -0.84466304 -0.19158681 -0.67930224 -0.94828469 -0.20314959 -0.6467498 -0.82404152 0.06590847 -0.25407821 -0.73454591 -0.73380559 -0.40145813 -0.77437884 -0.89303543 -0.29159806 -0.71088881 -0.93986319 0.22701827 2.42703504 0.13453551 -0.12950607 2.29145921 1.13565004 -0.28020564 -0.65960971 0.01863041 0.07066802 -0.0249816 -0.07787831 0.63652292 -0.2286711 -0.68373996 -0.10606584 -0.53528522 -0.35002725 -0.76024617 -0.94273054 -0.3107721 -0.73682509 -0.93089112 -0.23150613 -0.39838464 -0.34779996 -0.67482079 -0.37243115 -0.70588047 -0.91840861 -0.20436001 0.04145558 1.11922865] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.10767130316015067, 'kernel_inv_bw1': 0.5735987472433268, 'kernel_inv_bw2': 0.4501678262011913, 'kernel_inv_bw3': 3.847041571312879, 'kernel_inv_bw4': 0.43384795376286595, 'kernel_inv_bw5': 0.726479391236762, 'kernel_inv_bw6': 1.5954055912198324, 'kernel_inv_bw7': 0.0014678419708413885, 'kernel_inv_bw8': 0.7949160188252204, 'kernel_covariance_scale': 1.367033050795075, 'mean_mean_value': 1.8026277861637732} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.842823 ; Training: accuracy=0.657410 ; Validation: accuracy=0.807369 .. parsed-literal:: :class: output Update for config_id 39:1: reward = 0.8073687752355316, crit_val = 0.19263122476446837 config_id 39: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 7.860336 ; Training: accuracy=0.830902 ; Validation: accuracy=0.867429 Epoch 3 ; Time: 11.884933 ; Training: accuracy=0.874917 ; Validation: accuracy=0.875168 .. parsed-literal:: :class: output config_id 39: Terminating evaluation at 3 Update for config_id 39:3: reward = 0.8751682368775235, crit_val = 0.12483176312247646 Starting get_config[BO] for config_id 40 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.27516907967736626 - self.std = 0.22329649743884794 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 71 Current best is [0.06466667] [40: BO] (26 evaluations) batch_size: 8 dropout_1: 0.19742704332913752 dropout_2: 0.3252681176512454 learning_rate: 0.005219528352860058 n_units_1: 128 n_units_2: 108 scale_1: 0.7796167062520984 scale_2: 0.29756654929393117 Started BO from (top scorer): batch_size: 31 dropout_1: 0.4878886531255304 dropout_2: 0.18114926469664466 learning_rate: 0.0016224230284530277 n_units_1: 92 n_units_2: 116 scale_1: 0.060583648432504245 scale_2: 0.29827246813684094 Top score values: [-0.02123424 0.02655392 0.03792998 0.04227423 0.0445185 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 6:1, 6:3, 6:9, 7:1, 8:1, 8:3, 8:9, 9:1, 10:1, 10:3, 10:9, 11:1, 11:3, 11:9, 12:1, 12:3, 12:9, 13:1, 14:1, 14:3, 14:9, 15:1, 15:3, 15:9, 16:1, 16:3, 16:9, 17:1, 18:1, 19:1, 20:1, 21:1, 22:1, 23:1, 23:3, 24:1, 25:1, 26:1, 27:1, 28:1, 29:1, 29:3, 30:1, 30:3, 31:1, 31:3, 31:9, 32:1, 32:3, 32:9, 33:1, 33:3, 34:1, 34:3, 35:1, 35:3, 35:9, 36:1, 37:1, 38:1, 39:1, 39:3. Pending: Targets: [ 0.86692323 0.13500029 -0.18456116 2.83960341 1.59053593 3.09229027 2.66019892 0.94283067 0.36118756 -0.03995761 -0.27679346 -0.41117368 3.07567902 -0.31668339 -0.60551318 -0.77970435 1.14642262 -0.48892149 -0.6800339 -0.83804022 -0.17839844 -0.671017 -0.9427036 -0.19007746 -0.6381373 -0.81721138 0.08168551 -0.24151808 -0.72681605 -0.72606829 -0.39037965 -0.76704943 -0.88689891 -0.27941513 -0.70292112 -0.93419744 0.24441499 2.46654913 0.15100248 -0.11569359 2.32961031 1.16218149 -0.26790818 -0.6511265 0.03393215 0.08649291 -0.0101183 -0.06354679 0.6580365 -0.21585555 -0.67549934 -0.0920177 -0.52555215 -0.33843173 -0.75277469 -0.93709362 -0.29878193 -0.72911815 -0.92513517 -0.21871908 -0.38727526 -0.33618205 -0.6664905 -0.36106086 -0.69786243 -0.91252717 -0.19130005 0.05698679 1.145595 -0.36963345 -0.67326321] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.10341476367436149, 'kernel_inv_bw1': 0.5983479644078403, 'kernel_inv_bw2': 0.4179785036924045, 'kernel_inv_bw3': 3.91803420329974, 'kernel_inv_bw4': 0.4149104350412135, 'kernel_inv_bw5': 0.7649108916443379, 'kernel_inv_bw6': 1.610976592213731, 'kernel_inv_bw7': 0.002604672903894858, 'kernel_inv_bw8': 0.800340076458261, 'kernel_covariance_scale': 1.363105011961438, 'mean_mean_value': 1.8215236543541617} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.738302 ; Training: accuracy=0.602371 ; Validation: accuracy=0.794583 .. parsed-literal:: :class: output Update for config_id 40:1: reward = 0.7945827725437415, crit_val = 0.20541722745625846 config_id 40: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 7.337619 ; Training: accuracy=0.734251 ; Validation: accuracy=0.816117 Epoch 3 ; Time: 10.956102 ; Training: accuracy=0.763843 ; Validation: accuracy=0.851279 .. parsed-literal:: :class: output config_id 40: Terminating evaluation at 3 Update for config_id 40:3: reward = 0.851278600269179, crit_val = 0.148721399730821 Analysing the results ~~~~~~~~~~~~~~~~~~~~~ The training history is stored in the ``results_df``, the main fields are the runtime and ``'best'`` (the objective). **Note**: You will get slightly different curves for different pairs of scheduler/searcher, the ``time_out`` here is a bit too short to really see the difference in a significant way (it would be better to set it to >1000s). Generally speaking though, hyperband stopping / promotion + model will tend to significantly outperform other combinations given enough time. .. code:: python results_df.head() .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) .. raw:: html
bracket elapsed_time epoch error eval_time objective runtime searcher_data_size searcher_params_kernel_covariance_scale searcher_params_kernel_inv_bw0 ... searcher_params_kernel_inv_bw7 searcher_params_kernel_inv_bw8 searcher_params_mean_mean_value searcher_params_noise_variance target_epoch task_id time_since_start time_step time_this_iter best
0 0 0.484257 1 0.468750 0.479386 0.531250 0.571462 NaN 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.573196 1.603831e+09 0.513793 0.468750
1 0 0.925274 2 0.344753 0.434060 0.655247 1.012479 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.017026 1.603831e+09 0.441000 0.344753
2 0 1.347351 3 0.305314 0.416302 0.694686 1.434556 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.435394 1.603831e+09 0.422076 0.305314
3 0 1.773053 4 0.288937 0.421576 0.711063 1.860258 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.861271 1.603831e+09 0.425702 0.288937
4 0 2.192330 5 0.273061 0.417183 0.726939 2.279535 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 2.280613 1.603832e+09 0.419278 0.273061

5 rows × 26 columns

.. code:: python import matplotlib.pyplot as plt plt.figure(figsize=(12, 8)) runtime = results_df['runtime'].values objective = results_df['best'].values plt.plot(runtime, objective, lw=2) plt.xticks(fontsize=12) plt.xlim(0, 120) plt.ylim(0, 0.5) plt.yticks(fontsize=12) plt.xlabel("Runtime [s]", fontsize=14) plt.ylabel("Objective", fontsize=14) .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_14/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) .. parsed-literal:: :class: output Text(0, 0.5, 'Objective') .. figure:: output_mlp_cb387f_18_2.png Diving Deeper ------------- Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at :ref:`sec_customstorch`). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful. Logging the Search Progress ~~~~~~~~~~~~~~~~~~~~~~~~~~~ First, it is a good idea in general to switch on ``debug_log``, which outputs useful information about the search progress. This is already done in the example above. The outputs show which configurations are chosen, stopped, or promoted. For BO and BOHB, a range of information is displayed for every ``get_config`` decision. This log output is very useful in order to figure out what is going on during the search. Configuring ``HyperbandScheduler`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The most important knobs to turn with ``HyperbandScheduler`` are ``max_t``, ``grace_period``, ``reduction_factor``, ``brackets``, and ``type``. The first three determine the rung levels at which stopping or promotion decisions are being made. - The maximum resource level ``max_t`` (usually, resource equates to epochs, so ``max_t`` is the maximum number of training epochs) is typically hardcoded in ``train_fn`` passed to the scheduler (this is ``run_mlp_openml`` in the example above). As already noted above, the value is best fixed in the ``ag.args`` decorator as ``epochs=XYZ``, it can then be accessed as ``args.epochs`` in the ``train_fn`` code. If this is done, you do not have to pass ``max_t`` when creating the scheduler. - ``grace_period`` and ``reduction_factor`` determine the rung levels, which are ``grace_period``, ``grace_period * reduction_factor``, ``grace_period * (reduction_factor ** 2)``, etc. All rung levels must be less or equal than ``max_t``. It is recommended to make ``max_t`` equal to the largest rung level. For example, if ``grace_period = 1``, ``reduction_factor = 3``, it is in general recommended to use ``max_t = 9``, ``max_t = 27``, or ``max_t = 81``. Choosing a ``max_t`` value "off the grid" works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you set ``max_t = 10``, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch. - With ``reduction_factor``, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used. - Finally, ``grace_period`` should be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. While ``grace_period = 1`` should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung. - ``brackets`` sets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). For ``brackets = 1``, you are running successive halving (single bracket). Higher brackets have larger effective ``grace_period`` values (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (``brackets = 1``) in a comparison. - Finally, with ``type`` (values ``stopping``, ``promotion``) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the default ``stopping`` is simpler and seems to perform well, but ``promotion`` is more careful promoting configurations to higher resource levels, which can work better in some cases. Asynchronous BOHB ~~~~~~~~~~~~~~~~~ Finally, here are some ideas for tuning asynchronous BOHB, apart from tuning its ``HyperbandScheduling`` component. You need to pass these options in ``search_options``. - We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by ``gp_resource_kernel``, values are ``matern52``, ``matern52-res-warp``, ``exp-decay-sum``, ``exp-decay-combined``, ``exp-decay-delta1``. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found `here `__. - Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper ``train_fn`` and find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the options ``opt_skip_period`` and ``opt_skip_num_max_resource``. The basic idea is as follows. By far the most expensive part of a ``get_config`` call (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for most ``get_config`` calls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers -- we would love to learn about your use case.