.. _sec_custom_advancedhpo: Getting started with Advanced HPO Algorithms ============================================ Loading libraries ----------------- .. code:: python # Basic utils for folder manipulations etc import time import multiprocessing # to count the number of CPUs available # External tools to load and process data import numpy as np import pandas as pd # MXNet (NeuralNets) import mxnet as mx from mxnet import gluon, autograd from mxnet.gluon import nn # AutoGluon and HPO tools import autogluon as ag from autogluon.utils import load_and_split_openml_data Check the version of MxNet, you should be fine with version >= 1.5 .. code:: python mx.__version__ .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_15/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) .. parsed-literal:: :class: output '1.7.0' You can also check the version of AutoGluon and the specific commit and check that it matches what you want. .. code:: python ag.__version__ .. parsed-literal:: :class: output '0.0.15b20201208' Hyperparameter Optimization of a 2-layer MLP -------------------------------------------- Setting up the context ~~~~~~~~~~~~~~~~~~~~~~ Here we declare a few "environment variables" setting the context for what we're doing .. code:: python OPENML_TASK_ID = 6 # describes the problem we will tackle RATIO_TRAIN_VALID = 0.33 # split of the training data used for validation RESOURCE_ATTR_NAME = 'epoch' # how do we measure resources (will become clearer further) REWARD_ATTR_NAME = 'objective' # how do we measure performance (will become clearer further) NUM_CPUS = multiprocessing.cpu_count() Preparing the data ~~~~~~~~~~~~~~~~~~ We will use a multi-way classification task from OpenML. Data preparation includes: - Missing values are imputed, using the 'mean' strategy of ``sklearn.impute.SimpleImputer`` - Split training set into training and validation - Standardize inputs to mean 0, variance 1 .. code:: python X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data( OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False) n_classes .. parsed-literal:: :class: output Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff... 100%|██████████| 704/704 [00:00<00:00, 35778.82KB/s] Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3... 100%|██████████| 2521/2521 [00:00<00:00, 18565.58KB/s] Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml... 3KB [00:00, 3819.95KB/s] Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml... 8KB [00:00, 9165.37KB/s] Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml... 15KB [00:00, 15427.80KB/s] Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff... 2998KB [00:00, 19865.09KB/s] Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3... 881KB [00:00, 45336.32KB/s] Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml... 3KB [00:00, 3311.29KB/s] pickle load data letter .. parsed-literal:: :class: output 26 The problem has 26 classes. Declaring a model specifying a hyperparameter space with AutoGluon ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Two layer MLP where we optimize over: - the number of units on the first layer - the number of units on the second layer - the dropout rate after each layer - the learning rate - the scaling - the ``@ag.args`` decorator allows us to specify the space we will optimize over, this matches the `ConfigSpace `__ syntax The body of the function ``run_mlp_openml`` is pretty simple: - it reads the hyperparameters given via the decorator - it defines a 2 layer MLP with dropout - it declares a trainer with the 'adam' loss function and a provided learning rate - it trains the NN with a number of epochs (most of that is boilerplate code from ``mxnet``) - the ``reporter`` at the end is used to keep track of training history in the hyperparameter optimization **Note**: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment .. code:: python @ag.args(n_units_1=ag.space.Int(lower=16, upper=128), n_units_2=ag.space.Int(lower=16, upper=128), dropout_1=ag.space.Real(lower=0, upper=.75), dropout_2=ag.space.Real(lower=0, upper=.75), learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True), batch_size=ag.space.Int(lower=8, upper=128), scale_1=ag.space.Real(lower=0.001, upper=10, log=True), scale_2=ag.space.Real(lower=0.001, upper=10, log=True), epochs=9) def run_mlp_openml(args, reporter, **kwargs): # Time stamp for elapsed_time ts_start = time.time() # Unwrap hyperparameters n_units_1 = args.n_units_1 n_units_2 = args.n_units_2 dropout_1 = args.dropout_1 dropout_2 = args.dropout_2 scale_1 = args.scale_1 scale_2 = args.scale_2 batch_size = args.batch_size learning_rate = args.learning_rate ctx = mx.cpu() net = nn.Sequential() with net.name_scope(): # Layer 1 net.add(nn.Dense(n_units_1, activation='relu', weight_initializer=mx.initializer.Uniform(scale=scale_1))) # Dropout net.add(gluon.nn.Dropout(dropout_1)) # Layer 2 net.add(nn.Dense(n_units_2, activation='relu', weight_initializer=mx.initializer.Uniform(scale=scale_2))) # Dropout net.add(gluon.nn.Dropout(dropout_2)) # Output net.add(nn.Dense(n_classes)) net.initialize(ctx=ctx) trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate': learning_rate}) for epoch in range(args.epochs): ts_epoch = time.time() train_iter = mx.io.NDArrayIter( data={'data': X_train}, label={'label': y_train}, batch_size=batch_size, shuffle=True) valid_iter = mx.io.NDArrayIter( data={'data': X_valid}, label={'label': y_valid}, batch_size=batch_size, shuffle=False) metric = mx.metric.Accuracy() loss = gluon.loss.SoftmaxCrossEntropyLoss() for batch in train_iter: data = batch.data[0].as_in_context(ctx) label = batch.label[0].as_in_context(ctx) with autograd.record(): output = net(data) L = loss(output, label) L.backward() trainer.step(data.shape[0]) metric.update([label], [output]) name, train_acc = metric.get() metric = mx.metric.Accuracy() for batch in valid_iter: data = batch.data[0].as_in_context(ctx) label = batch.label[0].as_in_context(ctx) output = net(data) metric.update([label], [output]) name, val_acc = metric.get() print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % ( epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc)) ts_now = time.time() eval_time = ts_now - ts_epoch elapsed_time = ts_now - ts_start # The resource reported back (as 'epoch') is the number of epochs # done, starting at 1 reporter( epoch=epoch + 1, objective=float(val_acc), eval_time=eval_time, time_step=ts_now, elapsed_time=elapsed_time) .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_15/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) **Note**: The annotation ``epochs=9`` specifies the maximum number of epochs for training. It becomes available as ``args.epochs``. Importantly, it is also processed by ``HyperbandScheduler`` below in order to set its ``max_t`` attribute. **Recommendation**: Whenever writing training code to be passed as ``train_fn`` to a scheduler, if this training code reports a resource (or time) attribute, the corresponding maximum resource value should be included in ``train_fn.args``: - If the resource attribute (``time_attr`` of scheduler) in ``train_fn`` is ``epoch``, make sure to include ``epochs=XYZ`` in the annotation. This allows the scheduler to read ``max_t`` from ``train_fn.args.epochs``. This case corresponds to our example here. - If the resource attribute is something else than ``epoch``, you can also include the annotation ``max_t=XYZ``, which allows the scheduler to read ``max_t`` from ``train_fn.args.max_t``. Annotating the training function by the correct value for ``max_t`` simplifies scheduler creation (since ``max_t`` does not have to be passed), and avoids inconsistencies between ``train_fn`` and the scheduler. Running the Hyperparameter Optimization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can use the following schedulers: - FIFO (``fifo``) - Hyperband (either the stopping (``hbs``) or promotion (``hbp``) variant) And the following searchers: - Random search (``random``) - Gaussian process based Bayesian optimization (``bayesopt``) - SkOpt Bayesian optimization (``skopt``; only with FIFO scheduler) Note that the method known as (asynchronous) Hyperband is using random search. Combining Hyperband scheduling with the ``bayesopt`` searcher uses a novel method called asynchronous BOHB. Pick the combination you're interested in (doing the full experiment takes around 120 seconds, see the ``time_out`` parameter), running everything with multiple runs can take a fair bit of time. In real life, you will want to choose a larger ``time_out`` in order to obtain good performance. .. code:: python SCHEDULER = "hbs" SEARCHER = "bayesopt" .. code:: python def compute_error(df): return 1.0 - df["objective"] def compute_runtime(df, start_timestamp): return df["time_step"] - start_timestamp def process_training_history(task_dicts, start_timestamp, runtime_fn=compute_runtime, error_fn=compute_error): task_dfs = [] for task_id in task_dicts: task_df = pd.DataFrame(task_dicts[task_id]) task_df = task_df.assign(task_id=task_id, runtime=runtime_fn(task_df, start_timestamp), error=error_fn(task_df), target_epoch=task_df["epoch"].iloc[-1]) task_dfs.append(task_df) result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True) # re-order by runtime result = result.sort_values(by="runtime") # calculate incumbent best -- the cumulative minimum of the error. result = result.assign(best=result["error"].cummin()) return result resources = dict(num_cpus=NUM_CPUS, num_gpus=0) .. code:: python search_options = { 'num_init_random': 2, 'debug_log': True} if SCHEDULER == 'fifo': myscheduler = ag.scheduler.FIFOScheduler( run_mlp_openml, resource=resources, searcher=SEARCHER, search_options=search_options, time_out=120, time_attr=RESOURCE_ATTR_NAME, reward_attr=REWARD_ATTR_NAME) else: # This setup uses rung levels at 1, 3, 9 epochs. We just use a single # bracket, so this is in fact successive halving (Hyperband would use # more than 1 bracket). # Also note that since we do not use the max_t argument of # HyperbandScheduler, this value is obtained from train_fn.args.epochs. sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion' myscheduler = ag.scheduler.HyperbandScheduler( run_mlp_openml, resource=resources, searcher=SEARCHER, search_options=search_options, time_out=120, time_attr=RESOURCE_ATTR_NAME, reward_attr=REWARD_ATTR_NAME, type=sch_type, grace_period=1, reduction_factor=3, brackets=1) # run tasks myscheduler.run() myscheduler.join_jobs() results_df = process_training_history( myscheduler.training_history.copy(), start_timestamp=myscheduler._start_time) .. parsed-literal:: :class: output max_t = 9, as inferred from train_fn.args scheduler_options: Key 'resume': Imputing default value False scheduler_options: Key 'keep_size_ratios': Imputing default value False scheduler_options: Key 'maxt_pending': Imputing default value False scheduler_options: Key 'searcher_data': Imputing default value rungs scheduler_options: Key 'do_snapshots': Imputing default value False scheduler_options: Key 'visualizer': Imputing default value none scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60 scheduler_options: Key 'delay_get_config': Imputing default value True search_options: Key 'random_seed': Imputing default value 6623 search_options: Key 'opt_skip_init_length': Imputing default value 150 search_options: Key 'opt_skip_period': Imputing default value 1 search_options: Key 'profiler': Imputing default value False search_options: Key 'opt_maxiter': Imputing default value 50 search_options: Key 'opt_nstarts': Imputing default value 2 search_options: Key 'opt_warmstart': Imputing default value False search_options: Key 'opt_verbose': Imputing default value False search_options: Key 'opt_debug_writer': Imputing default value False search_options: Key 'num_fantasy_samples': Imputing default value 20 search_options: Key 'num_init_candidates': Imputing default value 250 search_options: Key 'initial_scoring': Imputing default value thompson_indep search_options: Key 'first_is_default': Imputing default value True search_options: Key 'opt_skip_num_max_resource': Imputing default value False search_options: Key 'gp_resource_kernel': Imputing default value matern52 search_options: Key 'resource_acq': Imputing default value bohb [GPMultiFidelitySearcher.__init__] - acquisition_class = - local_minimizer_class = - num_initial_candidates = 250 - num_initial_random_choices = 2 - initial_scoring = thompson_indep - first_is_default = True Starting Experiments Num of Finished Tasks is 0 Time out (secs) is 120 Starting get_config[random] for config_id 0 Start with default config: {'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1} [0: random] batch_size: 68 dropout_1: 0.375 dropout_2: 0.375 learning_rate: 0.001 n_units_1: 72 n_units_2: 72 scale_1: 0.1 scale_2: 0.1 /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_15/lib/python3.7/site-packages/distributed/worker.py:3382: UserWarning: Large object of size 1.30 MB detected in task graph: (, {'ar ... sReporter}, []) Consider scattering large objects ahead of time with client.scatter to reduce scheduler burden and keep data on workers future = client.submit(func, big_data) # bad big_future = client.scatter(big_data) # good future = client.submit(func, big_future) # good % (format_bytes(len(b)), s) .. parsed-literal:: :class: output Epoch 1 ; Time: 0.487472 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250 .. parsed-literal:: :class: output Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875 config_id 0: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.926579 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247 Epoch 3 ; Time: 1.356268 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686 .. parsed-literal:: :class: output Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464 config_id 0: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.845146 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063 Epoch 5 ; Time: 2.314168 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939 Epoch 6 ; Time: 2.740413 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321 Epoch 7 ; Time: 3.177248 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501 Epoch 8 ; Time: 3.643309 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202 Epoch 9 ; Time: 4.067426 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043 .. parsed-literal:: :class: output config_id 0: Terminating evaluation at 9 Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695 Starting get_config[random] for config_id 1 [1: random] batch_size: 115 dropout_1: 0.20259011582642464 dropout_2: 0.733834734066138 learning_rate: 0.05543264041163712 n_units_1: 16 n_units_2: 98 scale_1: 0.002572165463098296 scale_2: 0.10173245902918794 .. parsed-literal:: :class: output Epoch 1 ; Time: 0.329100 ; Training: accuracy=0.229979 ; Validation: accuracy=0.439465 .. parsed-literal:: :class: output config_id 1: Terminating evaluation at 1 Update for config_id 1:1: reward = 0.4394648829431438, crit_val = 0.5605351170568562 Starting get_config[BO] for config_id 2 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.39213912685779695 - self.std = 0.12921544434693305 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 4 Current best is [0.46875] [2: BO] (31 evaluations) batch_size: 8 dropout_1: 0.75 dropout_2: 0.0 learning_rate: 1.0000000000000004e-06 n_units_1: 104 n_units_2: 16 scale_1: 5.268003346484092 scale_2: 10.0 Started BO from (top scorer): batch_size: 121 dropout_1: 0.5574991411739321 dropout_2: 0.6004867652907465 learning_rate: 1.1442441844469983e-05 n_units_1: 104 n_units_2: 74 scale_1: 5.1796125582255845 scale_2: 4.062152221968767 Top score values: [0.04288078 0.09683621 0.10506474 0.16396388 0.16578847] Labeled: 0:1, 0:3, 0:9, 1:1. Pending: Targets: [ 0.59289254 -0.67193946 -1.22417183 1.30321875] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.6499287450947893, 'kernel_inv_bw1': 0.7868267238572046, 'kernel_inv_bw2': 0.6968644163688983, 'kernel_inv_bw3': 1.5741271749031467, 'kernel_inv_bw4': 0.006439399118472123, 'kernel_inv_bw5': 1.263427319135036, 'kernel_inv_bw6': 0.016976849029310993, 'kernel_inv_bw7': 7.446843139170003, 'kernel_inv_bw8': 77.487094491379, 'kernel_covariance_scale': 0.6952346587872724, 'mean_mean_value': -0.20198178577073855} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.706185 ; Training: accuracy=0.030670 ; Validation: accuracy=0.020525 .. parsed-literal:: :class: output config_id 2: Terminating evaluation at 1 Update for config_id 2:1: reward = 0.020524899057873486, crit_val = 0.9794751009421265 Starting get_config[BO] for config_id 3 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5096063216746629 - self.std = 0.26182336083473223 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 5 Current best is [0.46875] [3: BO] (1 evaluations) batch_size: 96 dropout_1: 0.5781081746991475 dropout_2: 0.72091413345427 learning_rate: 1.61137399838547e-06 n_units_1: 109 n_units_2: 22 scale_1: 0.008727376434682146 scale_2: 0.2756687103347264 Started BO from (top scorer): batch_size: 96 dropout_1: 0.5781081746991475 dropout_2: 0.72091413345427 learning_rate: 1.61137399838547e-06 n_units_1: 109 n_units_2: 22 scale_1: 0.008727376434682146 scale_2: 0.2756687103347264 Top score values: [-0.25782355 -0.02287811 -0.01564609 0.05535007 0.13022974] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending: Targets: [-0.15604536 -0.78026709 -1.05280561 0.19451586 1.7946022 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 1.3494943628982463, 'kernel_inv_bw1': 0.5787082424900308, 'kernel_inv_bw2': 1.0506610825670117, 'kernel_inv_bw3': 4.215044918879378, 'kernel_inv_bw4': 13.290378704778576, 'kernel_inv_bw5': 0.45832863654556477, 'kernel_inv_bw6': 100.00000000000004, 'kernel_inv_bw7': 0.21774850431320444, 'kernel_inv_bw8': 1.5400558722296638, 'kernel_covariance_scale': 0.8098233651957109, 'mean_mean_value': 0.29870471837022616} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.405849 ; Training: accuracy=0.048611 ; Validation: accuracy=0.081989 .. parsed-literal:: :class: output config_id 3: Terminating evaluation at 1 Update for config_id 3:1: reward = 0.08198924731182795, crit_val = 0.918010752688172 Starting get_config[BO] for config_id 4 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5776737268435811 - self.std = 0.2833585804791231 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 6 Current best is [0.46875] [4: BO] (2 evaluations) batch_size: 114 dropout_1: 0.02102008918174078 dropout_2: 0.7424384878196432 learning_rate: 0.009540252459329624 n_units_1: 77 n_units_2: 60 scale_1: 4.513630047489416 scale_2: 2.5774596161227126 Started BO from (top scorer): batch_size: 114 dropout_1: 0.02115203449262079 dropout_2: 0.7424374090277859 learning_rate: 0.009516071128311068 n_units_1: 77 n_units_2: 60 scale_1: 4.510177647703899 scale_2: 0.4090186411843695 Top score values: [-0.09495753 -0.059177 0.15157438 0.15623764 0.17638331] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending: Targets: [-0.38440243 -0.96118337 -1.213009 -0.06048382 1.41799614 1.20108248] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00791939840556937, 'kernel_inv_bw1': 0.4134976030920574, 'kernel_inv_bw2': 0.19020975634901863, 'kernel_inv_bw3': 0.45927321133324434, 'kernel_inv_bw4': 0.21024309992401136, 'kernel_inv_bw5': 0.5639434738758471, 'kernel_inv_bw6': 0.2973090901522113, 'kernel_inv_bw7': 58.02742522405625, 'kernel_inv_bw8': 1.3178422777681582, 'kernel_covariance_scale': 0.8149392022165727, 'mean_mean_value': 0.4540365480466013} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.321782 ; Training: accuracy=0.167825 ; Validation: accuracy=0.421384 .. parsed-literal:: :class: output config_id 4: Terminating evaluation at 1 Update for config_id 4:1: reward = 0.42138364779874216, crit_val = 0.5786163522012578 Starting get_config[BO] for config_id 5 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5778083876089635 - self.std = 0.26233927661788736 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 7 Current best is [0.46875] [5: BO] (8 evaluations) batch_size: 106 dropout_1: 0.0 dropout_2: 0.19489793058292 learning_rate: 0.0001917164114589211 n_units_1: 32 n_units_2: 62 scale_1: 0.8743649345893009 scale_2: 0.011718758917625331 Started BO from (top scorer): batch_size: 106 dropout_1: 0.027455281142636845 dropout_2: 0.1947445824338346 learning_rate: 0.00019353840383486696 n_units_1: 32 n_units_2: 62 scale_1: 0.8743650199122025 scale_2: 0.011718825652088465 Top score values: [0.23310624 0.24388219 0.2489686 0.26519887 0.28645084] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1. Pending: Targets: [-0.41571506 -1.03870919 -1.31071173 -0.06584325 1.53109637 1.29680302 0.00307985] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0011228852506329462, 'kernel_inv_bw1': 0.5389755119780264, 'kernel_inv_bw2': 0.035642439357343174, 'kernel_inv_bw3': 3.4290743735425813, 'kernel_inv_bw4': 0.0009578091245968871, 'kernel_inv_bw5': 0.0071188337593711135, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0006415358974401202, 'kernel_inv_bw8': 97.96343416545665, 'kernel_covariance_scale': 0.7653084445327711, 'mean_mean_value': -0.2651390598902953} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.370475 ; Training: accuracy=0.229229 ; Validation: accuracy=0.304038 .. parsed-literal:: :class: output config_id 5: Terminating evaluation at 1 Update for config_id 5:1: reward = 0.30403839788149617, crit_val = 0.6959616021185038 Starting get_config[BO] for config_id 6 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.592577539422656 - self.std = 0.24848753281888844 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 8 Current best is [0.46875] [6: BO] (8 evaluations) batch_size: 16 dropout_1: 0.3640756919610413 dropout_2: 0.5718969782781318 learning_rate: 0.0013273787481107593 n_units_1: 128 n_units_2: 100 scale_1: 1.5323135545345714 scale_2: 0.2711460391616208 Started BO from (top scorer): batch_size: 16 dropout_1: 0.36407558466849665 dropout_2: 0.5718969773157769 learning_rate: 0.0013273802663397456 n_units_1: 122 n_units_2: 110 scale_1: 1.5323510833732292 scale_2: 0.27114866856161 Top score values: [0.03998227 0.11977853 0.13245333 0.14247006 0.1801681 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1. Pending: Targets: [-0.49832496 -1.15604741 -1.44321253 -0.12894982 1.55700995 1.3096561 -0.05618466 0.41605332] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0006993602391388139, 'kernel_inv_bw1': 0.0002486573595518353, 'kernel_inv_bw2': 0.0001318698763768803, 'kernel_inv_bw3': 0.00031063667937819903, 'kernel_inv_bw4': 0.8479717587157058, 'kernel_inv_bw5': 1.7456862096511072, 'kernel_inv_bw6': 0.0007826063528641171, 'kernel_inv_bw7': 0.0006299031529609759, 'kernel_inv_bw8': 1.273414833431511, 'kernel_covariance_scale': 1.3373236492243967, 'mean_mean_value': 0.21836461194132425} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.854029 ; Training: accuracy=0.336124 ; Validation: accuracy=0.668683 .. parsed-literal:: :class: output Update for config_id 6:1: reward = 0.6686827956989247, crit_val = 0.33131720430107525 config_id 6: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 3.640819 ; Training: accuracy=0.500746 ; Validation: accuracy=0.728663 Epoch 3 ; Time: 5.410069 ; Training: accuracy=0.554874 ; Validation: accuracy=0.737735 .. parsed-literal:: :class: output Update for config_id 6:3: reward = 0.7377352150537635, crit_val = 0.2622647849462365 config_id 6: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 7.642351 ; Training: accuracy=0.601790 ; Validation: accuracy=0.779402 Epoch 5 ; Time: 9.450804 ; Training: accuracy=0.618369 ; Validation: accuracy=0.796875 Epoch 6 ; Time: 11.364773 ; Training: accuracy=0.639920 ; Validation: accuracy=0.812332 Epoch 7 ; Time: 13.246381 ; Training: accuracy=0.656582 ; Validation: accuracy=0.818716 Epoch 8 ; Time: 15.031076 ; Training: accuracy=0.672580 ; Validation: accuracy=0.838710 Epoch 9 ; Time: 16.789679 ; Training: accuracy=0.683107 ; Validation: accuracy=0.830981 .. parsed-literal:: :class: output config_id 6: Terminating evaluation at 9 Update for config_id 6:9: reward = 0.8309811827956989, crit_val = 0.16901881720430112 Starting get_config[BO] for config_id 7 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5002928292575328 - self.std = 0.26234133843029117 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 11 Current best is [0.3313172] [7: BO] (4 evaluations) batch_size: 12 dropout_1: 0.6745038543033849 dropout_2: 0.5392565199404769 learning_rate: 0.8563314945282694 n_units_1: 128 n_units_2: 128 scale_1: 0.00941255860057999 scale_2: 0.2646064285412933 Started BO from (top scorer): batch_size: 12 dropout_1: 0.6745027588016371 dropout_2: 0.5392565564737246 learning_rate: 0.8563260354947063 n_units_1: 84 n_units_2: 105 scale_1: 0.00942347303672847 scale_2: 0.2646066115886901 Top score values: [0.18302057 0.20010663 0.21453925 0.26892143 0.26953059] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9. Pending: Targets: [-0.12023583 -0.74322506 -1.01522548 0.22963323 1.82656029 1.59226878 0.29855578 0.74585566 -0.64410598 -0.90732191 -1.26275948] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00046721381575460976, 'kernel_inv_bw1': 0.00027945416294156703, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 0.000361818016098626, 'kernel_inv_bw4': 0.8054995260303396, 'kernel_inv_bw5': 1.8102707809066103, 'kernel_inv_bw6': 0.002517693024440981, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.114599391234888, 'kernel_covariance_scale': 1.0980804092858671, 'mean_mean_value': 0.48770984232856585} .. parsed-literal:: :class: output Epoch 1 ; Time: 2.948995 ; Training: accuracy=0.038806 ; Validation: accuracy=0.037879 .. parsed-literal:: :class: output config_id 7: Terminating evaluation at 1 Update for config_id 7:1: reward = 0.03787878787878788, crit_val = 0.9621212121212122 Starting get_config[BO] for config_id 8 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5387785278295061 - self.std = 0.281745219365974 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 12 Current best is [0.33131721] [8: BO] (16 evaluations) batch_size: 8 dropout_1: 0.2659272127964557 dropout_2: 0.6900363068623963 learning_rate: 0.005809894190138349 n_units_1: 128 n_units_2: 95 scale_1: 6.514260009328389 scale_2: 0.0407308493761842 Started BO from (top scorer): batch_size: 100 dropout_1: 0.10188329107858729 dropout_2: 0.6900361830485076 learning_rate: 0.005808309193037013 n_units_1: 126 n_units_2: 95 scale_1: 6.514197760883919 scale_2: 0.04073063082851759 Top score values: [0.1325505 0.15233528 0.191637 0.22464727 0.23966581] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1. Pending: Targets: [-0.24855267 -0.82863644 -1.0819041 0.07722079 1.56416699 1.34601121 0.14139663 0.55789083 -0.73634372 -0.98143189 -1.31239036 1.50257273] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.06701545817196791, 'kernel_inv_bw1': 3.1620401434912626, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 0.0005111348651985465, 'kernel_inv_bw4': 0.707297536733937, 'kernel_inv_bw5': 0.002050806496292287, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.5509672657087004, 'kernel_covariance_scale': 0.802545891685748, 'mean_mean_value': 0.2111618055911085} .. parsed-literal:: :class: output Epoch 1 ; Time: 4.405943 ; Training: accuracy=0.098226 ; Validation: accuracy=0.160162 .. parsed-literal:: :class: output config_id 8: Terminating evaluation at 1 Update for config_id 8:1: reward = 0.1601615074024226, crit_val = 0.8398384925975774 Starting get_config[BO] for config_id 9 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5619369866578192 - self.std = 0.2823295185000604 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 13 Current best is [0.3313172] [9: BO] (1 evaluations) batch_size: 22 dropout_1: 0.6060180058908871 dropout_2: 0.3694251698271682 learning_rate: 0.21464029125308434 n_units_1: 16 n_units_2: 52 scale_1: 0.0018357925258620815 scale_2: 1.462784943068434 Started BO from (top scorer): batch_size: 22 dropout_1: 0.6060180058908871 dropout_2: 0.3694251698271682 learning_rate: 0.21464029125308434 n_units_1: 16 n_units_2: 52 scale_1: 0.0018357925258620815 scale_2: 1.462784943068434 Top score values: [0.05141957 0.06756006 0.07343811 0.1288816 0.15526496] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1. Pending: Targets: [-0.33006463 -0.90894787 -1.16169138 -0.00496537 1.4789035 1.26119921 0.05907765 0.47470989 -0.81684616 -1.0614271 -1.39170063 1.41743672 0.98431615] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.12536897330209024, 'kernel_inv_bw1': 56.05143589172606, 'kernel_inv_bw2': 0.6891701888367768, 'kernel_inv_bw3': 99.97740071934261, 'kernel_inv_bw4': 0.07537196068684757, 'kernel_inv_bw5': 0.8049266978059991, 'kernel_inv_bw6': 0.2532172541571074, 'kernel_inv_bw7': 99.9650172872422, 'kernel_inv_bw8': 1.1172076352259694, 'kernel_covariance_scale': 0.7868042958143471, 'mean_mean_value': 0.35682399111693597} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.371782 ; Training: accuracy=0.038003 ; Validation: accuracy=0.038552 .. parsed-literal:: :class: output config_id 9: Terminating evaluation at 1 Update for config_id 9:1: reward = 0.03855218855218855, crit_val = 0.9614478114478114 Starting get_config[BO] for config_id 10 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5904734741428187 - self.std = 0.29086542110040536 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 14 Current best is [0.33131721] [10: BO] (38 evaluations) batch_size: 8 dropout_1: 0.39058248200567325 dropout_2: 0.75 learning_rate: 0.018352560186403126 n_units_1: 112 n_units_2: 114 scale_1: 0.3781195956947368 scale_2: 10.0 Started BO from (top scorer): batch_size: 67 dropout_1: 0.42180185164183004 dropout_2: 0.14768835668807428 learning_rate: 0.007382785030804872 n_units_1: 17 n_units_2: 107 scale_1: 0.23149492613519776 scale_2: 2.6437066234271045 Top score values: [-0.15693853 0.17195647 0.17419227 0.25711761 0.26617094] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1. Pending: Targets: [-0.41848726 -0.98038227 -1.22570862 -0.10292855 1.33739385 1.12607844 -0.04076498 0.36266988 -0.89098343 -1.12838676 -1.44896789 1.27773091 0.85732095 1.27541574] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.9537010450485282, 'kernel_inv_bw1': 12.897576685309335, 'kernel_inv_bw2': 0.046304772665198994, 'kernel_inv_bw3': 0.017375775199196232, 'kernel_inv_bw4': 0.03136432845599069, 'kernel_inv_bw5': 0.007643925695191919, 'kernel_inv_bw6': 0.007533859065587351, 'kernel_inv_bw7': 0.16990051031051623, 'kernel_inv_bw8': 1.3697360955805709, 'kernel_covariance_scale': 0.6491252043225703, 'mean_mean_value': 0.3329432783656786} .. parsed-literal:: :class: output Epoch 1 ; Time: 4.292739 ; Training: accuracy=0.085046 ; Validation: accuracy=0.073015 .. parsed-literal:: :class: output config_id 10: Terminating evaluation at 1 Update for config_id 10:1: reward = 0.07301480484522208, crit_val = 0.926985195154778 Starting get_config[BO] for config_id 11 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6129075888769493 - self.std = 0.29327217171161385 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 15 Current best is [0.33131721] [11: BO] (17 evaluations) batch_size: 8 dropout_1: 0.5200398610955415 dropout_2: 0.6746529820978835 learning_rate: 0.0007111511659553435 n_units_1: 20 n_units_2: 127 scale_1: 0.11925200634046369 scale_2: 1.416601084977511 Started BO from (top scorer): batch_size: 60 dropout_1: 0.6173446849627836 dropout_2: 0.6746530798671563 learning_rate: 1.8073792217344776e-05 n_units_1: 20 n_units_2: 128 scale_1: 0.11925197766089414 scale_2: 1.5592615231526614 Top score values: [0.2202283 0.23189342 0.23951484 0.25662481 0.30129138] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1. Pending: Targets: [-0.49154882 -1.04883261 -1.29214568 -0.17857975 1.24992259 1.04034134 -0.11692632 0.28319773 -0.96016742 -1.19562249 -1.51357276 1.19074927 0.77378942 1.1884531 1.07094241] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 1.4044370934439048, 'kernel_inv_bw1': 0.09921002427331542, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 9.22095360131539, 'kernel_inv_bw4': 0.004399620781082389, 'kernel_inv_bw5': 0.024535043977392212, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.033807460653923434, 'kernel_inv_bw8': 1.2036867934072941, 'kernel_covariance_scale': 0.6627478502737585, 'mean_mean_value': 0.37911881620584653} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.634546 ; Training: accuracy=0.187334 ; Validation: accuracy=0.562752 .. parsed-literal:: :class: output Update for config_id 11:1: reward = 0.5627523553162853, crit_val = 0.4372476446837147 config_id 11: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 7.186900 ; Training: accuracy=0.287550 ; Validation: accuracy=0.609354 Epoch 3 ; Time: 10.747527 ; Training: accuracy=0.324022 ; Validation: accuracy=0.646366 .. parsed-literal:: :class: output config_id 11: Terminating evaluation at 3 Update for config_id 11:3: reward = 0.6463660834454913, crit_val = 0.3536339165545087 Starting get_config[BO] for config_id 12 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5873232584936743 - self.std = 0.2846132844157344 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 17 Current best is [0.33131721] [12: BO] (10 evaluations) batch_size: 8 dropout_1: 0.7361153286837211 dropout_2: 0.3095766363524611 learning_rate: 3.845433057917802e-05 n_units_1: 98 n_units_2: 59 scale_1: 1.7405406580340526 scale_2: 0.5601206223681634 Started BO from (top scorer): batch_size: 12 dropout_1: 0.7361132626796868 dropout_2: 0.30957663579249306 learning_rate: 8.518737997663894e-06 n_units_1: 98 n_units_2: 59 scale_1: 1.7404698524703932 scale_2: 0.5601180658452514 Top score values: [0.11524898 0.22380164 0.30153974 0.31926266 0.3474178 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3. Pending: Targets: [-0.41661182 -0.99085005 -1.24156552 -0.09412119 1.37784096 1.16188355 -0.03059206 0.38170511 -0.89948737 -1.14210577 -1.46972915 1.31686739 0.88722223 1.31450137 1.19341561 -0.52729659 -0.82107672] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 1.3270225209850888, 'kernel_inv_bw1': 0.010397356922295495, 'kernel_inv_bw2': 0.0007221262406899736, 'kernel_inv_bw3': 10.500056136753114, 'kernel_inv_bw4': 0.07416146389339609, 'kernel_inv_bw5': 0.005070132310148894, 'kernel_inv_bw6': 0.0184692764104324, 'kernel_inv_bw7': 0.007977234368476308, 'kernel_inv_bw8': 1.170429723053323, 'kernel_covariance_scale': 0.6387803988286264, 'mean_mean_value': 0.5034450880907433} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.658152 ; Training: accuracy=0.038793 ; Validation: accuracy=0.056023 .. parsed-literal:: :class: output config_id 12: Terminating evaluation at 1 Update for config_id 12:1: reward = 0.056022880215343206, crit_val = 0.9439771197846568 Starting get_config[BO] for config_id 13 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6071373618987289 - self.std = 0.2884070684723663 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 18 Current best is [0.33131721] [13: BO] (50 evaluations) batch_size: 34 dropout_1: 0.0 dropout_2: 0.16531656565678315 learning_rate: 0.0015710151190437619 n_units_1: 16 n_units_2: 53 scale_1: 0.0010000000000000002 scale_2: 0.02515153076268347 Started BO from (top scorer): batch_size: 65 dropout_1: 0.14708276563581593 dropout_2: 0.17195192075723356 learning_rate: 0.0022913550028786415 n_units_1: 74 n_units_2: 57 scale_1: 0.33881181967715607 scale_2: 0.06287741099517685 Top score values: [0.24173118 0.32261171 0.32342942 0.35403063 0.37132952] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1. Pending: Targets: [-0.47983346 -1.04651801 -1.29393549 -0.16158496 1.29101461 1.07789796 -0.09889151 0.30798219 -0.95635713 -1.19578407 -1.51909781 1.2308431 0.80684961 1.2285082 1.10901524 -0.58906225 -0.87897792 1.1679317 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 1.3590575477767326, 'kernel_inv_bw1': 0.03936069903087651, 'kernel_inv_bw2': 0.0026984432414393494, 'kernel_inv_bw3': 10.296001231381487, 'kernel_inv_bw4': 0.04807502049559833, 'kernel_inv_bw5': 0.0075945343276932874, 'kernel_inv_bw6': 0.04822808438234503, 'kernel_inv_bw7': 0.02645952960098961, 'kernel_inv_bw8': 1.1509944808708752, 'kernel_covariance_scale': 0.6224491569300521, 'mean_mean_value': 0.5128076290185375} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.895618 ; Training: accuracy=0.285004 ; Validation: accuracy=0.484706 .. parsed-literal:: :class: output Update for config_id 13:1: reward = 0.48470588235294115, crit_val = 0.5152941176470589 config_id 13: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.719433 ; Training: accuracy=0.541591 ; Validation: accuracy=0.605882 Epoch 3 ; Time: 2.555429 ; Training: accuracy=0.613422 ; Validation: accuracy=0.655462 .. parsed-literal:: :class: output config_id 13: Terminating evaluation at 3 Update for config_id 13:3: reward = 0.6554621848739496, crit_val = 0.3445378151260504 Starting get_config[BO] for config_id 14 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5894152223475115 - self.std = 0.28002925628238085 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 20 Current best is [0.33131721] [14: BO] (11 evaluations) batch_size: 128 dropout_1: 0.3960821640083113 dropout_2: 0.042683254962728207 learning_rate: 0.0019572264290375644 n_units_1: 128 n_units_2: 67 scale_1: 0.0018443079577594394 scale_2: 0.21772892819706435 Started BO from (top scorer): batch_size: 96 dropout_1: 0.3960815012094535 dropout_2: 0.042683322497057974 learning_rate: 0.002198541885573249 n_units_1: 121 n_units_2: 67 scale_1: 0.0018447905151312408 scale_2: 0.21772889032173864 Top score values: [0.24839197 0.3059221 0.31899261 0.33188266 0.36762275] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3. Pending: Targets: [-0.4309022 -1.01454061 -1.26936024 -0.10313246 1.39292545 1.17343286 -0.03856336 0.38048303 -0.92168233 -1.16827235 -1.50125887 1.33095375 0.89427538 1.328549 1.20548109 -0.54339886 -0.84198812 1.26616019 -0.26469057 -0.8744708 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 1.1570959984938367, 'kernel_inv_bw1': 0.0013202869606903764, 'kernel_inv_bw2': 0.0003397239151057199, 'kernel_inv_bw3': 8.199525210323392, 'kernel_inv_bw4': 0.40806701165574927, 'kernel_inv_bw5': 0.0018460943947932122, 'kernel_inv_bw6': 0.005969169164110574, 'kernel_inv_bw7': 0.00019791409072134644, 'kernel_inv_bw8': 1.2666478406791895, 'kernel_covariance_scale': 0.701840781818308, 'mean_mean_value': 0.5419607149303785} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.293656 ; Training: accuracy=0.374836 ; Validation: accuracy=0.645778 .. parsed-literal:: :class: output Update for config_id 14:1: reward = 0.6457779255319149, crit_val = 0.35422207446808507 config_id 14: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.531325 ; Training: accuracy=0.634375 ; Validation: accuracy=0.724402 Epoch 3 ; Time: 0.780186 ; Training: accuracy=0.685197 ; Validation: accuracy=0.753657 .. parsed-literal:: :class: output Update for config_id 14:3: reward = 0.753656914893617, crit_val = 0.24634308510638303 config_id 14: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.024297 ; Training: accuracy=0.719819 ; Validation: accuracy=0.791722 Epoch 5 ; Time: 1.252722 ; Training: accuracy=0.738898 ; Validation: accuracy=0.804355 Epoch 6 ; Time: 1.483242 ; Training: accuracy=0.756908 ; Validation: accuracy=0.821975 Epoch 7 ; Time: 1.712579 ; Training: accuracy=0.769572 ; Validation: accuracy=0.833112 Epoch 8 ; Time: 1.942191 ; Training: accuracy=0.781497 ; Validation: accuracy=0.847573 Epoch 9 ; Time: 2.190483 ; Training: accuracy=0.788734 ; Validation: accuracy=0.855884 .. parsed-literal:: :class: output config_id 14: Terminating evaluation at 9 Update for config_id 14:9: reward = 0.8558843085106383, crit_val = 0.14411569148936165 Starting get_config[BO] for config_id 15 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5449124042614809 - self.std = 0.2869693324790331 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 23 Current best is [0.33131721] [15: BO] (20 evaluations) batch_size: 128 dropout_1: 0.5874699566298283 dropout_2: 0.019158429826698728 learning_rate: 0.0028143456348614813 n_units_1: 16 n_units_2: 65 scale_1: 0.007570194869186357 scale_2: 0.4176590013022225 Started BO from (top scorer): batch_size: 120 dropout_1: 0.5874698784709596 dropout_2: 0.019158458756236257 learning_rate: 0.004943278031654848 n_units_1: 45 n_units_2: 65 scale_1: 0.007570194707280391 scale_2: 0.41765883900238027 Top score values: [0.33024692 0.36369566 0.37554193 0.37856652 0.41267252] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9. Pending: Targets: [-0.26540259 -0.83492627 -1.08358333 0.05444036 1.51431755 1.30013317 0.11744791 0.52636007 -0.74431368 -0.98494016 -1.30987372 1.45384458 1.02772685 1.45149798 1.33140635 -0.37517863 -0.66654679 1.39061799 -0.10321063 -0.69824391 -0.66449724 -1.04042239 -1.39665347] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.8734634367949952, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 8.153205061445796, 'kernel_inv_bw4': 0.42861232208826666, 'kernel_inv_bw5': 0.00039202069260665104, 'kernel_inv_bw6': 0.0002390795781528369, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1579246089342603, 'kernel_covariance_scale': 0.7727887367465754, 'mean_mean_value': 0.5584941941324477} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.294466 ; Training: accuracy=0.152549 ; Validation: accuracy=0.472074 .. parsed-literal:: :class: output Update for config_id 15:1: reward = 0.4720744680851064, crit_val = 0.5279255319148937 config_id 15: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.537837 ; Training: accuracy=0.298109 ; Validation: accuracy=0.568816 Epoch 3 ; Time: 0.776367 ; Training: accuracy=0.353701 ; Validation: accuracy=0.629820 .. parsed-literal:: :class: output config_id 15: Terminating evaluation at 3 Update for config_id 15:3: reward = 0.6298204787234043, crit_val = 0.3701795212765957 Starting get_config[BO] for config_id 16 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.537243614048222 - self.std = 0.2773757036702802 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 25 Current best is [0.33131721] [16: BO] (19 evaluations) batch_size: 84 dropout_1: 0.29561829231685044 dropout_2: 0.04596460097673949 learning_rate: 5.724211615833766e-06 n_units_1: 16 n_units_2: 128 scale_1: 0.12674180899264345 scale_2: 0.05366256812884492 Started BO from (top scorer): batch_size: 84 dropout_1: 0.29561835745224185 dropout_2: 0.045964606867111624 learning_rate: 7.894147491592967e-06 n_units_1: 47 n_units_2: 89 scale_1: 0.1267418849373593 scale_2: 0.05726027896274499 Top score values: [0.30987005 0.3136461 0.31724314 0.32688824 0.32740173] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3. Pending: Targets: [-0.24693444 -0.8361563 -1.0934137 0.08397096 1.59434111 1.37274871 0.14915776 0.57221302 -0.74240969 -0.99135874 -1.32753083 1.53177655 1.09092063 1.52934879 1.40510353 -0.36050731 -0.66195307 1.46636313 -0.07913273 -0.6947465 -0.65983263 -1.04875995 -1.41731203 -0.03359372 -0.60230255] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.014383779703254456, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 8.832360400391776, 'kernel_inv_bw4': 1.3472447408705768, 'kernel_inv_bw5': 0.21165268977016086, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.03367028338858101, 'kernel_inv_bw8': 1.1091776477163353, 'kernel_covariance_scale': 0.895496207576915, 'mean_mean_value': 0.7364176704770193} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.407605 ; Training: accuracy=0.032242 ; Validation: accuracy=0.034205 .. parsed-literal:: :class: output config_id 16: Terminating evaluation at 1 Update for config_id 16:1: reward = 0.03420523138832998, crit_val = 0.96579476861167 Starting get_config[BO] for config_id 17 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5537263507622007 - self.std = 0.28420093250084527 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 26 Current best is [0.33131721] [17: BO] (23 evaluations) batch_size: 8 dropout_1: 0.42586366439452183 dropout_2: 0.35223538880391364 learning_rate: 0.0008740524364132574 n_units_1: 128 n_units_2: 115 scale_1: 0.003423700940392097 scale_2: 1.68546307271504 Started BO from (top scorer): batch_size: 11 dropout_1: 0.4258636765224537 dropout_2: 0.35223545110974847 learning_rate: 0.0008874104488487803 n_units_1: 69 n_units_2: 115 scale_1: 0.0034237046336428783 scale_2: 1.685459854482685 Top score values: [0.34829086 0.36348719 0.36570441 0.37972359 0.38632808] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1. Pending: Targets: [-0.29900096 -0.87407236 -1.12515159 0.02395758 1.49805543 1.28178468 0.08757889 0.50047426 -0.78257712 -1.02554754 -1.35364627 1.43699339 1.00672485 1.43462394 1.31336249 -0.40984632 -0.7040527 1.37315091 -0.13522909 -0.73605858 -0.70198319 -1.08157022 -1.44127134 -0.09078372 -0.64583472 1.4499193 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.7888083475148504, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.332875248743284, 'kernel_inv_bw4': 0.3771762783333212, 'kernel_inv_bw5': 0.0016658182432786528, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0002777776968666048, 'kernel_inv_bw8': 1.1887945351189257, 'kernel_covariance_scale': 0.8060677884471117, 'mean_mean_value': 0.5254497585338413} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.757869 ; Training: accuracy=0.485162 ; Validation: accuracy=0.740747 .. parsed-literal:: :class: output Update for config_id 17:1: reward = 0.7407469717362046, crit_val = 0.25925302826379537 config_id 17: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 7.342433 ; Training: accuracy=0.620441 ; Validation: accuracy=0.783479 Epoch 3 ; Time: 10.871896 ; Training: accuracy=0.666777 ; Validation: accuracy=0.805013 .. parsed-literal:: :class: output Update for config_id 17:3: reward = 0.8050134589502019, crit_val = 0.19498654104979807 config_id 17: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 14.472019 ; Training: accuracy=0.688578 ; Validation: accuracy=0.811406 Epoch 5 ; Time: 18.196518 ; Training: accuracy=0.701426 ; Validation: accuracy=0.815949 Epoch 6 ; Time: 21.747994 ; Training: accuracy=0.711207 ; Validation: accuracy=0.836137 Epoch 7 ; Time: 25.340872 ; Training: accuracy=0.725216 ; Validation: accuracy=0.839838 Epoch 8 ; Time: 29.057332 ; Training: accuracy=0.730355 ; Validation: accuracy=0.844549 Epoch 9 ; Time: 32.737144 ; Training: accuracy=0.740882 ; Validation: accuracy=0.851783 .. parsed-literal:: :class: output config_id 17: Terminating evaluation at 9 Update for config_id 17:9: reward = 0.8517833109017496, crit_val = 0.14821668909825036 Starting get_config[BO] for config_id 18 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5172186682147953 - self.std = 0.2901378851551728 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 29 Current best is [0.25925303] [18: BO] (25 evaluations) batch_size: 8 dropout_1: 0.6103372263926843 dropout_2: 0.75 learning_rate: 0.0006065386795024323 n_units_1: 128 n_units_2: 116 scale_1: 6.883514764953241 scale_2: 1.9951490582671303 Started BO from (top scorer): batch_size: 27 dropout_1: 0.6103372305742708 dropout_2: 0.7169201124332221 learning_rate: 0.0007434736984816291 n_units_1: 113 n_units_2: 116 scale_1: 6.883405660029565 scale_2: 1.9951489067176602 Top score values: [0.29140152 0.35405184 0.36892151 0.37652894 0.3779718 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 17:1, 17:3, 17:9. Pending: Targets: [-0.16705391 -0.7303579 -0.97629942 0.14929608 1.59323017 1.38138487 0.21161554 0.61606203 -0.64073488 -0.87873351 -1.20011853 1.53341762 1.11195346 1.53109665 1.41231652 -0.2756311 -0.56381727 1.47088151 -0.00663323 -0.59516824 -0.56179011 -0.93360984 -1.28595056 0.03690267 -0.50679058 1.54607903 -0.88911395 -1.11061721 -1.27181591] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.8212271192604788, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.1901591006465807, 'kernel_inv_bw3': 6.338459251879282, 'kernel_inv_bw4': 0.3895836008359322, 'kernel_inv_bw5': 0.0007520472230177056, 'kernel_inv_bw6': 0.0008055860353685261, 'kernel_inv_bw7': 0.00015286692670848332, 'kernel_inv_bw8': 1.0493843619970442, 'kernel_covariance_scale': 0.8464555264643454, 'mean_mean_value': 0.5698720264511019} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.634857 ; Training: accuracy=0.086621 ; Validation: accuracy=0.416218 .. parsed-literal:: :class: output config_id 18: Terminating evaluation at 1 Update for config_id 18:1: reward = 0.41621803499327054, crit_val = 0.5837819650067295 Starting get_config[BO] for config_id 19 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5194374447745264 - self.std = 0.2855113986670422 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 30 Current best is [0.25925303] [19: BO] (22 evaluations) batch_size: 57 dropout_1: 0.15094904546633964 dropout_2: 0.054486021482137714 learning_rate: 0.0012666545242820861 n_units_1: 59 n_units_2: 117 scale_1: 0.0010000000000000002 scale_2: 2.321079057850671 Started BO from (top scorer): batch_size: 57 dropout_1: 0.1509661326935938 dropout_2: 0.0544905189778816 learning_rate: 0.0018805444047928965 n_units_1: 59 n_units_2: 117 scale_1: 7.9386304987563845 scale_2: 3.675533754679733 Top score values: [0.15998046 0.16207085 0.24653442 0.30552353 0.3111285 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 17:1, 17:3, 17:9, 18:1. Pending: Targets: [-0.17753212 -0.74996401 -0.99989082 0.14394407 1.61127597 1.39599788 0.20727336 0.61827359 -0.65888872 -0.90074393 -1.22733673 1.5504942 1.12220055 1.54813562 1.42743075 -0.28786872 -0.58072472 1.48694475 -0.01451195 -0.6125837 -0.57866471 -0.95650948 -1.31455961 0.02972942 -0.52277396 1.56336078 -0.91129257 -1.13638512 -1.30019592 0.22536585] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00433111323493072, 'kernel_inv_bw1': 0.0024413041830750283, 'kernel_inv_bw2': 0.0009169872154755717, 'kernel_inv_bw3': 13.772939589553168, 'kernel_inv_bw4': 0.0006378068727762713, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.2946555693253991, 'kernel_inv_bw7': 1.9253478811176665, 'kernel_inv_bw8': 1.2453772959764744, 'kernel_covariance_scale': 0.7867773961362077, 'mean_mean_value': 0.8260903038304861} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.599440 ; Training: accuracy=0.531281 ; Validation: accuracy=0.729323 .. parsed-literal:: :class: output Update for config_id 19:1: reward = 0.7293233082706767, crit_val = 0.2706766917293233 config_id 19: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.115420 ; Training: accuracy=0.709037 ; Validation: accuracy=0.798830 Epoch 3 ; Time: 1.625280 ; Training: accuracy=0.751324 ; Validation: accuracy=0.819716 .. parsed-literal:: :class: output Update for config_id 19:3: reward = 0.8197159565580618, crit_val = 0.18028404344193816 config_id 19: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 2.136457 ; Training: accuracy=0.777557 ; Validation: accuracy=0.845113 Epoch 5 ; Time: 2.652630 ; Training: accuracy=0.796922 ; Validation: accuracy=0.851295 Epoch 6 ; Time: 3.163065 ; Training: accuracy=0.803956 ; Validation: accuracy=0.869173 Epoch 7 ; Time: 3.677748 ; Training: accuracy=0.813803 ; Validation: accuracy=0.870510 Epoch 8 ; Time: 4.186631 ; Training: accuracy=0.825472 ; Validation: accuracy=0.880535 Epoch 9 ; Time: 4.694727 ; Training: accuracy=0.830354 ; Validation: accuracy=0.886884 .. parsed-literal:: :class: output config_id 19: Terminating evaluation at 9 Update for config_id 19:9: reward = 0.8868838763575606, crit_val = 0.11311612364243939 Starting get_config[BO] for config_id 20 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4893090970318028 - self.std = 0.2890712425986082 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 33 Current best is [0.18028404] [20: BO] (19 evaluations) batch_size: 50 dropout_1: 0.49158334246503194 dropout_2: 0.6092455972564605 learning_rate: 0.0012226018662646772 n_units_1: 128 n_units_2: 101 scale_1: 0.0010000000000000002 scale_2: 8.841059210697582 Started BO from (top scorer): batch_size: 50 dropout_1: 0.4915833332725128 dropout_2: 0.6092455947750531 learning_rate: 0.0009896305015320154 n_units_1: 105 n_units_2: 104 scale_1: 0.004185509210264236 scale_2: 8.841058817468932 Top score values: [0.14444307 0.21323356 0.24522217 0.24625234 0.25256966] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 17:1, 17:3, 17:9, 18:1, 19:1, 19:3, 19:9. Pending: Targets: [-0.07112121 -0.63650374 -0.88335275 0.24639608 1.69565813 1.48303114 0.30894548 0.71488434 -0.54655002 -0.78542684 -1.10799773 1.63562488 1.21260556 1.63329534 1.51407693 -0.18009904 -0.46934859 1.57285802 0.08989141 -0.50081523 -0.46731394 -0.84050565 -1.19414648 0.13358795 -0.41211147 1.64833301 -0.79584557 -1.01816616 -1.17995967 0.32681517 -0.75632707 -1.06902731 -1.30138498] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0025147433668945606, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 10.228070087035343, 'kernel_inv_bw4': 0.3341694514951017, 'kernel_inv_bw5': 1.5345181583587304, 'kernel_inv_bw6': 0.42391206685457244, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9836551864837128, 'kernel_covariance_scale': 0.7751464871518577, 'mean_mean_value': 0.9277859812909629} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.659930 ; Training: accuracy=0.309752 ; Validation: accuracy=0.653277 .. parsed-literal:: :class: output Update for config_id 20:1: reward = 0.6532773109243698, crit_val = 0.3467226890756302 config_id 20: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.318232 ; Training: accuracy=0.458430 ; Validation: accuracy=0.712269 Epoch 3 ; Time: 1.887254 ; Training: accuracy=0.503554 ; Validation: accuracy=0.734118 .. parsed-literal:: :class: output config_id 20: Terminating evaluation at 3 Update for config_id 20:3: reward = 0.7341176470588235, crit_val = 0.26588235294117646 Starting get_config[BO] for config_id 21 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4788515784018943 - self.std = 0.28405104436502127 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 35 Current best is [0.18028404] [21: BO] (21 evaluations) batch_size: 69 dropout_1: 0.6846097998516558 dropout_2: 0.0 learning_rate: 0.00419794572293586 n_units_1: 128 n_units_2: 16 scale_1: 0.01280647308906677 scale_2: 1.4928556711581158 Started BO from (top scorer): batch_size: 69 dropout_1: 0.6846098063842428 dropout_2: 0.37503620187634523 learning_rate: 0.006604515646728909 n_units_1: 39 n_units_2: 16 scale_1: 0.012806773195459422 scale_2: 1.4928547138400632 Top score values: [0.07741982 0.1112656 0.13929171 0.16458562 0.17968934] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 17:1, 17:3, 17:9, 18:1, 19:1, 19:3, 19:9, 20:1, 20:3. Pending: Targets: [-0.03556255 -0.6109374 -0.86214912 0.28756641 1.76244211 1.54605724 0.35122129 0.76433454 -0.51939388 -0.76249251 -1.09076438 1.70134785 1.27085227 1.69897715 1.57765171 -0.1464664 -0.44082803 1.63747168 0.12829574 -0.4728508 -0.43875742 -0.81854476 -1.17843568 0.17276456 -0.38257933 1.71428058 -0.77309538 -0.99934516 -1.16399815 0.3694068 -0.73287844 -1.05110522 -1.28756948 -0.46515896 -0.74975688] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0027323470407812672, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.625258886948428, 'kernel_inv_bw3': 10.700609872387792, 'kernel_inv_bw4': 0.3590962475629683, 'kernel_inv_bw5': 1.5098067650168017, 'kernel_inv_bw6': 0.0004296334105178157, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.0249418832596129, 'kernel_covariance_scale': 0.7844650900066925, 'mean_mean_value': 0.9631358750589729} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.482841 ; Training: accuracy=0.390973 ; Validation: accuracy=0.650341 .. parsed-literal:: :class: output Update for config_id 21:1: reward = 0.6503414959187073, crit_val = 0.34965850408129273 config_id 21: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.899163 ; Training: accuracy=0.551056 ; Validation: accuracy=0.727136 Epoch 3 ; Time: 1.419126 ; Training: accuracy=0.599172 ; Validation: accuracy=0.741129 .. parsed-literal:: :class: output config_id 21: Terminating evaluation at 3 Update for config_id 21:3: reward = 0.7411294352823589, crit_val = 0.25887056471764114 Starting get_config[BO] for config_id 22 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4694144408882496 - self.std = 0.27927329180703736 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 37 Current best is [0.18028404] [22: BO] (31 evaluations) batch_size: 97 dropout_1: 0.03525520179231688 dropout_2: 0.0 learning_rate: 0.0010462362290686923 n_units_1: 128 n_units_2: 116 scale_1: 0.0089324290118041 scale_2: 0.002694886459580705 Started BO from (top scorer): batch_size: 97 dropout_1: 0.03525519267376742 dropout_2: 0.14112112793136056 learning_rate: 0.0008731605681165959 n_units_1: 34 n_units_2: 111 scale_1: 0.00893254334894475 scale_2: 0.0026966114746035533 Top score values: [0.13717338 0.16953916 0.20037068 0.20657606 0.2089187 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 17:1, 17:3, 17:9, 18:1, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3. Pending: Targets: [-0.00237918 -0.58759743 -0.84310684 0.3262778 1.82638539 1.60629865 0.39102168 0.81120239 -0.4944878 -0.74174532 -1.0756332 1.76424594 1.32638552 1.76183468 1.63843363 -0.11518035 -0.41457786 1.69927699 0.16428236 -0.44714847 -0.41247183 -0.7987565 -1.16480437 0.20951195 -0.35533265 1.77739992 -0.75252958 -0.98265 -1.15011983 0.4095183 -0.71162462 -1.03529555 -1.2758052 -0.43932505 -0.72879181 -0.42881271 -0.75389907] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.008237124952554261, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.5880912076104247, 'kernel_inv_bw3': 10.185589798340844, 'kernel_inv_bw4': 0.3653671308166189, 'kernel_inv_bw5': 1.3603653677848018, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.0027483490401810492, 'kernel_inv_bw8': 0.9967960318909171, 'kernel_covariance_scale': 0.8585744127880335, 'mean_mean_value': 0.9591701223363772} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.406953 ; Training: accuracy=0.283216 ; Validation: accuracy=0.534420 .. parsed-literal:: :class: output Update for config_id 22:1: reward = 0.5344196873960758, crit_val = 0.4655803126039242 config_id 22: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.708424 ; Training: accuracy=0.605938 ; Validation: accuracy=0.657965 Epoch 3 ; Time: 1.003700 ; Training: accuracy=0.696825 ; Validation: accuracy=0.724144 .. parsed-literal:: :class: output config_id 22: Terminating evaluation at 3 Update for config_id 22:3: reward = 0.724143664782175, crit_val = 0.27585633521782504 Starting get_config[BO] for config_id 23 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4643531015560764 - self.std = 0.27373215983793886 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 39 Current best is [0.18028404] [23: BO] (25 evaluations) batch_size: 9 dropout_1: 0.16072914348806375 dropout_2: 0.0 learning_rate: 0.001376475809627921 n_units_1: 128 n_units_2: 66 scale_1: 4.51193567432714 scale_2: 8.339615256729386 Started BO from (top scorer): batch_size: 70 dropout_1: 0.16072911814660834 dropout_2: 0.05925522155820254 learning_rate: 0.0019714398213228606 n_units_1: 88 n_units_2: 66 scale_1: 4.518191497775782 scale_2: 5.428016900058286 Top score values: [0.13008039 0.14359533 0.17553767 0.1820455 0.27343393] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 6:1, 6:3, 6:9, 7:1, 8:1, 9:1, 10:1, 11:1, 11:3, 12:1, 13:1, 13:3, 14:1, 14:3, 14:9, 15:1, 15:3, 16:1, 17:1, 17:3, 17:9, 18:1, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 22:1, 22:3. Pending: Targets: [ 0.01606278 -0.58100199 -0.84168365 0.35137273 1.88184684 1.6573049 0.41742721 0.84611359 -0.48600755 -0.73827027 -1.07891701 1.81844951 1.37172553 1.81598943 1.69009039 -0.09902182 -0.40448 1.7521654 0.18609803 -0.43770994 -0.40233134 -0.79643553 -1.16989326 0.23224319 -0.3440355 1.83186976 -0.74927284 -0.98405157 -1.15491147 0.43629825 -0.70753984 -1.03776282 -1.28314108 -0.42972814 -0.72505455 -0.419003 -0.75067006 0.00448325 -0.68861754] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.685268695112951, 'kernel_inv_bw1': 0.00018046869895978142, 'kernel_inv_bw2': 0.576333883706921, 'kernel_inv_bw3': 7.658553558436415, 'kernel_inv_bw4': 0.4084432750494162, 'kernel_inv_bw5': 0.002265293744615886, 'kernel_inv_bw6': 0.004653979744777945, 'kernel_inv_bw7': 0.7395935215764463, 'kernel_inv_bw8': 1.0480594101621694, 'kernel_covariance_scale': 0.941687947464726, 'mean_mean_value': 0.9217972942451371} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.162787 ; Training: accuracy=0.422222 ; Validation: accuracy=0.603704 .. parsed-literal:: :class: output Update for config_id 23:1: reward = 0.6037037037037037, crit_val = 0.39629629629629626 config_id 23: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 6.263769 ; Training: accuracy=0.518574 ; Validation: accuracy=0.604714 Epoch 3 ; Time: 9.469267 ; Training: accuracy=0.525373 ; Validation: accuracy=0.592424 .. parsed-literal:: :class: output config_id 23: Terminating evaluation at 3 Update for config_id 23:3: reward = 0.5924242424242424, crit_val = 0.4075757575757576 Analysing the results ~~~~~~~~~~~~~~~~~~~~~ The training history is stored in the ``results_df``, the main fields are the runtime and ``'best'`` (the objective). **Note**: You will get slightly different curves for different pairs of scheduler/searcher, the ``time_out`` here is a bit too short to really see the difference in a significant way (it would be better to set it to >1000s). Generally speaking though, hyperband stopping / promotion + model will tend to significantly outperform other combinations given enough time. .. code:: python results_df.head() .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_15/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) .. raw:: html
bracket elapsed_time epoch error eval_time objective runtime searcher_data_size searcher_params_kernel_covariance_scale searcher_params_kernel_inv_bw0 ... searcher_params_kernel_inv_bw7 searcher_params_kernel_inv_bw8 searcher_params_mean_mean_value searcher_params_noise_variance target_epoch task_id time_since_start time_step time_this_iter best
0 0 0.490147 1 0.468750 0.483471 0.531250 0.578926 NaN 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.581122 1.607457e+09 0.520655 0.468750
1 0 0.928254 2 0.344753 0.431203 0.655247 1.017033 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.017834 1.607457e+09 0.438082 0.344753
2 0 1.358146 3 0.305314 0.428079 0.694686 1.446926 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.447861 1.607457e+09 0.429894 0.305314
3 0 1.846933 4 0.288937 0.484237 0.711063 1.935713 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.936584 1.607457e+09 0.488787 0.288937
4 0 2.315858 5 0.273061 0.466913 0.726939 2.404638 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 2.405663 1.607457e+09 0.468925 0.273061

5 rows × 26 columns

.. code:: python import matplotlib.pyplot as plt plt.figure(figsize=(12, 8)) runtime = results_df['runtime'].values objective = results_df['best'].values plt.plot(runtime, objective, lw=2) plt.xticks(fontsize=12) plt.xlim(0, 120) plt.ylim(0, 0.5) plt.yticks(fontsize=12) plt.xlabel("Runtime [s]", fontsize=14) plt.ylabel("Objective", fontsize=14) .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs-v0_0_15/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) .. parsed-literal:: :class: output Text(0, 0.5, 'Objective') .. figure:: output_mlp_cb387f_18_2.png Diving Deeper ------------- Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at :ref:`sec_customstorch`). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful. Logging the Search Progress ~~~~~~~~~~~~~~~~~~~~~~~~~~~ First, it is a good idea in general to switch on ``debug_log``, which outputs useful information about the search progress. This is already done in the example above. The outputs show which configurations are chosen, stopped, or promoted. For BO and BOHB, a range of information is displayed for every ``get_config`` decision. This log output is very useful in order to figure out what is going on during the search. Configuring ``HyperbandScheduler`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The most important knobs to turn with ``HyperbandScheduler`` are ``max_t``, ``grace_period``, ``reduction_factor``, ``brackets``, and ``type``. The first three determine the rung levels at which stopping or promotion decisions are being made. - The maximum resource level ``max_t`` (usually, resource equates to epochs, so ``max_t`` is the maximum number of training epochs) is typically hardcoded in ``train_fn`` passed to the scheduler (this is ``run_mlp_openml`` in the example above). As already noted above, the value is best fixed in the ``ag.args`` decorator as ``epochs=XYZ``, it can then be accessed as ``args.epochs`` in the ``train_fn`` code. If this is done, you do not have to pass ``max_t`` when creating the scheduler. - ``grace_period`` and ``reduction_factor`` determine the rung levels, which are ``grace_period``, ``grace_period * reduction_factor``, ``grace_period * (reduction_factor ** 2)``, etc. All rung levels must be less or equal than ``max_t``. It is recommended to make ``max_t`` equal to the largest rung level. For example, if ``grace_period = 1``, ``reduction_factor = 3``, it is in general recommended to use ``max_t = 9``, ``max_t = 27``, or ``max_t = 81``. Choosing a ``max_t`` value "off the grid" works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you set ``max_t = 10``, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch. - With ``reduction_factor``, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used. - Finally, ``grace_period`` should be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. While ``grace_period = 1`` should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung. - ``brackets`` sets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). For ``brackets = 1``, you are running successive halving (single bracket). Higher brackets have larger effective ``grace_period`` values (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (``brackets = 1``) in a comparison. - Finally, with ``type`` (values ``stopping``, ``promotion``) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the default ``stopping`` is simpler and seems to perform well, but ``promotion`` is more careful promoting configurations to higher resource levels, which can work better in some cases. Asynchronous BOHB ~~~~~~~~~~~~~~~~~ Finally, here are some ideas for tuning asynchronous BOHB, apart from tuning its ``HyperbandScheduling`` component. You need to pass these options in ``search_options``. - We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by ``gp_resource_kernel``, values are ``matern52``, ``matern52-res-warp``, ``exp-decay-sum``, ``exp-decay-combined``, ``exp-decay-delta1``. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found `here `__. - Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper ``train_fn`` and find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the options ``opt_skip_period`` and ``opt_skip_num_max_resource``. The basic idea is as follows. By far the most expensive part of a ``get_config`` call (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for most ``get_config`` calls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers -- we would love to learn about your use case.