.. _sec_custom_advancedhpo:
Getting started with Advanced HPO Algorithms
============================================
This tutorial provides a complete example of how to use AutoGluon's
state-of-the-art hyperparameter optimization (HPO) algorithms to tune a
basic Multi-Layer Perceptron (MLP) model, which is the most basic type
of neural network.
Loading libraries
-----------------
.. code:: python
# Basic utils for folder manipulations etc
import time
import multiprocessing # to count the number of CPUs available
# External tools to load and process data
import numpy as np
import pandas as pd
# MXNet (NeuralNets)
import mxnet as mx
from mxnet import gluon, autograd
from mxnet.gluon import nn
# AutoGluon and HPO tools
import autogluon.core as ag
from autogluon.mxnet.utils import load_and_split_openml_data
Check the version of MxNet, you should be fine with version >= 1.5
.. code:: python
mx.__version__
.. parsed-literal::
:class: output
'1.7.0'
You can also check the version of AutoGluon and the specific commit and
check that it matches what you want.
.. code:: python
import autogluon.core.version
ag.version.__version__
.. parsed-literal::
:class: output
'0.3.2b20211122'
Hyperparameter Optimization of a 2-layer MLP
--------------------------------------------
Setting up the context
~~~~~~~~~~~~~~~~~~~~~~
Here we declare a few "environment variables" setting the context for
what we're doing
.. code:: python
OPENML_TASK_ID = 6 # describes the problem we will tackle
RATIO_TRAIN_VALID = 0.33 # split of the training data used for validation
RESOURCE_ATTR_NAME = 'epoch' # how do we measure resources (will become clearer further)
REWARD_ATTR_NAME = 'objective' # how do we measure performance (will become clearer further)
NUM_CPUS = multiprocessing.cpu_count()
Preparing the data
~~~~~~~~~~~~~~~~~~
We will use a multi-way classification task from OpenML. Data
preparation includes:
- Missing values are imputed, using the 'mean' strategy of
``sklearn.impute.SimpleImputer``
- Split training set into training and validation
- Standardize inputs to mean 0, variance 1
.. code:: python
X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data(
OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False)
n_classes
.. parsed-literal::
:class: output
100%|██████████| 704/704 [00:00<00:00, 70632.46KB/s]
100%|██████████| 2521/2521 [00:00<00:00, 78161.47KB/s]
3KB [00:00, 3776.38KB/s]
8KB [00:00, 11008.67KB/s]
15KB [00:00, 15333.79KB/s]
2998KB [00:00, 46318.93KB/s]
881KB [00:00, 69350.11KB/s]
3KB [00:00, 3578.76KB/s]
.. parsed-literal::
:class: output
26
The problem has 26 classes.
Declaring a model specifying a hyperparameter space with AutoGluon
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Two layer MLP where we optimize over:
- the number of units on the first layer
- the number of units on the second layer
- the dropout rate after each layer
- the learning rate
- the scaling
- the ``@ag.args`` decorator allows us to specify the space we will
optimize over, this matches the
`ConfigSpace `__ syntax
The body of the function ``run_mlp_openml`` is pretty simple:
- it reads the hyperparameters given via the decorator
- it defines a 2 layer MLP with dropout
- it declares a trainer with the 'adam' loss function and a provided
learning rate
- it trains the NN with a number of epochs (most of that is boilerplate
code from ``mxnet``)
- the ``reporter`` at the end is used to keep track of training history
in the hyperparameter optimization
**Note**: The number of epochs and the hyperparameter space are reduced
to make for a shorter experiment
.. code:: python
@ag.args(n_units_1=ag.space.Int(lower=16, upper=128),
n_units_2=ag.space.Int(lower=16, upper=128),
dropout_1=ag.space.Real(lower=0, upper=.75),
dropout_2=ag.space.Real(lower=0, upper=.75),
learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True),
batch_size=ag.space.Int(lower=8, upper=128),
scale_1=ag.space.Real(lower=0.001, upper=10, log=True),
scale_2=ag.space.Real(lower=0.001, upper=10, log=True),
epochs=9)
def run_mlp_openml(args, reporter, **kwargs):
# Time stamp for elapsed_time
ts_start = time.time()
# Unwrap hyperparameters
n_units_1 = args.n_units_1
n_units_2 = args.n_units_2
dropout_1 = args.dropout_1
dropout_2 = args.dropout_2
scale_1 = args.scale_1
scale_2 = args.scale_2
batch_size = args.batch_size
learning_rate = args.learning_rate
ctx = mx.cpu()
net = nn.Sequential()
with net.name_scope():
# Layer 1
net.add(nn.Dense(n_units_1, activation='relu',
weight_initializer=mx.initializer.Uniform(scale=scale_1)))
# Dropout
net.add(gluon.nn.Dropout(dropout_1))
# Layer 2
net.add(nn.Dense(n_units_2, activation='relu',
weight_initializer=mx.initializer.Uniform(scale=scale_2)))
# Dropout
net.add(gluon.nn.Dropout(dropout_2))
# Output
net.add(nn.Dense(n_classes))
net.initialize(ctx=ctx)
trainer = gluon.Trainer(net.collect_params(), 'adam',
{'learning_rate': learning_rate})
for epoch in range(args.epochs):
ts_epoch = time.time()
train_iter = mx.io.NDArrayIter(
data={'data': X_train},
label={'label': y_train},
batch_size=batch_size,
shuffle=True)
valid_iter = mx.io.NDArrayIter(
data={'data': X_valid},
label={'label': y_valid},
batch_size=batch_size,
shuffle=False)
metric = mx.metric.Accuracy()
loss = gluon.loss.SoftmaxCrossEntropyLoss()
for batch in train_iter:
data = batch.data[0].as_in_context(ctx)
label = batch.label[0].as_in_context(ctx)
with autograd.record():
output = net(data)
L = loss(output, label)
L.backward()
trainer.step(data.shape[0])
metric.update([label], [output])
name, train_acc = metric.get()
metric = mx.metric.Accuracy()
for batch in valid_iter:
data = batch.data[0].as_in_context(ctx)
label = batch.label[0].as_in_context(ctx)
output = net(data)
metric.update([label], [output])
name, val_acc = metric.get()
print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % (
epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc))
ts_now = time.time()
eval_time = ts_now - ts_epoch
elapsed_time = ts_now - ts_start
# The resource reported back (as 'epoch') is the number of epochs
# done, starting at 1
reporter(
epoch=epoch + 1,
objective=float(val_acc),
eval_time=eval_time,
time_step=ts_now,
elapsed_time=elapsed_time)
**Note**: The annotation ``epochs=9`` specifies the maximum number of
epochs for training. It becomes available as ``args.epochs``.
Importantly, it is also processed by ``HyperbandScheduler`` below in
order to set its ``max_t`` attribute.
**Recommendation**: Whenever writing training code to be passed as
``train_fn`` to a scheduler, if this training code reports a resource
(or time) attribute, the corresponding maximum resource value should be
included in ``train_fn.args``:
- If the resource attribute (``time_attr`` of scheduler) in
``train_fn`` is ``epoch``, make sure to include ``epochs=XYZ`` in the
annotation. This allows the scheduler to read ``max_t`` from
``train_fn.args.epochs``. This case corresponds to our example here.
- If the resource attribute is something else than ``epoch``, you can
also include the annotation ``max_t=XYZ``, which allows the scheduler
to read ``max_t`` from ``train_fn.args.max_t``.
Annotating the training function by the correct value for ``max_t``
simplifies scheduler creation (since ``max_t`` does not have to be
passed), and avoids inconsistencies between ``train_fn`` and the
scheduler.
Running the Hyperparameter Optimization
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can use the following schedulers:
- FIFO (``fifo``)
- Hyperband (either the stopping (``hbs``) or promotion (``hbp``)
variant)
And the following searchers:
- Random search (``random``)
- Gaussian process based Bayesian optimization (``bayesopt``)
- SkOpt Bayesian optimization (``skopt``; only with FIFO scheduler)
Note that the method known as (asynchronous) Hyperband is using random
search. Combining Hyperband scheduling with the ``bayesopt`` searcher
uses a novel method called asynchronous BOHB.
Pick the combination you're interested in (doing the full experiment
takes around 120 seconds, see the ``time_out`` parameter), running
everything with multiple runs can take a fair bit of time. In real life,
you will want to choose a larger ``time_out`` in order to obtain good
performance.
.. code:: python
SCHEDULER = "hbs"
SEARCHER = "bayesopt"
.. code:: python
def compute_error(df):
return 1.0 - df["objective"]
def compute_runtime(df, start_timestamp):
return df["time_step"] - start_timestamp
def process_training_history(task_dicts, start_timestamp,
runtime_fn=compute_runtime,
error_fn=compute_error):
task_dfs = []
for task_id in task_dicts:
task_df = pd.DataFrame(task_dicts[task_id])
task_df = task_df.assign(task_id=task_id,
runtime=runtime_fn(task_df, start_timestamp),
error=error_fn(task_df),
target_epoch=task_df["epoch"].iloc[-1])
task_dfs.append(task_df)
result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True)
# re-order by runtime
result = result.sort_values(by="runtime")
# calculate incumbent best -- the cumulative minimum of the error.
result = result.assign(best=result["error"].cummin())
return result
resources = dict(num_cpus=NUM_CPUS, num_gpus=0)
.. code:: python
search_options = {
'num_init_random': 2,
'debug_log': True}
if SCHEDULER == 'fifo':
myscheduler = ag.scheduler.FIFOScheduler(
run_mlp_openml,
resource=resources,
searcher=SEARCHER,
search_options=search_options,
time_out=120,
time_attr=RESOURCE_ATTR_NAME,
reward_attr=REWARD_ATTR_NAME)
else:
# This setup uses rung levels at 1, 3, 9 epochs. We just use a single
# bracket, so this is in fact successive halving (Hyperband would use
# more than 1 bracket).
# Also note that since we do not use the max_t argument of
# HyperbandScheduler, this value is obtained from train_fn.args.epochs.
sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion'
myscheduler = ag.scheduler.HyperbandScheduler(
run_mlp_openml,
resource=resources,
searcher=SEARCHER,
search_options=search_options,
time_out=120,
time_attr=RESOURCE_ATTR_NAME,
reward_attr=REWARD_ATTR_NAME,
type=sch_type,
grace_period=1,
reduction_factor=3,
brackets=1)
# run tasks
myscheduler.run()
myscheduler.join_jobs()
results_df = process_training_history(
myscheduler.training_history.copy(),
start_timestamp=myscheduler._start_time)
.. parsed-literal::
:class: output
The meaning of 'time_out' has changed. Previously, jobs started before
'time_out' were allowed to continue until stopped by other means. Now,
we stop jobs once 'time_out' is passed (at the next metric reporting).
If you like to keep the old behaviour, use
'stop_jobs_after_time_out=False'
/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/venv/lib/python3.9/site-packages/distributed/worker.py:4325: UserWarning: Large object of size 1.24 MiB detected in task graph:
(0, , { ... sReporter}, [])
Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and
keep data on workers
future = client.submit(func, big_data) # bad
big_future = client.scatter(big_data) # good
future = client.submit(func, big_future) # good
warnings.warn(
.. parsed-literal::
:class: output
Epoch 1 ; Time: 1.127097 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250
Epoch 2 ; Time: 1.559585 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247
Epoch 3 ; Time: 1.976377 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686
Epoch 4 ; Time: 2.410871 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063
Epoch 5 ; Time: 2.826730 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939
Epoch 6 ; Time: 3.258214 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321
Epoch 7 ; Time: 3.698543 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501
Epoch 8 ; Time: 4.239387 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202
Epoch 9 ; Time: 4.652758 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043
Epoch 1 ; Time: 0.921497 ; Training: accuracy=0.542928 ; Validation: accuracy=0.740500
Epoch 2 ; Time: 1.161204 ; Training: accuracy=0.759258 ; Validation: accuracy=0.803167
Epoch 3 ; Time: 1.396676 ; Training: accuracy=0.802557 ; Validation: accuracy=0.860667
Epoch 4 ; Time: 1.632215 ; Training: accuracy=0.834969 ; Validation: accuracy=0.866500
Epoch 5 ; Time: 1.873816 ; Training: accuracy=0.842639 ; Validation: accuracy=0.885500
Epoch 6 ; Time: 2.111499 ; Training: accuracy=0.851134 ; Validation: accuracy=0.882333
Epoch 7 ; Time: 2.364644 ; Training: accuracy=0.864082 ; Validation: accuracy=0.901500
Epoch 8 ; Time: 2.697722 ; Training: accuracy=0.867876 ; Validation: accuracy=0.882667
Epoch 9 ; Time: 2.935741 ; Training: accuracy=0.861443 ; Validation: accuracy=0.889333
Epoch 1 ; Time: 1.041602 ; Training: accuracy=0.061425 ; Validation: accuracy=0.216130
Epoch 1 ; Time: 2.214662 ; Training: accuracy=0.130017 ; Validation: accuracy=0.128283
Epoch 1 ; Time: 1.428753 ; Training: accuracy=0.129032 ; Validation: accuracy=0.203955
Epoch 1 ; Time: 1.443827 ; Training: accuracy=0.046733 ; Validation: accuracy=0.071057
Epoch 1 ; Time: 2.025383 ; Training: accuracy=0.522650 ; Validation: accuracy=0.737675
Epoch 2 ; Time: 3.473571 ; Training: accuracy=0.622195 ; Validation: accuracy=0.796231
Epoch 3 ; Time: 4.968731 ; Training: accuracy=0.654244 ; Validation: accuracy=0.799596
Epoch 1 ; Time: 1.226295 ; Training: accuracy=0.160826 ; Validation: accuracy=0.469529
Epoch 1 ; Time: 1.262127 ; Training: accuracy=0.475041 ; Validation: accuracy=0.693782
Epoch 2 ; Time: 1.830804 ; Training: accuracy=0.627025 ; Validation: accuracy=0.762857
Epoch 3 ; Time: 2.394689 ; Training: accuracy=0.669256 ; Validation: accuracy=0.794958
Epoch 1 ; Time: 1.009172 ; Training: accuracy=0.456132 ; Validation: accuracy=0.719807
Epoch 2 ; Time: 1.345231 ; Training: accuracy=0.627719 ; Validation: accuracy=0.755456
Epoch 3 ; Time: 1.674079 ; Training: accuracy=0.673199 ; Validation: accuracy=0.805597
Epoch 4 ; Time: 2.056237 ; Training: accuracy=0.701398 ; Validation: accuracy=0.822755
Epoch 5 ; Time: 2.383546 ; Training: accuracy=0.719838 ; Validation: accuracy=0.833583
Epoch 6 ; Time: 2.716469 ; Training: accuracy=0.730257 ; Validation: accuracy=0.835749
Epoch 7 ; Time: 3.042952 ; Training: accuracy=0.736955 ; Validation: accuracy=0.842079
Epoch 8 ; Time: 3.374295 ; Training: accuracy=0.749111 ; Validation: accuracy=0.853740
Epoch 9 ; Time: 3.703335 ; Training: accuracy=0.756223 ; Validation: accuracy=0.843578
Epoch 1 ; Time: 2.061541 ; Training: accuracy=0.648922 ; Validation: accuracy=0.767172
Epoch 2 ; Time: 3.472418 ; Training: accuracy=0.743118 ; Validation: accuracy=0.805892
Epoch 3 ; Time: 4.821994 ; Training: accuracy=0.778773 ; Validation: accuracy=0.818519
Epoch 4 ; Time: 6.167079 ; Training: accuracy=0.787396 ; Validation: accuracy=0.824916
Epoch 5 ; Time: 7.514087 ; Training: accuracy=0.802653 ; Validation: accuracy=0.818855
Epoch 6 ; Time: 8.878316 ; Training: accuracy=0.804892 ; Validation: accuracy=0.814141
Epoch 7 ; Time: 10.247409 ; Training: accuracy=0.819983 ; Validation: accuracy=0.826263
Epoch 8 ; Time: 11.594504 ; Training: accuracy=0.819486 ; Validation: accuracy=0.824579
Epoch 9 ; Time: 12.944206 ; Training: accuracy=0.815008 ; Validation: accuracy=0.835354
Epoch 1 ; Time: 1.637113 ; Training: accuracy=0.402405 ; Validation: accuracy=0.661279
Epoch 1 ; Time: 1.005991 ; Training: accuracy=0.303025 ; Validation: accuracy=0.583960
Epoch 1 ; Time: 0.991454 ; Training: accuracy=0.044878 ; Validation: accuracy=0.039773
Epoch 1 ; Time: 0.907944 ; Training: accuracy=0.475576 ; Validation: accuracy=0.674535
Epoch 1 ; Time: 1.147227 ; Training: accuracy=0.701094 ; Validation: accuracy=0.817710
Epoch 2 ; Time: 1.619035 ; Training: accuracy=0.831399 ; Validation: accuracy=0.844325
Epoch 3 ; Time: 2.076548 ; Training: accuracy=0.863229 ; Validation: accuracy=0.864078
Epoch 4 ; Time: 2.554046 ; Training: accuracy=0.884035 ; Validation: accuracy=0.858052
Epoch 5 ; Time: 3.052599 ; Training: accuracy=0.886025 ; Validation: accuracy=0.876799
Epoch 6 ; Time: 3.534283 ; Training: accuracy=0.908737 ; Validation: accuracy=0.886006
Epoch 7 ; Time: 4.025623 ; Training: accuracy=0.919844 ; Validation: accuracy=0.901071
Epoch 8 ; Time: 4.508651 ; Training: accuracy=0.919678 ; Validation: accuracy=0.876799
Epoch 9 ; Time: 4.998040 ; Training: accuracy=0.919015 ; Validation: accuracy=0.905758
Epoch 1 ; Time: 1.180452 ; Training: accuracy=0.164187 ; Validation: accuracy=0.095623
Epoch 1 ; Time: 0.933891 ; Training: accuracy=0.556908 ; Validation: accuracy=0.749335
Epoch 2 ; Time: 1.178343 ; Training: accuracy=0.703043 ; Validation: accuracy=0.761802
Epoch 3 ; Time: 1.422596 ; Training: accuracy=0.739638 ; Validation: accuracy=0.790891
Epoch 1 ; Time: 1.131916 ; Training: accuracy=0.502902 ; Validation: accuracy=0.732014
Epoch 2 ; Time: 1.564124 ; Training: accuracy=0.650746 ; Validation: accuracy=0.793560
Epoch 3 ; Time: 1.977382 ; Training: accuracy=0.707214 ; Validation: accuracy=0.814355
Epoch 4 ; Time: 2.392521 ; Training: accuracy=0.732338 ; Validation: accuracy=0.837330
Epoch 5 ; Time: 2.804134 ; Training: accuracy=0.754063 ; Validation: accuracy=0.848902
Epoch 6 ; Time: 3.219433 ; Training: accuracy=0.772886 ; Validation: accuracy=0.850579
Epoch 7 ; Time: 3.631443 ; Training: accuracy=0.780431 ; Validation: accuracy=0.861815
Epoch 8 ; Time: 4.052007 ; Training: accuracy=0.803814 ; Validation: accuracy=0.873721
Epoch 9 ; Time: 4.466076 ; Training: accuracy=0.803317 ; Validation: accuracy=0.891330
Epoch 1 ; Time: 0.921546 ; Training: accuracy=0.205447 ; Validation: accuracy=0.527252
Epoch 1 ; Time: 1.746557 ; Training: accuracy=0.686294 ; Validation: accuracy=0.796639
Epoch 2 ; Time: 3.063088 ; Training: accuracy=0.831884 ; Validation: accuracy=0.845882
Epoch 3 ; Time: 4.296646 ; Training: accuracy=0.867081 ; Validation: accuracy=0.861008
Epoch 4 ; Time: 5.591588 ; Training: accuracy=0.881242 ; Validation: accuracy=0.892941
Epoch 5 ; Time: 6.870993 ; Training: accuracy=0.897226 ; Validation: accuracy=0.889580
Epoch 6 ; Time: 8.142529 ; Training: accuracy=0.910311 ; Validation: accuracy=0.900504
Epoch 7 ; Time: 9.510922 ; Training: accuracy=0.910393 ; Validation: accuracy=0.894286
Epoch 8 ; Time: 10.784985 ; Training: accuracy=0.925383 ; Validation: accuracy=0.897143
Epoch 9 ; Time: 12.067234 ; Training: accuracy=0.919669 ; Validation: accuracy=0.895630
Epoch 1 ; Time: 0.967161 ; Training: accuracy=0.702620 ; Validation: accuracy=0.826522
Epoch 2 ; Time: 1.305281 ; Training: accuracy=0.864534 ; Validation: accuracy=0.862719
Epoch 3 ; Time: 1.576099 ; Training: accuracy=0.886850 ; Validation: accuracy=0.902919
Epoch 4 ; Time: 1.835132 ; Training: accuracy=0.910158 ; Validation: accuracy=0.920267
Epoch 5 ; Time: 2.098331 ; Training: accuracy=0.925614 ; Validation: accuracy=0.910425
Epoch 6 ; Time: 2.361005 ; Training: accuracy=0.931813 ; Validation: accuracy=0.926439
Epoch 7 ; Time: 2.632337 ; Training: accuracy=0.938259 ; Validation: accuracy=0.927273
Epoch 8 ; Time: 2.901258 ; Training: accuracy=0.940656 ; Validation: accuracy=0.919600
Epoch 9 ; Time: 3.158888 ; Training: accuracy=0.946359 ; Validation: accuracy=0.919600
Epoch 1 ; Time: 0.914596 ; Training: accuracy=0.387911 ; Validation: accuracy=0.659076
Epoch 1 ; Time: 1.109114 ; Training: accuracy=0.554286 ; Validation: accuracy=0.734300
Epoch 2 ; Time: 1.513431 ; Training: accuracy=0.763644 ; Validation: accuracy=0.793437
Epoch 3 ; Time: 1.903367 ; Training: accuracy=0.821863 ; Validation: accuracy=0.843412
Epoch 1 ; Time: 1.009637 ; Training: accuracy=0.491748 ; Validation: accuracy=0.679552
Epoch 1 ; Time: 1.013380 ; Training: accuracy=0.525903 ; Validation: accuracy=0.724775
Epoch 1 ; Time: 0.965356 ; Training: accuracy=0.645467 ; Validation: accuracy=0.796953
Epoch 2 ; Time: 1.300216 ; Training: accuracy=0.830838 ; Validation: accuracy=0.848510
Epoch 3 ; Time: 1.593046 ; Training: accuracy=0.873951 ; Validation: accuracy=0.855541
Epoch 4 ; Time: 1.870007 ; Training: accuracy=0.889831 ; Validation: accuracy=0.889019
Epoch 5 ; Time: 2.135961 ; Training: accuracy=0.907356 ; Validation: accuracy=0.897556
Epoch 6 ; Time: 2.407151 ; Training: accuracy=0.916159 ; Validation: accuracy=0.904084
Epoch 7 ; Time: 2.684765 ; Training: accuracy=0.922083 ; Validation: accuracy=0.907265
Epoch 8 ; Time: 3.011492 ; Training: accuracy=0.931463 ; Validation: accuracy=0.905089
Epoch 9 ; Time: 3.301439 ; Training: accuracy=0.940020 ; Validation: accuracy=0.908939
Epoch 1 ; Time: 0.987179 ; Training: accuracy=0.477987 ; Validation: accuracy=0.651771
Epoch 1 ; Time: 0.909132 ; Training: accuracy=0.549424 ; Validation: accuracy=0.750000
Analysing the results
~~~~~~~~~~~~~~~~~~~~~
The training history is stored in the ``results_df``, the main fields
are the runtime and ``'best'`` (the objective).
**Note**: You will get slightly different curves for different pairs of
scheduler/searcher, the ``time_out`` here is a bit too short to really
see the difference in a significant way (it would be better to set it to
>1000s). Generally speaking though, hyperband stopping / promotion +
model will tend to significantly outperform other combinations given
enough time.
.. code:: python
results_df.head()
.. raw:: html
|
bracket |
elapsed_time |
epoch |
error |
eval_time |
objective |
runtime |
target_epoch |
task_id |
terminated |
time_step |
best |
0 |
0 |
1.129621 |
1 |
0.468750 |
1.124675 |
0.531250 |
2.004078 |
9 |
0 |
NaN |
1.637612e+09 |
0.468750 |
1 |
0 |
1.561105 |
2 |
0.344753 |
0.427605 |
0.655247 |
2.435562 |
9 |
0 |
NaN |
1.637612e+09 |
0.344753 |
2 |
0 |
1.978117 |
3 |
0.305314 |
0.414816 |
0.694686 |
2.852574 |
9 |
0 |
NaN |
1.637612e+09 |
0.305314 |
3 |
0 |
2.412325 |
4 |
0.288937 |
0.431483 |
0.711063 |
3.286782 |
9 |
0 |
NaN |
1.637612e+09 |
0.288937 |
4 |
0 |
2.828440 |
5 |
0.273061 |
0.414081 |
0.726939 |
3.702898 |
9 |
0 |
NaN |
1.637612e+09 |
0.273061 |
.. code:: python
import matplotlib.pyplot as plt
plt.figure(figsize=(12, 8))
runtime = results_df['runtime'].values
objective = results_df['best'].values
plt.plot(runtime, objective, lw=2)
plt.xticks(fontsize=12)
plt.xlim(0, 120)
plt.ylim(0, 0.5)
plt.yticks(fontsize=12)
plt.xlabel("Runtime [s]", fontsize=14)
plt.ylabel("Objective", fontsize=14)
.. parsed-literal::
:class: output
Text(0, 0.5, 'Objective')
Diving Deeper
-------------
Now, you are ready to try HPO on your own machine learning models (if
you use PyTorch, have a look at :ref:`sec_customstorch`). While
AutoGluon comes with well-chosen defaults, it can pay off to tune it to
your specific needs. Here are some tips which may come useful.
Logging the Search Progress
~~~~~~~~~~~~~~~~~~~~~~~~~~~
First, it is a good idea in general to switch on ``debug_log``, which
outputs useful information about the search progress. This is already
done in the example above.
The outputs show which configurations are chosen, stopped, or promoted.
For BO and BOHB, a range of information is displayed for every
``get_config`` decision. This log output is very useful in order to
figure out what is going on during the search.
Configuring ``HyperbandScheduler``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The most important knobs to turn with ``HyperbandScheduler`` are
``max_t``, ``grace_period``, ``reduction_factor``, ``brackets``, and
``type``. The first three determine the rung levels at which stopping or
promotion decisions are being made.
- The maximum resource level ``max_t`` (usually, resource equates to
epochs, so ``max_t`` is the maximum number of training epochs) is
typically hardcoded in ``train_fn`` passed to the scheduler (this is
``run_mlp_openml`` in the example above). As already noted above, the
value is best fixed in the ``ag.args`` decorator as ``epochs=XYZ``,
it can then be accessed as ``args.epochs`` in the ``train_fn`` code.
If this is done, you do not have to pass ``max_t`` when creating the
scheduler.
- ``grace_period`` and ``reduction_factor`` determine the rung levels,
which are ``grace_period``, ``grace_period * reduction_factor``,
``grace_period * (reduction_factor ** 2)``, etc. All rung levels must
be less or equal than ``max_t``. It is recommended to make ``max_t``
equal to the largest rung level. For example, if
``grace_period = 1``, ``reduction_factor = 3``, it is in general
recommended to use ``max_t = 9``, ``max_t = 27``, or ``max_t = 81``.
Choosing a ``max_t`` value "off the grid" works against the
successive halving principle that the total resources spent in a rung
should be roughly equal between rungs. If in the example above, you
set ``max_t = 10``, about a third of configurations reaching 9 epochs
are allowed to proceed, but only for one more epoch.
- With ``reduction_factor``, you tune the extent to which successive
halving filtering is applied. The larger this integer, the fewer
configurations make it to higher number of epochs. Values 2, 3, 4 are
commonly used.
- Finally, ``grace_period`` should be set to the smallest resource
(number of epochs) for which you expect any meaningful
differentiation between configurations. While ``grace_period = 1``
should always be explored, it may be too low for any meaningful
stopping decisions to be made at the first rung.
- ``brackets`` sets the maximum number of brackets in Hyperband (make
sure to study the Hyperband paper or follow-ups for details). For
``brackets = 1``, you are running successive halving (single
bracket). Higher brackets have larger effective ``grace_period``
values (so runs are not stopped until later), yet are also chosen
with less probability. We recommend to always consider successive
halving (``brackets = 1``) in a comparison.
- Finally, with ``type`` (values ``stopping``, ``promotion``) you are
choosing different ways of extending successive halving scheduling to
the asynchronous case. The method for the default ``stopping`` is
simpler and seems to perform well, but ``promotion`` is more careful
promoting configurations to higher resource levels, which can work
better in some cases.
Asynchronous BOHB
~~~~~~~~~~~~~~~~~
Finally, here are some ideas for tuning asynchronous BOHB, apart from
tuning its ``HyperbandScheduling`` component. You need to pass these
options in ``search_options``.
- We support a range of different surrogate models over the criterion
functions across resource levels. All of them are jointly dependent
Gaussian process models, meaning that data collected at all resource
levels are modelled together. The surrogate model is selected by
``gp_resource_kernel``, values are ``matern52``,
``matern52-res-warp``, ``exp-decay-sum``, ``exp-decay-combined``,
``exp-decay-delta1``. These are variants of either a joint Matern 5/2
kernel over configuration and resource, or the exponential decay
model. Details about the latter can be found
`here `__.
- Fitting a Gaussian process surrogate model to data encurs a cost
which scales cubically with the number of datapoints. When applied to
expensive deep learning workloads, even multi-fidelity asynchronous
BOHB is rarely running up more than 100 observations or so (across
all rung levels and brackets), and the GP computations are
subdominant. However, if you apply it to cheaper ``train_fn`` and
find yourself beyond 2000 total evaluations, the cost of GP fitting
can become painful. In such a situation, you can explore the options
``opt_skip_period`` and ``opt_skip_num_max_resource``. The basic idea
is as follows. By far the most expensive part of a ``get_config``
call (picking the next configuration) is the refitting of the GP
model to past data (this entails re-optimizing hyperparameters of the
surrogate model itself). The options allow you to skip this expensive
step for most ``get_config`` calls, after some initial period. Check
the docstrings for details about these options. If you find yourself
in such a situation and gain experience with these skipping features,
make sure to contact the AutoGluon developers -- we would love to
learn about your use case.