Search Space and Decorator¶
This tutorial explains the supported search spaces and how to use them,
including simple search spaces (Int, Real, and Categorical) and nested
search spaces (Categorical, List, Dict). Each search space describes the
set of possible values for a hyperparameter, from which the searcher
will try particular values during hyperparameter optimization. AutoGluon
also enables search spaces in user-defined objects using the decorator
ag.obj
and user-defined functions using the decorator ag.func
.
Search Space¶
Simple Search Space¶
import autogluon.core as ag
Integer Space autogluon.core.space.Int
¶
An integer is chosen between lower and upper value during the searcher sampling.
a = ag.space.Int(lower=0, upper=10)
print(a)
Int: lower=0, upper=10
Get default value:
a.default
5
Change default value, which is the first configuration that a random
searcher autogluon.core.searcher.RandomSearcher
will try:
a = ag.space.Int(lower=0, upper=10, default=2)
print(a.default)
2
Pick a random value.
a.rand
2
Real Space autogluon.core.space.Real
¶
A real number is chosen between lower and upper value during the searcher sampling.
b = ag.space.Real(lower=1e-4, upper=1e-2)
print(b)
Real: lower=0.0001, upper=0.01
Real space in log scale:
c = ag.space.Real(lower=1e-4, upper=1e-2, log=True)
print(c)
Real: lower=0.0001, upper=0.01
Categorical Space autogluon.core.space.Categorical
¶
Categorical Space chooses one value from all the possible values during the searcher sampling.
d = ag.space.Categorical('Monday', 'Tuesday', 'Wednesday')
print(d)
Categorical['Monday', 'Tuesday', 'Wednesday']
Nested Search Space¶
Categorical Space autogluon.core.space.Categorical
¶
Categorical Space can also be used as a nested search space. For an example, see NestedExampleObj_.
List Space autogluon.core.space.List
¶
List Space returns a list of sampled results.
In this example, the first element of the list is a Int Space sampled
from 0 to 3, and the second element is a Categorical Space sampled from
the choices of 'alpha'
and 'beta'
.
f = ag.space.List(
ag.space.Int(0, 3),
ag.space.Categorical('alpha', 'beta'),
)
print(f)
List[Int: lower=0, upper=3, Categorical['alpha', 'beta']]
Get one example configuration:
f.rand
[3, 'alpha']
Dict Space autogluon.core.space.Dict
¶
Dict Space returns a dict of sampled results.
Similar to List Space, the resulting configuraton of Dict is a dict. In
this example, the value of 'key1'
is sampled from a Categorical
Space with the choices of 'alpha'
and 'beta'
, and the value of
'key2'
is sampled from an Int Space between 0 and 3.
g = ag.space.Dict(
key1=ag.space.Categorical('alpha', 'beta'),
key2=ag.space.Int(0, 3),
key3='constant'
)
print(g)
Dict{'key1': Categorical['alpha', 'beta'], 'key2': Int: lower=0, upper=3, 'key3': 'constant'}
Get one example configuration:
g.rand
{'key1': 'alpha', 'key2': 3, 'key3': 'constant'}
Decorators for Searchbale Object and Customized Training Scripts¶
In this section, we show how to insert search space into customized objects and training functions.
Searchable Space in Customized Class autogluon.obj()
¶
In AutoGluon searchable object can be returned by a user defined class with a decorator.
@ag.obj(
name=ag.space.Categorical('auto', 'gluon'),
static_value=10,
rank=ag.space.Int(2, 5),
)
class MyObj:
def __init__(self, name, rank, static_value):
self.name = name
self.rank = rank
self.static_value = static_value
def __repr__(self):
repr = 'MyObj -- name: {}, rank: {}, static_value: {}'.format(
self.name, self.rank, self.static_value)
return repr
h = MyObj()
print(h)
AutoGluonObject -- MyObj
Get one example random object:
h.rand
MyObj -- name: auto, rank: 2, static_value: 10
We can also use it within a Nested Space such as
autogluon.core.space.Categorical
. In this example, the
resulting nested space will be sampled from:
nested = ag.space.Categorical(
ag.space.Dict(
obj1='1',
obj2=ag.space.Categorical('a', 'b'),
),
MyObj(),
)
print(nested)
Categorical[Dict{'obj1': '1', 'obj2': Categorical['a', 'b']}, AutoGluonObject -- MyObj]
Get an example output:
for _ in range(5):
result = nested.rand
assert (isinstance(result, dict) and result['obj2'] in ['a', 'b']) or hasattr(result, 'name')
print(result)
MyObj -- name: auto, rank: 5, static_value: 10
MyObj -- name: auto, rank: 3, static_value: 10
MyObj -- name: auto, rank: 5, static_value: 10
{'obj1': '1', 'obj2': 'a'}
{'obj1': '1', 'obj2': 'a'}
Searchable Space in Customized Function autogluon.obj()
¶
We can also insert a searchable space in a customized function:
@ag.func(
framework=ag.space.Categorical('mxnet', 'pytorch'),
)
def myfunc(framework):
return framework
i = myfunc()
print(i)
AutoGluonObject
We can also put a searchable space inside a nested space:
j = ag.space.Dict(
a=ag.Real(0, 10),
obj1=MyObj(),
obj2=myfunc(),
)
print(j)
Dict{'a': Real: lower=0, upper=10, 'obj1': AutoGluonObject -- MyObj, 'obj2': AutoGluonObject}
Customized Train Script Using autogluon.args()
¶
train_func
is where to put your model training script, which takes
in various keyword args
as its hyperparameters and reports the
performance of the trained model using the provided reporter
. Here,
we show a dummy train_func that simply prints these objects.
@ag.args(
a=ag.space.Int(1, 10),
b=ag.space.Real(1e-3, 1e-2),
c=ag.space.Real(1e-3, 1e-2, log=True),
d=ag.space.Categorical('a', 'b', 'c', 'd'),
e=ag.space.Bool(),
f=ag.space.List(
ag.space.Int(1, 2),
ag.space.Categorical(4, 5),
),
g=ag.space.Dict(
a=ag.Real(0, 10),
obj=MyObj(),
),
h=ag.space.Categorical('test', MyObj()),
i = myfunc(),
)
def train_fn(args, reporter):
print('args: {}'.format(args))
Create Searcher and Sample a Configuration¶
In this section, we create a Searcher object, which orchestrates a particular hyperparameter-tuning strategy.
searcher = ag.searcher.RandomSearcher(train_fn.cs)
config = searcher.get_config()
print(config)
{'a': 6, 'b': 0.0055, 'c': 0.0031622777, 'd▁choice': 0, 'e': 0, 'f▁0': 2, 'f▁1▁choice': 0, 'g▁a': 5.0, 'g▁obj▁name▁choice': 0, 'g▁obj▁rank': 4, 'h▁1▁name▁choice': 0, 'h▁1▁rank': 4, 'h▁choice': 0, 'i▁framework▁choice': 0}
train_fn(train_fn.args, config)
args: {'a': 6, 'b': 0.0055, 'c': 0.0031622777, 'd': 'a', 'e': 0, 'f': [2, 4], 'g': {'a': 5.0, 'obj': MyObj -- name: auto, rank: 4, static_value: 10}, 'h': 'test', 'i': 'mxnet', '_default_config': {}}