The Sigmoid Toy Benchmark

Task: approximate a sigmoid curve at timestep t in each of one or multiple dimensions
Cost: distance between prediction and function
Number of hyperparameters to control: user specified starting at one integer with no fixed upper limit
State Information: Remaining budget, instance descriptions for each dimension (inflection point and slope), last action for each dimension
Noise Level: None
Instance space: one sigmoid curve consisting of inflection point and slope per dimension. Sampling notebook and example datasets in repository.

This benchmark is not built on top of an algorithm, but is simply a function approximation task. In each step until the cutoff, the DAC controller predicts one y-value for a given sigmoid curve per task dimension. The predictions are discrete, that means there is usually some distance between the true function value and the best prediction. This distance is used as a cost function. If multiple task dimensions are used, the total cost is computed by multiplying the costs of all dimensions.

The benchmark is very cheap to run and the instances can be sampled and shaped easily. Therefore it’s a good starting point for any new DAC method or to gain specific insights for which fine control over the instance distribution is required.

The Sigmoid benchmark was constructed by Biedenkapp et al. for the paper `”Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework” <https://www.tnt.uni-hannover.de/papers/data/1432/20-ECAI-DAC.pdf>`_ at ECAI 2020

class dacbench.benchmarks.sigmoid_benchmark.SigmoidBenchmark(config_path=None, config=None)

Bases: AbstractBenchmark

Benchmark with default configuration & relevant functions for Sigmoid

get_benchmark(dimension=None, seed=0)

Get Benchmark from DAC paper

Parameters

dimension (int) – Sigmoid dimension, was 1, 2, 3 or 5 in the paper
seed (int) – Environment seed

Returns

env – Sigmoid environment

Return type

SigmoidEnv

get_environment()

Return Sigmoid env with current configuration

Returns: Sigmoid environment
Return type: SigmoidEnv

read_instance_set(test=False): Read instance set from file

set_action_values(values)

Adapt action values and update dependencies

Parameters: values (list) – A list of possible actions per dimension

Sigmoid environment from “Dynamic Algorithm Configuration:Foundation of a New Meta-Algorithmic Framework” by A. Biedenkapp and H. F. Bozkurt and T. Eimer and F. Hutter and M. Lindauer. Original environment authors: André Biedenkapp, H. Furkan Bozkurt

class dacbench.envs.sigmoid.ContinuousSigmoidEnv(config)

Bases: SigmoidEnv

Environment for tracing sigmoid curves with a continuous state on the x-axis

step(action: ndarray)

Execute environment step. !!NOTE!! The action here is a list of floats and not a single number !!NOTE!!

Parameters: action (list of floats) – action(s) to execute
Returns: state, reward, done, info
Return type: np.array, float, bool, dict

class dacbench.envs.sigmoid.ContinuousStateSigmoidEnv(config)

Bases: SigmoidEnv

Environment for tracing sigmoid curves with a continuous state on the x-axis

step(action: int)

Execute environment step

Parameters: action (int) – action to execute
Returns: state, reward, done, info
Return type: np.array, float, bool, dict

class dacbench.envs.sigmoid.SigmoidEnv(config)

Bases: AbstractEnv

Environment for tracing sigmoid curves

close() → bool

Close Env

Returns: Closing confirmation
Return type: bool

render(mode: str) → None

Render env in human mode

Parameters: mode (str) – Execution mode

reset() → List[int]

Resets env

Returns: Environment state
Return type: numpy.array

step(action: int)

Execute environment step

Parameters: action (int) – action to execute
Returns: state, reward, done, info
Return type: np.array, float, bool, dict