The Luby Toy Benchmark

Task: lning the Luby sequence with variations
Cost: correctness of sequence element prediction
Number of hyperparameters to control: one integer
State Information: Actions and timesteps of the last three iterations
Noise Level: None
Instance space: the Luby sequence with possibilities to modify the starting point of the series (e.g. element 5 instead of 1) as well as the repetition fo each element

This benchmark is not built on top of an algorithm, instead it’s a pure sequence learning task. In each step until the cutoff, the DAC controller’s task is to predict the next element of the Luby sequence. If the prediction is correct, it is given a reward of 1 and else 0.

The benchmark is very cheap to run, but can be altered to be quite challenging nonetheless. In its basic form, it can serve to validate DAC methods and observe their prowess in learning a series of predictions correctly.

The Luby benchmark was constructed by Biedenkapp et al. for the paper `”Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework” <https://www.tnt.uni-hannover.de/papers/data/1432/20-ECAI-DAC.pdf>`_ at ECAI 2020

class dacbench.benchmarks.luby_benchmark.LubyBenchmark(config_path=None, config=None)

Bases: AbstractBenchmark

Benchmark with default configuration & relevant functions for Sigmoid

get_benchmark(L=8, fuzziness=1.5, seed=0)

Get Benchmark from DAC paper

Parameters

  • L (int) – Minimum sequence lenght, was 8, 16 or 32 in the paper

  • fuzziness (float) – Amount of noise applied. Was 1.5 for most of the experiments

  • seed (int) – Environment seed

Returns

env – Luby environment

Return type

LubyEnv

get_environment()

Return Luby env with current configuration

Returns

Luby environment

Return type

LubyEnv

read_instance_set(test=False)

Read instance set from file

set_cutoff(steps)

Set cutoff and adapt dependencies

Parameters

int – Maximum number of steps

set_history_length(length)

Set history length and adapt dependencies

Parameters

int – History length

Luby environment from “Dynamic Algorithm Configuration:Foundation of a New Meta-Algorithmic Framework” by A. Biedenkapp and H. F. Bozkurt and T. Eimer and F. Hutter and M. Lindauer. Original environment authors: André Biedenkapp, H. Furkan Bozkurt

class dacbench.envs.luby.LubyEnv(config)

Bases: AbstractEnv

Environment to learn Luby Sequence

close() → bool

Close Env

Returns

Closing confirmation

Return type

bool

render(mode: str = 'human') → None

Render env in human mode

Parameters

mode (str) – Execution mode

reset() → List[int]

Resets env

Returns

Environment state

Return type

numpy.array

step(action: int)

Execute environment step

Parameters

action (int) – action to execute

Returns

state, reward, done, info

Return type

np.array, float, bool, dict

dacbench.envs.luby.luby_gen(i)

Generator for the Luby Sequence