Logging Experiments
As there are many potentially interesting metrics involved in the analysis of DAC methods, DACBench includes the possibility to track and store them.
To log information on an environment, you need a logger object:
from dacbench.logger import Logger
from pathlib import Path
logger = Logger(experiment_name="example", output_path=Path("your/path"))
If you want to use any of our tracking wrappers, you can then create a logging module for them:
from dacbench.wrappers import PerformanceTrackingWrapper
performance_logger = logger.add_module(PerformanceTrackingWrapper)
env = PerformanceTrackingWrapper(env, logger=performance_logger)
logger.add_env()
Now the logger will store information in the specified directory in .jsonl files. By adding more wrappers, you will also be provided with more information. The stored data can then be loaded into pandas dataframes:
from dacbench.logger import load_logs, log2dataframe
logs = load_logs("your/path/PerformancyTrackingWrapper.jsonl")
df = log2dataframe(logs)
- class dacbench.logger.AbstractLogger(experiment_name: str, output_path: Path, step_write_frequency: Optional[int] = None, episode_write_frequency: int = 1)
Logger interface.
The logger classes provide a way of writing structured logs as jsonl files and also help to track information like current episode, step, time …
In the jsonl log file each row corresponds to a step.
- abstract close() None
Makes sure, that all remaining entries in the are written to file and the file is closed.
- abstract log_dict(data)
Alternative to log if more the one value should be logged at once.
- Parameters
data (dict) – a dict with key-value so that each value is a valid value for log
- abstract log_space(key: str, value: Union[ndarray, Dict], space_info=None)
Special for logging gym.spaces.
Currently three types are supported: * Numbers: e.g. samples from Discrete * Fixed length arrays like MultiDiscrete or Box * Dict: assuming each key has fixed length array
- Parameters
key – see log
value – see log
space_info – a list of column names. The length of this list must equal the resulting number of columns.
- abstract next_episode() None
Call at the end of episode.
See next_step
- abstract next_step() None
Call at the end of the step. Updates the internal state and dumps the information of the last step into a json
- set_env(env: AbstractEnv) None
Needed to infer automatically logged information like the instance id :param env: :type env: AbstractEnv
- abstract write() None
Writes buffered logs to file.
Invoke manually if you want to load logs during a run.
- class dacbench.logger.Logger(experiment_name: str, output_path: Path, step_write_frequency: Optional[int] = None, episode_write_frequency: int = 1)
A logger that manages the creation of the module loggers.
To get a ModuleLogger for you module (e.g. wrapper) call module_logger = Logger(…).add_module(“my_wrapper”). From now on module_logger.log(…) or logger.log(…, module=”my_wrapper”) can be used to log.
The logger module takes care of updating information like episode and step in the subloggers. To indicate to the loggers the end of the episode or the next_step simple call logger.next_episode() or logger.next_step().
- add_agent(agent: AbstractDACBenchAgent)
Writes information about the agent
- Parameters
agent (AbstractDACBenchAgent) –
- add_benchmark(benchmark: AbstractBenchmark) None
Writes the config to the experiment path :param benchmark:
- add_module(module: Union[str, type]) ModuleLogger
Creates a sub-logger. For more details see class level documentation :param module: The module name or Wrapper-Type to create a sub-logger for :type module: str or type
- Returns
- Return type
- close()
Makes sure, that all remaining entries (from all sublogger) are written to files and the files are closed.
- log_dict(data, module)
Alternative to log if more the one value should be logged at once.
- Parameters
data (dict) – a dict with key-value so that each value is a valid value for log
- log_space(key, value, module, space_info=None)
Special for logging gym.spaces.
Currently three types are supported: * Numbers: e.g. samples from Discrete * Fixed length arrays like MultiDiscrete or Box * Dict: assuming each key has fixed length array
- Parameters
key – see log
value – see log
space_info – a list of column names. The length of this list must equal the resulting number of columns.
- next_episode()
Call at the end of episode.
See next_step
- next_step()
Call at the end of the step. Updates the internal state of all subloggers and dumps the information of the last step into a json
- set_env(env: AbstractEnv) None
Needed to infer automatically logged information like the instance id :param env: :type env: AbstractEnv
- write()
Writes buffered logs to file.
Invoke manually if you want to load logs during a run.
- class dacbench.logger.ModuleLogger(output_path: Path, experiment_name: str, module: str, step_write_frequency: Optional[int] = None, episode_write_frequency: int = 1)
A logger for handling logging of one module. e.g. a wrapper or toplevel general logging.
Don’t create manually use Logger to manage ModuleLoggers
- close()
Makes sure, that all remaining entries in the are written to file and the file is closed.
- get_logfile() Path
- Returns
the path to the log file of this logger
- Return type
pathlib.Path
- log_dict(data: Dict) None
Alternative to log if more the one value should be logged at once.
- Parameters
data (dict) – a dict with key-value so that each value is a valid value for log
- log_space(key, value, space_info=None)
Special for logging gym.spaces.
Currently three types are supported: * Numbers: e.g. samples from Discrete * Fixed length arrays like MultiDiscrete or Box * Dict: assuming each key has fixed length array
- Parameters
key – see log
value – see log
space_info – a list of column names. The length of this list must equal the resulting number of columns.
- next_episode()
Writes buffered logs to file.
Invoke manually if you want to load logs during a run.
- next_step()
Call at the end of the step. Updates the internal state and dumps the information of the last step into a json
- reset_episode() None
Resets the episode and step.
Be aware that this can lead to ambitious keys if no instance or seed or other identifying additional info is set
- Returns
- set_additional_info(**kwargs)
Can be used to log additional information for each step e.g. for seed, and instance id. :param kwargs:
- write()
Writes buffered logs to file.
Invoke manually if you want to load logs during a run.
- dacbench.logger.flatten_log_entry(log_entry: Dict) List[Dict]
Transforms a log entry of format like
- {
‘step’: 0, ‘episode’: 2, ‘some_value’: {
‘values’ : [34, 45], ‘times’:[‘28-12-20 16:20:53’, ‘28-12-20 16:21:30’],
}
} into [
{ ‘step’: 0,’episode’: 2, ‘value’: 34, ‘time’: ‘28-12-20 16:20:53’}, { ‘step’: 0,’episode’: 2, ‘value’: 45, ‘time’: ‘28-12-20 16:21:30’}
]
- Parameters
log_entry (Dict) – A log entry
- dacbench.logger.list_to_tuple(list_: List) Tuple
Recursively transforms a list of lists into tuples of tuples :param list_: (nested) list
- Returns
- Return type
(nested) tuple
- dacbench.logger.load_logs(log_file: Path) List[Dict]
Loads the logs from a jsonl written by any logger.
The result is the list of dicts in the format: {
‘instance’: 0, ‘episode’: 0, ‘step’: 1, ‘example_log_val’: {
‘values’: [val1, val2, … valn], ‘times: [time1, time2, …, timen],
} :param log_file: The path to the log file :type log_file: pathlib.Path
- Returns
- Return type
[Dict, …]
- dacbench.logger.log2dataframe(logs: List[dict], wide: bool = False, drop_columns: List[str] = ['time']) DataFrame
Converts a list of log entries to a pandas dataframe.
Usually used in combination with load_dataframe.
- Parameters
logs (List) – List of log entries
wide (bool) – wide=False (default) produces a dataframe with columns (episode, step, time, name, value) wide=True returns a dataframe (episode, step, time, name_1, name_2, …) if the variable name_n has not been logged at (episode, step, time) name_n is NaN.
drop_columns (List[str]) – List of column names to be dropped (before reshaping the long dataframe) mostly used in combination with wide=True to reduce NaN values
- Returns
- Return type
dataframe
- dacbench.logger.split(predicate: Callable, iterable: Iterable) Tuple[List, List]
Splits the iterable into two list depending on the result of predicate.
- Parameters
predicate (Callable) – A function taking an element of the iterable and return Ture or False
iterable (Iterable) –
- Returns
- Return type
(positives, negatives)