boa.metrics.metrics.ModularMetric#

class boa.metrics.metrics.ModularMetric(metric_to_eval: Callable, metric_func_kwargs: Optional[dict] = None, noise_sd: Optional[float] = 0.0, name: Optional[str] = None, wrapper: Optional[BaseWrapper] = None, properties: Optional[dict[str]] = None, metric_type: Optional[str] = None, **kwargs)[source]#

Bases: NoisyFunctionMetric

A wrappable metric defined by a generic deterministic function with the ability to inject a wrapper for higher customizability. The metric function can have some known or unknown noise such that each evaluation may be different, they will be centered around a true value with some noise_sd

The deterministic metric function to compute is implemented by passing some callable (a function or class with __call__) to argument metric_to_eval.

You can further customize the behavior of your metric by passing a Wrapper, which has will run methods such as fetch_trial_data() before calling the specified metric to evaluate, which can allow you to preprocess/prepare model output data for your metric calculation.

Parameters

metric_to_eval (Callable) –
metric_func_kwargs (Optional[dict]) – dictionary of keyword arguments to pass to the metric to eval function
noise_sd (Optional[float]) – Scale of normal noise added to the function result. If None, interpret the function as noisy with unknown noise level.
name (Optional[str]) – name: Name of the metric, if not specified, defaults to name of metric_to_eval
wrapper (Optional[BaseWrapper]) – Boa wrapper to handle running the model and getting the data, allows injecting custom function in the middle of ModularMetric
properties (Optional[dict[str]]) – Arbitrary dictionary of properties to store. Properties need to be json serializable
kwargs –

Methods

`clone`	Create a copy of this Metric.
`deserialize_init_args`	Given a dictionary, deserialize the properties needed to initialize the object.
`f`	The deterministic function that produces the metric outcomes.
`fetch_experiment_data`	Fetch this metric's data for an experiment.
`fetch_experiment_data_multi`	Fetch multiple metrics data for an experiment.
`fetch_trial_data`	Fetch data for one trial.
`fetch_trial_data_multi`	Fetch multiple metrics data for one trial.
`is_available_while_running`	Whether metrics of this class are available while the trial is running.
`lookup_or_fetch_experiment_data_multi`	Fetch or lookup (with fallback to fetching) data for given metrics, depending on whether they are available while running.
`serialize_init_args`	Serialize the properties needed to initialize the object.
`to_dict`	Convert Ax experiment to a dictionary.

Attributes

`db_id`
`fetch_multi_group_by_metric`	Metric class, with which to group this metric in Experiment._metrics_by_class, which is used to combine metrics on experiment into groups and then fetch their data via Metric.fetch_trial_data_multi for each group.
`name`	Get name of metric.

clone() → Metric[source]#: Create a copy of this Metric.

data_constructor#: alias of Data

classmethod deserialize_init_args(args: Dict[str, Any]) → Dict[str, Any]#: Given a dictionary, deserialize the properties needed to initialize the object. Used for storage.

f(*args, **kwargs)[source]#: The deterministic function that produces the metric outcomes.

fetch_experiment_data(experiment: core.experiment.Experiment, **kwargs: Any) → Data#

Fetch this metric’s data for an experiment.

Default behavior is to fetch data from all trials expecting data and concatenate the results.

classmethod fetch_experiment_data_multi(experiment: core.experiment.Experiment, metrics: Iterable[Metric], trials: Optional[Iterable[core.base_trial.BaseTrial]] = None, **kwargs: Any) → Data#

Fetch multiple metrics data for an experiment.

Default behavior calls fetch_trial_data_multi for each trial. Subclasses should override to batch data computation across trials + metrics.

property fetch_multi_group_by_metric: Type[Metric]#

Metric class, with which to group this metric in Experiment._metrics_by_class, which is used to combine metrics on experiment into groups and then fetch their data via Metric.fetch_trial_data_multi for each group.

NOTE: By default, this property will just return the class on which it is defined; however, in some cases it is useful to group metrics by their superclass, in which case this property should return that superclass.

fetch_trial_data(trial: BaseTrial, **kwargs)[source]#: Fetch data for one trial.

classmethod fetch_trial_data_multi(trial: core.base_trial.BaseTrial, metrics: Iterable[Metric], **kwargs: Any) → Data#

Fetch multiple metrics data for one trial.

Default behavior calls fetch_trial_data for each metric. Subclasses should override this to trial data computation for multiple metrics.

classmethod is_available_while_running() → bool[source]#

Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.

NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.

classmethod lookup_or_fetch_experiment_data_multi(experiment: core.experiment.Experiment, metrics: Iterable[Metric], trials: Optional[Iterable[core.base_trial.BaseTrial]] = None, **kwargs: Any) → Tuple[Data, bool]#

Fetch or lookup (with fallback to fetching) data for given metrics, depending on whether they are available while running. Return a tuple containing the data, along with a boolean that will be True if new data was fetched, and False if all data was looked up from cache.

If metric is available while running, its data can change (and therefore we should always re-fetch it). If metric is available only upon trial completion, its data does not change, so we can look up that data on the experiment and only fetch the data that is not already attached to the experiment.

NOTE: If fetching data for a metrics class that is only available upon trial completion, data fetched in this function (data that was not yet available on experiment) will be attached to experiment.

property name: str#: Get name of metric.

classmethod serialize_init_args(obj: Any) → Dict[str, Any]#: Serialize the properties needed to initialize the object. Used for storage.

to_dict() → dict[source]#: Convert Ax experiment to a dictionary.