Modular Metric#

Overview Information Here: boa.metrics

class boa.metrics.modular_metric.ModularMetric(metric_to_eval: Union[Callable, str] = None, metric_func_kwargs: Optional[dict] = None, param_names: list[str] = None, noise_sd: Optional[float] = 0.0, name: Optional[str] = None, wrapper: Optional[BaseWrapper] = None, properties: Optional[dict[str]] = None, weight: Optional[float] = None, **kwargs)[source]#

Bases: NoisyFunctionMetric

A wrappable metric defined by a generic deterministic function with the ability to inject a wrapper for higher customizability. The metric function can have some known or unknown noise such that each evaluation may be different, they will be centered around a true value with some noise_sd

The deterministic metric function to compute is implemented by passing some callable (a function or class with __call__) to argument metric_to_eval.

You can further customize the behavior of your metric by passing a Wrapper, which has will run methods such as BaseWrapper.fetch_trial_data() before calling the specified metric to evaluate, which can allow you to preprocess/prepare model output data for your metric calculation.

Parameters

  • metric_to_eval (Callable | str) –

  • metric_func_kwargs (Optional[dict]) – dictionary of keyword arguments to pass to the metric to eval function

  • noise_sd (Optional[float]) – Scale of normal noise added to the function result. If None, interpret the function as noisy with unknown noise level.

  • param_names (list[str]) – A list of names of parameters to be passed to your wrapper. Useful for filtering out parameters before those parameters are passed to your metric

  • name (Optional[str]) – The name of the metric, if not specified, defaults to name of metric_to_eval

  • wrapper (Optional[BaseWrapper]) – Boa wrapper to handle running the model and getting the data, allows injecting custom function in the middle of ModularMetric

  • properties (Optional[dict[str]]) – Arbitrary dictionary of properties to store. Properties need to be json serializable

  • kwargs –

  • weight (Optional[float]) –

classmethod is_available_while_running() → bool[source]#

Whether metrics of this class are available while the trial is running. Metrics that are not available while the trial is running are assumed to be available only upon trial completion. For such metrics, data is assumed to never change once the trial is completed.

NOTE: If this method returns False, data-fetching via experiment.fetch_data will return the data cached on the experiment (for the metrics of the given class) whenever its available. Data is cached on experiment when attached via experiment.attach_data.

Return type

bool

property weight#
fetch_trial_data(trial: Trial, **kwargs)[source]#

Fetch data for one trial.

Parameters

trial (Trial) –

f(*args, **kwargs)[source]#

The deterministic function that produces the metric outcomes.

clone() → Metric[source]#

Create a copy of this Metric.

Return type

Metric

to_dict() → dict[source]#

Convert the Metric to a dictionary.

Return type

dict

classmethod serialize_init_args(obj: Any) → dict[str, Any][source]#

Serialize the properties needed to initialize the object. Used for storage.

Parameters

obj (Any) –

Return type

dict[str, Any]

classmethod deserialize_init_args(args: dict[str, Any]) → dict[str, Any][source]#

Given a dictionary, deserialize the properties needed to initialize the object. Used for storage.

Parameters

args (dict[str, Any]) –

Return type

dict[str, Any]