List of Metrics#

Metrics that are already defined in BOA:

PassThrough
MeanSquaredError
RootMeanSquaredError
RSquared
Mean
NormalizedRootMeanSquaredError

Any of these Metrics can be used directly in your configuration file.

Examples

# Single objective optimization config
objective:
    metrics:
        # List all of your metrics here,
        # only list 1 metric for a single objective optimization
        - metric: RootMeanSquaredError

# MultiObjective Optimization config
objective:
    metrics:
        # List all of your metrics here,
        # only list multiple objectives for a multi objective optimization
        - metric: RMSE
        - metric: R2

PassThrough Metric#

If your metric is computed elsewhere (say by your model code or your model wrapping code), then you can use the PassThrough metric. PassThrough metric defaults to minimizing. To mark as maximize, use minimize: False.

# Single objective optimization config
objective:
    metrics:
        # List all of your metrics here,
        # only list 1 metric for a single objective optimization
        - metric: PassThrough
          name: foo  # optional, any name will work
          minimize: False

You could also simply specify a name, with no metric type and it will default to a pass through metric:

# Single objective optimization config
objective:
    metrics:
        # List all of your metrics here,
        # only list 1 metric for a single objective optimization
        - name: foo  # optional, any name will work

If working in a language agnostic way, you can write out your output.json file like this (see more at Wrappers):

{
    "foo": 42
}

If working with the Python API, your fetch_trial_status function could be something like this

def fetch_trial_data(self, trial, *args, **kwargs):
    value = get_value_somehow()
    return value

Overview Information Here: boa.metrics

class boa.metrics.metrics.PassThrough(lower_is_better=True, *args, **kwargs)[source]#

Bases: ModularMetric

Metric that just passes the value it receives back through, for example:

{metric: 5}

will just return 5.

Defaults to minimizing, set minimize: False in your config if you wish to maximize.

boa.metrics.metrics.PassThroughMetric#: alias of PassThrough

boa.metrics.metrics.pass_through#: alias of PassThrough

boa.metrics.metrics.passthrough#: alias of PassThrough

boa.metrics.metrics.pass_through_metric#: alias of PassThrough

class boa.metrics.metrics.BOASklearnMetric(metric_to_eval: Optional[str] = None, *args, **kwargs)[source]#

Bases: ModularMetric

A subclass of ModularMetric where you can pass in a string name of a metric from sklrean.metrics, and BOA will grab that metric and create a BOA metric class for you.

See also

sklearn.metrics: for the list of metrics you can use.
ModularMetric: For information on all parameters various metrics in general can be supplied

Parameters: metric_to_eval (str) –

class boa.metrics.metrics.MeanSquaredError(lower_is_better=True, *args, **kwargs)[source]#

Bases: BOASklearnMetric

Mean squared error regression loss.

See also

sklearn.metrics.mean_squared_error(): for the function parameters to guide your json attribute-value pairs needed.
ModularMetric: For information on all parameters various metrics in general can be supplied

boa.metrics.metrics.MSE#: alias of MeanSquaredError

boa.metrics.metrics.mean_squared_error#: alias of MeanSquaredError

class boa.metrics.metrics.RootMeanSquaredError(lower_is_better=True, metric_func_kwargs=None, *args, **kwargs)[source]#

Bases: BOASklearnMetric

Root mean squared error regression loss.

See also

sklearn.metrics.mean_squared_error(): with squared=False for the function parameters to guide your json attribute-value pairs needed.
ModularMetric: For information on all parameters various metrics in general can be supplied

boa.metrics.metrics.RMSE#: alias of RootMeanSquaredError

boa.metrics.metrics.root_mean_squared_error#: alias of RootMeanSquaredError

class boa.metrics.metrics.RSquared(lower_is_better=True, *args, **kwargs)[source]#

Bases: BOASklearnMetric

\(R^2\) (coefficient of determination) regression score function.

Best possible score is 1.0, and it can be negative (because the model can be arbitrarily worse). In the general case when the true y is non-constant, a constant model that always predicts the average y disregarding the input features would get a \(R^2\) score of 0.0.

See also

sklearn.metrics.r2_score(): for the function parameters to guide your json attribute-value pairs needed.
ModularMetric: For information on all parameters various metrics in general can be supplied

boa.metrics.metrics.r2_score#: alias of RSquared

boa.metrics.metrics.R2#: alias of RSquared

class boa.metrics.metrics.Mean(lower_is_better=True, *args, **kwargs)[source]#

Bases: ModularMetric

Arithmetic mean along the specified axis for your metric, Defaults to minimization, if you want to maximize, specify lower_is_better: False or minimize: False in your configuration

See also

numpy.mean(): for the function parameters to guide your json attribute-value pairs needed.
ModularMetric: For information on all parameters various metrics in general can be supplied

boa.metrics.metrics.mean#: alias of Mean

class boa.metrics.metrics.NormalizedRootMeanSquaredError(lower_is_better=True, *args, **kwargs)[source]#

Bases: ModularMetric

Normalized root mean squared error. Like a normalized version of RMSE. Normalization defaults to IQR (inner quartile range).

normalized_root_mean_squared_error

See also

normalized_root_mean_squared_error(): for the function parameters to guide your json attribute-value pairs needed.
ModularMetric: For information on all parameters various metrics in general can be supplied

boa.metrics.metrics.NRMSE#: alias of NormalizedRootMeanSquaredError

boa.metrics.metrics.normalized_root_mean_squared_error#: alias of NormalizedRootMeanSquaredError

boa.metrics.metrics.get_metric_from_config(config: BOAMetric, instantiate=True, **kwargs) → ModularMetric[source]#

Parameters: config (BOAMetric) –
Return type: ModularMetric

boa.metrics.metrics.get_metric_by_class_name(metric_name, instantiate=True, sklearn_=False, **kwargs)[source]#

boa.metrics.metrics.get_boa_metric(name) → Type[ModularMetric][source]#

Return type: Type[ModularMetric]