Running BOA Optimization Directly in Python#
This notebook demonstrates how to:
Write a basic Wrapper in Python and launch a optimization from Python. If you wanted to launch it from command line, you would do a similar thing of defining the Wrapper, and then put in your configuration file the information about where the wrapper is, and use BOA’s CLI tools. See Running an Experiment from Command Line (Python Wrapper) for more information.
import pathlib
import yaml
import shutil
import boa
from wrapper import Wrapper
Show code cell content
# Remove old runs to have a clean slate for this example
old_runs = pathlib.Path().resolve().glob("boa_runs*")
for path in old_runs:
shutil.rmtree(path)
Loading the Config File#
config_path = pathlib.Path().resolve() / "single_config.yaml"
Here we can see what the configuration file looks like
with open(config_path, 'r') as f:
file_contents = f.read()
print (file_contents)
# Single objective optimization config
optimization_options:
objective_options:
objectives:
- name: Cosine8
trials: 50
append_timestamp: False
parameters:
x0:
type: range
bounds: [0.0, 1.0]
x1:
type: range
bounds: [0.0, 1.0]
x2:
type: range
bounds: [0.0, 1.0]
x3:
type: range
bounds: [0.0, 1.0]
x4:
type: range
bounds: [0.0, 1.0]
x5:
type: range
bounds: [0.0, 1.0]
x6:
type: range
bounds: [0.0, 1.0]
x7:
type: range
bounds: [0.0, 1.0]
# These are all defaults, so we don't need to specify them in this case
#script_options:
# wrapper_path: ./wrapper.py
# wrapper_name: Wrapper
# working_dir: .
# experiment_dir: ... # this is where boa will write logs to by default
# if not specified it will be working_dir/experiment_name
# append_timestamp: True
# This last option appends a timestamp to our output experiment directory.
# This is also the default (True)
we need the config normalized, which modifies the parameter section into a less user friendly form, but what the downstream libraries need
config = boa.load_jsonlike(config_path)
Define Our Wrapper#
We define our wrapper in wrapper.py and use a synthetic function that stands in for any black box model call
Wrapper.path()
PosixPath('/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/wrapper.py')
with open(Wrapper.path(), 'r') as f:
file_contents = f.read()
print (file_contents)
import numpy as np
from ax.utils.measurement.synthetic_functions import from_botorch
from botorch.test_functions.synthetic import Cosine8
import boa
cosine8 = from_botorch(Cosine8())
def black_box_model(X) -> float:
result = -cosine8(X)
return result
class Wrapper(boa.BaseWrapper):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.data = {}
def run_model(self, trial) -> None:
X = np.array([parameter for parameter in trial.arm.parameters.values()])
# This is a silly toy function, in reality,
# you could instead import your model main() function and use that, and then collect the results.
# You could also call an external script to start a model run from Bash or elsewhere.
self.data[trial.index] = black_box_model(X)
def set_trial_status(self, trial) -> None:
data_exists = self.data.get(trial.index)
if data_exists:
trial.mark_completed()
def fetch_trial_data(self, trial, *args, **kwargs):
return self.data[trial.index]
Initialize our Setup#
controller = boa.Controller(config_path=config_path, wrapper=Wrapper)
controller.initialize_scheduler()
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x0. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x1. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x2. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x3. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x4. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x5. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x6. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter x7. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 07-10 11:37:38] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='x0', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x7', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[]).
[INFO 07-10 11:37:38] ax.modelbridge.dispatch_utils: Using Models.GPEI since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 07-10 11:37:38] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=8 num_trials=None use_batch_trials=False
[INFO 07-10 11:37:38] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=16
[INFO 07-10 11:37:38] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=16
[INFO 07-10 11:37:38] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 16 trials, GPEI for subsequent trials]). Iterations after 16 will take longer to generate due to model-fitting.
[INFO 07-10 11:37:38] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.
(Scheduler(experiment=Experiment(boa_runs), generation_strategy=GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 16 trials, GPEI for subsequent trials]), options=SchedulerOptions(max_pending_trials=10, trial_type=<TrialType.TRIAL: 0>, batch_size=None, total_trials=None, tolerated_trial_failure_rate=0.5, min_failed_trials_for_failure_rate_check=5, log_filepath=None, logging_level=20, ttl_seconds_for_trials=None, init_seconds_between_polls=1, min_seconds_before_poll=1.0, seconds_between_polls_backoff_factor=1.5, timeout_hours=None, run_trials_in_batches=False, debug_log_run_metadata=False, early_stopping_strategy=None, global_stopping_strategy=None, suppress_storage_errors_after_retries=False)),
<wrapper.Wrapper at 0x158872d40>)
Run our Experiment#
The Controller will save our scheduler to JSON after it completes the run so we can reload it at a later time for analysis or to resume our experiment
scheduler = controller.run()
[INFO 2023-07-10 11:37:38,278 MainProcess] boa:
##############################################
BOA Experiment Run
Output Experiment Dir: /Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738
Start Time: 20230710T113738
Version: 0.8.3.dev1+g0636f51.d20230411
##############################################
[INFO 07-10 11:37:38] Scheduler: Running trials [0]...
[INFO 07-10 11:37:39] Scheduler: Running trials [1]...
[INFO 07-10 11:37:40] Scheduler: Running trials [2]...
[INFO 07-10 11:37:41] Scheduler: Running trials [3]...
[INFO 07-10 11:37:41] Scheduler: Running trials [4]...
[INFO 07-10 11:37:42] Scheduler: Running trials [5]...
[INFO 07-10 11:37:43] Scheduler: Running trials [6]...
[INFO 07-10 11:37:44] Scheduler: Running trials [7]...
[INFO 07-10 11:37:45] Scheduler: Running trials [8]...
[INFO 07-10 11:37:46] Scheduler: Running trials [9]...
[INFO 07-10 11:37:47] Scheduler: Retrieved COMPLETED trials: 0 - 9.
[INFO 07-10 11:37:47] Scheduler: Fetching data for trials: 0 - 9.
[ERROR 2023-07-10 11:37:47,573 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:37:47,588 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:37:47,616 MainProcess] boa: Trials so far: 10
Running trials:
Will Produce next trials from generation step: Sobol
Best trial so far: {1: {'Cosine8': 1.646672921687271}}
[INFO 07-10 11:37:47] Scheduler: Running trials [10]...
[INFO 07-10 11:37:48] Scheduler: Running trials [11]...
[INFO 07-10 11:37:49] Scheduler: Running trials [12]...
[INFO 07-10 11:37:50] Scheduler: Running trials [13]...
[INFO 07-10 11:37:51] Scheduler: Running trials [14]...
[INFO 07-10 11:37:52] Scheduler: Running trials [15]...
[INFO 07-10 11:37:54] Scheduler: Running trials [16]...
[INFO 07-10 11:37:56] Scheduler: Running trials [17]...
[INFO 07-10 11:37:58] Scheduler: Running trials [18]...
[INFO 07-10 11:37:59] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:37:59] Scheduler: Retrieved COMPLETED trials: 10 - 18.
[INFO 07-10 11:37:59] Scheduler: Fetching data for trials: 10 - 18.
[ERROR 2023-07-10 11:37:59,293 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:37:59,313 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:37:59,339 MainProcess] boa: Trials so far: 19
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {16: {'Cosine8': 1.47902090330447}}
[INFO 07-10 11:37:59] Scheduler: Running trials [19]...
[INFO 07-10 11:38:01] Scheduler: Running trials [20]...
[INFO 07-10 11:38:03] Scheduler: Running trials [21]...
[INFO 07-10 11:38:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:04] Scheduler: Retrieved COMPLETED trials: 19 - 21.
[INFO 07-10 11:38:04] Scheduler: Fetching data for trials: 19 - 21.
[ERROR 2023-07-10 11:38:04,536 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:04,558 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:04,584 MainProcess] boa: Trials so far: 22
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {19: {'Cosine8': 1.046563782988446}}
[INFO 07-10 11:38:05] Scheduler: Running trials [22]...
[INFO 07-10 11:38:06] Scheduler: Running trials [23]...
[INFO 07-10 11:38:09] Scheduler: Running trials [24]...
[INFO 07-10 11:38:09] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:09] Scheduler: Retrieved COMPLETED trials: 22 - 24.
[INFO 07-10 11:38:09] Scheduler: Fetching data for trials: 22 - 24.
[ERROR 2023-07-10 11:38:09,336 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:09,360 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:09,387 MainProcess] boa: Trials so far: 25
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {22: {'Cosine8': 0.6945835739165599}}
[INFO 07-10 11:38:10] Scheduler: Running trials [25]...
[INFO 07-10 11:38:12] Scheduler: Running trials [26]...
[INFO 07-10 11:38:16] Scheduler: Running trials [27]...
[INFO 07-10 11:38:17] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:17] Scheduler: Retrieved COMPLETED trials: 25 - 27.
[INFO 07-10 11:38:17] Scheduler: Fetching data for trials: 25 - 27.
[ERROR 2023-07-10 11:38:17,303 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:17,329 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:17,357 MainProcess] boa: Trials so far: 28
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {25: {'Cosine8': 0.4806960605264868}}
[INFO 07-10 11:38:17] Scheduler: Running trials [28]...
[INFO 07-10 11:38:20] Scheduler: Running trials [29]...
[INFO 07-10 11:38:22] Scheduler: Running trials [30]...
[INFO 07-10 11:38:23] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:23] Scheduler: Retrieved COMPLETED trials: 28 - 30.
[INFO 07-10 11:38:23] Scheduler: Fetching data for trials: 28 - 30.
[ERROR 2023-07-10 11:38:23,726 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:23,756 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:23,785 MainProcess] boa: Trials so far: 31
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {29: {'Cosine8': 0.41281812319182754}}
[INFO 07-10 11:38:24] Scheduler: Running trials [31]...
[INFO 07-10 11:38:25] Scheduler: Running trials [32]...
[INFO 07-10 11:38:28] Scheduler: Running trials [33]...
[INFO 07-10 11:38:29] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:29] Scheduler: Retrieved COMPLETED trials: 31 - 33.
[INFO 07-10 11:38:29] Scheduler: Fetching data for trials: 31 - 33.
[ERROR 2023-07-10 11:38:29,616 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:29,650 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:29,684 MainProcess] boa: Trials so far: 34
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {31: {'Cosine8': 0.36052556506256594}}
[INFO 07-10 11:38:30] Scheduler: Running trials [34]...
[INFO 07-10 11:38:31] Scheduler: Running trials [35]...
[INFO 07-10 11:38:33] Scheduler: Running trials [36]...
[INFO 07-10 11:38:34] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:34] Scheduler: Retrieved COMPLETED trials: 34 - 36.
[INFO 07-10 11:38:34] Scheduler: Fetching data for trials: 34 - 36.
[ERROR 2023-07-10 11:38:34,712 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:34,745 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:34,776 MainProcess] boa: Trials so far: 37
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {36: {'Cosine8': 0.30693978745638906}}
[INFO 07-10 11:38:34] Scheduler: Running trials [37]...
[INFO 07-10 11:38:36] Scheduler: Running trials [38]...
[INFO 07-10 11:38:37] Scheduler: Running trials [39]...
[INFO 07-10 11:38:38] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:38] Scheduler: Retrieved COMPLETED trials: 37 - 39.
[INFO 07-10 11:38:38] Scheduler: Fetching data for trials: 37 - 39.
[ERROR 2023-07-10 11:38:38,459 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:38,496 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:38,536 MainProcess] boa: Trials so far: 40
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {37: {'Cosine8': 0.2773584545190131}}
[INFO 07-10 11:38:39] Scheduler: Running trials [40]...
[INFO 07-10 11:38:40] Scheduler: Running trials [41]...
[INFO 07-10 11:38:42] Scheduler: Running trials [42]...
[INFO 07-10 11:38:43] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:43] Scheduler: Retrieved COMPLETED trials: 40 - 42.
[INFO 07-10 11:38:43] Scheduler: Fetching data for trials: 40 - 42.
[ERROR 2023-07-10 11:38:43,173 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:43,211 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:43,245 MainProcess] boa: Trials so far: 43
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {37: {'Cosine8': 0.2773584545190131}}
[INFO 07-10 11:38:47] Scheduler: Running trials [43]...
[INFO 07-10 11:38:48] Scheduler: Running trials [44]...
[INFO 07-10 11:38:50] Scheduler: Running trials [45]...
[INFO 07-10 11:38:51] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:51] Scheduler: Retrieved COMPLETED trials: 43 - 45.
[INFO 07-10 11:38:51] Scheduler: Fetching data for trials: 43 - 45.
[ERROR 2023-07-10 11:38:51,320 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:51,367 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:51,412 MainProcess] boa: Trials so far: 46
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {37: {'Cosine8': 0.2773584545190131}}
[INFO 07-10 11:38:51] Scheduler: Running trials [46]...
[INFO 07-10 11:38:53] Scheduler: Running trials [47]...
[INFO 07-10 11:38:54] Scheduler: Running trials [48]...
[INFO 07-10 11:38:55] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 07-10 11:38:55] Scheduler: Retrieved COMPLETED trials: 46 - 48.
[INFO 07-10 11:38:55] Scheduler: Fetching data for trials: 46 - 48.
[ERROR 2023-07-10 11:38:55,626 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:55,669 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:55,710 MainProcess] boa: Trials so far: 49
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {46: {'Cosine8': 0.24594534280361183}}
/Users/madelinescyphers/miniconda3/envs/boa-dev/lib/python3.10/site-packages/botorch/optim/initializers.py:224: BadInitialCandidatesWarning: Unable to find non-zero acquisition function values - initial conditions are being selected randomly.
warnings.warn(
[INFO 07-10 11:38:56] Scheduler: Running trials [49]...
[INFO 07-10 11:38:57] Scheduler: Retrieved COMPLETED trials: [49].
[INFO 07-10 11:38:57] Scheduler: Fetching data for trials: [49].
[ERROR 2023-07-10 11:38:57,314 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:57,359 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:57,400 MainProcess] boa: Trials so far: 50
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {46: {'Cosine8': 0.24594534280361183}}
[ERROR 2023-07-10 11:38:57,401 MainProcess] boa: Object <wrapper.Wrapper object at 0x158872d40> passed to `object_to_json` (of type <class 'wrapper.Wrapper'>, module: wrapper) is not registered with a corresponding encoder in ENCODER_REGISTRY.
[INFO 2023-07-10 11:38:57,448 MainProcess] boa: Saved JSON-serialized state of optimization to `/Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738/scheduler.json`.
Boa version: 0.8.3.dev1+g0636f51.d20230411
[INFO 2023-07-10 11:38:57,489 MainProcess] boa: Trials so far: 50
Running trials:
Will Produce next trials from generation step: GPEI
Best trial so far: {46: {'Cosine8': 0.24594534280361183}}
[INFO 2023-07-10 11:38:57,514 MainProcess] boa:
##############################################
Trials Completed!
BOA Experiment Run
Output Experiment Dir: /Users/madelinescyphers/Documents/projs_.nosync/boa/docs/examples/boa_runs_20230710T113738
Start Time: 20230710T113738
Version: 0.8.3.dev1+g0636f51.d20230411
End Time: 20230710T113857
Total Run Time: 79.21129965782166
trial_index arm_name trial_status generation_method Cosine8 x0 \
0 0 0_0 COMPLETED Sobol 2.029324 0.990547
1 1 1_0 COMPLETED Sobol 1.646673 0.262968
2 2 2_0 COMPLETED Sobol 2.144039 0.449725
3 3 3_0 COMPLETED Sobol 2.753313 0.431875
4 4 4_0 COMPLETED Sobol 3.049808 0.417872
5 5 5_0 COMPLETED Sobol 3.438582 0.895702
6 6 6_0 COMPLETED Sobol 2.069711 0.730564
7 7 7_0 COMPLETED Sobol 3.978929 0.119162
8 8 8_0 COMPLETED Sobol 2.448196 0.872235
9 9 9_0 COMPLETED Sobol 2.355544 0.286572
10 10 10_0 COMPLETED Sobol 3.009791 0.878739
11 11 11_0 COMPLETED Sobol 4.328626 0.923367
12 12 12_0 COMPLETED Sobol 1.768276 0.243641
13 13 13_0 COMPLETED Sobol 3.453395 0.687077
14 14 14_0 COMPLETED Sobol 3.690702 0.975360
15 15 15_0 COMPLETED Sobol 3.142783 0.429568
16 16 16_0 COMPLETED GPEI 1.479021 0.372412
17 17 17_0 COMPLETED GPEI 1.848654 0.133412
18 18 18_0 COMPLETED GPEI 1.719800 0.325406
19 19 19_0 COMPLETED GPEI 1.046564 0.402912
20 20 20_0 COMPLETED GPEI 1.218009 0.258951
21 21 21_0 COMPLETED GPEI 1.568101 0.481007
22 22 22_0 COMPLETED GPEI 0.694584 0.336467
23 23 23_0 COMPLETED GPEI 0.825017 0.352038
24 24 24_0 COMPLETED GPEI 1.557443 0.395534
25 25 25_0 COMPLETED GPEI 0.480696 0.315712
26 26 26_0 COMPLETED GPEI 0.666511 0.339306
27 27 27_0 COMPLETED GPEI 0.706937 0.291560
28 28 28_0 COMPLETED GPEI 0.462260 0.294458
29 29 29_0 COMPLETED GPEI 0.412818 0.381948
30 30 30_0 COMPLETED GPEI 0.779420 0.201541
31 31 31_0 COMPLETED GPEI 0.360526 0.384370
32 32 32_0 COMPLETED GPEI 0.496104 0.423386
33 33 33_0 COMPLETED GPEI 0.484507 0.316908
34 34 34_0 COMPLETED GPEI 0.511228 0.422185
35 35 35_0 COMPLETED GPEI 0.412813 0.391179
36 36 36_0 COMPLETED GPEI 0.306940 0.363556
37 37 37_0 COMPLETED GPEI 0.277358 0.347890
38 38 38_0 COMPLETED GPEI 1.136160 0.112641
39 39 39_0 COMPLETED GPEI 0.308610 0.341814
40 40 40_0 COMPLETED GPEI 0.301817 0.334515
41 41 41_0 COMPLETED GPEI 0.516885 0.264182
42 42 42_0 COMPLETED GPEI 0.343410 0.354746
43 43 43_0 COMPLETED GPEI 0.290229 0.348906
44 44 44_0 COMPLETED GPEI 0.526956 0.342680
45 45 45_0 COMPLETED GPEI 1.662781 0.399885
46 46 46_0 COMPLETED GPEI 0.245945 0.351724
47 47 47_0 COMPLETED GPEI 0.307105 0.380847
48 48 48_0 COMPLETED GPEI 0.484492 0.357150
49 49 49_0 COMPLETED GPEI 3.524827 0.675317
x1 x2 x3 x4 x5 x6 x7
0 0.288610 0.346135 0.301916 0.392528 0.110616 0.251962 0.694829
1 0.633469 0.616036 0.388401 0.427373 0.075654 0.736025 0.068805
2 0.480685 0.448159 0.548387 0.476969 0.054146 0.188638 0.978092
3 0.352470 0.052401 0.387660 0.960082 0.576974 0.864202 0.656100
4 0.647828 0.534399 0.174281 0.122930 0.771529 0.943236 0.778014
5 0.123685 0.017906 0.944492 0.357977 0.397257 0.967300 0.830473
6 0.433245 0.129033 0.007862 0.276506 0.861777 0.722698 0.493578
7 0.841748 0.990021 0.173180 0.671806 0.980801 0.351636 0.768843
8 0.815817 0.038919 0.380534 0.389911 0.711213 0.233549 0.776188
9 0.689979 0.266366 0.717407 0.132874 0.584265 0.825217 0.396867
10 0.405202 0.912348 0.580230 0.005852 0.909818 0.364704 0.253842
11 0.330759 0.834218 0.927327 0.997183 0.092835 0.869460 0.131116
12 0.347330 0.265216 0.186192 0.356840 0.319708 0.540039 0.912528
13 0.397473 0.776234 0.949205 0.041172 0.667479 0.779359 0.674563
14 0.064095 0.197106 0.757204 0.497103 0.936394 0.569573 0.615107
15 0.688117 0.034250 0.896887 0.059263 0.958642 0.881870 0.458131
16 0.593584 0.499630 0.420313 0.380171 0.086078 0.618598 0.190225
17 0.684324 0.681570 0.411335 0.436329 0.068753 0.885155 0.000000
18 0.594220 0.725250 0.252905 0.509522 0.030746 0.666139 0.000000
19 0.494737 0.378010 0.276616 0.380545 0.153921 0.527507 0.442987
20 0.508938 0.575936 0.340772 0.332884 0.000000 0.529066 0.350962
21 0.621265 0.352696 0.290243 0.383432 0.214157 0.624025 0.128180
22 0.445526 0.348530 0.228832 0.347404 0.025912 0.501000 0.434758
23 0.451447 0.468967 0.271408 0.324219 0.205404 0.402803 0.416627
24 0.549492 0.458918 0.248066 0.392106 0.047248 0.617300 0.554689
25 0.389980 0.319921 0.210277 0.304849 0.065001 0.386094 0.359033
26 0.340017 0.384153 0.180095 0.305986 0.086064 0.496477 0.330224
27 0.464385 0.268035 0.237875 0.283826 0.036284 0.310615 0.422044
28 0.372932 0.331051 0.213052 0.406540 0.045543 0.338189 0.331316
29 0.360701 0.308135 0.281827 0.271955 0.029180 0.377959 0.373382
30 0.402730 0.284366 0.215171 0.325871 0.130716 0.428317 0.285005
31 0.348921 0.333884 0.256066 0.351508 0.000000 0.323477 0.369886
32 0.378361 0.328701 0.251978 0.341967 0.000000 0.314916 0.250011
33 0.313341 0.327963 0.270480 0.343131 0.000000 0.330986 0.450200
34 0.353153 0.259158 0.230322 0.373841 0.044939 0.324527 0.406461
35 0.366165 0.387284 0.245393 0.318190 0.000000 0.297713 0.397175
36 0.368205 0.333058 0.286906 0.374388 0.000000 0.351965 0.367086
37 0.362567 0.370842 0.317156 0.376454 0.000000 0.351853 0.343095
38 0.189161 0.303689 0.489141 0.372734 0.511617 0.477609 0.238445
39 0.374499 0.338032 0.340505 0.376037 0.000000 0.299552 0.351262
40 0.340547 0.343663 0.377408 0.394207 0.000000 0.370891 0.312737
41 0.317618 0.370233 0.449533 0.350750 0.000000 0.364791 0.229735
42 0.345775 0.366915 0.346625 0.380737 0.078507 0.338985 0.337162
43 0.361392 0.379225 0.351250 0.419578 0.000000 0.344883 0.344597
44 0.344620 0.411267 0.366610 0.502446 0.000000 0.311529 0.329995
45 0.441332 0.581806 0.777294 0.626267 0.051651 0.416699 0.374300
46 0.369852 0.366687 0.389105 0.373427 0.000000 0.360029 0.356265
47 0.364766 0.371232 0.441443 0.358401 0.000000 0.356899 0.365381
48 0.284339 0.304645 0.428924 0.365574 0.001600 0.306166 0.270031
49 0.286011 0.860863 0.223397 0.281086 0.897188 0.753581 0.865855
##############################################
Get the Best Trial and Output All Trials#
best_fitted_trials uses the data to do a fitting from all trials and with the noise levels you provided (or if no noise levels was provided, it assumed an unknown level of noise and inferred the noise level from the trial runs)
trial = scheduler.best_fitted_trials()
trial
/Users/madelinescyphers/miniconda3/envs/boa-dev/lib/python3.10/site-packages/botorch/models/utils/assorted.py:201: InputDataWarning: Input data is not standardized. Please consider scaling the input to zero mean and unit variance.
warnings.warn(msg, InputDataWarning)
/Users/madelinescyphers/miniconda3/envs/boa-dev/lib/python3.10/site-packages/botorch/models/utils/assorted.py:201: InputDataWarning: Input data is not standardized. Please consider scaling the input to zero mean and unit variance.
warnings.warn(msg, InputDataWarning)
{49: {'params': {'x0': 0.35172402518319096,
'x1': 0.3698520404111157,
'x2': 0.3666874377748964,
'x3': 0.3891053518012792,
'x4': 0.3734271631500215,
'x5': 0.0,
'x6': 0.36002879332283894,
'x7': 0.35626455543974167},
'means': {'Cosine8': 0.2462324826494231},
'cov_matrix': {'Cosine8': {'Cosine8': 1.3891737938407046e-06}}}}
if you need the exact points of the best trial, maybe because you need the trial number of the best trial to plot results, or for any other reason, best_raw_trails does not do any fitting
trial = scheduler.best_raw_trials()
trial
{46: {'params': {'x0': 0.35172402518319096,
'x1': 0.3698520404111157,
'x2': 0.3666874377748964,
'x3': 0.3891053518012792,
'x4': 0.3734271631500215,
'x5': 0.0,
'x6': 0.36002879332283894,
'x7': 0.35626455543974167},
'means': {'Cosine8': 0.24594534280361183},
'cov_matrix': {'Cosine8': {'Cosine8': 0.0}}}}
Output a DataFrame of All Trials#
boa.scheduler_to_df(scheduler)
| trial_index | arm_name | trial_status | generation_method | Cosine8 | x0 | x1 | x2 | x3 | x4 | x5 | x6 | x7 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0_0 | COMPLETED | Sobol | 2.029324 | 0.990547 | 0.288610 | 0.346135 | 0.301916 | 0.392528 | 0.110616 | 0.251962 | 0.694829 |
| 1 | 1 | 1_0 | COMPLETED | Sobol | 1.646673 | 0.262968 | 0.633469 | 0.616036 | 0.388401 | 0.427373 | 0.075654 | 0.736025 | 0.068805 |
| 2 | 2 | 2_0 | COMPLETED | Sobol | 2.144039 | 0.449725 | 0.480685 | 0.448159 | 0.548387 | 0.476969 | 0.054146 | 0.188638 | 0.978092 |
| 3 | 3 | 3_0 | COMPLETED | Sobol | 2.753313 | 0.431875 | 0.352470 | 0.052401 | 0.387660 | 0.960082 | 0.576974 | 0.864202 | 0.656100 |
| 4 | 4 | 4_0 | COMPLETED | Sobol | 3.049808 | 0.417872 | 0.647828 | 0.534399 | 0.174281 | 0.122930 | 0.771529 | 0.943236 | 0.778014 |
| 5 | 5 | 5_0 | COMPLETED | Sobol | 3.438582 | 0.895702 | 0.123685 | 0.017906 | 0.944492 | 0.357977 | 0.397257 | 0.967300 | 0.830473 |
| 6 | 6 | 6_0 | COMPLETED | Sobol | 2.069711 | 0.730564 | 0.433245 | 0.129033 | 0.007862 | 0.276506 | 0.861777 | 0.722698 | 0.493578 |
| 7 | 7 | 7_0 | COMPLETED | Sobol | 3.978929 | 0.119162 | 0.841748 | 0.990021 | 0.173180 | 0.671806 | 0.980801 | 0.351636 | 0.768843 |
| 8 | 8 | 8_0 | COMPLETED | Sobol | 2.448196 | 0.872235 | 0.815817 | 0.038919 | 0.380534 | 0.389911 | 0.711213 | 0.233549 | 0.776188 |
| 9 | 9 | 9_0 | COMPLETED | Sobol | 2.355544 | 0.286572 | 0.689979 | 0.266366 | 0.717407 | 0.132874 | 0.584265 | 0.825217 | 0.396867 |
| 10 | 10 | 10_0 | COMPLETED | Sobol | 3.009791 | 0.878739 | 0.405202 | 0.912348 | 0.580230 | 0.005852 | 0.909818 | 0.364704 | 0.253842 |
| 11 | 11 | 11_0 | COMPLETED | Sobol | 4.328626 | 0.923367 | 0.330759 | 0.834218 | 0.927327 | 0.997183 | 0.092835 | 0.869460 | 0.131116 |
| 12 | 12 | 12_0 | COMPLETED | Sobol | 1.768276 | 0.243641 | 0.347330 | 0.265216 | 0.186192 | 0.356840 | 0.319708 | 0.540039 | 0.912528 |
| 13 | 13 | 13_0 | COMPLETED | Sobol | 3.453395 | 0.687077 | 0.397473 | 0.776234 | 0.949205 | 0.041172 | 0.667479 | 0.779359 | 0.674563 |
| 14 | 14 | 14_0 | COMPLETED | Sobol | 3.690702 | 0.975360 | 0.064095 | 0.197106 | 0.757204 | 0.497103 | 0.936394 | 0.569573 | 0.615107 |
| 15 | 15 | 15_0 | COMPLETED | Sobol | 3.142783 | 0.429568 | 0.688117 | 0.034250 | 0.896887 | 0.059263 | 0.958642 | 0.881870 | 0.458131 |
| 16 | 16 | 16_0 | COMPLETED | GPEI | 1.479021 | 0.372412 | 0.593584 | 0.499630 | 0.420313 | 0.380171 | 0.086078 | 0.618598 | 0.190225 |
| 17 | 17 | 17_0 | COMPLETED | GPEI | 1.848654 | 0.133412 | 0.684324 | 0.681570 | 0.411335 | 0.436329 | 0.068753 | 0.885155 | 0.000000 |
| 18 | 18 | 18_0 | COMPLETED | GPEI | 1.719800 | 0.325406 | 0.594220 | 0.725250 | 0.252905 | 0.509522 | 0.030746 | 0.666139 | 0.000000 |
| 19 | 19 | 19_0 | COMPLETED | GPEI | 1.046564 | 0.402912 | 0.494737 | 0.378010 | 0.276616 | 0.380545 | 0.153921 | 0.527507 | 0.442987 |
| 20 | 20 | 20_0 | COMPLETED | GPEI | 1.218009 | 0.258951 | 0.508938 | 0.575936 | 0.340772 | 0.332884 | 0.000000 | 0.529066 | 0.350962 |
| 21 | 21 | 21_0 | COMPLETED | GPEI | 1.568101 | 0.481007 | 0.621265 | 0.352696 | 0.290243 | 0.383432 | 0.214157 | 0.624025 | 0.128180 |
| 22 | 22 | 22_0 | COMPLETED | GPEI | 0.694584 | 0.336467 | 0.445526 | 0.348530 | 0.228832 | 0.347404 | 0.025912 | 0.501000 | 0.434758 |
| 23 | 23 | 23_0 | COMPLETED | GPEI | 0.825017 | 0.352038 | 0.451447 | 0.468967 | 0.271408 | 0.324219 | 0.205404 | 0.402803 | 0.416627 |
| 24 | 24 | 24_0 | COMPLETED | GPEI | 1.557443 | 0.395534 | 0.549492 | 0.458918 | 0.248066 | 0.392106 | 0.047248 | 0.617300 | 0.554689 |
| 25 | 25 | 25_0 | COMPLETED | GPEI | 0.480696 | 0.315712 | 0.389980 | 0.319921 | 0.210277 | 0.304849 | 0.065001 | 0.386094 | 0.359033 |
| 26 | 26 | 26_0 | COMPLETED | GPEI | 0.666511 | 0.339306 | 0.340017 | 0.384153 | 0.180095 | 0.305986 | 0.086064 | 0.496477 | 0.330224 |
| 27 | 27 | 27_0 | COMPLETED | GPEI | 0.706937 | 0.291560 | 0.464385 | 0.268035 | 0.237875 | 0.283826 | 0.036284 | 0.310615 | 0.422044 |
| 28 | 28 | 28_0 | COMPLETED | GPEI | 0.462260 | 0.294458 | 0.372932 | 0.331051 | 0.213052 | 0.406540 | 0.045543 | 0.338189 | 0.331316 |
| 29 | 29 | 29_0 | COMPLETED | GPEI | 0.412818 | 0.381948 | 0.360701 | 0.308135 | 0.281827 | 0.271955 | 0.029180 | 0.377959 | 0.373382 |
| 30 | 30 | 30_0 | COMPLETED | GPEI | 0.779420 | 0.201541 | 0.402730 | 0.284366 | 0.215171 | 0.325871 | 0.130716 | 0.428317 | 0.285005 |
| 31 | 31 | 31_0 | COMPLETED | GPEI | 0.360526 | 0.384370 | 0.348921 | 0.333884 | 0.256066 | 0.351508 | 0.000000 | 0.323477 | 0.369886 |
| 32 | 32 | 32_0 | COMPLETED | GPEI | 0.496104 | 0.423386 | 0.378361 | 0.328701 | 0.251978 | 0.341967 | 0.000000 | 0.314916 | 0.250011 |
| 33 | 33 | 33_0 | COMPLETED | GPEI | 0.484507 | 0.316908 | 0.313341 | 0.327963 | 0.270480 | 0.343131 | 0.000000 | 0.330986 | 0.450200 |
| 34 | 34 | 34_0 | COMPLETED | GPEI | 0.511228 | 0.422185 | 0.353153 | 0.259158 | 0.230322 | 0.373841 | 0.044939 | 0.324527 | 0.406461 |
| 35 | 35 | 35_0 | COMPLETED | GPEI | 0.412813 | 0.391179 | 0.366165 | 0.387284 | 0.245393 | 0.318190 | 0.000000 | 0.297713 | 0.397175 |
| 36 | 36 | 36_0 | COMPLETED | GPEI | 0.306940 | 0.363556 | 0.368205 | 0.333058 | 0.286906 | 0.374388 | 0.000000 | 0.351965 | 0.367086 |
| 37 | 37 | 37_0 | COMPLETED | GPEI | 0.277358 | 0.347890 | 0.362567 | 0.370842 | 0.317156 | 0.376454 | 0.000000 | 0.351853 | 0.343095 |
| 38 | 38 | 38_0 | COMPLETED | GPEI | 1.136160 | 0.112641 | 0.189161 | 0.303689 | 0.489141 | 0.372734 | 0.511617 | 0.477609 | 0.238445 |
| 39 | 39 | 39_0 | COMPLETED | GPEI | 0.308610 | 0.341814 | 0.374499 | 0.338032 | 0.340505 | 0.376037 | 0.000000 | 0.299552 | 0.351262 |
| 40 | 40 | 40_0 | COMPLETED | GPEI | 0.301817 | 0.334515 | 0.340547 | 0.343663 | 0.377408 | 0.394207 | 0.000000 | 0.370891 | 0.312737 |
| 41 | 41 | 41_0 | COMPLETED | GPEI | 0.516885 | 0.264182 | 0.317618 | 0.370233 | 0.449533 | 0.350750 | 0.000000 | 0.364791 | 0.229735 |
| 42 | 42 | 42_0 | COMPLETED | GPEI | 0.343410 | 0.354746 | 0.345775 | 0.366915 | 0.346625 | 0.380737 | 0.078507 | 0.338985 | 0.337162 |
| 43 | 43 | 43_0 | COMPLETED | GPEI | 0.290229 | 0.348906 | 0.361392 | 0.379225 | 0.351250 | 0.419578 | 0.000000 | 0.344883 | 0.344597 |
| 44 | 44 | 44_0 | COMPLETED | GPEI | 0.526956 | 0.342680 | 0.344620 | 0.411267 | 0.366610 | 0.502446 | 0.000000 | 0.311529 | 0.329995 |
| 45 | 45 | 45_0 | COMPLETED | GPEI | 1.662781 | 0.399885 | 0.441332 | 0.581806 | 0.777294 | 0.626267 | 0.051651 | 0.416699 | 0.374300 |
| 46 | 46 | 46_0 | COMPLETED | GPEI | 0.245945 | 0.351724 | 0.369852 | 0.366687 | 0.389105 | 0.373427 | 0.000000 | 0.360029 | 0.356265 |
| 47 | 47 | 47_0 | COMPLETED | GPEI | 0.307105 | 0.380847 | 0.364766 | 0.371232 | 0.441443 | 0.358401 | 0.000000 | 0.356899 | 0.365381 |
| 48 | 48 | 48_0 | COMPLETED | GPEI | 0.484492 | 0.357150 | 0.284339 | 0.304645 | 0.428924 | 0.365574 | 0.001600 | 0.306166 | 0.270031 |
| 49 | 49 | 49_0 | COMPLETED | GPEI | 3.524827 | 0.675317 | 0.286011 | 0.860863 | 0.223397 | 0.281086 | 0.897188 | 0.753581 | 0.865855 |