Xopt Parallel Examples¶

Xopt provides methods to parallelize optimizations using Processes, Threads, MPI, and Dask using the concurrent.futures interface as defined in https://www.python.org/dev/peps/pep-3148/ .

In [1]:

from xopt import AsynchronousXopt as Xopt

from xopt import AsynchronousXopt as Xopt

In [2]:

# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)

# Helpers for this notebook import multiprocessing from concurrent.futures import ProcessPoolExecutor from dask.distributed import Client import matplotlib.pyplot as plt import pandas as pd from concurrent.futures import ThreadPoolExecutor import os # Notebook printing output # from xopt import output_notebook # output_notebook() N_CPUS = multiprocessing.cpu_count() N_CPUS # directory for data. os.makedirs("temp", exist_ok=True)

The Xopt object can be instantiated from a JSON or YAML file, or a dict, with the proper structure.

Here we will make one

In [3]:

# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X

# Make a proper input file. YAML = """ max_evaluations: 1000 generator: name: cnsga output_path: temp population_size: 64 evaluator: function: xopt.resources.test_functions.tnk.evaluate_TNK function_kwargs: sleep: 0 random_sleep: 0.1 vocs: variables: x1: [0, 3.14159] x2: [0, 3.14159] objectives: {y1: MINIMIZE, y2: MINIMIZE} constraints: c1: [GREATER_THAN, 0] c2: [LESS_THAN, 0.5] constants: {a: dummy_constant} """ X = Xopt(YAML) X

Out[3]:

            Xopt
________________________________
Version: 2.6.8.dev18+g6fb143c55.d20251203
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
  supports_single_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [4]:

%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)

%%timeit # Check that the average time is close to random_sleep X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)

95.2 ms ± 7.99 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]:

%%time
X.run()

%%time X.run()

CPU times: user 3.54 s, sys: 22.8 ms, total: 3.56 s
Wall time: 1min 44s

Processes¶

In [6]:

%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

%%time X = Xopt(YAML) with ProcessPoolExecutor(max_workers=N_CPUS) as executor: X.evaluator.executor = executor X.evaluator.max_workers = N_CPUS X.run() len(X.data)

CPU times: user 3.51 s, sys: 143 ms, total: 3.66 s
Wall time: 26.4 s

Out[6]:

1000

Threads¶

Continue running, this time with threads.

In [7]:

%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

%%time X = Xopt(YAML) with ThreadPoolExecutor(max_workers=N_CPUS) as executor: X.evaluator.executor = executor X.evaluator.max_workers = N_CPUS X.run() len(X.data)

CPU times: user 3.26 s, sys: 70.3 ms, total: 3.33 s
Wall time: 25.6 s

Out[7]:

1000

MPI¶

The test.yaml file completely defines the problem. We will also direct the logging to an xopt.log file. The following invocation recruits 4 MPI workers to solve this problem.

We can also continue by calling .save with a JSON filename. This will write all of previous results into the file.

In [8]:

X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml

X = Xopt(YAML) X.dump("test.yaml") # Write this input to file !cat test.yaml

data: null
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
  supports_single_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [9]:

%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml

%%time !mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 8
slots that were requested by the application:

  python

Either request fewer procs for your application, or make more slots
available for use.

A "slot" is the PRRTE term for an allocatable unit where we can
launch a process.  The number of slots available are defined by the
environment in which PRRTE processes are run:

  1. Hostfile, via "slots=N" clauses (N defaults to number of
     processor cores if not provided)
  2. The --host command line parameter, via a ":N" suffix on the
     hostname (N defaults to 1 if not provided)
  3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
  4. If none of a hostfile, the --host command line parameter, or an
     RM is present, PRRTE defaults to the number of processor cores

In all the above cases, if you want PRRTE to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.

Alternatively, you can use the --map-by :OVERSUBSCRIBE option to ignore the
number of available slots when deciding the number of processes to
launch.
--------------------------------------------------------------------------

CPU times: user 467 μs, sys: 8.09 ms, total: 8.56 ms
Wall time: 148 ms

In [10]:

!tail xopt.log

!tail xopt.log

tail: cannot open 'xopt.log' for reading: No such file or directory

Dask¶

In [11]:

client = Client()
executor = client.get_executor()
client

client = Client() executor = client.get_executor() client

Out[11]:

Client

Client-a9ff5bb6-d077-11f0-9ca9-000d3a317e0e

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

20ce8f20

Dashboard: http://127.0.0.1:8787/status	Workers: 4
Total threads: 4	Total memory: 15.62 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-fa0e7223-f6ce-4874-8852-450cd4190ab8

Comm: tcp://127.0.0.1:37315	Workers: 0
Dashboard: http://127.0.0.1:8787/status	Total threads: 0
Started: Just now	Total memory: 0 B

Workers

Worker: 0

Comm: tcp://127.0.0.1:37457	Total threads: 1
Dashboard: http://127.0.0.1:38745/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:42825
Local directory: /tmp/dask-scratch-space/worker-ccqz2im8

Worker: 1

Comm: tcp://127.0.0.1:45127	Total threads: 1
Dashboard: http://127.0.0.1:46061/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:35901
Local directory: /tmp/dask-scratch-space/worker-7dfctg3u

Worker: 2

Comm: tcp://127.0.0.1:42117	Total threads: 1
Dashboard: http://127.0.0.1:39577/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:44981
Local directory: /tmp/dask-scratch-space/worker-vz6q5krq

Worker: 3

Comm: tcp://127.0.0.1:35439	Total threads: 1
Dashboard: http://127.0.0.1:37215/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:43687
Local directory: /tmp/dask-scratch-space/worker-xua43_5i

In [12]:

%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)

%%time X = Xopt(YAML) X.evaluator.executor = executor X.evaluator.max_workers = N_CPUS X.run() len(X.data)

CPU times: user 7.17 s, sys: 647 ms, total: 7.82 s
Wall time: 30.4 s

Out[12]:

1000

Load output into Pandas¶

This algorithm writes two types of files: gen_{i}.json with all of the new individuals evaluated in a generation, and pop_{i}.json with the latest best population. Xopt provides some functions to load these easily into a Pandas dataframe for further analysis.

In [13]:

X.data

X.data

Out[13]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error
2	1.089640	2.230736	dummy_constant	1.089640	2.230736	5.108371	3.343124	0.104015	False
1	0.772432	0.180022	dummy_constant	0.772432	0.180022	-0.284256	0.176605	0.145912	False
2	0.586120	0.632615	dummy_constant	0.586120	0.632615	-0.338221	0.025003	0.156089	False
3	2.921642	2.560153	dummy_constant	2.921642	2.560153	14.040933	10.108584	0.182092	False
4	2.725453	1.170002	dummy_constant	2.725453	1.170002	7.699089	5.401545	0.021157	False
...	...	...	...	...	...	...	...	...	...
995	0.522976	0.743470	dummy_constant	0.522976	0.743470	-0.081014	0.059806	0.092954	False
996	1.073184	0.184681	dummy_constant	1.073184	0.184681	0.277347	0.427966	0.046406	False
997	0.981384	0.168502	dummy_constant	0.981384	0.168502	0.082777	0.341621	0.036686	False
998	0.730416	0.155436	dummy_constant	0.730416	0.155436	-0.344597	0.171816	0.096178	False
999	0.522976	0.836591	dummy_constant	0.522976	0.836591	0.061823	0.113822	0.198677	False

1000 rows × 9 columns

In [14]:

df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]

df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1) df[df["feasible"]]

Out[14]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error	feasible_c1	feasible_c2	feasible
57	1.002910	0.294021	dummy_constant	1.002910	0.294021	0.107176	0.295346	0.152849	False	True	True	True
192	1.095427	0.367512	dummy_constant	1.095427	0.367512	0.290027	0.372087	0.043260	False	True	True	True
194	1.007448	0.831033	dummy_constant	1.007448	0.831033	0.701551	0.367086	0.161134	False	True	True	True
235	1.046172	0.664853	dummy_constant	1.046172	0.664853	0.629851	0.325480	0.161129	False	True	True	True
237	0.526827	0.986664	dummy_constant	0.526827	0.986664	0.250352	0.237562	0.008427	False	True	True	True
...	...	...	...	...	...	...	...	...	...	...	...	...
991	0.522976	1.126629	dummy_constant	0.522976	1.126629	0.464436	0.393191	0.002795	False	True	True	True
993	0.522976	0.885269	dummy_constant	0.522976	0.885269	0.120357	0.148960	0.071302	False	True	True	True
996	1.073184	0.184681	dummy_constant	1.073184	0.184681	0.277347	0.427966	0.046406	False	True	True	True
997	0.981384	0.168502	dummy_constant	0.981384	0.168502	0.082777	0.341621	0.036686	False	True	True	True
999	0.522976	0.836591	dummy_constant	0.522976	0.836591	0.061823	0.113822	0.198677	False	True	True	True

386 rows × 12 columns

In [15]:

# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

# Plot the feasible ones feasible_df = df[df["feasible"]] feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

In [16]:

# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

# Plot the infeasible ones infeasible_df = df[~df["feasible"]] infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

In [17]:

# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")

# This is the final population df1 = X.generator.population df1.plot("y1", "y2", kind="scatter").set_aspect("equal")

matplotlib plotting¶

You can always use matplotlib for customizable plotting

In [18]:

# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()

# Extract objectives from output k1, k2 = "y1", "y2" fig, ax = plt.subplots(figsize=(6, 6)) ax.scatter( infeasible_df[k1], infeasible_df[k2], color="blue", marker=".", alpha=0.5, label="infeasible", ) ax.scatter( feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible" ) ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population") ax.set_xlabel(k1) ax.set_ylabel(k2) ax.set_aspect("auto") ax.set_title("Xopt's CNSGA algorithm") plt.legend()

Out[18]:

<matplotlib.legend.Legend at 0x7f44c8096e40>

In [19]:

# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml

# Cleanup #!rm -r dask-worker-space !rm -r temp !rm xopt.log* !rm test.yaml

rm: cannot remove 'xopt.log*': No such file or directory