Xopt Parallel Examples¶

Xopt provides methods to parallelize optimizations using Processes, Threads, MPI, and Dask using the concurrent.futures interface as defined in https://www.python.org/dev/peps/pep-3148/ .

In [1]:

Copied!

from xopt import AsynchronousXopt as Xopt
from xopt import AsynchronousXopt as Xopt

In [2]:

Copied!





# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)
# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)

The Xopt object can be instantiated from a JSON or YAML file, or a dict, with the proper structure.

Here we will make one

In [3]:

Copied!





# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X
# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X

Out[3]:

            Xopt
________________________________
Version: 0.1.dev1947+g7831d28.d20250426
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [4]:

Copied!

%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)
%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)

100 ms ± 20.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]:

Copied!

%%time
X.run()
%%time
X.run()

CPU times: user 3.27 s, sys: 26.4 ms, total: 3.3 s
Wall time: 1min 41s

Processes¶

In [6]:

Copied!





%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)
%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

CPU times: user 3.28 s, sys: 147 ms, total: 3.43 s
Wall time: 25.1 s

Out[6]:

Threads¶

Continue running, this time with threads.

In [7]:

Copied!





%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)
%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

CPU times: user 3.17 s, sys: 57.1 ms, total: 3.23 s
Wall time: 26.8 s

Out[7]:

MPI¶

The test.yaml file completely defines the problem. We will also direct the logging to an xopt.log file. The following invocation recruits 4 MPI workers to solve this problem.

We can also continue by calling .save with a JSON filename. This will write all of previous results into the file.

In [8]:

Copied!

X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml
X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml

data: null
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [9]:

Copied!

%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml
%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml

Namespace(input_file='test.yaml', logfile='xopt.log', verbose=2, asynchronous=True)
Parallel execution with 8 workers
Enabling async mode

Initialized generator cnsga
Created toolbox with 2 variables, 2 constraints, and 2 objectives.
    Using selection algorithm: nsga2

            Xopt
________________________________
Version: 0.1.dev1947+g7831d28.d20250426
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159


Running Xopt

Xopt is done. Max evaluations 1000 reached.

CPU times: user 169 ms, sys: 50.5 ms, total: 220 ms
Wall time: 23 s

In [10]:

Copied!

!tail xopt.log
!tail xopt.log

2025-04-26T21:04:44+0000 - xopt - INFO - Parallel execution with 8 workers
2025-04-26T21:04:44+0000 - xopt - INFO - Enabling async mode
2025-04-26T21:04:44+0000 - xopt.generator - INFO - Initialized generator cnsga
2025-04-26T21:04:44+0000 - xopt.generators.ga.cnsga - INFO - Created toolbox with 2 variables, 2 constraints, and 2 objectives.
2025-04-26T21:04:44+0000 - xopt.generators.ga.cnsga - INFO -     Using selection algorithm: nsga2
2025-04-26T21:04:44+0000 - xopt.base - INFO - Running Xopt
2025-04-26T21:05:04+0000 - xopt.base - INFO - Xopt is done. Max evaluations 1000 reached.

Dask¶

In [11]:

Copied!

client = Client()
executor = client.get_executor()
client
client = Client()
executor = client.get_executor()
client

Out[11]:

Client

Client-20e23659-22e2-11f0-9b79-6045bdb65fec

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

0448311b

Dashboard: http://127.0.0.1:8787/status	Workers: 4
Total threads: 4	Total memory: 15.61 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-f471143b-1139-4ce3-a25d-6d036ec643df

Comm: tcp://127.0.0.1:46433	Workers: 0
Dashboard: http://127.0.0.1:8787/status	Total threads: 0
Started: Just now	Total memory: 0 B

Workers

Worker: 0

Comm: tcp://127.0.0.1:43271	Total threads: 1
Dashboard: http://127.0.0.1:32875/status	Memory: 3.90 GiB
Nanny: tcp://127.0.0.1:38377
Local directory: /tmp/dask-scratch-space/worker-k1rb1h8t

Worker: 1

Comm: tcp://127.0.0.1:43499	Total threads: 1
Dashboard: http://127.0.0.1:33501/status	Memory: 3.90 GiB
Nanny: tcp://127.0.0.1:39357
Local directory: /tmp/dask-scratch-space/worker-iwzsn8fy

Worker: 2

Comm: tcp://127.0.0.1:44665	Total threads: 1
Dashboard: http://127.0.0.1:36729/status	Memory: 3.90 GiB
Nanny: tcp://127.0.0.1:37055
Local directory: /tmp/dask-scratch-space/worker-jb7q42ig

Worker: 3

Comm: tcp://127.0.0.1:46731	Total threads: 1
Dashboard: http://127.0.0.1:43191/status	Memory: 3.90 GiB
Nanny: tcp://127.0.0.1:43375
Local directory: /tmp/dask-scratch-space/worker-lpwvzrk4

In [12]:

Copied!





%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)
%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)

CPU times: user 6.85 s, sys: 636 ms, total: 7.49 s
Wall time: 30.8 s

Out[12]:

Load output into Pandas¶

This algorithm writes two types of files: gen_{i}.json with all of the new individuals evaluated in a generation, and pop_{i}.json with the latest best population. Xopt provides some functions to load these easily into a Pandas dataframe for further analysis.

In [13]:

Copied!

X.data
X.data

Out[13]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error
1	0.227790	0.549314	dummy_constant	0.227790	0.549314	-0.746364	0.076530	0.086456	False
1	2.738727	2.605098	dummy_constant	2.738727	2.605098	13.195051	9.443333	0.109034	False
2	0.600929	1.603917	dummy_constant	0.600929	1.603917	1.848286	1.228819	0.024060	False
3	1.880928	2.250522	dummy_constant	1.880928	2.250522	7.588460	4.971287	0.166876	False
4	3.099413	0.840913	dummy_constant	3.099413	0.840913	9.359088	6.873169	0.163600	False
...	...	...	...	...	...	...	...	...	...
995	0.937704	0.464901	dummy_constant	0.937704	0.464901	0.048389	0.192816	0.109793	False
996	0.840804	0.669984	dummy_constant	0.840804	0.669984	0.178688	0.145042	0.151432	False
997	1.106431	0.079145	dummy_constant	1.106431	0.079145	0.188927	0.544877	0.012535	False
998	0.316210	0.929427	dummy_constant	0.316210	0.929427	-0.087124	0.218187	0.065119	False
999	0.977654	0.289925	dummy_constant	0.977654	0.289925	0.049820	0.272284	0.121174	False

1000 rows × 9 columns

In [14]:

Copied!

df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]
df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]

Out[14]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error	feasible_c1	feasible_c2	feasible
13	0.548036	0.799371	dummy_constant	0.548036	0.799371	0.037518	0.091930	0.005263	False	True	True	True
27	1.101708	0.547513	dummy_constant	1.101708	0.547513	0.467843	0.364310	0.052608	False	True	True	True
123	0.570708	1.175711	dummy_constant	0.570708	1.175711	0.649625	0.461585	0.099300	False	True	True	True
129	1.121069	0.766973	dummy_constant	1.121069	0.766973	0.943510	0.457001	0.115202	False	True	True	True
193	1.106431	0.547513	dummy_constant	1.106431	0.547513	0.475865	0.370016	0.032470	False	True	True	True
...	...	...	...	...	...	...	...	...	...	...	...	...
993	1.097911	0.803067	dummy_constant	1.097911	0.803067	0.928109	0.449348	0.178250	False	True	True	True
994	0.749514	0.780963	dummy_constant	0.749514	0.780963	0.077030	0.141198	0.122507	False	True	True	True
995	0.937704	0.464901	dummy_constant	0.937704	0.464901	0.048389	0.192816	0.109793	False	True	True	True
996	0.840804	0.669984	dummy_constant	0.840804	0.669984	0.178688	0.145042	0.151432	False	True	True	True
999	0.977654	0.289925	dummy_constant	0.977654	0.289925	0.049820	0.272284	0.121174	False	True	True	True

437 rows × 12 columns

In [15]:

Copied!

# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")
# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

No description has been provided for this image

In [16]:

Copied!

# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")
# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

In [17]:

Copied!

# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")
# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")

matplotlib plotting¶

You can always use matplotlib for customizable plotting

In [18]:

Copied!





# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()
# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()

Out[18]:

<matplotlib.legend.Legend at 0x7f0cfe33b380>

In [19]:

Copied!





# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml
# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml