Xopt Parallel Examples¶

Xopt provides methods to parallelize optimizations using Processes, Threads, MPI, and Dask using the concurrent.futures interface as defined in https://www.python.org/dev/peps/pep-3148/ .

In [1]:

Copied!

from xopt import AsynchronousXopt as Xopt
from xopt import AsynchronousXopt as Xopt

In [2]:

Copied!





# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)
# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)

The Xopt object can be instantiated from a JSON or YAML file, or a dict, with the proper structure.

Here we will make one

In [3]:

Copied!





# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X
# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X

Out[3]:

            Xopt
________________________________
Version: 2.6.4.dev6+g92995519.d20250623
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [4]:

Copied!

%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)
%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)

96.8 ms ± 18.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]:

Copied!

%%time
X.run()
%%time
X.run()

CPU times: user 3.33 s, sys: 25.6 ms, total: 3.35 s
Wall time: 1min 45s

Processes¶

In [6]:

Copied!





%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)
%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

CPU times: user 3.33 s, sys: 116 ms, total: 3.45 s
Wall time: 26.2 s

Out[6]:

Threads¶

Continue running, this time with threads.

In [7]:

Copied!





%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)
%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

CPU times: user 3.13 s, sys: 55.8 ms, total: 3.19 s
Wall time: 26 s

Out[7]:

MPI¶

The test.yaml file completely defines the problem. We will also direct the logging to an xopt.log file. The following invocation recruits 4 MPI workers to solve this problem.

We can also continue by calling .save with a JSON filename. This will write all of previous results into the file.

In [8]:

Copied!

X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml
X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml

data: null
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [9]:

Copied!

%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml
%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml

Namespace(input_file='test.yaml', logfile='xopt.log', verbose=2, asynchronous=True)
Parallel execution with 8 workers
Enabling async mode

Initialized generator cnsga
Created toolbox with 2 variables, 2 constraints, and 2 objectives.
    Using selection algorithm: nsga2

            Xopt
________________________________
Version: 2.6.4.dev6+g92995519.d20250623
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159


Running Xopt

Xopt is done. Max evaluations 1000 reached.

CPU times: user 183 ms, sys: 42.2 ms, total: 225 ms
Wall time: 24.1 s

In [10]:

Copied!

!tail xopt.log
!tail xopt.log

2025-06-23T19:51:21+0000 - xopt - INFO - Parallel execution with 8 workers
2025-06-23T19:51:21+0000 - xopt - INFO - Enabling async mode
2025-06-23T19:51:21+0000 - xopt.generator - INFO - Initialized generator cnsga
2025-06-23T19:51:21+0000 - xopt.generators.ga.cnsga - INFO - Created toolbox with 2 variables, 2 constraints, and 2 objectives.
2025-06-23T19:51:21+0000 - xopt.generators.ga.cnsga - INFO -     Using selection algorithm: nsga2
2025-06-23T19:51:21+0000 - xopt.base - INFO - Running Xopt
2025-06-23T19:51:41+0000 - xopt.base - INFO - Xopt is done. Max evaluations 1000 reached.

Dask¶

In [11]:

Copied!

client = Client()
executor = client.get_executor()
client
client = Client()
executor = client.get_executor()
client

Out[11]:

Client

Client-7cdc7347-506b-11f0-9c35-7ced8d55c0f9

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

17e98acc

Dashboard: http://127.0.0.1:8787/status	Workers: 4
Total threads: 4	Total memory: 15.62 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-a62c5a0b-3d9e-4d9f-bf0f-f70cf7b0573a

Comm: tcp://127.0.0.1:38015	Workers: 0
Dashboard: http://127.0.0.1:8787/status	Total threads: 0
Started: Just now	Total memory: 0 B

Workers

Worker: 0

Comm: tcp://127.0.0.1:39817	Total threads: 1
Dashboard: http://127.0.0.1:46145/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:39031
Local directory: /tmp/dask-scratch-space/worker-ufohfmvi

Worker: 1

Comm: tcp://127.0.0.1:41515	Total threads: 1
Dashboard: http://127.0.0.1:44349/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:39675
Local directory: /tmp/dask-scratch-space/worker-zwrbz56h

Worker: 2

Comm: tcp://127.0.0.1:38783	Total threads: 1
Dashboard: http://127.0.0.1:38979/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:38465
Local directory: /tmp/dask-scratch-space/worker-r3rb0out

Worker: 3

Comm: tcp://127.0.0.1:39521	Total threads: 1
Dashboard: http://127.0.0.1:45605/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:39877
Local directory: /tmp/dask-scratch-space/worker-qnyph2pr

In [12]:

Copied!





%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)
%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)

CPU times: user 6.57 s, sys: 648 ms, total: 7.22 s
Wall time: 29.9 s

Out[12]:

Load output into Pandas¶

This algorithm writes two types of files: gen_{i}.json with all of the new individuals evaluated in a generation, and pop_{i}.json with the latest best population. Xopt provides some functions to load these easily into a Pandas dataframe for further analysis.

In [13]:

Copied!

X.data
X.data

Out[13]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error
4	0.589552	0.229676	dummy_constant	0.589552	0.229676	-0.693974	0.081095	0.005751	False
1	2.980390	1.551153	dummy_constant	2.980390	1.551153	10.271294	7.257257	0.077717	False
2	1.747979	2.018535	dummy_constant	1.747979	2.018535	6.088821	3.863399	0.029138	False
3	0.739680	1.483073	dummy_constant	0.739680	1.483073	1.702987	1.023879	0.153759	False
4	1.448283	0.551983	dummy_constant	1.448283	0.551983	1.312472	0.901942	0.118093	False
...	...	...	...	...	...	...	...	...	...
995	0.543416	0.792679	dummy_constant	0.543416	0.792679	0.021831	0.087546	0.081642	False
996	0.524598	1.128755	dummy_constant	0.524598	1.128755	0.471392	0.395938	0.161382	False
997	0.650015	0.792103	dummy_constant	0.650015	0.792103	0.050002	0.107828	0.093142	False
998	1.002114	0.307375	dummy_constant	1.002114	0.307375	0.093767	0.289223	0.031110	False
999	1.057709	0.198935	dummy_constant	1.057709	0.198935	0.256931	0.401679	0.093314	False

1000 rows × 9 columns

In [14]:

Copied!

df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]
df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]

Out[14]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error	feasible_c1	feasible_c2	feasible
37	0.832326	1.038107	dummy_constant	0.832326	1.038107	0.788575	0.399999	0.154506	False	True	True	True
128	0.557104	1.041890	dummy_constant	0.557104	1.041890	0.396140	0.296905	0.032237	False	True	True	True
141	0.598626	1.039270	dummy_constant	0.598626	1.039270	0.487031	0.300539	0.005584	False	True	True	True
181	0.739280	1.083291	dummy_constant	0.739280	1.083291	0.818830	0.397484	0.065250	False	True	True	True
193	0.551207	1.034344	dummy_constant	0.551207	1.034344	0.371698	0.288146	0.094793	False	True	True	True
...	...	...	...	...	...	...	...	...	...	...	...	...
995	0.543416	0.792679	dummy_constant	0.543416	0.792679	0.021831	0.087546	0.081642	False	True	True	True
996	0.524598	1.128755	dummy_constant	0.524598	1.128755	0.471392	0.395938	0.161382	False	True	True	True
997	0.650015	0.792103	dummy_constant	0.650015	0.792103	0.050002	0.107828	0.093142	False	True	True	True
998	1.002114	0.307375	dummy_constant	1.002114	0.307375	0.093767	0.289223	0.031110	False	True	True	True
999	1.057709	0.198935	dummy_constant	1.057709	0.198935	0.256931	0.401679	0.093314	False	True	True	True

422 rows × 12 columns

In [15]:

Copied!

# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")
# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

No description has been provided for this image

In [16]:

Copied!

# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")
# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

In [17]:

Copied!

# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")
# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")

matplotlib plotting¶

You can always use matplotlib for customizable plotting

In [18]:

Copied!





# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()
# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()

Out[18]:

<matplotlib.legend.Legend at 0x7f9507ac5fd0>

In [19]:

Copied!





# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml
# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml