Xopt Parallel Examples¶

Xopt provides methods to parallelize optimizations using Processes, Threads, MPI, and Dask using the concurrent.futures interface as defined in https://www.python.org/dev/peps/pep-3148/ .

In [1]:

Copied!

from xopt import AsynchronousXopt as Xopt
from xopt import AsynchronousXopt as Xopt

In [2]:

Copied!





# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)
# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)

The Xopt object can be instantiated from a JSON or YAML file, or a dict, with the proper structure.

Here we will make one

In [3]:

Copied!





# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X
# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X

Out[3]:

            Xopt
________________________________
Version: 2.6.7.dev55+g7aa2f3618.d20250930
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
  supports_single_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [4]:

Copied!

%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)
%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)

94.1 ms ± 11.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]:

Copied!

%%time
X.run()
%%time
X.run()

CPU times: user 3.16 s, sys: 19.1 ms, total: 3.18 s
Wall time: 1min 44s

Processes¶

In [6]:

Copied!





%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)
%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

CPU times: user 3.3 s, sys: 133 ms, total: 3.43 s
Wall time: 26.7 s

Out[6]:

Threads¶

Continue running, this time with threads.

In [7]:

Copied!





%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)
%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

CPU times: user 3.12 s, sys: 76.4 ms, total: 3.2 s
Wall time: 25.8 s

Out[7]:

MPI¶

The test.yaml file completely defines the problem. We will also direct the logging to an xopt.log file. The following invocation recruits 4 MPI workers to solve this problem.

We can also continue by calling .save with a JSON filename. This will write all of previous results into the file.

In [8]:

Copied!

X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml
X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml

data: null
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
  supports_single_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [9]:

Copied!

%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml
%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml

Namespace(input_file='test.yaml', logfile='xopt.log', verbose=2, asynchronous=True)
Parallel execution with 8 workers
Enabling async mode
Initialized generator cnsga
Created toolbox with 2 variables, 2 constraints, and 2 objectives.
    Using selection algorithm: nsga2

            Xopt
________________________________
Version: 2.6.7.dev55+g7aa2f3618.d20250930
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
  supports_single_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159


Running Xopt

Xopt is done. Max evaluations 1000 reached.

CPU times: user 185 ms, sys: 36.3 ms, total: 221 ms
Wall time: 24 s

In [10]:

Copied!

!tail xopt.log
!tail xopt.log

2025-09-30T19:49:17+0000 - xopt - INFO - Parallel execution with 8 workers
2025-09-30T19:49:17+0000 - xopt - INFO - Enabling async mode
2025-09-30T19:49:17+0000 - xopt.generator - INFO - Initialized generator cnsga
2025-09-30T19:49:17+0000 - xopt.generators.ga.cnsga - INFO - Created toolbox with 2 variables, 2 constraints, and 2 objectives.
2025-09-30T19:49:17+0000 - xopt.generators.ga.cnsga - INFO -     Using selection algorithm: nsga2
2025-09-30T19:49:17+0000 - xopt.base - INFO - Running Xopt
2025-09-30T19:49:37+0000 - xopt.base - INFO - Xopt is done. Max evaluations 1000 reached.

Dask¶

In [11]:

Copied!

client = Client()
executor = client.get_executor()
client
client = Client()
executor = client.get_executor()
client

Out[11]:

Client

Client-99d365fa-9e36-11f0-9cea-7ced8d824150

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

8a6d3465

Dashboard: http://127.0.0.1:8787/status	Workers: 4
Total threads: 4	Total memory: 15.62 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-242444cf-060e-458f-8169-2c2c01d258f8

Comm: tcp://127.0.0.1:34517	Workers: 0
Dashboard: http://127.0.0.1:8787/status	Total threads: 0
Started: Just now	Total memory: 0 B

Workers

Worker: 0

Comm: tcp://127.0.0.1:35037	Total threads: 1
Dashboard: http://127.0.0.1:42729/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:35237
Local directory: /tmp/dask-scratch-space/worker-l5yayyho

Worker: 1

Comm: tcp://127.0.0.1:44409	Total threads: 1
Dashboard: http://127.0.0.1:32863/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:33883
Local directory: /tmp/dask-scratch-space/worker-t_dfe243

Worker: 2

Comm: tcp://127.0.0.1:39499	Total threads: 1
Dashboard: http://127.0.0.1:45871/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:46113
Local directory: /tmp/dask-scratch-space/worker-e80q6lpj

Worker: 3

Comm: tcp://127.0.0.1:39135	Total threads: 1
Dashboard: http://127.0.0.1:42005/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:40617
Local directory: /tmp/dask-scratch-space/worker-kxf_q9m6

In [12]:

Copied!





%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)
%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)

CPU times: user 6.69 s, sys: 642 ms, total: 7.33 s
Wall time: 29.6 s

Out[12]:

Load output into Pandas¶

This algorithm writes two types of files: gen_{i}.json with all of the new individuals evaluated in a generation, and pop_{i}.json with the latest best population. Xopt provides some functions to load these easily into a Pandas dataframe for further analysis.

In [13]:

Copied!

X.data
X.data

Out[13]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error
2	0.313553	0.283635	dummy_constant	0.313553	0.283635	-0.890842	0.081576	0.005680	False
1	0.810693	0.366143	dummy_constant	0.810693	0.366143	-0.296267	0.114448	0.071065	False
2	2.326680	0.869248	dummy_constant	2.326680	0.869248	5.084439	3.473104	0.070665	False
3	2.958467	1.503797	dummy_constant	2.958467	1.503797	9.981556	7.051668	0.065563	False
4	1.710095	2.024934	dummy_constant	1.710095	2.024934	6.002443	3.789752	0.000235	False
...	...	...	...	...	...	...	...	...	...
995	0.842582	0.733350	dummy_constant	0.842582	0.733350	0.203034	0.171815	0.006779	False
996	0.787073	0.695054	dummy_constant	0.787073	0.695054	0.047891	0.120457	0.055584	False
997	0.615313	0.537051	dummy_constant	0.615313	0.537051	-0.379660	0.014670	0.189305	False
998	0.779663	0.752189	dummy_constant	0.779663	0.752189	0.077751	0.141811	0.038135	False
999	0.779680	0.679445	dummy_constant	0.779680	0.679445	0.023956	0.110421	0.181631	False

1000 rows × 9 columns

In [14]:

Copied!

df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]
df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]

Out[14]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error	feasible_c1	feasible_c2	feasible
24	1.053626	0.604337	dummy_constant	1.053626	0.604337	0.521375	0.317388	0.179802	False	True	True	True
36	0.575200	1.142523	dummy_constant	0.575200	1.142523	0.598051	0.418491	0.071042	False	True	True	True
64	0.933658	0.694204	dummy_constant	0.933658	0.694204	0.422961	0.225774	0.031912	False	True	True	True
99	1.010428	0.352324	dummy_constant	1.010428	0.352324	0.084125	0.282345	0.022322	False	True	True	True
130	0.549467	1.142523	dummy_constant	0.549467	1.142523	0.544267	0.415283	0.150917	False	True	True	True
...	...	...	...	...	...	...	...	...	...	...	...	...
992	0.874430	0.496559	dummy_constant	0.874430	0.496559	0.050997	0.140210	0.137792	False	True	True	True
995	0.842582	0.733350	dummy_constant	0.842582	0.733350	0.203034	0.171815	0.006779	False	True	True	True
996	0.787073	0.695054	dummy_constant	0.787073	0.695054	0.047891	0.120457	0.055584	False	True	True	True
998	0.779663	0.752189	dummy_constant	0.779663	0.752189	0.077751	0.141811	0.038135	False	True	True	True
999	0.779680	0.679445	dummy_constant	0.779680	0.679445	0.023956	0.110421	0.181631	False	True	True	True

410 rows × 12 columns

In [15]:

Copied!

# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")
# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

No description has been provided for this image

In [16]:

Copied!

# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")
# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

In [17]:

Copied!

# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")
# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")

matplotlib plotting¶

You can always use matplotlib for customizable plotting

In [18]:

Copied!





# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()
# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()

Out[18]:

<matplotlib.legend.Legend at 0x7f4723b1fcb0>

In [19]:

Copied!





# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml
# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml