Xopt Parallel Examples¶

Xopt provides methods to parallelize optimizations using Processes, Threads, MPI, and Dask using the concurrent.futures interface as defined in https://www.python.org/dev/peps/pep-3148/ .

In [1]:

Copied!

from xopt import AsynchronousXopt as Xopt
from xopt import AsynchronousXopt as Xopt

In [2]:

Copied!





# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)
# Helpers for this notebook
import multiprocessing
from concurrent.futures import ProcessPoolExecutor
from dask.distributed import Client

import matplotlib.pyplot as plt

import pandas as pd
from concurrent.futures import ThreadPoolExecutor

import os


# Notebook printing output
# from xopt import output_notebook
# output_notebook()

N_CPUS = multiprocessing.cpu_count()
N_CPUS

# directory for data.
os.makedirs("temp", exist_ok=True)

The Xopt object can be instantiated from a JSON or YAML file, or a dict, with the proper structure.

Here we will make one

In [3]:

Copied!





# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X
# Make a proper input file.
YAML = """

max_evaluations: 1000

generator:
  name: cnsga
  output_path: temp
  population_size:  64
  
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    sleep: 0
    random_sleep: 0.1
  
vocs:
  variables:
    x1: [0, 3.14159]
    x2: [0, 3.14159]
  objectives: {y1: MINIMIZE, y2: MINIMIZE}
  constraints:
    c1: [GREATER_THAN, 0]
    c2: [LESS_THAN, 0.5]
  constants: {a: dummy_constant}

"""
X = Xopt(YAML)
X

Out[3]:

            Xopt
________________________________
Version: 2.6.6.dev25+g041b76137.d20250827
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
  supports_single_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [4]:

Copied!

%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)
%%timeit
# Check that the average time is close to random_sleep
X.evaluator.function({"x1": 0.5, "x2": 0.5}, random_sleep=0.1)

105 ms ± 25.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]:

Copied!

%%time
X.run()
%%time
X.run()

CPU times: user 3.61 s, sys: 23.3 ms, total: 3.63 s
Wall time: 1min 41s

Processes¶

In [6]:

Copied!





%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)
%%time
X = Xopt(YAML)

with ProcessPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

CPU times: user 3.61 s, sys: 163 ms, total: 3.77 s
Wall time: 26.9 s

Out[6]:

Threads¶

Continue running, this time with threads.

In [7]:

Copied!





%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)
%%time
X = Xopt(YAML)

with ThreadPoolExecutor(max_workers=N_CPUS) as executor:
    X.evaluator.executor = executor
    X.evaluator.max_workers = N_CPUS
    X.run()
len(X.data)

CPU times: user 3.44 s, sys: 67.6 ms, total: 3.51 s
Wall time: 25.9 s

Out[7]:

MPI¶

The test.yaml file completely defines the problem. We will also direct the logging to an xopt.log file. The following invocation recruits 4 MPI workers to solve this problem.

We can also continue by calling .save with a JSON filename. This will write all of previous results into the file.

In [8]:

Copied!

X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml
X = Xopt(YAML)
X.dump("test.yaml")  # Write this input to file
!cat test.yaml

data: null
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
  supports_single_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159

In [9]:

Copied!

%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml
%%time
!mpirun -n 8 python -m mpi4py.futures -m xopt.mpi.run -vv --logfile xopt.log test.yaml

Namespace(input_file='test.yaml', logfile='xopt.log', verbose=2, asynchronous=True)
Parallel execution with 8 workers
Enabling async mode

Initialized generator cnsga
Created toolbox with 2 variables, 2 constraints, and 2 objectives.
    Using selection algorithm: nsga2

            Xopt
________________________________
Version: 2.6.6.dev25+g041b76137.d20250827
Data size: 0
Config as YAML:
dump_file: null
evaluator:
  function: xopt.resources.test_functions.tnk.evaluate_TNK
  function_kwargs:
    raise_probability: 0
    random_sleep: 0.1
    sleep: 0
  max_workers: 1
  vectorized: false
generator:
  crossover_probability: 0.9
  mutation_probability: 1.0
  name: cnsga
  output_path: temp
  population: null
  population_file: null
  population_size: 64
  supports_constraints: true
  supports_multi_objective: true
  supports_single_objective: true
is_done: false
max_evaluations: 1000
serialize_inline: false
serialize_torch: false
strict: true
vocs:
  constants:
    a: dummy_constant
  constraints:
    c1:
    - GREATER_THAN
    - 0.0
    c2:
    - LESS_THAN
    - 0.5
  objectives:
    y1: MINIMIZE
    y2: MINIMIZE
  observables: []
  variables:
    x1:
    - 0.0
    - 3.14159
    x2:
    - 0.0
    - 3.14159


Running Xopt

Xopt is done. Max evaluations 1000 reached.

CPU times: user 258 ms, sys: 53.2 ms, total: 311 ms
Wall time: 24.4 s

In [10]:

Copied!

!tail xopt.log
!tail xopt.log

2025-08-27T00:29:12+0000 - xopt - INFO - Parallel execution with 8 workers
2025-08-27T00:29:12+0000 - xopt - INFO - Enabling async mode
2025-08-27T00:29:12+0000 - xopt.generator - INFO - Initialized generator cnsga
2025-08-27T00:29:12+0000 - xopt.generators.ga.cnsga - INFO - Created toolbox with 2 variables, 2 constraints, and 2 objectives.
2025-08-27T00:29:12+0000 - xopt.generators.ga.cnsga - INFO -     Using selection algorithm: nsga2
2025-08-27T00:29:12+0000 - xopt.base - INFO - Running Xopt
2025-08-27T00:29:33+0000 - xopt.base - INFO - Xopt is done. Max evaluations 1000 reached.

Dask¶

In [11]:

Copied!

client = Client()
executor = client.get_executor()
client
client = Client()
executor = client.get_executor()
client

Out[11]:

Client

Client-e87598a9-82dc-11f0-9c8f-000d3a310fba

Connection method: Cluster object	Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

2e477805

Dashboard: http://127.0.0.1:8787/status	Workers: 4
Total threads: 4	Total memory: 15.62 GiB
Status: running	Using processes: True

Scheduler Info

Scheduler

Scheduler-a13073ba-2f64-4c86-96ae-388eaa8f3bc2

Comm: tcp://127.0.0.1:45871	Workers: 0
Dashboard: http://127.0.0.1:8787/status	Total threads: 0
Started: Just now	Total memory: 0 B

Workers

Worker: 0

Comm: tcp://127.0.0.1:40589	Total threads: 1
Dashboard: http://127.0.0.1:46641/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:33541
Local directory: /tmp/dask-scratch-space/worker-un9i8ar7

Worker: 1

Comm: tcp://127.0.0.1:43261	Total threads: 1
Dashboard: http://127.0.0.1:40893/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:37273
Local directory: /tmp/dask-scratch-space/worker-0qbfc24k

Worker: 2

Comm: tcp://127.0.0.1:41531	Total threads: 1
Dashboard: http://127.0.0.1:45717/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:46379
Local directory: /tmp/dask-scratch-space/worker-l5z016d3

Worker: 3

Comm: tcp://127.0.0.1:46001	Total threads: 1
Dashboard: http://127.0.0.1:33767/status	Memory: 3.91 GiB
Nanny: tcp://127.0.0.1:33097
Local directory: /tmp/dask-scratch-space/worker-6hw63h87

In [12]:

Copied!





%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)
%%time
X = Xopt(YAML)
X.evaluator.executor = executor
X.evaluator.max_workers = N_CPUS
X.run()
len(X.data)

CPU times: user 7.98 s, sys: 789 ms, total: 8.77 s
Wall time: 30.9 s

Out[12]:

Load output into Pandas¶

This algorithm writes two types of files: gen_{i}.json with all of the new individuals evaluated in a generation, and pop_{i}.json with the latest best population. Xopt provides some functions to load these easily into a Pandas dataframe for further analysis.

In [13]:

Copied!

X.data
X.data

Out[13]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error
2	1.345250	2.256591	dummy_constant	1.345250	2.256591	5.969865	3.800059	0.023129	False
1	0.419612	0.003625	dummy_constant	0.419612	0.003625	-0.922959	0.252851	0.045755	False
2	2.565581	1.345071	dummy_constant	2.565581	1.345071	7.378663	4.980769	0.125744	False
3	0.319622	1.233825	dummy_constant	0.319622	1.233825	0.685536	0.571035	0.134023	False
4	0.902869	2.858760	dummy_constant	0.902869	2.858760	7.969562	5.726052	0.151403	False
...	...	...	...	...	...	...	...	...	...
995	0.847930	0.536194	dummy_constant	0.847930	0.536194	0.098485	0.122366	0.058739	False
996	0.629734	0.800884	dummy_constant	0.629734	0.800884	0.070790	0.107362	0.042648	False
997	0.778102	0.823962	dummy_constant	0.778102	0.823962	0.194658	0.182292	0.002657	False
998	0.774511	0.720065	dummy_constant	0.774511	0.720065	0.034857	0.123785	0.110031	False
999	0.632058	0.580131	dummy_constant	0.632058	0.580131	-0.341394	0.023860	0.151577	False

1000 rows × 9 columns

In [14]:

Copied!

df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]
df = pd.concat([X.data, X.vocs.feasibility_data(X.data)], axis=1)
df[df["feasible"]]

Out[14]:

	x1	x2	a	y1	y2	c1	c2	xopt_runtime	xopt_error	feasible_c1	feasible_c2	feasible
8	0.726456	1.059708	dummy_constant	0.726456	1.059708	0.748914	0.364555	0.102579	False	True	True	True
18	1.001494	0.217218	dummy_constant	1.001494	0.217218	0.146395	0.331462	0.157711	False	True	True	True
39	0.960397	0.186391	dummy_constant	0.960397	0.186391	0.056827	0.310316	0.080137	False	True	True	True
47	0.305490	0.973169	dummy_constant	0.305490	0.973169	0.025009	0.261723	0.095710	False	True	True	True
48	0.895954	0.829067	dummy_constant	0.895954	0.829067	0.408703	0.265065	0.197866	False	True	True	True
...	...	...	...	...	...	...	...	...	...	...	...	...
994	1.124018	0.220521	dummy_constant	1.124018	0.220521	0.411959	0.467507	0.013580	False	True	True	True
995	0.847930	0.536194	dummy_constant	0.847930	0.536194	0.098485	0.122366	0.058739	False	True	True	True
996	0.629734	0.800884	dummy_constant	0.629734	0.800884	0.070790	0.107362	0.042648	False	True	True	True
997	0.778102	0.823962	dummy_constant	0.778102	0.823962	0.194658	0.182292	0.002657	False	True	True	True
998	0.774511	0.720065	dummy_constant	0.774511	0.720065	0.034857	0.123785	0.110031	False	True	True	True

468 rows × 12 columns

In [15]:

Copied!

# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")
# Plot the feasible ones
feasible_df = df[df["feasible"]]
feasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

No description has been provided for this image

In [16]:

Copied!

# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")
# Plot the infeasible ones
infeasible_df = df[~df["feasible"]]
infeasible_df.plot("y1", "y2", kind="scatter").set_aspect("equal")

In [17]:

Copied!

# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")
# This is the final population
df1 = X.generator.population
df1.plot("y1", "y2", kind="scatter").set_aspect("equal")

matplotlib plotting¶

You can always use matplotlib for customizable plotting

In [18]:

Copied!





# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()
# Extract objectives from output
k1, k2 = "y1", "y2"

fig, ax = plt.subplots(figsize=(6, 6))

ax.scatter(
    infeasible_df[k1],
    infeasible_df[k2],
    color="blue",
    marker=".",
    alpha=0.5,
    label="infeasible",
)
ax.scatter(
    feasible_df[k1], feasible_df[k2], color="orange", marker=".", label="feasible"
)
ax.scatter(df1[k1], df1[k2], color="red", marker=".", label="final population")
ax.set_xlabel(k1)
ax.set_ylabel(k2)
ax.set_aspect("auto")
ax.set_title("Xopt's CNSGA algorithm")
plt.legend()

Out[18]:

<matplotlib.legend.Legend at 0x7f5ff80b12b0>

In [19]:

Copied!





# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml
# Cleanup
#!rm -r dask-worker-space
!rm -r temp
!rm xopt.log*
!rm test.yaml