Numerical Optimizers

GridOptimizer

Bases: NumericalOptimizer

Numerical optimizer that uses a brute-force grid search to find the optimum.

Attributes:

Name	Type	Description
name	str	The name of the optimizer. Default is "grid".
n_grid_points	PositiveInt	Number of mesh points per axis to sample. Algorithm time scales as n_grid_points^input_dimension.

Methods:

Name	Description
optimize	Optimize the given function within the specified bounds.

Parameters:

Name	Type	Description	Default
function	callable	The acquisition function to be optimized.	required
bounds	Tensor	The bounds within which to optimize the acquisition function. Must have shape [2, ndim].	required
n_candidates	int	Number of candidates to return, default is 1.	required

Returns:

Name	Type	Description
candidates	Tensor	The optimized candidates.

Source code in xopt/numerical_optimizer.py

class GridOptimizer(NumericalOptimizer):
    """
    Numerical optimizer that uses a brute-force grid search to find the optimum.

    Attributes
    ----------
    name : str
        The name of the optimizer. Default is "grid".
    n_grid_points : PositiveInt
        Number of mesh points per axis to sample. Algorithm time scales as `n_grid_points`^`input_dimension`.

    Methods
    -------
    optimize
        Optimize the given function within the specified bounds.

    Parameters
    ----------
    function : callable
        The acquisition function to be optimized.
    bounds : Tensor
        The bounds within which to optimize the acquisition function. Must have shape [2, ndim].
    n_candidates : int, optional
        Number of candidates to return, default is 1.

    Returns
    -------
    candidates : Tensor
        The optimized candidates.
    """

    name: str = Field("grid", frozen=True)
    n_grid_points: PositiveInt = Field(
        10, description="number of grid points per axis used for optimization"
    )

    def optimize(self, function, bounds, n_candidates=1):
        """
        Optimize the given function within the specified bounds using a brute-force grid search.

        Parameters
        ----------
        function : Callable
            The function to be optimized.
        bounds : Tensor
            A tensor specifying the bounds for the optimization. It must have the shape [2, ndim].
        n_candidates : int, optional
            The number of candidates to generate (default is 1).

        Returns
        -------
        candidates : Tensor
            The optimized candidates.
        """
        assert isinstance(bounds, Tensor)
        # create mesh
        if len(bounds) != 2:
            raise ValueError("bounds must have the shape [2, ndim]")

        dim = len(bounds[0])
        # add in a machine eps
        eps = 1e-5
        linspace_list = [
            torch.linspace(
                bounds.T[i][0] + eps, bounds.T[i][1] - eps, self.n_grid_points
            )
            for i in range(dim)
        ]

        xx = torch.meshgrid(*linspace_list, indexing="ij")
        mesh_pts = torch.stack(xx).flatten(start_dim=1).T.double()

        # evaluate the function on grid points
        f_values = function(mesh_pts.unsqueeze(1))

        # get the best n_candidates
        _, indicies = torch.sort(f_values)
        x_min = mesh_pts[indicies.squeeze().flipud()]
        return x_min[:n_candidates]

optimize(function, bounds, n_candidates=1)

Optimize the given function within the specified bounds using a brute-force grid search.

Parameters:

Name	Type	Description	Default
function	Callable	The function to be optimized.	required
bounds	Tensor	A tensor specifying the bounds for the optimization. It must have the shape [2, ndim].	required
n_candidates	int	The number of candidates to generate (default is 1).	1

Returns:

Name	Type	Description
candidates	Tensor	The optimized candidates.

Source code in xopt/numerical_optimizer.py

def optimize(self, function, bounds, n_candidates=1):
    """
    Optimize the given function within the specified bounds using a brute-force grid search.

    Parameters
    ----------
    function : Callable
        The function to be optimized.
    bounds : Tensor
        A tensor specifying the bounds for the optimization. It must have the shape [2, ndim].
    n_candidates : int, optional
        The number of candidates to generate (default is 1).

    Returns
    -------
    candidates : Tensor
        The optimized candidates.
    """
    assert isinstance(bounds, Tensor)
    # create mesh
    if len(bounds) != 2:
        raise ValueError("bounds must have the shape [2, ndim]")

    dim = len(bounds[0])
    # add in a machine eps
    eps = 1e-5
    linspace_list = [
        torch.linspace(
            bounds.T[i][0] + eps, bounds.T[i][1] - eps, self.n_grid_points
        )
        for i in range(dim)
    ]

    xx = torch.meshgrid(*linspace_list, indexing="ij")
    mesh_pts = torch.stack(xx).flatten(start_dim=1).T.double()

    # evaluate the function on grid points
    f_values = function(mesh_pts.unsqueeze(1))

    # get the best n_candidates
    _, indicies = torch.sort(f_values)
    x_min = mesh_pts[indicies.squeeze().flipud()]
    return x_min[:n_candidates]

yaml(**kwargs)

serialize first then dump to yaml string

Source code in xopt/pydantic.py

def yaml(self, **kwargs):
    """serialize first then dump to yaml string"""
    output = json.loads(
        self.to_json(
            **kwargs,
        )
    )
    return yaml.dump(output)

LBFGSOptimizer

Bases: NumericalOptimizer

LBFGSOptimizer is a numerical optimizer that uses the Limited-memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) algorithm.

Attributes:

Name	Type	Description
n_restarts	PositiveInt	Number of restarts during acquisition function optimization, default is 20.
max_iter	PositiveInt	Maximum number of iterations for the optimizer, default is 1000.
max_time	Optional[PositiveFloat]	Maximum time allowed for optimization, default is None (no time limit).

Methods:

Name	Description
optimize	Optimize the given acquisition function within the specified bounds.

Parameters:

Name	Type	Description	Default
function	callable	The acquisition function to be optimized.	required
bounds	Tensor	The bounds within which to optimize the acquisition function. Must have shape [2, ndim].	required
n_candidates	int	Number of candidates to return, default is 1.	required
**kwargs	dict	Additional keyword arguments to pass to the optimizer.	required

Returns:

Name	Type	Description
candidates	Tensor	The optimized candidates.

Source code in xopt/numerical_optimizer.py

class LBFGSOptimizer(NumericalOptimizer):
    """
    LBFGSOptimizer is a numerical optimizer that uses the Limited-memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) algorithm.

    Attributes
    ----------
    n_restarts : PositiveInt
        Number of restarts during acquisition function optimization, default is 20.
    max_iter : PositiveInt
        Maximum number of iterations for the optimizer, default is 1000.
    max_time : Optional[PositiveFloat]
        Maximum time allowed for optimization, default is None (no time limit).

    Methods
    -------
    optimize
        Optimize the given acquisition function within the specified bounds.

    Parameters
    ----------
    function : callable
        The acquisition function to be optimized.
    bounds : Tensor
        The bounds within which to optimize the acquisition function. Must have shape [2, ndim].
    n_candidates : int, optional
        Number of candidates to return, default is 1.
    **kwargs : dict
        Additional keyword arguments to pass to the optimizer.

    Returns
    -------
    candidates : Tensor
        The optimized candidates.
    """

    name: str = Field("LBFGS", frozen=True)
    n_restarts: PositiveInt = Field(
        20, description="number of restarts during acquisition function optimization"
    )
    max_iter: PositiveInt = Field(
        1000, description="maximum number of optimization steps"
    )
    max_time: Optional[PositiveFloat] = Field(
        5.0, description="maximum time for optimization in seconds"
    )
    mixed_max_discrete_configurations: PositiveInt = Field(
        512,
        description=(
            "maximum number of discrete configurations to enumerate in mixed"
            " optimization"
        ),
    )
    discrete_max_choices: PositiveInt = Field(
        4096,
        description="maximum number of discrete choices to evaluate directly",
    )
    discrete_max_batch_size: PositiveInt = Field(
        2048,
        description="maximum batch size used by botorch discrete optimization",
    )

    def optimize(
        self,
        function: AcquisitionFunction,
        bounds: Tensor,
        n_candidates: int = 1,
        **kwargs: Any,
    ):
        """
        Optimize the given function within the specified bounds using LBFGS.

        Parameters
        ----------
        function : Callable
            The function to be optimized.
        bounds : Tensor
            A tensor specifying the bounds for the optimization. It must have the shape [2, ndim].
        n_candidates : int, optional
            The number of candidates to generate (default is 1).
        **kwargs : dict
            Additional keyword arguments to be passed to the function optimizer.

        Returns
        -------
        candidates : Tensor
            The optimized candidates.
        """

        assert isinstance(bounds, Tensor)
        if len(bounds) != 2:
            raise ValueError("bounds must have the shape [2, ndim]")

        # empirical testing showed that max time is overrun slightly on the botorch side
        # fix by reducing the max time passed to botorch
        if self.max_time is not None:
            max_time = self.max_time * 0.8 - 0.01
        else:
            max_time = None

        fixed_features_list = kwargs.pop("fixed_features_list", None)
        discrete_choices = kwargs.pop("discrete_choices", None)

        if fixed_features_list is not None and discrete_choices is not None:
            raise ValueError(
                "cannot specify both fixed_features_list and discrete_choices"
            )

        if fixed_features_list is not None:
            original_count = len(fixed_features_list)
            fixed_features_list = fixed_features_list[
                : self.mixed_max_discrete_configurations
            ]
            if len(fixed_features_list) == 0:
                raise ValueError("fixed_features_list cannot be empty")

            if original_count > self.mixed_max_discrete_configurations:
                logger.warning(
                    "truncating mixed discrete configurations from %d to %d",
                    original_count,
                    self.mixed_max_discrete_configurations,
                )

            candidates, _ = optimize_acqf_mixed(
                acq_function=function,
                bounds=bounds,
                q=n_candidates,
                num_restarts=self.n_restarts,
                fixed_features_list=fixed_features_list,
                raw_samples=self.n_restarts,
                timeout_sec=max_time,
                options={"maxiter": self.max_iter},
                **kwargs,
            )
            return candidates

        if discrete_choices is not None:
            if "batch_initial_conditions" in kwargs:
                logger.warning(
                    "batch_initial_conditions are not used by optimize_acqf_discrete"
                )
                kwargs.pop("batch_initial_conditions", None)

            if discrete_choices.shape[0] > self.discrete_max_choices:
                logger.warning(
                    "truncating discrete choices from %d to %d",
                    discrete_choices.shape[0],
                    self.discrete_max_choices,
                )
                discrete_choices = discrete_choices[: self.discrete_max_choices]

            # if the user does not set unique, set it to False if we want more than 1 candidate
            unique = kwargs.pop("unique", n_candidates == 1)
            candidates, _ = optimize_acqf_discrete(
                acq_function=function,
                q=n_candidates,
                choices=discrete_choices,
                max_batch_size=self.discrete_max_batch_size,
                unique=unique,
                **kwargs,
            )
            return candidates

        candidates, _ = optimize_acqf(
            acq_function=function,
            bounds=bounds,
            q=n_candidates,
            raw_samples=self.n_restarts,
            num_restarts=self.n_restarts,
            timeout_sec=max_time,
            options={"maxiter": self.max_iter},
            **kwargs,
        )
        return candidates

optimize(function, bounds, n_candidates=1, **kwargs)

Optimize the given function within the specified bounds using LBFGS.

Parameters:

Name	Type	Description	Default
function	Callable	The function to be optimized.	required
bounds	Tensor	A tensor specifying the bounds for the optimization. It must have the shape [2, ndim].	required
n_candidates	int	The number of candidates to generate (default is 1).	1
**kwargs	dict	Additional keyword arguments to be passed to the function optimizer.	{}

Returns:

Name	Type	Description
candidates	Tensor	The optimized candidates.

Source code in xopt/numerical_optimizer.py

def optimize(
    self,
    function: AcquisitionFunction,
    bounds: Tensor,
    n_candidates: int = 1,
    **kwargs: Any,
):
    """
    Optimize the given function within the specified bounds using LBFGS.

    Parameters
    ----------
    function : Callable
        The function to be optimized.
    bounds : Tensor
        A tensor specifying the bounds for the optimization. It must have the shape [2, ndim].
    n_candidates : int, optional
        The number of candidates to generate (default is 1).
    **kwargs : dict
        Additional keyword arguments to be passed to the function optimizer.

    Returns
    -------
    candidates : Tensor
        The optimized candidates.
    """

    assert isinstance(bounds, Tensor)
    if len(bounds) != 2:
        raise ValueError("bounds must have the shape [2, ndim]")

    # empirical testing showed that max time is overrun slightly on the botorch side
    # fix by reducing the max time passed to botorch
    if self.max_time is not None:
        max_time = self.max_time * 0.8 - 0.01
    else:
        max_time = None

    fixed_features_list = kwargs.pop("fixed_features_list", None)
    discrete_choices = kwargs.pop("discrete_choices", None)

    if fixed_features_list is not None and discrete_choices is not None:
        raise ValueError(
            "cannot specify both fixed_features_list and discrete_choices"
        )

    if fixed_features_list is not None:
        original_count = len(fixed_features_list)
        fixed_features_list = fixed_features_list[
            : self.mixed_max_discrete_configurations
        ]
        if len(fixed_features_list) == 0:
            raise ValueError("fixed_features_list cannot be empty")

        if original_count > self.mixed_max_discrete_configurations:
            logger.warning(
                "truncating mixed discrete configurations from %d to %d",
                original_count,
                self.mixed_max_discrete_configurations,
            )

        candidates, _ = optimize_acqf_mixed(
            acq_function=function,
            bounds=bounds,
            q=n_candidates,
            num_restarts=self.n_restarts,
            fixed_features_list=fixed_features_list,
            raw_samples=self.n_restarts,
            timeout_sec=max_time,
            options={"maxiter": self.max_iter},
            **kwargs,
        )
        return candidates

    if discrete_choices is not None:
        if "batch_initial_conditions" in kwargs:
            logger.warning(
                "batch_initial_conditions are not used by optimize_acqf_discrete"
            )
            kwargs.pop("batch_initial_conditions", None)

        if discrete_choices.shape[0] > self.discrete_max_choices:
            logger.warning(
                "truncating discrete choices from %d to %d",
                discrete_choices.shape[0],
                self.discrete_max_choices,
            )
            discrete_choices = discrete_choices[: self.discrete_max_choices]

        # if the user does not set unique, set it to False if we want more than 1 candidate
        unique = kwargs.pop("unique", n_candidates == 1)
        candidates, _ = optimize_acqf_discrete(
            acq_function=function,
            q=n_candidates,
            choices=discrete_choices,
            max_batch_size=self.discrete_max_batch_size,
            unique=unique,
            **kwargs,
        )
        return candidates

    candidates, _ = optimize_acqf(
        acq_function=function,
        bounds=bounds,
        q=n_candidates,
        raw_samples=self.n_restarts,
        num_restarts=self.n_restarts,
        timeout_sec=max_time,
        options={"maxiter": self.max_iter},
        **kwargs,
    )
    return candidates

yaml(**kwargs)

serialize first then dump to yaml string

Source code in xopt/pydantic.py

def yaml(self, **kwargs):
    """serialize first then dump to yaml string"""
    output = json.loads(
        self.to_json(
            **kwargs,
        )
    )
    return yaml.dump(output)

NumericalOptimizer

Bases: XoptBaseModel, ABC

Base class for numerical optimizers.

Attributes:

Name	Type	Description
name	str	The name of the optimizer. Default is "base_numerical_optimizer".
model_config	ConfigDict	Configuration dictionary with extra fields forbidden.

Methods:

Name	Description
optimize	Abstract method to optimize a function to produce a number of candidate points that minimize the function.

Source code in xopt/numerical_optimizer.py

class NumericalOptimizer(XoptBaseModel, ABC):
    """
    Base class for numerical optimizers.

    Attributes
    ----------
    name : str
        The name of the optimizer. Default is "base_numerical_optimizer".
    model_config : ConfigDict
        Configuration dictionary with extra fields forbidden.

    Methods
    -------
    optimize
        Abstract method to optimize a function to produce a number of candidate points that minimize the function.
    """

    @abstractmethod
    def optimize(
        self,
        function: AcquisitionFunction,
        bounds: Tensor,
        n_candidates: int = 1,
        **kwargs: Any,
    ) -> Tensor:
        """Optimize a function to produce a number of candidate points that minimize the function."""
        raise NotImplementedError

optimize(function, bounds, n_candidates=1, **kwargs) abstractmethod

Optimize a function to produce a number of candidate points that minimize the function.

Source code in xopt/numerical_optimizer.py

@abstractmethod
def optimize(
    self,
    function: AcquisitionFunction,
    bounds: Tensor,
    n_candidates: int = 1,
    **kwargs: Any,
) -> Tensor:
    """Optimize a function to produce a number of candidate points that minimize the function."""
    raise NotImplementedError

yaml(**kwargs)

serialize first then dump to yaml string

Source code in xopt/pydantic.py

def yaml(self, **kwargs):
    """serialize first then dump to yaml string"""
    output = json.loads(
        self.to_json(
            **kwargs,
        )
    )
    return yaml.dump(output)