A manifold objective

AbstractManifoldObjective{E<:AbstractEvaluationType}

Describe the collection of the optimization function $f: \mathcal M → ℝ$ (or even a vectorial range) and its corresponding elements, which might for example be a gradient or (one or more) proximal maps.

All these elements should usually be implemented as functions (M, p) -> ..., or (M, X, p) -> ... that is

• the first argument of these functions should be the manifold M they are defined on
• the argument X is present, if the computation is performed in-place of X (see InplaceEvaluation)
• the argument p is the place the function ($f$ or one of its elements) is evaluated at.

the type T indicates the global AbstractEvaluationType.

AbstractDecoratedManifoldObjective{E<:AbstractEvaluationType,O<:AbstractManifoldObjective}

A common supertype for all decorators of AbstractManifoldObjectives to simplify dispatch. The second parameter should refer to the undecorated objective (the most inner one).

AbstractEvaluationType

An abstract type to specify the kind of evaluation a AbstractManifoldObjective supports.

AllocatingEvaluation <: AbstractEvaluationType

A parameter for a AbstractManoptProblem or a Function indicating that the problem contains or the function(s) allocate memory for their result, they work out of place.

AllocatingInplaceEvaluation <: AbstractEvaluationType

A parameter for a AbstractManoptProblem or a Function indicating that the problem contains or the function(s) that provides both an allocating variant and one, that does not allocate memory but work on their input, in place.

InplaceEvaluation <: AbstractEvaluationType

A parameter for a AbstractManoptProblem or a Function indicating that the problem contains or the function(s) do not allocate memory but work on their input, in place.

ParentEvaluationType <: AbstractEvaluationType

A parameter for a AbstractManoptProblem or a Function indicating that the problem contains or the function(s) do inherit their property from a parent AbstractManoptProblem or function.

evaluation_type(mp::AbstractManoptProblem)

Get the AbstractEvaluationType of the objective in AbstractManoptProblem mp.

evaluation_type(::AbstractManifoldObjective{Teval})

Get the AbstractEvaluationType of the objective.

dispatch_objective_decorator(o::AbstractManoptSolverState)

Indicate internally, whether an AbstractManifoldObjective o to be of decorating type, it stores (encapsulates) an object in itself, by default in the field o.objective.

Decorators indicate this by returning Val{true} for further dispatch.

The default is Val{false}, so by default an state is not decorated.

is_object_decorator(s::AbstractManifoldObjective)

Indicate, whether AbstractManifoldObjective s are of decorator type.

decorate_objective!(M, o::AbstractManifoldObjective)

decorate the AbstractManifoldObjectiveo with specific decorators.

Optional arguments

optional arguments provide necessary details on the decorators. A specific one is used to activate certain decorators.

• cache=missing: specify a cache. Currently :Simple is supported and :LRU if you load LRUCache.jl. For this case a tuple specifying what to cache and how many can be provided, has to be specified. For example (:LRU, [:Cost, :Gradient], 10) states that the last 10 used cost function evaluations and gradient evaluations should be stored. See objective_cache_factory for details.
• count=missing: specify calls to the objective to be called, see ManifoldCountObjective for the full list
• objective_type=:Riemannian: specify that an objective is :Riemannian or :Euclidean. The :Euclidean symbol is equivalent to specifying it as :Embedded, since in the end, both refer to converting an objective from the embedding (whether its Euclidean or not) to the Riemannian one.

See also

objective_cache_factory

EmbeddedManifoldObjective{P, T, E, O2, O1<:AbstractManifoldObjective{E}} <:
   AbstractDecoratedManifoldObjective{E,O2}

Declare an objective to be defined in the embedding. This also declares the gradient to be defined in the embedding, and especially being the Riesz representer with respect to the metric in the embedding. The types can be used to still dispatch on also the undecorated objective type O2.

Fields

• objective: the objective that is defined in the embedding
• p=nothing: a point in the embedding.
• X=nothing: a tangent vector in the embedding

When a point in the embedding p is provided, embed! is used in place of this point to reduce memory allocations. Similarly X is used when embedding tangent vectors

ScaledManifoldObjective{E, O2, O1<:AbstractManifoldObjective{E},F} <:
   AbstractDecoratedManifoldObjective{E,O2}

Declare an objective to be defined as a scaled version of an existing objective.

This rescales all involved functions.

For now the functions rescaled are

• the cost
• the gradient
• the Hessian

Fields

• objective: the objective that is defined in the embedding
• scale=1: the scaling applied

Constructors

ScaledManifoldObjective(objective, scale::Real=1)
-objective
scale*objective

Generate a scaled manifold objective based on objective with scale being 1 by default in the first, scale=-1 in the second case. The multiplication from the left with a scalar is also overloaded.

reset_counters(co::ManifoldCountObjective, value::Integer=0)

Reset all values in the count objective to value.

objective_cache_factory(M::AbstractManifold, o::AbstractManifoldObjective, cache::Symbol)

Generate a cached variant of the AbstractManifoldObjective o on the AbstractManifold M based on the symbol cache.

The following caches are available

• :Simple generates a SimpleManifoldCachedObjective
• :LRU generates a ManifoldCachedObjective where you should use the form (:LRU, [:Cost, :Gradient]) to specify what should be cached or (:LRU, [:Cost, :Gradient], 100) to specify the cache size. Here this variant defaults to (:LRU, [:Cost, :Gradient], 100), caching up to 100 cost and gradient values.^[1]

objective_cache_factory(M::AbstractManifold, o::AbstractManifoldObjective, cache::Tuple{Symbol, Array, Array})
objective_cache_factory(M::AbstractManifold, o::AbstractManifoldObjective, cache::Tuple{Symbol, Array})

Generate a cached variant of the AbstractManifoldObjective o on the AbstractManifold M based on the symbol cache[1], where the second element cache[2] are further arguments to the cache and the optional third is passed down as keyword arguments.

For all available caches see the simpler variant with symbols.

 SimpleManifoldCachedObjective{O<:AbstractManifoldFirstOrderObjective{E}, P, T,C} <: AbstractDecoratedManifoldObjective{E,O}

Provide a simple cache for an AbstractManifoldFirstOrderObjective that is, this cache stores a point p and a gradient $\operatorname{grad} f(p)$ in X as well as a cost value $f(p)$ in c. It can also easily evaluate the differential based on the cached gradient.

Both X and c are accompanied by booleans to keep track of their validity.

While this does not provide a cache for the differential, it uses the cached gradient as a help to evaluate the differential, if an up-to-date gradient is available. It otherwise does call the original differential.

This simple cache does not take into account, that some first order objectives have a common function for cost & grad. It only caches the function that is actually called.

Constructor

SimpleManifoldCachedObjective(M::AbstractManifold, obj::AbstractManifoldFirstOrderObjective; kwargs...)

Keyword arguments

• p=rand(M): a point on the manifold to initialize the cache with
• X=get_gradient(M, obj, p) or zero_vector(M,p): a tangent vector to store the gradient in, see also initialize=
• c=[get_cost](@ref)(M, obj, p)or0.0: a value to store the cost function ininitialize`
• initialized=true: whether to initialize the cached X and c or not.

ManifoldCachedObjective{E,P,O<:AbstractManifoldObjective{<:E},C<:NamedTuple{}} <: AbstractDecoratedManifoldObjective{E,P}

Create a cache for an objective, based on a NamedTuple that stores some kind of cache.

Constructor

ManifoldCachedObjective(M, o::AbstractManifoldObjective, caches::Vector{Symbol}; kwargs...)

Create a cache for the AbstractManifoldObjective where the Symbols in caches indicate, which function evaluations to cache.

Supported symbols

Keyword arguments

• p=rand(M): the type of the keys to be used in the caches. Defaults to the default representation on M.
• value=get_cost(M, objective, p): the type of values for numeric values in the cache
• X=zero_vector(M,p): the type of values to be cached for gradient and Hessian calls.
• cache=[:Cost]: a vector of symbols indicating which function calls should be cached.
• cache_size=10: number of (least recently used) calls to cache
• cache_sizes=Dict{Symbol,Int}(): a named tuple or dictionary specifying the sizes individually for each cache.

init_caches(caches, T::Type{LRU}; kwargs...)

Given a vector of symbols caches, this function sets up the NamedTuple of caches, where T is the type of cache to use.

Keyword arguments

• p=rand(M): a point on a manifold, to both infer its type for keys and initialize caches
• value=0.0: a value both typing and initialising number-caches, the default is for (Float) values like the cost.
• X=zero_vector(M, p): a tangent vector at p to both type and initialize tangent vector caches
• cache_size=10: a default cache size to use
• cache_sizes=Dict{Symbol,Int}(): a dictionary of sizes for the caches to specify different (non-default) sizes

init_caches(M::AbstractManifold, caches, T; kwargs...)

Given a vector of symbols caches, this function sets up the NamedTuple of caches for points/vectors on M, where T is the type of cache to use.

ManifoldCountObjective{E,P,O<:AbstractManifoldObjective,I<:Integer} <: AbstractDecoratedManifoldObjective{E,P}

A wrapper for any AbstractManifoldObjective of type O to count different calls to parts of the objective.

Fields

• counts a dictionary of symbols mapping to integers keeping the counted values
• objective the wrapped objective

Supported symbols

Constructors

ManifoldCountObjective(objective::AbstractManifoldObjective, counts::Dict{Symbol, <:Integer})

Initialise the ManifoldCountObjective to wrap objective initializing the set of counts

ManifoldCountObjective(M::AbstractManifold, objective::AbstractManifoldObjective, count::AbstractVecor{Symbol}, init=0)

Count function calls on objective using the symbols in count initialising all entries to init.

ReturnManifoldObjective{E,O2,O1<:AbstractManifoldObjective{E}} <:
   AbstractDecoratedManifoldObjective{E,O2}

A wrapper to indicate that get_solver_result should return the inner objective.

The types are such that one can still dispatch on the undecorated type O2 of the original objective as well.

AbstractManifoldCostObjective{T<:AbstractEvaluationType} <: AbstractManifoldObjective{T}

Representing objectives on manifolds with a cost function implemented.

ManifoldCostObjective{T, TC} <: AbstractManifoldCostObjective{T, TC}

specify an AbstractManifoldObjective that does only have information about the cost function $f: \mathbb M → ℝ$ implemented as a function (M, p) -> c to compute the cost value c at p on the manifold M.

• cost: a function $f: \mathcal M → ℝ$ to minimize

Constructors

ManifoldCostObjective(f)

Generate a problem. While this Problem does not have any allocating functions, the type T can be set for consistency reasons with other problems.

Used with

NelderMead, particle_swarm

get_cost(amp::AbstractManoptProblem, p)

evaluate the cost function f stored within the AbstractManifoldObjective of an AbstractManoptProblem amp at the point p.

get_cost(M::AbstractManifold, obj::AbstractManifoldObjective, p)

evaluate the cost function f defined on M stored within the AbstractManifoldObjective at the point p.

get_cost(M::AbstractManifold, mco::AbstractManifoldCostObjective, p)

Evaluate the cost function from within the AbstractManifoldCostObjective on M at p.

By default this implementation assumed that the cost is stored within mco.cost.

get_cost(TpM, trmo::TrustRegionModelObjective, X)

Evaluate the tangent space TrustRegionModelObjective

\[m(X) = f(p) + ⟨\operatorname{grad} f(p), X ⟩_p + \frac{1}{2} ⟨\operatorname{Hess} f(p)[X], X⟩_p.\]

get_cost(TpM, trmo::AdaptiveRagularizationWithCubicsModelObjective, X)

Evaluate the tangent space AdaptiveRagularizationWithCubicsModelObjective

\[m(X) = f(p) + ⟨\operatorname{grad} f(p), X ⟩_p + \frac{1}{2} ⟨\operatorname{Hess} f(p)[X], X⟩_p + \frac{σ}{3} \lVert X \rVert^3,\]

at X, cf. Eq. (33) in [ABBC20].

get_cost(TpM::TangentSpace, slso::SymmetricLinearSystemObjective, X)

evaluate the cost

\[f(X) = \frac{1}{2} \lVert \mathcal A[X] + b \rVert_{p}^2,\qquad X ∈ T_{p}\mathcal M,\]

at X.

get_cost(M::AbstractManifold, sgo::ManifoldStochasticGradientObjective, p, i)

Evaluate the ith summand of the cost.

If you use a single function for the stochastic cost, then only the index ì=1` is available to evaluate the whole cost.

get_cost(M::AbstractManifold,emo::EmbeddedManifoldObjective, p)

Evaluate the cost function of an objective defined in the embedding by first embedding p before calling the cost function stored in the EmbeddedManifoldObjective.

get_cost(M::AbstractManifold, scaled_objective::ScaledManifoldObjective, p)

Evaluate the scaled objective. $s*f(p)$

get_cost_function(amco::AbstractManifoldCostObjective; recursive=false)

return the function to evaluate (just) the cost $f(p)=c$ as a function (M,p) -> c. If amco has more than one decorator, recursive determines whether just one (false) or all wrappers (true) should be “unwrapped” at once.

AbstractManifoldFirstOrderObjective{E<:AbstractEvaluationType, FGD} <: AbstractManifoldCostObjective{E, FGD}

An abstract type for all objectives that provide

• a cost
• first order information, so either a (full) gradient or a differential, where

E is a AbstractEvaluationType for the gradient function.

ManifoldFirstOrderObjective{E<:AbstractEvaluationType, F} <: AbstractManifoldFirstOrderObjective{E, F}

specify an objective containing a cost and its gradient or differential, where the AbstractEvaluationType E indicates the type of evaluation for a gradient.

Fields

• functions::F: a function or a tuple of functions containing the cost and first order information.

Currently the following cases are covered, sorted by their popularity

1. a single function fg, i.e. a function or a functor, represents a combined function

(M, p) -> (c, X) that computes the cost c=cost(M,p) and gradient X=grad_f(M,p);

1. a single function fdf, i.e. a function or a functor, represents a combined function

(M, p) -> (c, d) that computes the cost c=cost(M,p) and differential d=diff_f(M,p);

1. pairs of single functions (f, g), (f, df) of a cost function f and either its gradient g or its differential d, respectively
2. The function (fg, d) and (fdf, g) from 1 and 2, respectively joined by the other missing third information,

the differential for the first or the gradient for the second

1. a tuple (f, g, d) of three functions, computing cost, f, gradient g, and differentiald` separately
2. a (f, gd) of a cost function and a combined function (X, d) = gd(M, p, X) to compute gradient and differential together
3. a single function (c, X, d) = fgd(M, p,X)

For all cases where a gradient is present, also an in-place variant is possible, where the signature has the result Y in second place.

The cases of a common fg function for cost and gradient and the tuple (f,g) are the most common one. They can also be addressed by their alternate constructors ManifoldCostGradientObjective(fg) and ManifoldGradientObjective(f,g), respectively.

Constructors

ManifoldFirstOrderObjective(; kwargs...)

Keyword arguments

• cost = nothing the cost function c = f(M,p)
• differential = nothing the differential d = df(M, p, X)
• gradient=nothing the gradient function g(M, p) or in-place g!(M, X, p)
• costgradient = nothing the combined cost and gradient function fg(M,p) or in-place fg!(M, X, p))
• costdifferential = nothing the combined cost and differential function fdf(M, p, X)
• evaluation=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating its result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.

Where:

• At least one of cost, costgradient or costdifferential must be provided.
• Either gradient, costgradient, differential or costdifferential must be provided.
• If more than one function provides the same thing (e.g. cost), it is assumed that all such functions return the same value. Optimization algorithms will attempt to make the most efficient use of provided functions.

Used with

gradient_descent, conjugate_gradient_descent, quasi_Newton

ManifoldAlternatingGradientObjective{E<:AbstractEvaluationType,F,G} <: AbstractManifoldFirstOrderObjective{E, Tuple{F,G}}

An alternating gradient objective consists of

• a cost function $F(x)$
• a gradient $\operatorname{grad}F$ that is either
  • given as one function $\operatorname{grad}F$ returning a tangent vector X on M or
  • an array of gradient functions $\operatorname{grad}F_i$, ì=1,…,n s each returning a component of the gradient
  which might be allocating or mutating variants, but not a mix of both.

Constructors

ManifoldAlternatingGradientObjective(F, gradF::Function;
    evaluation=AllocatingEvaluation()
)
ManifoldAlternatingGradientObjective(F, gradF::AbstractVector{<:Function};
    evaluation=AllocatingEvaluation()
)

Create a alternating gradient problem with an optional cost and the gradient either as one function (returning an array) or a vector of functions.

ManifoldStochasticGradientObjective{E<:AbstractEvaluationType, F, G} <: AbstractManifoldFirstOrderObjective{E, Tuple{F,G}}

A stochastic gradient objective consists of

• a(n optional) cost function $f(p) = \displaystyle\sum_{i=1}^{n} f_i(p)$
• an array of gradients, $\operatorname{grad} f_i(p), i=1,…,n$ which can be given in two forms
  • as one single function $(\mathcal M, p) ↦ (X_1,…,X_n) ∈ (T_{p}\mathcal M)^n$
  • as a vector of functions $\bigl( (\mathcal M, p) ↦ X_1, …, (\mathcal M, p) ↦ X_n\bigr)$.

Where both variants can also be provided as InplaceEvaluation functions (M, X, p) -> X, where X is the vector of X1,...,Xn and (M, X1, p) -> X1, ..., (M, Xn, p) -> Xn, respectively.

Constructors

ManifoldStochasticGradientObjective(
    grad_f::Function;
    cost=Missing(),
    evaluation=AllocatingEvaluation()
)
ManifoldStochasticGradientObjective(
    grad_f::AbstractVector{<:Function};
    cost=Missing(), evaluation=AllocatingEvaluation()
)

Create a Stochastic gradient problem with the gradient either as one function (returning an array of tangent vectors) or a vector of functions (each returning one tangent vector).

The optional cost can also be given as either a single function (returning a number) pr a vector of functions, each returning a value.

Used with

stochastic_gradient_descent

Note that this can also be used with a gradient_descent, since the (complete) gradient is just the sums of the single gradients.

NonlinearLeastSquaresObjective{E<:AbstractEvaluationType} <: AbstractManifoldObjective{T}

An objective to model the nonlinear least squares problem

\[\operatorname*{arg\,min}_{p ∈ \mathcal M} \frac{1}{2} \sum_{}^{}_{i=1}^m \lvert f_i(p) \rvert^2\]

where $f: \mathcal M → ℝ^m$ is written with component functions $f_i: \mathcal M → ℝ$, $i=1,…,m$, and each component function is continuously differentiable.

Specify a nonlinear least squares problem

Fields

• objective: a AbstractVectorGradientFunction{E} containing both the vector of cost functions $f_i$ (or a function returning a vector of costs) as well as their gradients $\operatorname{grad} f_i$ (or Jacobian of the vector-valued function).

This NonlinearLeastSquaresObjective then has the same AbstractEvaluationType T as the (inner) objective.

Constructors

NonlinearLeastSquaresObjective(f, jacobian, range_dimension::Integer; kwargs...)
NonlinearLeastSquaresObjective(vf::AbstractVectorGradientFunction)

Arguments

• f the vectorial cost function $f: \mathcal M → ℝ^m$
• jacobian the Jacobian, might also be a vector of gradients of the component functions of f
• range_dimension::Integer the number of dimensions m the function f maps into

These three can also be passed as a AbstractVectorGradientFunction vf already.

Keyword arguments

• evaluation=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating its result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.
• function_type::AbstractVectorialType=FunctionVectorialType(): specify the format the residuals are given in. By default a function returning a vector.
• jacobian_tangent_basis::AbstractBasis=DefaultOrthonormalBasis(); shortcut to specify the basis the Jacobian matrix is build with.
• jacobian_type::AbstractVectorialType=CoordinateVectorialType(jacobian_tangent_basis): specify the format the Jacobian is given in. By default a matrix of the differential with respect to a certain basis of the tangent space.

See also

LevenbergMarquardt, LevenbergMarquardtState

ManifoldGradientObjective(cost, gradient; evaluation::E=AllocatingEvaluation() kwargs...)

Generate an objective with a function cost and its gradient. Depending on the AbstractEvaluationType E the gradient can have to forms

• as a function (M, p) -> X that allocates memory for X, an AllocatingEvaluation
• as a function (M, X, p) -> X that work in place of X, an InplaceEvaluation

Internally this is stored in a ManifoldFirstOrderObjective. The kwargs... are also passed to this representation, which allows to add a special function to evaluate the differential.

Used with

gradient_descent, conjugate_gradient_descent, quasi_Newton

ManifoldCostGradientObjective(costgrad; evaluation::E=AllocatingEvaluation(), kwargs...)

create an objective containing one function to perform a combined computation of cost and its gradient

Depending on the AbstractEvaluationType E the gradient can have to forms

• as a function (M, p) -> (c, X) that allocates memory for the gradient X, an AllocatingEvaluation
• as a function (M, X, p) -> (c, X) that work in place of X, an InplaceEvaluation

Internally this is stored in a ManifoldFirstOrderObjective. The kwargs... are also passed to this representation, which allows to add a special function to evaluate the differential.

Used with

gradient_descent, conjugate_gradient_descent, quasi_Newton

get_gradient(s::AbstractManoptSolverState)

return the (last stored) gradient within AbstractManoptSolverStates`. By default also undecorates the state beforehand

get_gradient(amp::AbstractManoptProblem, p)
get_gradient!(amp::AbstractManoptProblem, X, p)

evaluate the gradient of an AbstractManoptProblem amp at the point p.

The evaluation is done in place of X for the !-variant.

get_gradient(agst::AbstractGradientSolverState)

return the gradient stored within gradient options. THe default returns agst.X.

get_gradient(M::AbstractManifold, mgo::ManifoldProximalGradientObjective, p)
get_gradient!(M::AbstractManifold, X, mgo::ManifoldProximalGradientObjective, p)

Evaluate the gradient of the smooth part of a ManifoldProximalGradientObjective mgo at p.

get_gradient(M::AbstractManifold, vgf::VectorGradientFunction, p, i)
get_gradient(M::AbstractManifold, vgf::VectorGradientFunction, p, i, range)
get_gradient!(M::AbstractManifold, X, vgf::VectorGradientFunction, p, i)
get_gradient!(M::AbstractManifold, X, vgf::VectorGradientFunction, p, i, range)

Evaluate the gradients of the vector function vgf on the manifold M at p and the values given in range, specifying the representation of the gradients.

Since i is assumed to be a linear index, you can provide

• a single integer
• a UnitRange to specify a range to be returned like 1:3
• a BitVector specifying a selection
• a AbstractVector{<:Integer} to specify indices
• : to return the vector of all gradients

get_gradient(TpM, trmo::TrustRegionModelObjective, X)

Evaluate the gradient of the TrustRegionModelObjective

\[\operatorname{grad} m(X) = \operatorname{grad} f(p) + \operatorname{Hess} f(p)[X].\]

get_gradient(TpM, trmo::AdaptiveRagularizationWithCubicsModelObjective, X)

Evaluate the gradient of the AdaptiveRagularizationWithCubicsModelObjective

\[\operatorname{grad} m(X) = \operatorname{grad} f(p) + \operatorname{Hess} f(p)[X] + σ\lVert X \rVert X,\]

at X, cf. Eq. (37) in [ABBC20].

get_gradient(TpM::TangentSpace, slso::SymmetricLinearSystemObjective, X)
get_gradient!(TpM::TangentSpace, Y, slso::SymmetricLinearSystemObjective, X)

evaluate the gradient of

\[f(X) = \frac{1}{2} \lVert \mathcal A[X] + b \rVert_{p}^2,\qquad X ∈ T_{p}\mathcal M,\]

Which is $\operatorname{grad} f(X) = \mathcal A[X]+b$. This can be computed in-place of Y.

get_gradient(M::AbstractManifold, sgo::ManifoldStochasticGradientObjective, p, k)
get_gradient!(M::AbstractManifold, sgo::ManifoldStochasticGradientObjective, Y, p, k)

Evaluate one of the summands gradients $\operatorname{grad}f_k$, $k∈\{1,…,n\}$, at x (in place of Y).

If you use a single function for the stochastic gradient, that works in-place, then get_gradient is not available, since the length (or number of elements of the gradient required for allocation) can not be determined.

get_gradient(M::AbstractManifold, sgo::ManifoldStochasticGradientObjective, p)
get_gradient!(M::AbstractManifold, sgo::ManifoldStochasticGradientObjective, X, p)

Evaluate the complete gradient $\operatorname{grad} f = \displaystyle\sum_{i=1}^n \operatorname{grad} f_i(p)$ at p (in place of X).

If you use a single function for the stochastic gradient, that works in-place, then get_gradient is not available, since the length (or number of elements of the gradient required for allocation) can not be determined.

get_gradient(M::AbstractManifold, emo::EmbeddedManifoldObjective, p)
get_gradient!(M::AbstractManifold, X, emo::EmbeddedManifoldObjective, p)

Evaluate the gradient function of an objective defined in the embedding, that is embed p before calling the gradient function stored in the EmbeddedManifoldObjective.

The returned gradient is then converted to a Riemannian gradient calling riemannian_gradient.

get_gradient(M::AbstractManifold, scaled_objective::ScaledManifoldObjective, p)
get_gradient!(M::AbstractManifold, X, scaled_objective::ScaledManifoldObjective, p)

Evaluate the scaled gradient. $s*\operatorname{grad}f(p)$

get_gradients(M::AbstractManifold, sgo::ManifoldStochasticGradientObjective, p)
get_gradients!(M::AbstractManifold, X, sgo::ManifoldStochasticGradientObjective, p)

Evaluate all summands gradients $\{\operatorname{grad}f_i\}_{i=1}^n$ at p (in place of X).

If you use a single function for the stochastic gradient, that works in-place, then get_gradient is not available, since the length (or number of elements of the gradient) can not be determined.

 get_differential(amp::AbstractManoptProblem, p, X; kwargs...)
 get_differential(M::AbstractManifold, amfo:AbstractManifoldFirstOrderObjective, p, X; kwargs...)
 get_differential(M::AbstractManifold, amfo:AbstractDecoratedManifoldObjective, p, X; kwargs...)

Evaluate the differential $Df(p)[X]$ of the function $f$ represented by the AbstractManifoldFirstOrderObjective. For AbstractManoptProblem the inner manifold and objectives are used, similarly, any objective decorator would “pass though” to its inner objective. By default this falls back to ``Df(p)[X] = ⟨\operatorname{grad}f(p), X⟩

Keyword arguments

• gradient=nothing – pass a tangent vector to be used internally as interims memory, e.g. in the default variant to evaluate the gradient in-place in.
• evaluated=false – indicate whether gradient is just memory (false, default) or already contains the evaluated gradient (true).

get_residuals(M::AbstractManifold, nlso::NonlinearLeastSquaresObjective, p)
get_residuals!(M::AbstractManifold, V, nlso::NonlinearLeastSquaresObjective, p)

Compute the vector of residuals $f_i(p)$, $i=1,…,m$ given the manifold M, the NonlinearLeastSquaresObjective nlso and a current point $p$ on M.

get_residuals(M::AbstractManifold, nlso::NonlinearLeastSquaresObjective, p)
get_residuals!(M::AbstractManifold, V, nlso::NonlinearLeastSquaresObjective, p)

Compute the vector of residuals $f_i(p)$, $i=1,…,m$ given the manifold M, the NonlinearLeastSquaresObjective nlso and a current point $p$ on M.

 get_differential_function(admo::AbstractManifoldFirstOrderObjective, recursive::Bool=false)

Return the function to evaluate (just) the differential $Df(p)[X]$. For a decorated objective, the recursive positional parameter determines whether to directly call this function on the next decorator or whether to get the “most inner” objective.

get_gradient_function(amgo::AbstractManifoldFirstOrderObjective, recursive=false)

return the function to evaluate (just) the gradient $\operatorname{grad} f(p)$, where either the gradient function using the decorator or without the decorator is used.

By default recursive is set to false, since usually to just pass the gradient function somewhere, one still wants for example the cached one or the one that still counts calls.

Depending on the AbstractEvaluationType E this is a function

• (M, p) -> X for the AllocatingEvaluation case
• (M, X, p) -> X for the InplaceEvaluation working in-place of X.

ManifoldSubgradientObjective{T<:AbstractEvaluationType,C,S} <:AbstractManifoldCostObjective{T, C}

A structure to store information about a objective for a subgradient based optimization problem

Fields

• cost: the function $f$ to be minimized
• subgradient: a function returning a subgradient $∂f$ of $f$

Constructor

ManifoldSubgradientObjective(f, ∂f)

Generate the ManifoldSubgradientObjective for a subgradient objective, consisting of a (cost) function f(M, p) and a function ∂f(M, p) that returns a not necessarily deterministic element from the subdifferential at p on a manifold M.

X = get_subgradient(M::AbstractManifold, sgo::AbstractManifoldFirstOrderObjective, p)
get_subgradient!(M::AbstractManifold, X, sgo::AbstractManifoldFirstOrderObjective, p)

Evaluate the subgradient, which for the case of a objective having a gradient, means evaluating the gradient itself.

While in general, the result might not be deterministic, for this case it is.

get_subgradient(amp::AbstractManoptProblem, p)
get_subgradient!(amp::AbstractManoptProblem, X, p)

evaluate the subgradient of an AbstractManoptProblem amp at point p.

The evaluation is done in place of X for the !-variant. The result might not be deterministic, one element of the subdifferential is returned.

X = get_subgradient(M;;AbstractManifold, sgo::ManifoldSubgradientObjective, p)
get_subgradient!(M;;AbstractManifold, X, sgo::ManifoldSubgradientObjective, p)

Evaluate the (sub)gradient of a ManifoldSubgradientObjective sgo at the point p.

The evaluation is done in place of X for the !-variant. The result might not be deterministic, one element of the subdifferential is returned.

get_subgradient_function(amgo::ManifoldSubgradientObjective, recursive=false)

return the function to evaluate (just) the gradient $\operatorname{grad} f(p)$, where either the gradient function using the decorator or without the decorator is used.

By default recursive is set to false, since usually to just pass the gradient function somewhere, one still wants for example the cached one or the one that still counts calls.

Depending on the AbstractEvaluationType E this is a function

• (M, p) -> X for the AllocatingEvaluation case
• (M, X, p) -> X for the InplaceEvaluation working in-place of X.

ManifoldProximalMapObjective{E<:AbstractEvaluationType, TC, TP, V <: Vector{<:Integer}} <: AbstractManifoldCostObjective{E, TC}

specify a problem for solvers based on the evaluation of proximal maps, which represents proximal maps $\operatorname{prox}_{λf_i}$ for summands $f = f_1 + f_2+ … + f_N$ of the cost function $f$.

Fields

• cost: a function $f:\mathcal M→ℝ$ to minimize
• proxes: proximal maps $\operatorname{prox}_{λf_i}:\mathcal M → \mathcal M$ as functions (M, λ, p) -> q or in-place (M, q, λ, p).
• number_of_proxes: number of proximal maps per function, to specify when one of the maps is a combined one such that the proximal maps functions return more than one entry per function, you have to adapt this value. if not specified, it is set to one prox per function.

Constructor

ManifoldProximalMapObjective(f, proxes_f::Union{Tuple,AbstractVector}, numer_of_proxes=onex(length(proxes));
   evaluation=Allocating)

Generate a proximal problem with a tuple or vector of funtions, where by default every function computes a single prox of one component of $f$.

ManifoldProximalMapObjective(f, prox_f); evaluation=Allocating)

Generate a proximal objective for $f$ and its proxial map $\operatorname{prox}_{λf}$

See also

cyclic_proximal_point, get_cost, get_proximal_map

q = get_proximal_map(M::AbstractManifold, mpo::ManifoldProximalMapObjective, λ, p)
get_proximal_map!(M::AbstractManifold, q, mpo::ManifoldProximalMapObjective, λ, p)
q = get_proximal_map(M::AbstractManifold, mpo::ManifoldProximalMapObjective, λ, p, i)
get_proximal_map!(M::AbstractManifold, q, mpo::ManifoldProximalMapObjective, λ, p, i)

evaluate the (ith) proximal map of the ManifoldProximalMapObjectivempo at the point p of M with parameter $λ>0$.

q = get_proximal_map(M::AbstractManifold, mpo::ManifoldProximalGradientObjective, λ, p)
get_proximal_map!(M::AbstractManifold, q, mpo::ManifoldProximalGradientObjective, λ, p)

Evaluate proximal map of the nonsmooth component $h$ of the ManifoldProximalGradientObjectivempo at the point p on M with parameter $λ>0$.

AbstractManifoldHessianObjective{E<:AbstractEvaluationType,F, G, H} <: AbstractManifoldFirstOrderObjective{E,Tuple{F,G}}

An abstract type for all objectives that provide a (full) Hessian, where T is a AbstractEvaluationType for the gradient and Hessian functions.

ManifoldHessianObjective{T<:AbstractEvaluationType,C,G,H,Pre} <: AbstractManifoldHessianObjective{T,C,G,H}

specify a problem for Hessian based algorithms.

Fields

• cost: a function $f:\mathcal M→ℝ$ to minimize
• gradient: the gradient $\operatorname{grad}f:\mathcal M → \mathcal T\mathcal M$ of the cost function $f$
• hessian: the Hessian $\operatorname{Hess}f(x)[⋅]: \mathcal T_{x} \mathcal M → \mathcal T_{x} \mathcal M$ of the cost function $f$
• preconditioner: the symmetric, positive definite preconditioner as an approximation of the inverse of the Hessian of $f$, a map with the same input variables as the hessian to numerically stabilize iterations when the Hessian is ill-conditioned

Depending on the AbstractEvaluationType T the gradient and can have to forms

• as a function (M, p) -> X and (M, p, X) -> Y, resp., an AllocatingEvaluation
• as a function (M, X, p) -> X and (M, Y, p, X), resp., an InplaceEvaluation

Constructor

ManifoldHessianObjective(f, grad_f, Hess_f, preconditioner = (M, p, X) -> X;
    evaluation=AllocatingEvaluation())

See also

truncated_conjugate_gradient_descent, trust_regions

Y = get_hessian(amp::AbstractManoptProblem{T}, p, X)
get_hessian!(amp::AbstractManoptProblem{T}, Y, p, X)

evaluate the Hessian of an AbstractManoptProblem amp at p applied to a tangent vector X, computing $\operatorname{Hess}f(q)[X]$, which can also happen in-place of Y.

get_hessian(M::AbstractManifold, vgf::VectorHessianFunction, p, X, i)
get_hessian(M::AbstractManifold, vgf::VectorHessianFunction, p, X, i, range)
get_hessian!(M::AbstractManifold, X, vgf::VectorHessianFunction, p, X, i)
get_hessian!(M::AbstractManifold, X, vgf::VectorHessianFunction, p, X, i, range)

Evaluate the Hessians of the vector function vgf on the manifold M at p in direction X and the values given in range, specifying the representation of the gradients.

Since i is assumed to be a linear index, you can provide

• a single integer
• a UnitRange to specify a range to be returned like 1:3
• a BitVector specifying a selection
• a AbstractVector{<:Integer} to specify indices
• : to return the vector of all Hessian evaluations

get_hessian(TpM, trmo::TrustRegionModelObjective, X)

Evaluate the Hessian of the TrustRegionModelObjective

\[\operatorname{Hess} m(X)[Y] = \operatorname{Hess} f(p)[Y].\]

get_Hessian(TpM::TangentSpace, slso::SymmetricLinearSystemObjective, X, V)
get_Hessian!(TpM::TangentSpace, W, slso::SymmetricLinearSystemObjective, X, V)

evaluate the Hessian of

\[f(X) = \frac{1}{2} \lVert \mathcal A[X] + b \rVert_{p}^2,\qquad X ∈ T_{p}\mathcal M,\]

Which is $\operatorname{Hess} f(X)[Y] = \mathcal A[V]$. This can be computed in-place of W.

get_hessian(M::AbstractManifold, emo::EmbeddedManifoldObjective, p, X)
get_hessian!(M::AbstractManifold, Y, emo::EmbeddedManifoldObjective, p, X)

Evaluate the Hessian of an objective defined in the embedding, that is embed p and X before calling the Hessian function stored in the EmbeddedManifoldObjective.

The returned Hessian is then converted to a Riemannian Hessian calling riemannian_Hessian.

get_hessian(M::AbstractManifold, scaled_objective::ScaledManifoldObjective, p, X)
get_hessian!(M::AbstractManifold, Y, scaled_objective::ScaledManifoldObjective, p, X)

Evaluate the scaled Hessian $s*\operatorname{Hess}f(p)$

get_preconditioner(amp::AbstractManoptProblem, p, X)

evaluate the symmetric, positive definite preconditioner (approximation of the inverse of the Hessian of the cost function f) of a AbstractManoptProblem amps objective at the point p applied to a tangent vector X.

get_preconditioner(M::AbstractManifold, mho::ManifoldHessianObjective, p, X)

evaluate the symmetric, positive definite preconditioner (approximation of the inverse of the Hessian of the cost function F) of a ManifoldHessianObjective mho at the point p applied to a tangent vector X.

get_hessian_function(amgo::ManifoldHessianObjective{E<:AbstractEvaluationType})

return the function to evaluate (just) the Hessian $\operatorname{Hess} f(p)$. Depending on the AbstractEvaluationType E this is a function

• (M, p, X) -> Y for the AllocatingEvaluation case
• (M, Y, p, X) -> X for the InplaceEvaluation, working in-place of Y.

AbstractPrimalDualManifoldObjective{E<:AbstractEvaluationType,C,P} <: AbstractManifoldCostObjective{E,C}

A common abstract super type for objectives that consider primal-dual problems.

PrimalDualManifoldObjective{T<:AbstractEvaluationType} <: AbstractPrimalDualManifoldObjective{T}

Describes an Objective linearized or exact Chambolle-Pock algorithm, cf. [BHS+21], [CP11]

Fields

All fields with !! can either be in-place or allocating functions, which should be set depending on the evaluation= keyword in the constructor and stored in T <: AbstractEvaluationType.

• cost: $F + G(Λ(⋅))$ to evaluate interim cost function values
• linearized_forward_operator!!: linearized operator for the forward operation in the algorithm $DΛ$
• linearized_adjoint_operator!!: the adjoint differential $(DΛ)^* : \mathcal N → T\mathcal M$
• prox_f!!: the proximal map belonging to $f$
• prox_G_dual!!: the proximal map belonging to $g_n^*$
• Λ!!: the forward operator (if given) $Λ: \mathcal M → \mathcal N$

Either the linearized operator $DΛ$ or $Λ$ are required usually.

Constructor

PrimalDualManifoldObjective(cost, prox_f, prox_G_dual, adjoint_linearized_operator;
    linearized_forward_operator::Union{Function,Missing}=missing,
    Λ::Union{Function,Missing}=missing,
    evaluation::AbstractEvaluationType=AllocatingEvaluation()
)

The last optional argument can be used to provide the 4 or 5 functions as allocating or mutating (in place computation) ones. Note that the first argument is always the manifold under consideration, the mutated one is the second.

PrimalDualManifoldSemismoothNewtonObjective{E<:AbstractEvaluationType, TC, LO, ALO, PF, DPF, PG, DPG, L} <: AbstractPrimalDualManifoldObjective{E, TC, PF}

Describes a Problem for the Primal-dual Riemannian semismooth Newton algorithm. [DL21]

Fields

• cost: $F + G(Λ(⋅))$ to evaluate interim cost function values
• linearized_operator: the linearization $DΛ(⋅)[⋅]$ of the operator $Λ(⋅)$.
• linearized_adjoint_operator: the adjoint differential $(DΛ)^* : \mathcal N → T\mathcal M$
• prox_F: the proximal map belonging to $F$
• diff_prox_F: the (Clarke Generalized) differential of the proximal maps of $F$
• prox_G_dual: the proximal map belonging to G^\ast_n`
• diff_prox_dual_G: the (Clarke Generalized) differential of the proximal maps of $G^\ast_n$
• Λ: the exact forward operator. This operator is required if Λ(m)=n does not hold.

Constructor

PrimalDualManifoldSemismoothNewtonObjective(cost, prox_F, prox_G_dual, forward_operator, adjoint_linearized_operator,Λ)

X = adjoint_linearized_operator(N::AbstractManifold, apdmo::AbstractPrimalDualManifoldObjective, m, n, Y)
adjoint_linearized_operator(N::AbstractManifold, X, apdmo::AbstractPrimalDualManifoldObjective, m, n, Y)

Evaluate the adjoint of the linearized forward operator of $(DΛ(m))^*[Y]$ stored within the AbstractPrimalDualManifoldObjective (in place of X). Since $Y∈T_n\mathcal N$, both $m$ and $n=Λ(m)$ are necessary arguments, mainly because the forward operator $Λ$ might be missing in p.

q = forward_operator(M::AbstractManifold, N::AbstractManifold, apdmo::AbstractPrimalDualManifoldObjective, p)
forward_operator!(M::AbstractManifold, N::AbstractManifold, q, apdmo::AbstractPrimalDualManifoldObjective, p)

Evaluate the forward operator of $Λ(x)$ stored within the TwoManifoldProblem (in place of q).

η = get_differential_dual_prox(N::AbstractManifold, pdsno::PrimalDualManifoldSemismoothNewtonObjective, n, τ, X, ξ)
get_differential_dual_prox!(N::AbstractManifold, pdsno::PrimalDualManifoldSemismoothNewtonObjective, η, n, τ, X, ξ)

Evaluate the differential proximal map of $G_n^*$ stored within PrimalDualManifoldSemismoothNewtonObjective

\[D\operatorname{prox}_{τG_n^*}(X)[ξ]\]

which can also be computed in place of η.

y = get_differential_primal_prox(M::AbstractManifold, pdsno::PrimalDualManifoldSemismoothNewtonObjective σ, x)
get_differential_primal_prox!(p::TwoManifoldProblem, y, σ, x)

Evaluate the differential proximal map of $F$ stored within AbstractPrimalDualManifoldObjective

\[D\operatorname{prox}_{σF}(x)[X]\]

which can also be computed in place of y.

Y = get_dual_prox(N::AbstractManifold, apdmo::AbstractPrimalDualManifoldObjective, n, τ, X)
get_dual_prox!(N::AbstractManifold, apdmo::AbstractPrimalDualManifoldObjective, Y, n, τ, X)

Evaluate the proximal map of $g_n^*$ stored within AbstractPrimalDualManifoldObjective

\[ Y = \operatorname{prox}}_{τG_n^*}(X)\]

which can also be computed in place of Y.

q = get_primal_prox(M::AbstractManifold, p::AbstractPrimalDualManifoldObjective, σ, p)
get_primal_prox!(M::AbstractManifold, p::AbstractPrimalDualManifoldObjective, q, σ, p)

Evaluate the proximal map of $F$ stored within AbstractPrimalDualManifoldObjective

\[\operatorname{prox}_{σF}(x)\]

which can also be computed in place of y.

Y = linearized_forward_operator(M::AbstractManifold, N::AbstractManifold, apdmo::AbstractPrimalDualManifoldObjective, m, X, n)
linearized_forward_operator!(M::AbstractManifold, N::AbstractManifold, Y, apdmo::AbstractPrimalDualManifoldObjective, m, X, n)

Evaluate the linearized operator (differential) $DΛ(m)[X]$ stored within the AbstractPrimalDualManifoldObjective (in place of Y), where n = Λ(m).

ConstrainedManifoldObjective{T<:AbstractEvaluationType, C<:ConstraintType} <: AbstractManifoldObjective{T}

Describes the constrained objective

\[\begin{aligned} \operatorname*{arg\,min}_{p ∈\mathcal{M}} & f(p)\\ \text{subject to } &g_i(p)\leq0 \quad \text{ for all } i=1,…,m,\\ \quad &h_j(p)=0 \quad \text{ for all } j=1,…,n. \end{aligned}\]

Fields

• objective: an AbstractManifoldObjective representing the unconstrained objective, that is containing cost $f$, the gradient of the cost $f$ and maybe the Hessian.
• equality_constraints: an AbstractManifoldObjective representing the equality constraints

$h: \mathcal M → \mathbb R^n$ also possibly containing its gradient and/or Hessian

• equality_constraints: an AbstractManifoldObjective representing the equality constraints

$h: \mathcal M → \mathbb R^n$ also possibly containing its gradient and/or Hessian

Constructors

ConstrainedManifoldObjective(M::AbstractManifold, f, grad_f;
    g=nothing,
    grad_g=nothing,
    h=nothing,
    grad_h=nothing;
    hess_f=nothing,
    hess_g=nothing,
    hess_h=nothing,
    equality_constraints=nothing,
    inequality_constraints=nothing,
    evaluation=AllocatingEvaluation(),
    M = nothing,
    p = isnothing(M) ? nothing : rand(M),
)

Generate the constrained objective based on all involved single functions f, grad_f, g, grad_g, h, grad_h, and optionally a Hessian for each of these. With equality_constraints and inequality_constraints you have to provide the dimension of the ranges of h and g, respectively. You can also provide a manifold M and a point p to use one evaluation of the constraints to automatically try to determine these sizes.

ConstrainedManifoldObjective(M::AbstractManifold, mho::AbstractManifoldObjective;
    equality_constraints = nothing,
    inequality_constraints = nothing
)

Generate the constrained objective either with explicit constraints $g$ and $h$, and their gradients, or in the form where these are already encapsulated in VectorGradientFunctions.

Both variants require that at least one of the constraints (and its gradient) is provided. If any of the three parts provides a Hessian, the corresponding object, that is a ManifoldHessianObjective for f or a VectorHessianFunction for g or h, respectively, is created.

ManifoldConstrainedSetObjective{E, MO, PF, IF} <: AbstractManifoldObjective{E}

Model a constrained objective restricted to a set

\[\operatorname*{arg\,min}_{p ∈ \mathcal C} f(p)\]

where $\mathcal C ⊂ \mathcal M$ is a convex closed subset.

Fields

• objective::AbstractManifoldObjective the (unconstrained) objective, which contains $f$ and for example ist gradient $\operatorname{grad} f$.
• project!!::PF a projection function $\operatorname{proj}_{\mathcal C}: \mathcal M → \mathcal C$ that projects onto the set $\mathcal C$.
• indicator::IF the indicator function ``ι_{\mathcal C}(p) = \begin{cases} 0 &\text{ for }p∈\mathcal C\\ ∞ &\text{ else.}\end{cases}

Constructor

ManifoldConstrainedSetObjective(f, grad_f, project!!; kwargs...)

Generate the constrained objective for a given function f its gradient grad_f and a projection project!! $\operatorname{proj}_{\mathcal C}$.

Keyword arguments

• evaluation=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating its result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.
• indicator=nothing: the indicator function $ι_{\mathcal C}(p)$. If not provided a test, whether the projection yields the same point is performed. For the InplaceEvaluation this required one allocation.

ConstrainedProblem{
    TM <: AbstractManifold,
    O <: AbstractManifoldObjective
    HR<:Union{AbstractPowerRepresentation,Nothing},
    GR<:Union{AbstractPowerRepresentation,Nothing},
    HHR<:Union{AbstractPowerRepresentation,Nothing},
    GHR<:Union{AbstractPowerRepresentation,Nothing},
} <: AbstractManoptProblem{TM}

A constrained problem might feature different ranges for the (vectors of) gradients of the equality and inequality constraints.

The ranges are required in a few places to allocate memory and access elements correctly, they work as follows:

Assume the objective is

\[\begin{aligned} \operatorname*{arg\,min}_{p ∈\mathcal{M}} & f(p)\\ \text{subject to } &g_i(p)\leq0 \quad \text{ for all } i=1,…,m,\\ \quad &h_j(p)=0 \quad \text{ for all } j=1,…,n. \end{aligned}\]

then the gradients can (classically) be considered as vectors of the components gradients, for example $\bigl(\operatorname{grad} g_1(p), \operatorname{grad} g_2(p), …, \operatorname{grad} g_m(p) \bigr)$.

In another interpretation, this can be considered a point on the tangent space at $P = (p,…,p) \in \mathcal M^m$, so in the tangent space to the PowerManifold $\mathcal M^m$. The case where this is a NestedPowerRepresentation this agrees with the interpretation from before, but on power manifolds, more efficient representations exist.

To then access the elements, the range has to be specified. That is what this problem is for.

Constructor

ConstrainedManoptProblem(
    M::AbstractManifold,
    co::ConstrainedManifoldObjective;
    range=NestedPowerRepresentation(),
    gradient_equality_range=range,
    gradient_inequality_range=range
    hessian_equality_range=range,
    hessian_inequality_range=range
)

Creates a constrained Manopt problem specifying an AbstractPowerRepresentation for both the gradient_equality_range and the gradient_inequality_range, respectively.

AbstractConstrainedFunctor{T}

A common supertype for functors that model constraint functions.

This supertype provides access for the fields $λ$ and $μ$, the dual variables of constraints of type T.

AbstractConstrainedSlackFunctor{T,R}

A common supertype for functors that model constraint functions with slack.

This supertype additionally provides access for the fields

• μ::T the dual for the inequality constraints
• s::T the slack parameter, and
• β::R the the barrier parameter

which is also of type T.

LagrangianCost{CO,T} <: AbstractConstrainedFunctor{T}

Implement the Lagrangian of a ConstrainedManifoldObjective co.

\[\mathcal L(p; μ, λ) = f(p) + \sum_{i=1}^m μ_ig_i(p) + \sum_{j=1}^n λ_jh_j(p)\]

Fields

• co::CO, μ::T, λ::T as mentioned, where T represents a vector type.

Constructor

LagrangianCost(co, μ, λ)

Create a functor for the Lagrangian with fixed dual variables.

Example

When you directly want to evaluate the Lagrangian $\mathcal L$ you can also call

LagrangianCost(co, μ, λ)(M,p)

LagrangianGradient{CO,T}

The gradient of the Lagrangian of a ConstrainedManifoldObjective co with respect to the variable $p$. The formula reads

\[\operatorname{grad}_p \mathcal L(p; μ, λ) = \operatorname{grad} f(p) + \sum_{i=1}^m μ_i \operatorname{grad} g_i(p) + \sum_{j=1}^n λ_j \operatorname{grad} h_j(p)\]

Fields

• co::CO, μ::T, λ::T as mentioned, where T represents a vector type.

Constructor

LagrangianGradient(co, μ, λ)

Create a functor for the Lagrangian with fixed dual variables.

Example

When you directly want to evaluate the gradient of the Lagrangian $\operatorname{grad}_p \mathcal L$ you can also call LagrangianGradient(co, μ, λ)(M,p) or LagrangianGradient(co, μ, λ)(M,X,p) for the in-place variant.

LagrangianHessian{CO, V, T}

The Hesian of the Lagrangian of a ConstrainedManifoldObjective co with respect to the variable $p$. The formula reads

\[\operatorname{Hess}_p \mathcal L(p; μ, λ)[X] = \operatorname{Hess} f(p) + \sum_{i=1}^m μ_i \operatorname{Hess} g_i(p)[X] + \sum_{j=1}^n λ_j \operatorname{Hess} h_j(p)[X]\]

Fields

• co::CO, μ::T, λ::T as mentioned, where T represents a vector type.

Constructor

LagrangianHessian(co, μ, λ)

Create a functor for the Lagrangian with fixed dual variables.

Example

When you directly want to evaluate the Hessian of the Lagrangian $\operatorname{Hess}_p \mathcal L$ you can also call LagrangianHessian(co, μ, λ)(M, p, X) or LagrangianHessian(co, μ, λ)(M, Y, p, X) for the in-place variant.

equality_constraints_length(co::ConstrainedManifoldObjective)

Return the number of equality constraints of an ConstrainedManifoldObjective. This acts transparently through AbstractDecoratedManifoldObjectives

inequality_constraints_length(cmo::ConstrainedManifoldObjective)

Return the number of inequality constraints of an ConstrainedManifoldObjective cmo. This acts transparently through AbstractDecoratedManifoldObjectives

get_equality_constraint(amp::AbstractManoptProblem, p, j=:)
get_equality_constraint(M::AbstractManifold, objective, p, j=:)

Evaluate equality constraints of a ConstrainedManifoldObjective objective at point p and indices j (by default : which corresponds to all indices).

get_grad_equality_constraint(amp::AbstractManoptProblem, p, j)
get_grad_equality_constraint(M::AbstractManifold, co::ConstrainedManifoldObjective, p, j, range=NestedPowerRepresentation())
get_grad_equality_constraint!(amp::AbstractManoptProblem, X, p, j)
get_grad_equality_constraint!(M::AbstractManifold, X, co::ConstrainedManifoldObjective, p, j, range=NestedPowerRepresentation())

Evaluate the gradient or gradients of the equality constraint $(\operatorname{grad} h(p))_j$ or $\operatorname{grad} h_j(p)$,

See also the ConstrainedManoptProblem to specify the range of the gradient.

get_grad_inequality_constraint(amp::AbstractManoptProblem, p, j=:)
get_grad_inequality_constraint(M::AbstractManifold, co::ConstrainedManifoldObjective, p, j=:, range=NestedPowerRepresentation())
get_grad_inequality_constraint!(amp::AbstractManoptProblem, X, p, j=:)
get_grad_inequality_constraint!(M::AbstractManifold, X, co::ConstrainedManifoldObjective, p, j=:, range=NestedPowerRepresentation())

Evaluate the gradient or gradients of the inequality constraint $(\operatorname{grad} g(p))_j$ or $\operatorname{grad} g_j(p)$,

See also the ConstrainedManoptProblem to specify the range of the gradient.

get_hess_equality_constraint(amp::AbstractManoptProblem, p, j=:)
get_hess_equality_constraint(M::AbstractManifold, co::ConstrainedManifoldObjective, p, j, range=NestedPowerRepresentation())
get_hess_equality_constraint!(amp::AbstractManoptProblem, X, p, j=:)
get_hess_equality_constraint!(M::AbstractManifold, X, co::ConstrainedManifoldObjective, p, j, range=NestedPowerRepresentation())

Evaluate the Hessian or Hessians of the equality constraint $(\operatorname{Hess} h(p))_j$ or $\operatorname{Hess} h_j(p)$,

See also the ConstrainedManoptProblem to specify the range of the Hessian.

get_hess_inequality_constraint(amp::AbstractManoptProblem, p, X, j=:)
get_hess_inequality_constraint(M::AbstractManifold, co::ConstrainedManifoldObjective, p, j=:, range=NestedPowerRepresentation())
get_hess_inequality_constraint!(amp::AbstractManoptProblem, Y, p, j=:)
get_hess_inequality_constraint!(M::AbstractManifold, Y, co::ConstrainedManifoldObjective, p, X, j=:, range=NestedPowerRepresentation())

Evaluate the Hessian or Hessians of the inequality constraint $(\operatorname{Hess} g(p)[X])_j$ or $\operatorname{Hess} g_j(p)[X]$,

See also the ConstrainedManoptProblem to specify the range of the Hessian.

get_inequality_constraint(amp::AbstractManoptProblem, p, j=:)
get_inequality_constraint(M::AbstractManifold, co::ConstrainedManifoldObjective, p, j=:, range=NestedPowerRepresentation())

Evaluate inequality constraints of a ConstrainedManifoldObjective objective at point p and indices j (by default : which corresponds to all indices).

get_projected_point(amp::AbstractManoptProblem, p)
get_projected_point!(amp::AbstractManoptProblem, q, p)
get_projected_point(M::AbstractManifold, cso::ManifoldConstrainedSetObjective, p)
get_projected_point!(M::AbstractManifold, q, cso::ManifoldConstrainedSetObjective, p)

Project p with the projection that is stored within the ManifoldConstrainedSetObjective. This can be done in-place of q.

get_projected_point(amp::AbstractManoptProblem, p)
get_projected_point!(amp::AbstractManoptProblem, q, p)
get_projected_point(M::AbstractManifold, cso::ManifoldConstrainedSetObjective, p)
get_projected_point!(M::AbstractManifold, q, cso::ManifoldConstrainedSetObjective, p)

Project p with the projection that is stored within the ManifoldConstrainedSetObjective. This can be done in-place of q.

get_projected_point(amp::AbstractManoptProblem, p)
get_projected_point!(amp::AbstractManoptProblem, q, p)
get_projected_point(M::AbstractManifold, cso::ManifoldConstrainedSetObjective, p)
get_projected_point!(M::AbstractManifold, q, cso::ManifoldConstrainedSetObjective, p)

Project p with the projection that is stored within the ManifoldConstrainedSetObjective. This can be done in-place of q.

get_projected_point(amp::AbstractManoptProblem, p)
get_projected_point!(amp::AbstractManoptProblem, q, p)
get_projected_point(M::AbstractManifold, cso::ManifoldConstrainedSetObjective, p)
get_projected_point!(M::AbstractManifold, q, cso::ManifoldConstrainedSetObjective, p)

Project p with the projection that is stored within the ManifoldConstrainedSetObjective. This can be done in-place of q.

get_unconstrained_objective(co::ConstrainedManifoldObjective)

Returns the internally stored unconstrained AbstractManifoldObjective within the ConstrainedManifoldObjective.

is_feasible(M::AbstractManifold, cmo::ConstrainedManifoldObjective, p, kwargs...)

Evaluate whether a boint p on M is feasible with respect to the ConstrainedManifoldObjective cmo. That is for the provided inequality constaints $g: \mathcal M → ℝ^m$ and equality constaints $h: \mathcal M \to ℝ^m$ from within cmo, the point $p ∈ \mathcal M$ is feasible if

\[g_i(p) ≤ 0, \text{ for all } i=1,…,m\quad\text{ and }\quad h_j(p) = 0, \text{ for all } j=1,…,n.\]

Keyword arguments

• check_point::Bool=true: whether to also verify that `p∈\mathcal M holds, using is_point
• error::Symbol=:none: if the point is not feasible, this symbol determines how to report the error.
  • :error: throws an error
  • :info: displays the error message as an @info
  • :none: (default) the function just returns true/false
  • :warn: displays the error message as a @warning.

The keyword error= and all other kwargs... are passed on to is_point if the point is verfied (see check_point).

All other keywords are passed on to is_poi

get_feasibility_status(
    M::AbstractManifold,
    cmo::ConstrainedManifoldObjective,
    g = get_inequality_constraints(M, cmo, p),
    h = get_equality_constraints(M, cmo, p),
)

Generate a message about the feasibiliy of p with respect to the ConstrainedManifoldObjective. You can also provide the evaluated vectors for the values of g and h as keyword arguments, in case you had them evaluated before.

AbstractVectorFunction{E, FT} <: Function

Represent an abstract vectorial function $f:\mathcal M → ℝ^n$ with an AbstractEvaluationType E and an AbstractVectorialType to specify the format $f$ is implemented as.

Representations of $f$

There are three different representations of $f$, which might be beneficial in one or the other situation:

• the FunctionVectorialType storing a single function $f$ that returns a vector,
• the ComponentVectorialType storing a vector of functions $f_i$ that return a single value each,
• the CoordinateVectorialType storing functions with respect to a specific basis of the tangent space for gradients and Hessians. Gradients of this type are usually referred to as Jacobians.

For the ComponentVectorialType imagine that $f$ could also be written using its component functions,

\[f(p) = \bigl( f_1(p), f_2(p), \ldots, f_n(p) \bigr)^{\mathrm{T}}\]

In this representation f is given as a vector [f1(M,p), f2(M,p), ..., fn(M,p)] of its component functions. An advantage is that the single components can be evaluated and from this representation one even can directly read of the number n. A disadvantage might be, that one has to implement a lot of individual (component) functions.

For the FunctionVectorialType $f$ is implemented as a single function f(M, p), that returns an AbstractArray. And advantage here is, that this is a single function. A disadvantage might be, that if this is expensive even to compute a single component, all of f has to be evaluated

VectorGradientFunction{E, FT, JT, F, J, I} <: AbstractManifoldObjective{E}

Represent an abstract vectorial function $f:\mathcal M → ℝ^n$ that provides a (component wise) gradient. The AbstractEvaluationType E indicates the evaluation type, and the AbstractVectorialTypes FT and JT the formats in which the function and the gradient are provided, see AbstractVectorFunction for an explanation.

VectorGradientFunction{E, FT, JT, F, J, I} <: AbstractVectorGradientFunction{E, FT, JT}

Represent a function $f:\mathcal M → ℝ^n$ including it first derivative, either as a vector of gradients of a Jacobian

And hence has a gradient `\operatorname{grad} f_i(p) ∈ T_{p}\mathcal M. Putting these gradients into a vector the same way as the functions, yields a ComponentVectorialType

\[\operatorname{grad} f(p) = \Bigl( \operatorname{grad} f_1(p), \operatorname{grad} f_2(p), …, \operatorname{grad} f_n(p) \Bigr)^\mathrm{T} ∈ (T_{p}\mathcal M)^n\]

And advantage here is, that again the single components can be evaluated individually

Fields

• value!!::F: the cost function $f$, which can take different formats
• cost_type::AbstractVectorialType: indicating / storing data for the type of f
• jacobian!!::G: the Jacobian of $f$
• jacobian_type::AbstractVectorialType: indicating / storing data for the type of $J_f$
• parameters: the number n from, the size of the vector $f$ returns.

Constructor

VectorGradientFunction(f, Jf, range_dimension;
    evaluation::AbstractEvaluationType=AllocatingEvaluation(),
    function_type::AbstractVectorialType=FunctionVectorialType(),
    jacobian_type::AbstractVectorialType=FunctionVectorialType(),
)

Create a VectorGradientFunction of f and its Jacobian (vector of gradients) Jf, where f maps into the Euclidean space of dimension range_dimension. Their types are specified by the function_type, and jacobian_type, respectively. The Jacobian can further be given as an allocating variant or an in-place variant, specified by the evaluation= keyword.

VectorHessianFunction{E, FT, JT, HT, F, J, H, I} <: AbstractVectorGradientFunction{E, FT, JT}

Represent a function $f:\mathcal M M → ℝ^n$ including it first derivative, either as a vector of gradients of a Jacobian, and the Hessian, as a vector of Hessians of the component functions.

Both the Jacobian and the Hessian can map into either a sequence of tangent spaces or a single tangent space of the power manifold of length n.

Fields

• value!!::F: the cost function $f$, which can take different formats
• cost_type::AbstractVectorialType: indicating / string data for the type of f
• jacobian!!::G: the Jacobian $J_f$ of $f$
• jacobian_type::AbstractVectorialType: indicating / storing data for the type of $J_f$
• hessians!!::H: the Hessians of $f$ (in a component wise sense)
• hessian_type::AbstractVectorialType: indicating / storing data for the type of $H_f$
• range_dimension: the number n from, the size of the vector $f$ returns.

Constructor

VectorHessianFunction(f, Jf, Hess_f, range_dimension;
    evaluation::AbstractEvaluationType=AllocatingEvaluation(),
    function_type::AbstractVectorialType=FunctionVectorialType(),
    jacobian_type::AbstractVectorialType=FunctionVectorialType(),
    hessian_type::AbstractVectorialType=FunctionVectorialType(),
)

Create a VectorHessianFunction of f and its Jacobian (vector of gradients) Jf and (vector of) Hessians, where f maps into the Euclidean space of dimension range_dimension. Their types are specified by the function_type, and jacobian_type, and hessian_type, respectively. The Jacobian and Hessian can further be given as an allocating variant or an inplace-variant, specified by the evaluation= keyword.

AbstractVectorialType

An abstract type for different representations of a vectorial function $f: \mathcal M → ℝ^m$ and its (component-wise) gradient/Jacobian

CoordinateVectorialType{B<:AbstractBasis} <: AbstractVectorialType

A type to indicate that gradient of the constraints is implemented as a Jacobian matrix with respect to a certain basis, that is if the vector function is $f: \mathcal M → ℝ^m$ and we have a basis $\mathcal B$ of $T_p\mathcal M$, at $p∈ \mathcal M$ This can be written as $J_g(p) = (c_1^{\mathrm{T}},…,c_m^{\mathrm{T}})^{\mathrm{T}} \in ℝ^{m,d}$, that is, every row $c_i$ of this matrix is a set of coefficients such that get_coefficients(M, p, c, B) is the tangent vector $\oepratorname{grad} g_i(p)$ for example $g_i(p) ∈ ℝ^m$ or $\operatorname{grad} g_i(p) ∈ T_p\mathcal M$, $i=1,…,m$.

Fields

• basis an AbstractBasis to indicate the basis in which Jacobian is expressed.

Constructor

CoordinateVectorialType(basis=DefaultOrthonormalBasis())

ComponentVectorialType <: AbstractVectorialType

A type to indicate that constraints are implemented as component functions, for example $g_i(p) ∈ ℝ^m$ or $\operatorname{grad} g_i(p) ∈ T_p\mathcal M$, $i=1,…,m$.

FunctionVectorialType{P<:AbstractPowerRepresentation} <: AbstractVectorialType

A type to indicate that constraints are implemented one whole functions, for example $g(p) ∈ ℝ^m$ or $\operatorname{grad} g(p) ∈ (T_p\mathcal M)^m$.

This type internally stores the AbstractPowerRepresentation, when it makes sense, especially for Hessian and gradient functions.

get_jacobian(M::AbstractManifold, vgf::AbstractVectorGradientFunction, p; kwargs...)
get_jacobian(M::AbstractManifold, J, vgf::AbstractVectorGradientFunction, p; kwargs...)

Compute the Jacobian $J_F ∈ ℝ^{m×n}$ of the AbstractVectorGradientFunction $F$ at p on the M.

There are two interpretations of the Jacobian of a vectorial function $F: \mathcal M → ℝ^m$ on a manifold. Both depend on choosing a basis on the tangent space $T_{p}\mathcal M$ which we denote by $Y_1,…,Y_n$, where n is the manifold_dimension(M)(M). We can write any tangent vector $X = \displaystyle\sum_{}^{}_i c_iY_i$

1. The Jacobian $J_F$ is the matrix with respect to the basis $Y_1,…,Y_n$ such that

for any $X∈T_{p}\mathcal M$ we have the equality of the differential $DF(p)[X] = Jc$. In other words, the jth column of $J$ is given by $DF(p)[Y_j]$

1. Given the gradients $\operatorname{grad} F_i(p)$ of the component functions $F_i: \mathcal M → ℝ$,

we define the jacobian function as

$math J(X) = \begin{pmatrix} ⟨\operatorname{grad} F_1, X⟩_p\\ ⟨\operatorname{grad} F_1, X⟩_p\\ ⋮\\ ⟨\operatorname{grad} F_1, X⟩_p\end{pmatrix}$

Then either the $j$th column of $J_F$ is given by $J(Y_i)$ or the $i$th row is given by all inner products $\operatorname{grad} F_1, Y_j⟩_p$ of the $i$th gradient function with all basis vectors $Y_j$.

The computation can be computed in-place of J.

Keyword arguments

• basis::AbstractBasis =get_basis(vgf) for the CoordinateVectorialType of the vectorial functions gradient, this might lead to a change of basis, if this basis and the one the coordinates are given in do not agree.
• range::AbstractPowerRepresentation =get_range(vgf.jacobian_type) specify the range of the gradients in the case of a FunctionVectorialType, that is, on which type of power manifold the gradient is given on.

get_jacobian(M::AbstractManifold, vgf::AbstractVectorGradientFunction, p; kwargs...)
get_jacobian(M::AbstractManifold, J, vgf::AbstractVectorGradientFunction, p; kwargs...)

Compute the Jacobian $J_F ∈ ℝ^{m×n}$ of the AbstractVectorGradientFunction $F$ at p on the M.

There are two interpretations of the Jacobian of a vectorial function $F: \mathcal M → ℝ^m$ on a manifold. Both depend on choosing a basis on the tangent space $T_{p}\mathcal M$ which we denote by $Y_1,…,Y_n$, where n is the manifold_dimension(M)(M). We can write any tangent vector $X = \displaystyle\sum_{}^{}_i c_iY_i$

1. The Jacobian $J_F$ is the matrix with respect to the basis $Y_1,…,Y_n$ such that

for any $X∈T_{p}\mathcal M$ we have the equality of the differential $DF(p)[X] = Jc$. In other words, the jth column of $J$ is given by $DF(p)[Y_j]$

1. Given the gradients $\operatorname{grad} F_i(p)$ of the component functions $F_i: \mathcal M → ℝ$,

we define the jacobian function as

$math J(X) = \begin{pmatrix} ⟨\operatorname{grad} F_1, X⟩_p\\ ⟨\operatorname{grad} F_1, X⟩_p\\ ⋮\\ ⟨\operatorname{grad} F_1, X⟩_p\end{pmatrix}$

Then either the $j$th column of $J_F$ is given by $J(Y_i)$ or the $i$th row is given by all inner products $\operatorname{grad} F_1, Y_j⟩_p$ of the $i$th gradient function with all basis vectors $Y_j$.

The computation can be computed in-place of J.

Keyword arguments

• basis::AbstractBasis =get_basis(vgf) for the CoordinateVectorialType of the vectorial functions gradient, this might lead to a change of basis, if this basis and the one the coordinates are given in do not agree.
• range::AbstractPowerRepresentation =get_range(vgf.jacobian_type) specify the range of the gradients in the case of a FunctionVectorialType, that is, on which type of power manifold the gradient is given on.

get_value(M::AbstractManifold, vgf::AbstractVectorFunction, p[, i=:])
get_value!(M::AbstractManifold, V, vgf::AbstractVectorFunction, p[, i=:])

Evaluate the vector function VectorGradientFunction vgf at p. The range can be used to specify a potential range, but is currently only present for consistency.

The i can be a linear index, you can provide

• a single integer
• a UnitRange to specify a range to be returned like 1:3
• a BitVector specifying a selection
• a AbstractVector{<:Integer} to specify indices
• : to return the vector of all gradients, which is also the default

This function can perform the evaluation inplace of V.

get_value_function(vgf::VectorGradientFunction, recursive=false)

return the internally stored function computing get_value.

length(vgf::AbstractVectorFunction)

Return the length of the vector the function $f: \mathcal M → ℝ^n$ maps into, that is the number n.

_to_iterable_indices(A::AbstractVector, i)

Convert index i (integer, colon, vector of indices, etc.) for array A into an iterable structure of indices.

_change_basis!(M::AbstractManifold, JF, p, from_basis::B1, to_basis::B; X=zero_vector(M,p))

Given a jacobian matrix JF on a manifold M at p with respect to the from_basis in the tangent space of p on M. Change the basis of the Jacobian to to_basis in place of JF.

Keyword Arguments

• X a temporary vector to store a generated vector, before decomposing it again with respect to the new basis

get_basis(::AbstractVectorialType)

Return a basis that fits a vector function representation.

For the case, where some vectorial data is stored with respect to a basis, this function returns the corresponding basis, most prominently for the CoordinateVectorialType.

If a type is not with respect to a certain basis, the DefaultOrthonormalBasis is returned.

get_range(::AbstractVectorialType)

Return an abstract power manifold representation that fits a vector function's range. Most prominently a FunctionVectorialType returns its internal range.

Otherwise the default NestedPowerRepresentation() is used to work on a vector of data.

AbstractManifoldSubObjective{O<:AbstractManifoldObjective} <: AbstractManifoldObjective

An abstract type for objectives of sub problems within a solver but still store the original objective internally to generate generic objectives for sub solvers.

get_objective_cost(M, amso::AbstractManifoldSubObjective, p)

Evaluate the cost of the (original) objective stored within the sub objective.

X = get_objective_gradient(M, amso::AbstractManifoldSubObjective, p)
get_objective_gradient!(M, X, amso::AbstractManifoldSubObjective, p)

Evaluate the gradient of the (original) objective stored within the sub objective amso.

Y = get_objective_Hessian(M, amso::AbstractManifoldSubObjective, p, X)
get_objective_Hessian!(M, Y, amso::AbstractManifoldSubObjective, p, X)

Evaluate the Hessian of the (original) objective stored within the sub objective amso.

Y = get_objective_preconditioner(M, amso::AbstractManifoldSubObjective, p, X)
get_objective_preconditioner(M, Y, amso::AbstractManifoldSubObjective, p, X)

Evaluate the Hessian of the (original) objective stored within the sub objective amso.

ManifoldProximalGradientObjective{E,<:AbstractEvaluationType, TC, TG, TGG, TP} <: AbstractManifoldObjective{E,TC,TGG}

Model an objective of the form

\[ f(p) = g(p) + h(p),\qquad p \in \mathcal M,\]

where $g: \mathcal M → \bar{ℝ}$ is a differentiable function and $h: → \bar{ℝ}$ is a (possibly) lower semicontinous, and proper function.

This objective provides the total cost $f$, its smooth component $g$, as well as $\operatorname{grad} g$ and $\operatorname{prox}_{λ} h$.

Fields

• cost: the overall cost $f = g + h$
• cost_smooth: the smooth cost component $g$
• gradient_g!!: the gradient $\operatorname{grad} g$
• proximal_map_h!!: the proximal map $\operatorname{prox}_{λ} h$

Constructor

ManifoldProximalGradientObjective(f, g, grad_g, prox_h;
    evalauation=[`AllocatingEvaluation`](@ref)
)

Generate the proximal gradient objective given the total cost f = g + h, smooth cost g, the gradient of the smooth component grad_g, and the proximal map of the nonsmooth component prox_h.

Keyword arguments

• evaluation=AllocatingEvaluation: whether the gradient and proximal map is given as an allocation function or an in-place (InplaceEvaluation).