Gradient descent

gradient_descent(M, f, grad_f, p=rand(M); kwargs...)
gradient_descent(M, gradient_objective, p=rand(M); kwargs...)
gradient_descent!(M, f, grad_f, p; kwargs...)
gradient_descent!(M, gradient_objective, p; kwargs...)

perform the gradient descent algorithm

\[p_{k+1} = \operatorname{retr}_{p_k}\bigl( -s_k\operatorname{grad}f(p_k) \bigr), \qquad k=0,1,…\]

where $s_k > 0$ denotes a step size.

The algorithm can be performed in-place of p.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$
• f: a cost function $f: \mathcal{M}→ ℝ$ implemented as (M, p) -> v
• grad_f: the (Riemannian) gradient $\operatorname{grad}f: \mathcal{M} → T\mathcal{M}$ of f as a function (M, p) -> X or a function (M, X, p) -> X computing X in-place
• p::P: a point on the manifold $\mathcal{M}$

Alternatively to f and grad_f you can provide the corresponding AbstractManifoldFirstOrderObjective gradient_objective directly.

Keyword arguments

• callbacks::D = Dict{Symbol,Function}(): provided callback functions given either as a single function (symbol, problem, state, k) called in every hook or as a (vector of) pairs :hook => function, which are processed by process_callbacks_arg. As key you can either pass single symbol or an array of symbols to indicate a callback should be added in multiple places
• differential = missing: specify a specific function to evaluate the differential. By default, $Df(p)[X] = ⟨\operatorname{grad}f(p),X⟩$ is used
• direction=IdentityUpdateRule(): specify to perform a certain processing of the direction, for example Nesterov, MomentumGradient or AverageGradient.
• evaluation::AbstractEvaluationType=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating their result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second. For example grad_f(M,p) allocates, but grad_f!(M, X, p) computes the result in-place of X.
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions
• stepsize::Stepsize=default_stepsize(M,GradientDescentState; retraction_method=retraction_method): a functor inheriting from Stepsize to determine a step size
• stopping_criterion::StoppingCriterion=StopAfterIteration(200)|StopWhenGradientNormLess(1e-8): a functor indicating that the stopping criterion is fulfilled
• X::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$ storing the gradient at the current iterate

All other keyword arguments are passed to decorate_state! for state decorators or decorate_objective! for objective decorators, respectively.

If you provide the ManifoldFirstOrderObjective directly, the evaluation= keyword is ignored. The decorations are still applied to the objective.

If you activate tutorial mode (cf. is_tutorial_mode), this solver provides additional debug warnings.

Output

The obtained approximate minimizer $p^*$. To obtain the whole final state of the solver, see get_solver_return for details, especially the return_state= keyword.

gradient_descent(M, f, grad_f, p=rand(M); kwargs...)
gradient_descent(M, gradient_objective, p=rand(M); kwargs...)
gradient_descent!(M, f, grad_f, p; kwargs...)
gradient_descent!(M, gradient_objective, p; kwargs...)

perform the gradient descent algorithm

\[p_{k+1} = \operatorname{retr}_{p_k}\bigl( -s_k\operatorname{grad}f(p_k) \bigr), \qquad k=0,1,…\]

where $s_k > 0$ denotes a step size.

The algorithm can be performed in-place of p.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$
• f: a cost function $f: \mathcal{M}→ ℝ$ implemented as (M, p) -> v
• grad_f: the (Riemannian) gradient $\operatorname{grad}f: \mathcal{M} → T\mathcal{M}$ of f as a function (M, p) -> X or a function (M, X, p) -> X computing X in-place
• p::P: a point on the manifold $\mathcal{M}$

Alternatively to f and grad_f you can provide the corresponding AbstractManifoldFirstOrderObjective gradient_objective directly.

Keyword arguments

• callbacks::D = Dict{Symbol,Function}(): provided callback functions given either as a single function (symbol, problem, state, k) called in every hook or as a (vector of) pairs :hook => function, which are processed by process_callbacks_arg. As key you can either pass single symbol or an array of symbols to indicate a callback should be added in multiple places
• differential = missing: specify a specific function to evaluate the differential. By default, $Df(p)[X] = ⟨\operatorname{grad}f(p),X⟩$ is used
• direction=IdentityUpdateRule(): specify to perform a certain processing of the direction, for example Nesterov, MomentumGradient or AverageGradient.
• evaluation::AbstractEvaluationType=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating their result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second. For example grad_f(M,p) allocates, but grad_f!(M, X, p) computes the result in-place of X.
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions
• stepsize::Stepsize=default_stepsize(M,GradientDescentState; retraction_method=retraction_method): a functor inheriting from Stepsize to determine a step size
• stopping_criterion::StoppingCriterion=StopAfterIteration(200)|StopWhenGradientNormLess(1e-8): a functor indicating that the stopping criterion is fulfilled
• X::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$ storing the gradient at the current iterate

All other keyword arguments are passed to decorate_state! for state decorators or decorate_objective! for objective decorators, respectively.

If you provide the ManifoldFirstOrderObjective directly, the evaluation= keyword is ignored. The decorations are still applied to the objective.

If you activate tutorial mode (cf. is_tutorial_mode), this solver provides additional debug warnings.

Output

The obtained approximate minimizer $p^*$. To obtain the whole final state of the solver, see get_solver_return for details, especially the return_state= keyword.

GradientDescentState{P,T} <: AbstractGradientSolverState

Describes the state of a gradient based descent algorithm.

Fields

• callbacks::D: provided callback functions given as a dictionary with symbols as keys
• p::P: a point on the manifold $\mathcal{M}$ storing the current iterate
• X::T: a tangent vector at the point $p$ on the manifold $\mathcal{M}$ storing the gradient at the current iterate
• stop::StoppingCriterion: a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize: a functor inheriting from Stepsize to determine a step size
• direction::DirectionUpdateRule : a processor to handle the obtained gradient and compute a direction to “walk into”.
• retraction_method::AbstractRetractionMethod: a retraction $\operatorname{retr}$ to use, see the section on retractions

Constructor

GradientDescentState(M::AbstractManifold; kwargs...)

Initialize the gradient descent solver state, where

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$

Keyword arguments

• direction=IdentityUpdateRule() specify a processor to modify the gradient direction
• callbacks::D = Dict{Symbol,Function}(): provided callback functions given as a dictionary with symbols as keys
• p::P =rand(M): a point on the manifold $\mathcal{M}$ to specify the initial value
• stopping_criterion::StoppingCriterion=StopAfterIteration(200)|StopWhenGradientNormLess(1e-8): a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize=default_stepsize(M,GradientDescentState; retraction_method=retraction_method): a functor inheriting from Stepsize to determine a step size
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions
• X::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$ to specify the representation of a tangent vector

See also

gradient_descent

AverageGradient(; kwargs...)
AverageGradient(M::AbstractManifold; kwargs...)

Add an average of gradients to a gradient processor. A set of previous directions (from the inner processor) and the last iterate are stored, average is taken after vector transporting them to the current iterate's tangent space.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$ (optional)

Keyword arguments

• p::P =rand(M): a point on the manifold $\mathcal{M}$ to specify the initial value
• direction=IdentityUpdateRule preprocess the actual gradient before averaging
• gradients=[zero_vector(M, p) for _ in 1:n] how to initialize the internal storage
• n=10 number of gradient evaluations to take the mean over
• vector_transport_method::AbstractVectorTransportMethod=default_vector_transport_method(M, typeof(p)): a vector transport $\mathcal T_{⋅←⋅}$ to use, see the section on vector transports

MomentumGradient(args...; kwargs...)

Append a momentum to a gradient processor.

The last direction and last iterate are stored and the new one is composed as $η_k = m η_{k-1}' + d_k$, where $d_k$ is the current (inner) direction, $η_{k-1}'$ is the last direction transported to the current iterate, and $m$ is the momentum. The step size is not folded into $η_i$, the solver applies it to the returned direction. This is the Riemannian version of gradient descent with momentum, first used in [RMH18, Section 3.1]; see [LS26, Section 6] for a convergence analysis.

Input

• M (optional)

Keyword arguments

• p::P =rand(M): a point on the manifold $\mathcal{M}$
• direction=IdentityUpdateRule preprocess the actual gradient before adding momentum
• X::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$
• momentum=0.2 amount of momentum to use
• vector_transport_method::AbstractVectorTransportMethod=default_vector_transport_method(M, typeof(p)): a vector transport $\mathcal T_{⋅←⋅}$ to use, see the section on vector transports

Nesterov(; kwargs...)
Nesterov(M::AbstractManifold; kwargs...)

Assume $f$ is $L$-Lipschitz and $μ$-strongly convex. Given

• a step size $h_k<\frac{1}{L}$ (from the GradientDescentState)
• a shrinkage parameter $β_k$
• and a current iterate $p_k$
• as well as the interim values $γ_k$ and $v_k$ from the previous iterate.

This computes a Nesterov type update using the following steps, see [ZS18]

1. Compute the positive root $α_k∈(0,1)$ of $α_k^2 = h_k\bigl((1-α_k)γ_k+α_k μ\bigr)$.
2. Set $\bar{γ}_{k+1} = (1-α_k)γ_k + α_kμ$
3. $y_k = \operatorname{retr}_{p_k}\Bigl(\frac{α_kγ_k}{γ_k + α_kμ}\operatorname{retr}^{-1}_{p_k}v_k \Bigr)$
4. $p_{k+1} = \operatorname{retr}_{y_k}(-h_k \operatorname{grad}f(y_k))$
5. $v_{k+1} = \operatorname{retr}_{y_k}\Bigl(\frac{(1-α_k)γ_k}{\bar{γ}_{k+1}}\operatorname{retr}_{y_k}^{-1}(v_k) - \frac{α_k}{\bar{γ}_{k+1}}\operatorname{grad}f(y_k) \Bigr)$
6. $γ_{k+1} = \frac{1}{1+β_k}\bar{γ}_{k+1}$

Then, with $d = \operatorname{retr}^{-1}_{p_k}p_{k+1}$ the direction from $p_k$ to $p_{k+1}$, the pair $(h_k, -d/h_k)$ is returned, so that the solver's step $\operatorname{retr}_{p_k}(-h_k ⋅ (-d/h_k))$ yields $p_{k+1}$.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$ (optional)

Keyword arguments

• p::P =rand(M): a point on the manifold $\mathcal{M}$ to specify the initial value
• γ=0.001
• μ=0.9
• shrinkage = k -> 0.8
• inverse_retraction_method::AbstractInverseRetractionMethod=default_inverse_retraction_method(M, typeof(p)): an inverse retraction $\operatorname{retr}^{-1}$ to use, see the section on retractions and their inverses
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions

PreconditionedDirection(preconditioner; kwargs...)
PreconditionedDirection(M::AbstractManifold, preconditioner; kwargs...)

Add a preconditioner to a gradient processor following the motivation for optimization, as a linear invertible map $P: T_{p}\mathcal{M} → T_{p}\mathcal{M}$ that usually should be

• symmetric: $⟨X, P(Y)⟩ = ⟨P(X), Y⟩$
• positive definite $⟨X, P(X)⟩ > 0$ for $X$ not the zero-vector

The gradient is then preconditioned as $P(X)$, where $X$ is either the gradient of the objective or the result of a previous (internally stored) gradient processor.

For example if you provide as the preconditioner the inverse of the Hessian $\operatorname{Hess}^{-1} f$, you turn a gradient descent into a Newton method.

Arguments

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$ (optional)
• preconditioner: preconditioner function, either as a (M, p, X) -> Y allocating or (M, Y, p, X) -> Y mutating function

Keyword arguments

• direction=IdentityUpdateRule internal DirectionUpdateRule to determine the gradients to store or a ManifoldDefaultsFactory generating one
• evaluation::AbstractEvaluationType=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating their result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.

AverageGradientRule <: DirectionUpdateRule

Add an average of gradients to a gradient processor.

A set of previous directions (from the inner processor) and the last iterate are stored. The average is taken after vector transporting them to the current iterate's tangent space.

Fields

• gradients: the last n gradient/direction updates
• last_iterate: last iterate (needed to transport the gradients)
• direction: internal DirectionUpdateRule to determine directions to apply the averaging to
• vector_transport_method::AbstractVectorTransportMethod: a vector transport $\mathcal T_{⋅←⋅}$ to use, see the section on vector transports

Constructors

AverageGradientRule(    M::AbstractManifold;    p=rand(M),    n::Int=10,    direction::Union{<:DirectionUpdateRule,ManifoldDefaultsFactory}=Gradient(),    gradients=[zero_vector(M, p) for _ in 1:n],    vector_transport_method=default_vector_transport_method(M, typeof(p)))AverageGradientRule(M::AbstractManifold, p; kwargs...)

Add average to a gradient problem, where

• n: determines the size of averaging
• direction: is the internal DirectionUpdateRule to determine the gradients to store
• gradients: can be pre-filled with some history
• vector_transport_method::AbstractVectorTransportMethod=default_vector_transport_method(M, typeof(p)): a vector transport $\mathcal T_{⋅←⋅}$ to use, see the section on vector transports

IdentityUpdateRule <: DirectionUpdateRule

The default gradient direction update is the identity, usually it just evaluates the gradient.

You can also use Gradient() to create the corresponding factory, though this only delays this parameter-free instantiation to later.

MomentumGradientRule <: DirectionUpdateRule

Store the necessary information to compute the MomentumGradient direction update, see [RMH18, LS26].

Fields

• p_old::P: a point on the manifold $\mathcal{M}$
• momentum::Real: factor for the momentum
• direction: internal DirectionUpdateRule to determine directions to add the momentum to.
• vector_transport_method::AbstractVectorTransportMethod: a vector transport $\mathcal T_{⋅←⋅}$ to use, see the section on vector transports
• η_old::T: the momentum direction accumulated so far, that is the direction the rule returned in the last step. Note that this is not the last gradient.

Constructors

MomentumGradientRule(M::AbstractManifold; kwargs...)MomentumGradientRule(M::AbstractManifold, p; kwargs...)

Initialize a momentum gradient rule, where p and X are memory for interim values, the latter for the accumulated momentum direction η_old.

Keyword arguments

• p::P =rand(M): a point on the manifold $\mathcal{M}$
• direction=IdentityUpdateRule()
• momentum=0.2
• vector_transport_method::AbstractVectorTransportMethod=default_vector_transport_method(M, typeof(p)): a vector transport $\mathcal T_{⋅←⋅}$ to use, see the section on vector transports
• X::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$

See also

MomentumGradient

NesterovRule <: DirectionUpdateRule

Compute a Nesterov inspired direction update rule. See Nesterov for details

Fields

• γ::Real, μ::Real: coefficients from the last iterate
• v::P: an interim point to compute the next gradient evaluation point y_k
• shrinkage: a function k -> ... to compute the shrinkage $β_k$ per iterate k.
• inverse_retraction_method::AbstractInverseRetractionMethod: an inverse retraction $\operatorname{retr}^{-1}$ to use, see the section on retractions and their inverses
• retraction_method::AbstractRetractionMethod: a retraction $\operatorname{retr}$ to use, see the section on retractions

Constructor

NesterovRule(M::AbstractManifold; kwargs...)NesterovRule(M::AbstractManifold, p; kwargs...)

Keyword arguments

• p::P =rand(M): a point on the manifold $\mathcal{M}$ to specify the initial value
• γ=0.001
• μ=0.9
• shrinkage = k -> 0.8
• inverse_retraction_method::AbstractInverseRetractionMethod=default_inverse_retraction_method(M, typeof(p)): an inverse retraction $\operatorname{retr}^{-1}$ to use, see the section on retractions and their inverses
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions

See also

Nesterov

PreconditionedDirectionRule <: DirectionUpdateRule

Add a preconditioning as gradient processor, see PreconditionedDirection for more mathematical background.

Fields

• direction: internal DirectionUpdateRule to determine directions to apply the preconditioning to
• preconditioner: the preconditioner function

Constructors

PreconditionedDirectionRule(    M::AbstractManifold,    preconditioner;    direction::Union{<:DirectionUpdateRule,ManifoldDefaultsFactory}=IdentityUpdateRule(),    evaluation::AbstractEvaluationType=AllocatingEvaluation())

Add preconditioning to a gradient problem.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$
• preconditioner: preconditioner function, either as a (M, p, X) -> Y allocating or (M, Y, p, X) -> Y mutating function

Keyword arguments

• direction=IdentityUpdateRule internal DirectionUpdateRule to determine the gradients to store or a ManifoldDefaultsFactory generating one
• evaluation::AbstractEvaluationType=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating their result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.