Stochastic gradient descent

stochastic_gradient_descent(M, grad_f, p=rand(M); kwargs...)
stochastic_gradient_descent(M, msgo; kwargs...)
stochastic_gradient_descent!(M, grad_f, p; kwargs...)
stochastic_gradient_descent!(M, msgo, p; kwargs...)

perform a stochastic gradient descent. This can be performed in-place of p.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$
• grad_f: a gradient function, that either returns a vector of the gradients or is a vector of gradient functions
• p::P: a point on the manifold $\mathcal{M}$

alternatively to the gradient you can provide a ManifoldStochasticGradientObjective msgo, then using the cost= keyword does not have any effect since if so, the cost is already within the objective.

Keyword arguments

• callbacks::D = Dict{Symbol,Function}(): provided callback functions given either as a single function (symbol, problem, state, k) called in every hook or as a (vector of) pairs :hook => function, which are processed by process_callbacks_arg. As key you can either pass single symbol or an array of symbols to indicate a callback should be added in multiple places
• cost=missing: you can provide a cost function for example to track the function value
• direction=StochasticGradient(; p=p) add a post-processor to the direction obtained from evaluating the stochastic gradient.
• evaluation::AbstractEvaluationType=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating their result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.
• order_type=:Random: whether to use a randomly permuted sequence that is drawn anew at the start of every epoch (:FixedRandom), the sequence as given in order (:Linear), or the default :Random one, which chooses a random gradient in every step.
• stopping_criterion::StoppingCriterion=StopAfterIteration(10000)|StopWhenGradientNormLess(1.0e-9): a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize=default_stepsize(M,StochasticGradientDescentState): a functor inheriting from Stepsize to determine a step size
• order=collect(1:n): the initial permutation, where n is the number of gradients in grad_f.
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions

All other keyword arguments are passed to decorate_state! for state decorators or decorate_objective! for objective decorators, respectively.

Output

The obtained approximate minimizer $p^*$. To obtain the whole final state of the solver, see get_solver_return for details, especially the return_state= keyword.

stochastic_gradient_descent(M, grad_f, p=rand(M); kwargs...)
stochastic_gradient_descent(M, msgo; kwargs...)
stochastic_gradient_descent!(M, grad_f, p; kwargs...)
stochastic_gradient_descent!(M, msgo, p; kwargs...)

perform a stochastic gradient descent. This can be performed in-place of p.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$
• grad_f: a gradient function, that either returns a vector of the gradients or is a vector of gradient functions
• p::P: a point on the manifold $\mathcal{M}$

alternatively to the gradient you can provide a ManifoldStochasticGradientObjective msgo, then using the cost= keyword does not have any effect since if so, the cost is already within the objective.

Keyword arguments

• callbacks::D = Dict{Symbol,Function}(): provided callback functions given either as a single function (symbol, problem, state, k) called in every hook or as a (vector of) pairs :hook => function, which are processed by process_callbacks_arg. As key you can either pass single symbol or an array of symbols to indicate a callback should be added in multiple places
• cost=missing: you can provide a cost function for example to track the function value
• direction=StochasticGradient(; p=p) add a post-processor to the direction obtained from evaluating the stochastic gradient.
• evaluation::AbstractEvaluationType=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating their result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.
• order_type=:Random: whether to use a randomly permuted sequence that is drawn anew at the start of every epoch (:FixedRandom), the sequence as given in order (:Linear), or the default :Random one, which chooses a random gradient in every step.
• stopping_criterion::StoppingCriterion=StopAfterIteration(10000)|StopWhenGradientNormLess(1.0e-9): a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize=default_stepsize(M,StochasticGradientDescentState): a functor inheriting from Stepsize to determine a step size
• order=collect(1:n): the initial permutation, where n is the number of gradients in grad_f.
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions

All other keyword arguments are passed to decorate_state! for state decorators or decorate_objective! for objective decorators, respectively.

Output

The obtained approximate minimizer $p^*$. To obtain the whole final state of the solver, see get_solver_return for details, especially the return_state= keyword.

StochasticGradientDescentState <: AbstractGradientSolverState

Store the following fields for a default stochastic gradient descent algorithm, see also ManifoldStochasticGradientObjective and stochastic_gradient_descent.

Fields

• callbacks::D: provided callback functions given as a dictionary with symbols as keys
• p::P: a point on the manifold $\mathcal{M}$ storing the current iterate
• direction: a direction update to use
• stop::StoppingCriterion: a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize: a functor inheriting from Stepsize to determine a step size
• order_type: specify whether to use a randomly permuted sequence that is drawn anew at the start of every epoch (:FixedRandom), the sequence as given in order (:Linear), or the default :Random one, which chooses a random gradient in every step.
• order: stores the current permutation
• retraction_method::AbstractRetractionMethod: a retraction $\operatorname{retr}$ to use, see the section on retractions
• X::T: a tangent vector at the point $p$ on the manifold $\mathcal{M}$ storing the gradient at the current iterate

Constructor

StochasticGradientDescentState(M::AbstractManifold; kwargs...)

Create a StochasticGradientDescentState with start point p.

Keyword arguments

• callbacks::D = Dict{Symbol,Function}(): provided callback functions given as a dictionary with symbols as keys
• direction=StochasticGradientRule(M; X=zero_vector(M, p))
• order_type=:Random
• order=Int[]: specify how to store the order of indices for the next epoch
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions
• p::P =rand(M): a point on the manifold $\mathcal{M}$ to specify the initial value
• stopping_criterion::StoppingCriterion=StopAfterIteration(1000): a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize=default_stepsize(M,StochasticGradientDescentState): a functor inheriting from Stepsize to determine a step size
• X::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$ to specify the representation of a tangent vector

default_stepsize(M::AbstractManifold, ::Type{StochasticGradientDescentState})

Define the default step size computed for the StochasticGradientDescentState, which is ConstantStepsize(M).

StochasticGradient(; kwargs...)
StochasticGradient(M::AbstractManifold; kwargs...)

Keyword arguments

• X::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$
• p::P =rand(M): a point on the manifold $\mathcal{M}$ to specify the initial value

StochasticGradientRule <: AbstractGradientGroupDirectionRule

Create a functor (problem, state, k) -> (s, X) to evaluate the stochastic gradient, that is, choose a random index from the state and use the internal field for evaluation of the gradient in-place.

The default gradient processor, which just evaluates the (stochastic) gradient or a subset thereof.

Fields

• X::T: a tangent vector at the point $p$ on the manifold $\mathcal{M}$

Constructor

StochasticGradientRule(M::AbstractManifold; p=rand(M), X=zero_vector(M, p))

Initialize the stochastic gradient processor with tangent vector type of X, where both M and p are just help variables.

See also

stochastic_gradient_descent, StochasticGradient