Stochastic gradient descent

stochastic_gradient_descent(M, grad_f, p=rand(M); kwargs...)
stochastic_gradient_descent(M, msgo; kwargs...)
stochastic_gradient_descent!(M, grad_f, p; kwargs...)
stochastic_gradient_descent!(M, msgo, p; kwargs...)

perform a stochastic gradient descent. This can be performed in-place of p.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$
• grad_f: a gradient function, that either returns a vector of the gradients or is a vector of gradient functions
• p::P: a point on the manifold $\mathcal{M}$

alternatively to the gradient you can provide an ManifoldStochasticGradientObjective msgo, then using the cost= keyword does not have any effect since if so, the cost is already within the objective.

Keyword arguments

• cost=missing: you can provide a cost function for example to track the function value
• direction=StochasticGradient(zero_vector(M, p)) add a post-processor to the direction obtained from evaluating the sub-gradient.
• evaluation::AbstractEvaluationType=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating its result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.
• evaluation_order=:Random: specify whether to use a randomly permuted sequence (:FixedRandom:, a per cycle permuted sequence (:Linear) or the default :Random one.
• order_type=:RandomOrder: a type of ordering of gradient evaluations. Possible values are :RandomOrder, a :FixedPermutation, :LinearOrder
• stopping_criterion::StoppingCriterion=StopAfterIteration(1000): a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize=default_stepsize(M,StochasticGradientDescentState): a functor inheriting from Stepsize to determine a step size
• order=[1:n]: the initial permutation, where n is the number of gradients in gradF.
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions

All other keyword arguments are passed to decorate_state! for state decorators or decorate_objective! for objective decorators, respectively.

Output

The obtained approximate minimizer $p^*$. To obtain the whole final state of the solver, see get_solver_return for details, especially the return_state= keyword.

stochastic_gradient_descent(M, grad_f, p=rand(M); kwargs...)
stochastic_gradient_descent(M, msgo; kwargs...)
stochastic_gradient_descent!(M, grad_f, p; kwargs...)
stochastic_gradient_descent!(M, msgo, p; kwargs...)

perform a stochastic gradient descent. This can be performed in-place of p.

Input

• M::AbstractManifold: a Riemannian manifold $\mathcal{M}$
• grad_f: a gradient function, that either returns a vector of the gradients or is a vector of gradient functions
• p::P: a point on the manifold $\mathcal{M}$

alternatively to the gradient you can provide an ManifoldStochasticGradientObjective msgo, then using the cost= keyword does not have any effect since if so, the cost is already within the objective.

Keyword arguments

• cost=missing: you can provide a cost function for example to track the function value
• direction=StochasticGradient(zero_vector(M, p)) add a post-processor to the direction obtained from evaluating the sub-gradient.
• evaluation::AbstractEvaluationType=AllocatingEvaluation(): specify whether the functions that return an array, for example a point or a tangent vector, work by allocating its result (AllocatingEvaluation) or whether they modify their input argument to return the result therein (InplaceEvaluation). Since usually the first argument is the manifold, the modified argument is the second.
• evaluation_order=:Random: specify whether to use a randomly permuted sequence (:FixedRandom:, a per cycle permuted sequence (:Linear) or the default :Random one.
• order_type=:RandomOrder: a type of ordering of gradient evaluations. Possible values are :RandomOrder, a :FixedPermutation, :LinearOrder
• stopping_criterion::StoppingCriterion=StopAfterIteration(1000): a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize=default_stepsize(M,StochasticGradientDescentState): a functor inheriting from Stepsize to determine a step size
• order=[1:n]: the initial permutation, where n is the number of gradients in gradF.
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions

All other keyword arguments are passed to decorate_state! for state decorators or decorate_objective! for objective decorators, respectively.

Output

The obtained approximate minimizer $p^*$. To obtain the whole final state of the solver, see get_solver_return for details, especially the return_state= keyword.

StochasticGradientDescentState <: AbstractGradientDescentSolverState

Store the following fields for a default stochastic gradient descent algorithm, see also ManifoldStochasticGradientObjective and stochastic_gradient_descent.

Fields

• p::P: a point on the manifold $\mathcal{M}$ storing the current iterate
• direction: a direction update to use
• stop::StoppingCriterion: a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize: a functor inheriting from Stepsize to determine a step size
• evaluation_order: specify whether to use a randomly permuted sequence (:FixedRandom:), a per cycle permuted sequence (:Linear) or the default, a :Random sequence.
• order: stores the current permutation
• retraction_method::AbstractRetractionMethod: a retraction $\operatorname{retr}$ to use, see the section on retractions

Constructor

StochasticGradientDescentState(M::AbstractManifold; kwargs...)

Create a StochasticGradientDescentState with start point p.

Keyword arguments

• direction=StochasticGradientRule(M,zero_vector(M, p))
• order_type=:RandomOrder`
• order=Int[]: specify how to store the order of indices for the next epoche
• retraction_method::AbstractRetractionMethod=default_retraction_method(M, typeof(p)): a retraction $\operatorname{retr}$ to use, see the section on retractions
• p::P =rand(M): a point on the manifold $\mathcal{M}$ to specify the initial value
• stopping_criterion::StoppingCriterion=StopAfterIteration(1000): a functor indicating that the stopping criterion is fulfilled
• stepsize::Stepsize=default_stepsize(M,StochasticGradientDescentState): a functor inheriting from Stepsize to determine a step size
• X::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$ to specify the representation of a tangent vector

default_stepsize(M::AbstractManifold, ::Type{StochasticGradientDescentState})

Deinfe the default step size computed for the StochasticGradientDescentState, which is ConstantStepsizeM.

StochasticGradient(; kwargs...)
StochasticGradient(M::AbstractManifold; kwargs...)

Keyword arguments

• initial_gradient::T =zero_vector(M, p): a tangent vector at the point $p$ on the manifold $\mathcal{M}$
• p::P =rand(M): a point on the manifold $\mathcal{M}$ to specify the initial value

AbstractStochasticGradientDescentSolverState <: AbstractManoptSolverState

A generic type for all options related to gradient descent methods working with parts of the total gradient

StochasticGradientRule<: AbstractGradientGroupDirectionRule

Create a functor (problem, state k) -> (s,X) to evaluate the stochatsic gradient, that is chose a random index from the state and use the internal field for evaluation of the gradient in-place.

The default gradient processor, which just evaluates the (stochastic) gradient or a subset thereof.

Fields

• X::T: a tangent vector at the point $p$ on the manifold $\mathcal{M}$

Constructor

StochasticGradientRule(M::AbstractManifold; p=rand(M), X=zero_vector(M, p))

Initialize the stochastic gradient processor with tangent vector type of X, where both M and p are just help variables.

See also

stochastic_gradient_descent, [StochasticGradient])@ref)