GitHub

How to record data during the iterations

Ronny Bergmann

The recording and debugging features make it possible to record nearly any data during the iterations. This tutorial illustrates how to:

  • record one value during the iterations;
  • record multiple values during the iterations and access them afterwards;
  • record within a sub solver;
  • define an own RecordAction to perform individual recordings.

Several predefined recordings exist, for example RecordCost or RecordGradient, if the problem the solver uses provides a gradient. For fields of the State the recording can also be done using RecordEntry. For other recordings, for example more advanced computations before storing a value, an own RecordAction can be defined.

We illustrate these using the gradient descent from the 🏔️ Get started with Manopt.jl tutorial.

Here the focus is put on ways to investigate the behavior during iterations by using recording techniques.

Let’s first load the necessary packages.

using Manopt, Manifolds, Random, ManifoldDiff, LinearAlgebrausing ManifoldDiff: grad_distanceRandom.seed!(42);

The objective

We generate data and define our cost and gradient:

Random.seed!(42)m = 30M = Sphere(m)n = 800σ = π / 8p = zeros(Float64, m + 1)p[2] = 1.0data = [exp(M, p, σ * rand(M; vector_at = p)) for i in 1:n]f(M, p) = sum(1 / (2 * n) * distance.(Ref(M), Ref(p), data) .^ 2)grad_f(M, p) = sum(1 / n * grad_distance.(Ref(M), data, Ref(p)))
grad_f (generic function with 1 method)

First examples

For the high-level interfaces of the solvers, like gradient_descent we have to set return_state to true to obtain the whole solver state and not only the resulting minimizer.

Then we can easily use the record= option to add recorded values. This keyword accepts RecordActions as well as several symbols as shortcuts, for example :Cost to record the cost, or if your state has a field f, :f would record that entry. An overview of the symbols that can be used is given here.

We first just record the cost after every iteration

R = gradient_descent(M, f, grad_f, data[1]; record = :Cost, return_state = true)
# Solver state for `Manopt.jl`s Gradient Descent
After 58 iterations

## Parameters
* direction: A gradient processor that evaluates the gradient
* retraction method: StabilizedRetraction()

## Stepsize
  Armijo backtracking line search
  A line search based on sufficient decrease backtracking (last stepsize: 2.517532610308425)

  ### Parameters
  * contraction_factor:    0.95
  * initial guess:         Manopt.ArmijoInitialGuess()
  * initial stepsize:      1.0
  * retraction method:     StabilizedRetraction()
  * sufficient decrease:   0.1


## Stopping criterion
  Stop when _one_ of the following is fulfilled:
    *   stopped after 200 iterations:  not reached
    *   |grad f| < 1.0e-8:  reached
  Overall: reached
The algorithm converged: Yes

## Record
(Iteration = RecordCost(),)

From the returned state, we see that the GradientDescentState is encapsulated (decorated) within a RecordSolverState.

For such a state, one can attach different recorders to some operations, currently to :Start, :Stop, and :Iteration, where :Iteration is the default when using the record= keyword with a RecordAction or a Symbol as we just did. We can access all values recorded during the iterations by calling get_record(R, :Iteration) or since this is the default even shorter

get_record(R)
58-element Vector{Float64}:
 0.6870172325261713
 0.6239221496686211
 0.5900244338953805
 0.569312079535616
 0.5518048258655455
 0.5429045359832493
 0.5383847696671531
 0.5360322830268693
 0.5348144739486791
 0.5341773307679921
 ⋮
 0.5334801024530127
 0.5334801024530104
 0.5334801024530084
 0.5334801024530073
 0.5334801024530067
 0.5334801024530064
 0.5334801024530063
 0.533480102453006
 0.533480102453006

To record more than one value, you can pass an array of a mix of symbols and RecordActions which formally introduces RecordGroup. Such a group records a tuple of values in every iteration:

R2 = gradient_descent(M, f, grad_f, data[1]; record = [:Iteration, :Cost], return_state = true)
# Solver state for `Manopt.jl`s Gradient Descent
After 58 iterations

## Parameters
* direction: A gradient processor that evaluates the gradient
* retraction method: StabilizedRetraction()

## Stepsize
  Armijo backtracking line search
  A line search based on sufficient decrease backtracking (last stepsize: 2.517532610308425)

  ### Parameters
  * contraction_factor:    0.95
  * initial guess:         Manopt.ArmijoInitialGuess()
  * initial stepsize:      1.0
  * retraction method:     StabilizedRetraction()
  * sufficient decrease:   0.1


## Stopping criterion
  Stop when _one_ of the following is fulfilled:
    *   stopped after 200 iterations:  not reached
    *   |grad f| < 1.0e-8:  reached
  Overall: reached
The algorithm converged: Yes

## Record
(Iteration = RecordGroup([RecordIteration(), RecordCost()]),)

Here, the symbol :Cost is mapped to using the RecordCost action. The same holds for :Iteration, which obviously records the current iteration number k. To access these you can first extract the group of records (that is where the :Iterations are recorded; note the plural) and then access the :Cost

get_record_action(R2, :Iteration)
A group of 2 RecordActions:
* A RecordAction to record the current iteration number
* A RecordAction to record the cost value

Since :Iteration is the default, we can also omit it here again. To access single recorded values, one can use

get_record_action(R2)[:Cost]
58-element Vector{Float64}:
 0.6870172325261713
 0.6239221496686211
 0.5900244338953805
 0.569312079535616
 0.5518048258655455
 0.5429045359832493
 0.5383847696671531
 0.5360322830268693
 0.5348144739486791
 0.5341773307679921
 ⋮
 0.5334801024530127
 0.5334801024530104
 0.5334801024530084
 0.5334801024530073
 0.5334801024530067
 0.5334801024530064
 0.5334801024530063
 0.533480102453006
 0.533480102453006

This can also be done by using the high-level interface get_record

get_record(R2, :Iteration, :Cost)
58-element Vector{Float64}:
 0.6870172325261713
 0.6239221496686211
 0.5900244338953805
 0.569312079535616
 0.5518048258655455
 0.5429045359832493
 0.5383847696671531
 0.5360322830268693
 0.5348144739486791
 0.5341773307679921
 ⋮
 0.5334801024530127
 0.5334801024530104
 0.5334801024530084
 0.5334801024530073
 0.5334801024530067
 0.5334801024530064
 0.5334801024530063
 0.533480102453006
 0.533480102453006

Note that the first symbol again refers to the point where we record (not to the thing we record). We can also pass a tuple as second argument to have our own order within the tuples returned. Switching the order of recorded cost and Iteration can be done using

get_record(R2, :Iteration, (:Cost, :Iteration))
58-element Vector{Tuple{Float64, Int64}}:
 (0.6870172325261713, 1)
 (0.6239221496686211, 2)
 (0.5900244338953805, 3)
 (0.569312079535616, 4)
 (0.5518048258655455, 5)
 (0.5429045359832493, 6)
 (0.5383847696671531, 7)
 (0.5360322830268693, 8)
 (0.5348144739486791, 9)
 (0.5341773307679921, 10)
 ⋮
 (0.5334801024530127, 50)
 (0.5334801024530104, 51)
 (0.5334801024530084, 52)
 (0.5334801024530073, 53)
 (0.5334801024530067, 54)
 (0.5334801024530064, 55)
 (0.5334801024530063, 56)
 (0.533480102453006, 57)
 (0.533480102453006, 58)

A more complex example

To illustrate a complicated example let’s record:

  • the iteration number, cost and gradient field, but only every sixth iteration;
  • the iteration at which we stop.

We first generate the problem and the state, to also illustrate how the low-level works when not using the high-level interface gradient_descent.

dmp = DefaultManoptProblem(M, ManifoldGradientObjective(f, grad_f))s = GradientDescentState(    M;    p = copy(data[1]),    stopping_criterion = StopAfterIteration(200) | StopWhenGradientNormLess(10.0^-9),)
# Solver state for `Manopt.jl`s Gradient Descent

## Parameters
* direction: A gradient processor that evaluates the gradient
* retraction method: StabilizedRetraction()

## Stepsize
  Armijo backtracking line search
  A line search based on sufficient decrease backtracking (last stepsize: 1.0)

  ### Parameters
  * contraction_factor:    0.95
  * initial guess:         Manopt.ArmijoInitialGuess()
  * initial stepsize:      1.0
  * retraction method:     StabilizedRetraction()
  * sufficient decrease:   0.1


## Stopping criterion
  Stop when _one_ of the following is fulfilled:
    *   stopped after 200 iterations:  not reached
    *   |grad f| < 1.0e-9:  not reached
  Overall: not reached
The algorithm converged: No

We now first build a RecordGroup to group the three entries we want to record per iteration. We then put this into a RecordEvery to only record this every sixth iteration

rI = RecordEvery(    RecordGroup(        [            RecordIteration() => :Iteration,            RecordCost() => :Cost,            RecordEntry(similar(data[1]), :X) => :Gradient,        ]    ),    6,)
A RecordAction that records every 6th iteration with
  A group of 3 RecordActions:
  * A RecordAction to record the current iteration number
  * A RecordAction to record the cost value
  * A RecordAction to record the solver state field :X

where the notation as a pair with the symbol can be read as “Is accessible by”. The record= keyword with the symbol :Iteration is actually the same as we specified here for the first group entry. For recording the final iteration number

sI = RecordIteration()
A RecordAction to record the current iteration number

We now combine both into the RecordSolverState decorator. It acts completely the same as any AbstractManoptSolverState but records something in every iteration additionally. This is stored in a dictionary of RecordActions, where :Iteration is the key for the group rI (recording only every sixth iteration) and :Stop the key for sI, which is executed at stop.

Note that the keyword record= in the high-level interface gradient_descent only would fill the :Iteration symbol of said dictionary, but we could also pass pairs like in the following, that is in the form Symbol => RecordAction into that keyword to obtain the same as in

r = RecordSolverState(s, Dict(:Iteration => rI, :Stop => sI))
# Solver state for `Manopt.jl`s Gradient Descent

## Parameters
* direction: A gradient processor that evaluates the gradient
* retraction method: StabilizedRetraction()

## Stepsize
  Armijo backtracking line search
  A line search based on sufficient decrease backtracking (last stepsize: 1.0)

  ### Parameters
  * contraction_factor:    0.95
  * initial guess:         Manopt.ArmijoInitialGuess()
  * initial stepsize:      1.0
  * retraction method:     StabilizedRetraction()
  * sufficient decrease:   0.1


## Stopping criterion
  Stop when _one_ of the following is fulfilled:
    *   stopped after 200 iterations:  not reached
    *   |grad f| < 1.0e-9:  not reached
  Overall: not reached
The algorithm converged: No

## Record
(Iteration = RecordEvery(RecordGroup([RecordIteration(), RecordCost(), RecordEntry(:X)]), 6, true), Stop = RecordIteration())

We now call the solver

res = solve!(dmp, r)
# Solver state for `Manopt.jl`s Gradient Descent
After 62 iterations

## Parameters
* direction: A gradient processor that evaluates the gradient
* retraction method: StabilizedRetraction()

## Stepsize
  Armijo backtracking line search
  A line search based on sufficient decrease backtracking (last stepsize: 2.3916559797930033)

  ### Parameters
  * contraction_factor:    0.95
  * initial guess:         Manopt.ArmijoInitialGuess()
  * initial stepsize:      1.0
  * retraction method:     StabilizedRetraction()
  * sufficient decrease:   0.1


## Stopping criterion
  Stop when _one_ of the following is fulfilled:
    *   stopped after 200 iterations:  not reached
    *   |grad f| < 1.0e-9:  reached
  Overall: reached
The algorithm converged: Yes

## Record
(Iteration = RecordEvery(RecordGroup([RecordIteration(), RecordCost(), RecordEntry(:X)]), 6, true), Stop = RecordIteration())

And we can look at the recorded value at :Stop to see how many iterations were performed

get_record(res, :Stop)
1-element Vector{Int64}:
 62

and the other values during the iterations are

get_record(res, :Iteration, (:Iteration, :Cost))
10-element Vector{Tuple{Int64, Float64}}:
 (6, 0.5429045359832493)
 (12, 0.5336712822308556)
 (18, 0.533484098624334)
 (24, 0.5334801877032025)
 (30, 0.533480104312984)
 (36, 0.5334801024945819)
 (42, 0.5334801024539587)
 (48, 0.5334801024530285)
 (54, 0.5334801024530067)
 (60, 0.5334801024530059)

where the last tuple contains the names from the pairs when we generated the record group. So similarly we can use :Gradient as specified before to access the recorded gradient.

Recording from a sub solver

One can also record from a sub solver. For that we need a problem that actually requires a sub solver. We take the constraint example from the How to print debug tutorial. Maybe read that part for more details on the problem

d = 4M2 = Sphere(d - 1)v0 = project(M2, [ones(2)..., zeros(d - 2)...])Z = v0 * v0'#Cost and gradientf2(M, p) = -tr(transpose(p) * Z * p) / 2grad_f2(M, p) = project(M, p, -transpose(Z) * p / 2 - Z * p / 2)# Constraintsg(M, p) = -p # now p ≥ 0mI = -Matrix{Float64}(I, d, d)# Vector of gradients of the constraint componentsgrad_g(M, p) = [project(M, p, mI[:, i]) for i in 1:d]p0 = project(M2, [ones(2)..., zeros(d - 3)..., 0.1])

We directly start with recording the sub solver’s iteration. We can specify what to record in the subsolver using the sub_kwargs keyword argument with a Symbol => value pair. Here we specify to record the iteration and the cost in every sub solver’s step.

Furthermore, we have to “collect” this recording after every sub solver run. This is done with the :Subsolver keyword in the main record= keyword.

s1 = exact_penalty_method(    M2,    f2,    grad_f2,    p0;    g = g,    grad_g = grad_g,    record = [:Iteration, :Cost, :Subsolver],    sub_kwargs = [:record => [:Iteration, :Cost]],    return_state = true,);

Then the first entry of the record contains the iteration k, the (main solver’s) cost, and the third entry is the recording of the subsolver.

get_record(s1)[1]
(1, -0.4733019623455377, [(1, -0.4288382393589549), (2, -0.43669534259556914), (3, -0.4374036673499917), (4, -0.4374408718086294)])

When adding a number to not record on every iteration, the :Subsolver keyword of course still also only “copies over” the subsolver recordings when active. But one could avoid allocations on the other runs. This is done by specifying the sub solver as :WhenActive

s2 = exact_penalty_method(    M2, f2, grad_f2, p0;    g = g, grad_g = grad_g,    record = [:Iteration, :Cost, :Subsolver, 25],    sub_kwargs = [:record => [:Iteration, :Cost, :WhenActive]],    return_state = true,);

Then

get_record(s2)
4-element Vector{Tuple{Int64, Float64, Vector{Tuple{Int64, Float64}}}}:
 (25, -0.4994494108530985, [(1, -0.4991469152295235)])
 (50, -0.49999564261147317, [(1, -0.49999366842932896)])
 (75, -0.49999997420136083, [(1, -0.4999999614701454)])
 (100, -0.49999999983370474, [(1, -0.49999999981081683)])

Finally, instead of recording iterations, we can also specify to record the stopping criterion and final cost by adding that to :Stop of the sub solver’s record. Then we can specify, as usual in a tuple, that the :Subsolver should record :Stop (by default it takes over :Iteration)

s3 = exact_penalty_method(    M2,    f2,    grad_f2,    p0;    g = g,    grad_g = grad_g,    record = [:Iteration, :Cost, (:Subsolver, :Stop), 25],    sub_kwargs = [:record => [:Stop => [:Stop, :Cost]]],    return_state = true,);

Then the following displays also the reasons why each of the recorded sub solvers stopped and the corresponding cost

get_record(s3)
4-element Vector{Tuple{Int64, Float64, Vector{Tuple{String, Float64}}}}:
 (25, -0.4994494108530985, [("The algorithm reached approximately critical point after 1 iterations; the gradient norm (0.00031307624887101047) is less than 0.001.\n", -0.4991469152295235)])
 (50, -0.49999564261147317, [("The algorithm reached approximately critical point after 1 iterations; the gradient norm (0.0009767910400237663) is less than 0.001.\n", -0.49999366842932896)])
 (75, -0.49999997420136083, [("The algorithm reached approximately critical point after 1 iterations; the gradient norm (0.00022396291196612712) is less than 0.001.\n", -0.4999999614701454)])
 (100, -0.49999999983370474, [("The algorithm reached approximately critical point after 1 iterations; the gradient norm (3.8129640908105984e-6) is less than 0.001.\n", -0.49999999981081683)])

Writing own RecordActions

Let’s investigate where we want to count the number of function evaluations, again just to illustrate, since for the gradient this is just one evaluation per iteration. We first define a cost, that counts its own calls.

mutable struct MyCost{T}    data::T    count::IntendMyCost(data::T) where {T} = MyCost{T}(data, 0)function (c::MyCost)(M, p)    c.count += 1    return sum(1 / (2 * length(c.data)) * distance.(Ref(M), Ref(p), c.data) .^ 2)end

and we define an own, new RecordAction, which is a functor, that is a struct that is also a function. The function we have to implement is similar to a single solver step in signature, since it might get called every iteration:

mutable struct RecordCount <: RecordAction    recorded_values::Vector{Int}    RecordCount() = new(Vector{Int}())endfunction (r::RecordCount)(amp::AbstractManoptProblem, ::AbstractManoptSolverState, k)    return if k > 0        push!(r.recorded_values, Manopt.get_cost_function(get_objective(amp)).count)    elseif k < 0 # reset if negative        r.recorded_values = Vector{Int}()    endend

Now we can initialize the new cost and call the gradient descent. Note that this also illustrates the last use case since you can pass symbol-action pairs into the record= array.

f3 = MyCost(data)

Now for the plain gradient descent, we have to modify the step to a constant step size. We also only look at the first 20 iterations to keep this example small in recorded values. We call

R3 = gradient_descent(    M, f3, grad_f, data[1];    record = [        :Iteration => [            :Iteration, RecordCount() => :Count, :Cost,        ],    ],    stepsize = ConstantLength(1.0),    stopping_criterion = StopAfterIteration(20),    return_state = true,)
# Solver state for `Manopt.jl`s Gradient Descent
After 20 iterations

## Parameters
* direction: A gradient processor that evaluates the gradient
* retraction method: StabilizedRetraction()

## Stepsize
  A relative constant step size of length 1.0

## Stopping criterion
  A stopping criterion to stop after 20 iterations
    reached
The algorithm converged: No

## Record
(Iteration = RecordGroup([RecordIteration(), Main.Notebook.RecordCount([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]), RecordCost()]),)

For :Cost we already learned how to access them, the => :Count introduces an action to obtain the :Count symbol as its access. We can again access the whole sets of records

get_record(R3)
20-element Vector{Tuple{Int64, Int64, Float64}}:
 (1, 1, 0.5823814423113638)
 (2, 2, 0.5408049802340039)
 (3, 3, 0.5345550944722897)
 (4, 4, 0.5336375289938885)
 (5, 5, 0.5335031591890169)
 (6, 6, 0.5334834802310251)
 (7, 7, 0.5334805973984544)
 (8, 8, 0.5334801749902925)
 (9, 9, 0.5334801130855076)
 (10, 10, 0.5334801040117542)
 (11, 11, 0.5334801026815555)
 (12, 12, 0.5334801024865217)
 (13, 13, 0.5334801024579214)
 (14, 14, 0.5334801024537271)
 (15, 15, 0.5334801024531117)
 (16, 16, 0.5334801024530216)
 (17, 17, 0.5334801024530085)
 (18, 18, 0.5334801024530064)
 (19, 19, 0.5334801024530063)
 (20, 20, 0.5334801024530059)

this is equivalent to calling R3[:Iteration]. Note that since we introduced :Count we can also access a single recorded value using

R3[:Iteration, :Count]
20-element Vector{Int64}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20

and we see that the cost function is called once per iteration.

If we use this counting cost and run the default gradient descent with Armijo line search, we can infer how many Armijo line search backtracks are performed:

f4 = MyCost(data)
MyCost{Vector{Vector{Float64}}}([[-0.054658825167894595, -0.5592077846510423, -0.04738273828111257, -0.04682080720921302, 0.12279468849667038, 0.07171438895366239, -0.12930045409417057, -0.22102081626380404, -0.31805333254577767, 0.0065859500152017645  …  -0.21999168261518043, 0.19570142227077295, 0.340909965798364, -0.0310802190082894, -0.04674431076254687, -0.006088297671169996, 0.01576037011323387, -0.14523596850249543, 0.14526158060820338, 0.1972125856685378], [-0.08192376929745249, -0.5097715132187676, -0.008339904915541005, 0.07289741328038676, 0.11422036270613797, -0.11546739299835748, 0.2296996932628472, 0.1490467170835958, -0.11124820565850364, -0.11790721606521781  …  -0.16421249630470344, -0.2450575844467715, -0.07570080850379841, -0.07426218324072491, -0.026520181327346338, 0.11555341205250205, -0.0292955762365121, -0.09012096853677576, -0.23470556634911574, -0.026214242996704013], [-0.22951484264859257, -0.6083825348640186, 0.14273766477054015, -0.11947823367023377, 0.05984293499234536, 0.058820835498203126, 0.07577331705863266, 0.1632847202946857, 0.20244385489915745, 0.04389826920203656  …  0.3222365119325929, 0.009728730325524067, -0.12094785371632395, -0.36322323926212824, -0.0689253407939657, 0.23356953371702974, 0.23489531397909744, 0.078303336494718, -0.14272984135578806, 0.07844539956202407], [-0.0012588500237817606, -0.29958740415089763, 0.036738459489123514, 0.20567651907595125, -0.1131046432541904, -0.06032435985370224, 0.3366633723165895, -0.1694687746143405, -0.001987171245125281, 0.04933779858684409  …  -0.2399584473006256, 0.19889267065775063, 0.22468755918787048, 0.1780090580180643, 0.023703860700539356, -0.10212737517121755, 0.03807004103115319, -0.20569120952458983, -0.03257704254233959, 0.06925473452536687], [-0.035534309946938375, -0.06645560787329002, 0.14823972268208874, -0.23913346587232426, 0.038347027875883496, 0.10453333143286662, 0.050933995140290705, -0.12319549375687473, 0.12956684644537844, -0.23540367869989412  …  -0.41471772859912864, -0.1418984610380257, 0.0038321446836859334, 0.23655566917750157, -0.17500681300994742, -0.039189751036839374, -0.08687860620942896, -0.11509948162959047, 0.11378233994840942, 0.38739450723013735], [-0.3122539912469438, -0.3101935557860296, 0.1733113629107006, 0.08968593616209351, -0.1836344261367962, -0.06480023695256802, 0.18165070013886545, 0.19618275767992124, -0.07956460275570058, 0.0325997354656551  …  0.2845492418767769, 0.17406455870721682, -0.053101230371568706, -0.1382082812981627, 0.005830071475508364, 0.16739264037923055, 0.034365814374995335, 0.09107702398753297, -0.1877250428700409, 0.05116494897806923], [-0.04159442361185588, -0.7768029783272633, 0.06303616666722486, 0.08070518925253539, -0.07396265237309446, -0.06008109299719321, 0.07977141629715745, 0.019511027129056415, 0.08629917589924847, -0.11156298867318722  …  0.0792587504128044, -0.016444383900170008, -0.181746064577005, -0.01888129512990984, -0.13523922089388968, 0.11358102175659832, 0.07929049608459493, 0.1689565359083833, 0.07673657951723721, -0.1128480905648813], [-0.21221814304651335, -0.5031823821503253, 0.010326342133992458, -0.12438192100961257, 0.04004758695231872, 0.2280527500843805, -0.2096243232022162, -0.16564828762420294, -0.28325749481138984, 0.17033534605245823  …  -0.13599096505924074, 0.28437770540525625, 0.08424426798544583, -0.1266207606984139, 0.04917635557603396, -0.00012608938533809706, -0.04283220254770056, -0.08771365647566572, 0.14750169103093985, 0.11601120086036351], [0.10683290707435536, -0.17680836277740156, 0.23767458301899405, 0.12011180867097299, -0.029404774462600154, 0.11522028383799933, -0.3318174480974519, -0.17859266746938374, 0.04352373642537759, 0.2530382802667988  …  0.08879861736692073, -0.004412506987801729, 0.19786810509925895, -0.1397104682727044, 0.09482328498485094, 0.05108149065160893, -0.14578343506951633, 0.3167479772660438, 0.10422673169182732, 0.21573150015891313], [-0.024895624707466164, -0.7473912016432697, -0.1392537238944721, -0.14948896791465557, -0.09765393283580377, 0.04413059403279867, -0.13865379004720355, -0.071032040283992, 0.15604054722246585, -0.10744260463413555  …  -0.14748067081342833, -0.14743635071251024, 0.0643591937981352, 0.16138827697852615, -0.12656652133603935, -0.06463635704869083, 0.14329582429103488, -0.01113113793821713, 0.29295387893749997, 0.06774523575259782]  …  [0.011874845316569967, -0.6910596618389588, 0.21275741439477827, -0.014042545524367437, -0.07883613103495014, -0.0021900966696246776, -0.033836430464220496, 0.2925813113264835, -0.04718187201980008, 0.03949680289730036  …  0.0867736586603294, 0.0404682510051544, -0.24779813848587257, -0.28631514602877145, -0.07211767532456789, -0.15072898498180473, 0.017855923621826746, -0.09795357710255254, -0.14755229203084924, 0.1305005778855436], [0.013457629515450426, -0.3750353654626534, 0.12349883726772073, 0.3521803555005319, 0.2475921439420274, 0.006088649842999206, 0.31203183112392907, -0.036869203979483754, -0.07475746464056504, -0.029297797064479717  …  0.16867368684091563, -0.09450564983271922, -0.0587273302122711, -0.1326667940553803, -0.25530237980444614, 0.37556905374043376, 0.04922612067677609, 0.2605362549983866, -0.21871556587505667, -0.22915883767386164], [0.03295085436260177, -0.971861604433394, 0.034748713521512035, -0.0494065013245799, -0.01767479281403355, 0.0465459739459587, 0.007470494722096038, 0.003227960072276129, 0.0058328596338402365, -0.037591237446692356  …  0.03205152122876297, 0.11331109854742015, 0.03044900529526686, 0.017971704993311105, -0.009329252062960229, -0.02939354719650879, 0.022088835776251863, -0.02546111553658854, -0.0026257225461427582, 0.005702111697172774], [0.06968243992532257, -0.7119502191435176, -0.18136614593117445, -0.1695926215673451, 0.01725015359973796, -0.00694164951158388, -0.34621134287344574, 0.024709256792651912, -0.1632255805999673, -0.2158226433583082  …  -0.14153772108081458, -0.11256850346909901, 0.045109821764180706, -0.1162754336222613, -0.13221711766357983, 0.005365354776191061, 0.012750671705879105, -0.018208207549835407, 0.12458753932455452, -0.31843587960340897], [-0.19830349374441875, -0.6086693423968884, 0.08552341811170468, 0.35781519334042255, 0.15790663648524367, 0.02712571268324985, 0.09855601327331667, -0.05840653973421127, -0.09546429767790429, -0.13414717696055448  …  -0.0430935804718714, 0.2678584478951765, 0.08780994289014614, 0.01613469379498457, 0.0516187906322884, -0.07383067566731401, -0.1481272738354552, -0.010532317187265649, 0.06555344745952187, -0.1506167863762911], [-0.04347524125197773, -0.6327981074196994, -0.221116680035191, 0.0282207467940456, -0.0855024881522933, 0.12821801740178346, 0.1779499563280024, -0.10247384887512365, 0.0396432464100116, -0.0582580338112627  …  0.1253893207083573, 0.09628202269764763, 0.3165295473947355, -0.14915034201394833, -0.1376727867817772, -0.004153096613530293, 0.09277957650773738, 0.05917264554031624, -0.12230262590034507, -0.19655728521529914], [-0.10173946348675116, -0.6475660153977272, 0.1260284619729566, -0.11933160462857616, -0.04774310633937567, 0.09093928358804217, 0.041662676324043114, -0.1264739543938265, 0.09605293126911392, -0.16790474428001648  …  -0.04056684573478108, 0.09351665120940456, 0.15259195558799882, 0.0009949298312580497, 0.09461980828206303, 0.3067004514287283, 0.16129258773733715, -0.18893664085007542, -0.1806865244492513, 0.029319680436405825], [-0.251780954320053, -0.39147463259941456, -0.24359579328578626, 0.30179309757665723, 0.21658893985206484, 0.12304585275893232, 0.28281133086451704, 0.029187615341955325, 0.03616243507191924, 0.029375588909979152  …  -0.08071746662465404, -0.2176101928258658, 0.20944684921170825, 0.043033273425352715, -0.040505542460853576, 0.17935596149079197, -0.08454569418519972, 0.0545941597033932, 0.12471741052450099, -0.24314124407858329], [0.28156471341150974, -0.6708572780452595, -0.1410302363738465, -0.08322589397277698, -0.022772599832907418, -0.04447265789199677, -0.016448068022011157, -0.07490911512503738, 0.2778432295769144, -0.10191899088372378  …  -0.057272155080983836, 0.12817478092201395, 0.04623814480781884, -0.12184190164369117, 0.1987855635987229, -0.14533603246124993, -0.16334072868597016, -0.052369977381939437, 0.014904286931394959, -0.2440882678882144], [0.12108727495744157, -0.714787344982596, 0.01632521838262752, 0.04437570556908449, -0.041199280304144284, 0.052984488452616, 0.03796520200156107, 0.2791785910964288, 0.11530429924056099, 0.12178223160398421  …  -0.07621847481721669, 0.18353870423743013, -0.19066653731436745, -0.09423224997242206, 0.14596847781388494, -0.09747986927777111, 0.16041150122587072, -0.02296513951256738, 0.06786878373578588, 0.15296635978447756]], 0)

To not get too many entries let’s just look at the first 20 iterations again

R4 = gradient_descent(    M, f4, grad_f, data[1];    record = [RecordCount()],    stopping_criterion = StopAfterIteration(20),    return_state = true,)
# Solver state for `Manopt.jl`s Gradient Descent
After 20 iterations

## Parameters
* direction: A gradient processor that evaluates the gradient
* retraction method: StabilizedRetraction()

## Stepsize
  Armijo backtracking line search
  A line search based on sufficient decrease backtracking (last stepsize: 2.7895098175162607)

  ### Parameters
  * contraction_factor:    0.95
  * initial guess:         Manopt.ArmijoInitialGuess()
  * initial stepsize:      1.0
  * retraction method:     StabilizedRetraction()
  * sufficient decrease:   0.1


## Stopping criterion
  A stopping criterion to stop after 20 iterations
    reached
The algorithm converged: No

## Record
(Iteration = Main.Notebook.RecordCount([25, 29, 33, 37, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84, 88, 92, 96, 100]),)
get_record(R4)
20-element Vector{Int64}:
  25
  29
  33
  37
  40
  44
  48
  52
  56
  60
  64
  68
  72
  76
  80
  84
  88
  92
  96
 100

We can see that the number of cost function calls varies, depending on how many line search backtrack steps were required to obtain a good stepsize.

Technical Details

This tutorial is cached. It was last run on the following package versions.

Status `~/work/Manopt.jl/Manopt.jl/tutorials/Project.toml`
  [47edcb42] ADTypes v1.24.0
  [6e4b80f9] BenchmarkTools v1.8.0
  [5ae59095] Colors v0.13.1
  [a0c0ee7d] DifferentiationInterface v0.7.21
  [31c24e10] Distributions v0.25.131
  [26cc04aa] FiniteDifferences v0.12.34
  [f6369f11] ForwardDiff v1.4.5
  [8ac3fa9e] LRUCache v1.6.2
  [af67fdf4] ManifoldDiff v0.4.5
  [1cead3c2] Manifolds v0.11.29
  [3362f125] ManifoldsBase v2.5.1
  [0fc0a36d] Manopt v0.6.7 `.`
  [91a5bcdd] Plots v1.41.7
  [731186ca] RecursiveArrayTools v4.5.1
  [37e2e46d] LinearAlgebra v1.12.0
  [9a3f8284] Random v1.11.0

This tutorial was last rendered September 9, 2026, 5:26:55.