Dissertation - HQ

A general modelling framework for larval behaviour 123
instantaneous gains (there are no gains or costs along the trajectories as
long as they satisfy the given criterion at the last time step), which are
thus defined by the function
L(θ, x, d, t) = 0 ∀ θ, x, d, t (6.1)
and a final gain equal to the amount of energy reserves of a larva when
it reaches the nursery (x = 0), and zero elsewhere
Φ(θ, x, T ) = θ · 1 {x=0} (6.2)
The function 1 {x=0} equals one when the condition is satisfied (x = 0),
zero otherwise.
Eventually, from any initial point in time (t = t i) and state (θ ti ,x ti ),
the optimisation problem can be written as the value function
V (θ ti , x ti , t i) =
max
d ti ,...,d T −1
E
TX
−1
= max
d ti ,...,d T −1
E `θ T · 1 {xT =0}
L(θ τ , x τ , d τ , τ) + Φ(θ T , x T , T )
τ=t i
´
(6.3)
!
The value function
meaning that, over all possible future decisions (d ti , . . . , d T −1), the final
energy (θ T ) is maximised but only if the larva reaches the nursery (i.e.
only if x T = 0).
6.2.2 Stochastic dynamic programming equation
Backward induction of decisions
Now that the evolution of the state is described (by transition matrices)
and that an optimisation criterion is specified (maximise energy
resources at recruitment), optimal strategies have to be found. Optimal
strategies are functions of state and time which give a sequence of
optimal decisions (d # 0 , ..., d# T −1
) for each state. They are computed by
means of the stochastic dynamic programming equation (or Bellman’s
equation) 243,244 which is the backward induction
8
V (θ, x, T ) = θ · 1 {x=0}
0
1
(1 − p)V (0, x, t + 1) +
pV (θ + ∆θ 0 , x + ∆x 0 , t + 1) ,
V (θ, x, t) = max
B
><
@ (1 − p)V (0, x, t + 1) +
C
A
pV (θ − ∆θ 1 , x − ∆x 1 , t + 1) (6.4)
0
1
(1 − p)V (0, x, t + 1) +
pV (θ + ∆θ 0 , x + ∆x 0 , t + 1) ,
d # (θ, x, t) ∈ argmax
B
@ (1 − p)V (0, x, t + 1) +
C
A
>:
pV (θ − ∆θ 1 , x − ∆x 1 , t + 1)
Computation of
the value function
from the final gain
where V (θ, x, T ) is the final gain and the first argument (i.e. the first
two lines) of the max and argmax functions is the mean gain associated