Centralized Versus Decentralized Control - A ... - Infoscience - EPFL
Centralized Versus Decentralized Control - A Solvable Stylized Model in Transportation M.-O. Hongler, O. Gallay, R. Colmorn, P. Cordes and M. Hülsmann Ecole Polytechnique Fédérale de Lausanne (EPFL), (CH) TRANSP-OR Jacobs University - Bremen, (D) Systems Management - International Logistics EURO XXIV Lisbon - July 14th 2010
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
1 / 15
Smart Parts Dynamics - A Fashionable Trend in Logistics
Smart Parts Dynamics - A Fashionable Trend in Logistics Highly complex decision issues ⇒ tendency to decentralize the management •
Huge number of control parameters
•
Feedback (i.e. non-linearity) in the underlying dynamics
•
Ubiquitous presence of randomness in the dynamics
•
...
⇓ Decisions based on limited rationality ⇒ Rigid pre-planning offers poor performance mutual interactions
⇓
self-organization
Autonomous agents might better perform than an effective central controller
⇓
goal of today’s presentation
Exhibit a solvable model showing performance of decentralized control
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
2 / 15
Stylized Model for Smart Parts Dynamics
A Simple Model for Competitive Dynamics X(t), Xk (t)) + qk (vk (t))dBk,t , X˙ k (t) = vk (t) + γk Ik (~ | {z } {z } | |{z} velocity
multi−agent interactions
k = 1, 2, ..., N.
noise sources
Multi-agent interactions: Ik (~ X(t), Xk (t)) =
Nk 1 X Ik (Xj (t)), Nk j6=k
8 0 > > > > < Ik (Xj (t)) = 1 > > > > : 0 Olivier Gallay (EPFL)
Nk := neighbourhood of agent k,
if
0 ≤ Xj (t) < Xk (t),
if
Xk (t) ≤ Xj (t) < Xk (t) + U, (U > 0),
if
Xj (t) > Xk (t) + U,
(velocity unchanged), (accelerate),
(velocity unchanged).
(U := "mutual influence" interval) Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
3 / 15
Stylized Model for Smart Parts Dynamics
A Simple Model for Competitive Dynamics - Applications
Logistics
Economy
Human Mimetism
... Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
4 / 15
Stylized Model for Smart Parts Dynamics
Homogeneous Population of Agents
i h X(t), Xk (t)) dt dXk (t) = v(t) + γI(~ | {z } := drift field Dk,v (x,t)
⇓
+
q dBk,t . | {z }
indep. White Gaussian Noise
diffusion process
Fokker - Planck diffusion equation: X ∂ ˆ ˜ 1 X ∂2 ∂ [P(~x, t)] , Dk,v(~x,t) P(~x, t) + q2 P(~x, t) = − ∂t ∂xk 2 ∂x2k k k P(~x, t) := conditional probability density
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
5 / 15
Stylized Model for Smart Parts Dynamics
Mean-Field Dynamics for Homogeneous Agents Nk ≡ N → ∞ ⇒ Mean-Field Dynamics (MFD)
⇓ trajectories point of view N 1 X I(Xj (t)) N j6=k | {z }
dynamics for a representative effective agent
probabilistic point of view
Z
≈
|x
x+U
P(x, t) dx {z }
proportion of representative agents located in [x,x+U]
proportion of velocity−active agents acting on k
⇓ Effective Fokker-Planck equation: » „Z x+U «– ff ∂ ∂ 1 ∂2 P(x, t) = − v(t) + γ P(x, t)dx P(x, t) + q2 2 [P(x, t)] , ∂t ∂x 2 ∂x x {z } | non−linear and non−local field equation
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
6 / 15
Stylized Model for Smart Parts Dynamics
Small Influence Region - Burgers’ Equation Dynamics Small values of U ⇒ Taylor expand up to 1st order in U
⇓
R x+U x
P(x, t)dx ≃ U P(x, t)
∂ ∂ 1 ∂2 P(x, t) = − {[v(t) + γ U P(x, t)] P(x, t)} + q2 2 [P(x, t)] ∂t ∂x | {z } 2 ∂x non−linear but local drift field
t 7→ τ = γt
⇓
x 7→ z =
R x− 0t v(s) ds 2U
Burgers’ Equation (to be solved with initial condition P(z, t) = δ(z)Θ(z))
˙ t) = P(z,
Olivier Gallay (EPFL)
1 ∂ 2 ∂z
P(z, t)
2
+
h
q2 8U 2 γ
Centralized Versus Decentralized Control
i
∂2 ∂z2
[P(z, t)]
EURO XXIV, 14/07/2010
7 / 15
Stylized Model for Smart Parts Dynamics
⇐
Burgers’ Eq.
logarithmic transformation (Hopf - Cole )
⇓
Heat Eq.
exact integration
" ` R ´ „ «# e −1 y q2 ∂ √ = − ln 1 + Erfc 4γU 2 ∂y 2 q t 2 3 2 ` R ´ 1 − y2 q t √ e e − 1 R 7 16 πq2 t 6 7 := 1 (e − 1)G(y, t) ” “ 5 (eR −1) R4 R E(y, t) y 1 + 2 Erfc q√ t
=
P(y, t)
⇒
=
# !)*
R +,-,")""! U =>0.0004
!)'
R +,-,")! U =>4
!)%
R U =>100 +,-,")&
R +,-,")"% U =>0.64
2 Typical shape of P(y, t) for various R := 4U 2 γ factors q
R U =>16 +,-,")# R U =>1600 +,-,#
!)#
(viewed from the relative moving frame) P (y, t) =! ")* ")'
Normalization and positivity are visually manifest !!
")% ")# " !!
"
!
#
$
%
&
'
(
y
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
8 / 15
Stylized Model for Smart Parts Dynamics
Benefit of Competition - Noise Induced Transport Enhancement 1.8
t=0 1.6 1.4
t = 20
1.2 1
P (y, t)
t = 40
0.8
t = 60 0.6 0.4 0.2 0
0
10
20
30
40
50
60
70
traveled distance y
Position probability distribution: without interaction, with interactions 4U √ γt, 3
Additional traveled distance when R =
4γU 2 q2
→ ∞: hX(t)it→∞ ≃
Additional traveled distance when R =
4γU 2 q2
→ 0: hX(t)it→∞ ≃ 0.
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
9 / 15
Average Costs Estimation
Optimal Effective Centralized Control Controlled diffusion process: dYt =
c(Y, t) dt | {z }
+ q dBt ,
Y =0 , | 0 {z }
(0 ≤ t ≤ T) ,
initial condition
effective central controller
⇓
(Fokker-Planck equation)
∂ q2 ∂ 2 ∂ Pc (y, t) = − [c(y, t)Pc (y, t)] + Pc (y, t) ∂t ∂y 2 ∂y2
Construct a drift controller c(Y, t) which, for time T, fulfills Pc (y, T) | {z }
=
Prob. density with central controller
Olivier Gallay (EPFL)
P(y, T) | {z }
Prob. density due to agent interactions Burgers’ exact solution
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
10 / 15
Average Costs Estimation
Optimal Effective Centralized Control (continued) Introduce a utility function Jcentral,T [c(y, t; T)] defined as:
Jcentral,T [c(y, t; T)] = h
Z
T 0
c2 (y, s; T) ds i, 2q2 | {z }
cost rate ρ(y,s)
(h·i := average over the realization of underlying stochastic process)
————————————————————————— Optimal Control Problem Construct an
optimal drift | {z }
c∗ (y, t; T)
such that:
i.e. yielding minimal cost
Jcentral,T [c∗ (y, t; T)] ≤ Jcentral,T [c(y, t; T)]
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
11 / 15
Average Costs Estimation
The Dai Pra Solution of the Optimal Control Problem Optimal drift controller: c∗ (y, t; T) = h(y, t) =
Z
R
∂ ∂y
ln [h(y, t)] ,
G [(z − y), (T − t)]
P(z, T) dz. G(z, t)
Paolo Dai Pra, "A Stochastic Control Approach to Reciprocal Diffusion Processes", Appl. Math. Optim. 23, (1991), 313-329.
———————————————————— Minimal cost:
Jcentral,T [c∗ (y, t; T)] =
N · |{z}
D(P|G) | {z }
♯ population Kullback−Leibler
Olivier Gallay (EPFL)
=
8 > < 0
> : N R + N ln 2
Centralized Versus Decentralized Control
h
(eR −1) R
i
for
t = 0,
for
t > 0.
EURO XXIV, 14/07/2010
12 / 15
Average Costs Estimation
Decentralized Agent Control - Cost Estimation Cost Jagents,T for decentralized evolution during time horizon T: Z T Jagents,T := N · Φ(s) , ·ρ ds |{z} |{z} 0 ♯ population
interacting agents
kinetic energy
•
z }| { γ 2 U 2 /2 ρ= := individual cost rate function, q2 |{z} diffusion rate
•
Φ(t) ∈ [0, 1] := proportion of interacting agents at time t.
******************************************************************************** Cost upper-bound, reached when Φ(t) ≡ 1
⇓ Jagents,T ≤ NρT Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
13 / 15
Average Costs Estimation
Costs Comparison - Centralized vs Decentralized
cumulative costs upper-bounded decentralized costs actual decentralized costs
Kullback-Leibler entropy
centralized costs
0
Olivier Gallay (EPFL)
T < Tc time horizons for which agent interactions beat the optimal effective centralized controller
Centralized Versus Decentralized Control
time horizon T
EURO XXIV, 14/07/2010
14 / 15
Conclusion
To Summarize and to Somehow "Philosophically" Conclude
The stylized model cartoons basic and somehow "universal" features:
•
Agents’ mimetic interactions produce an emergent structure - (here a "shock"- like wave),
•
Competition enhances global transport flow -
•
Self-organization via autonomous agents interactions can reduce costs.
Olivier Gallay (EPFL)
(here a
√
t-increase of the traveled distance),
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
15 / 15
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
1 / 15
Smart Parts Dynamics - A Fashionable Trend in Logistics
Smart Parts Dynamics - A Fashionable Trend in Logistics Highly complex decision issues ⇒ tendency to decentralize the management •
Huge number of control parameters
•
Feedback (i.e. non-linearity) in the underlying dynamics
•
Ubiquitous presence of randomness in the dynamics
•
...
⇓ Decisions based on limited rationality ⇒ Rigid pre-planning offers poor performance mutual interactions
⇓
self-organization
Autonomous agents might better perform than an effective central controller
⇓
goal of today’s presentation
Exhibit a solvable model showing performance of decentralized control
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
2 / 15
Stylized Model for Smart Parts Dynamics
A Simple Model for Competitive Dynamics X(t), Xk (t)) + qk (vk (t))dBk,t , X˙ k (t) = vk (t) + γk Ik (~ | {z } {z } | |{z} velocity
multi−agent interactions
k = 1, 2, ..., N.
noise sources
Multi-agent interactions: Ik (~ X(t), Xk (t)) =
Nk 1 X Ik (Xj (t)), Nk j6=k
8 0 > > > > < Ik (Xj (t)) = 1 > > > > : 0 Olivier Gallay (EPFL)
Nk := neighbourhood of agent k,
if
0 ≤ Xj (t) < Xk (t),
if
Xk (t) ≤ Xj (t) < Xk (t) + U, (U > 0),
if
Xj (t) > Xk (t) + U,
(velocity unchanged), (accelerate),
(velocity unchanged).
(U := "mutual influence" interval) Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
3 / 15
Stylized Model for Smart Parts Dynamics
A Simple Model for Competitive Dynamics - Applications
Logistics
Economy
Human Mimetism
... Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
4 / 15
Stylized Model for Smart Parts Dynamics
Homogeneous Population of Agents
i h X(t), Xk (t)) dt dXk (t) = v(t) + γI(~ | {z } := drift field Dk,v (x,t)
⇓
+
q dBk,t . | {z }
indep. White Gaussian Noise
diffusion process
Fokker - Planck diffusion equation: X ∂ ˆ ˜ 1 X ∂2 ∂ [P(~x, t)] , Dk,v(~x,t) P(~x, t) + q2 P(~x, t) = − ∂t ∂xk 2 ∂x2k k k P(~x, t) := conditional probability density
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
5 / 15
Stylized Model for Smart Parts Dynamics
Mean-Field Dynamics for Homogeneous Agents Nk ≡ N → ∞ ⇒ Mean-Field Dynamics (MFD)
⇓ trajectories point of view N 1 X I(Xj (t)) N j6=k | {z }
dynamics for a representative effective agent
probabilistic point of view
Z
≈
|x
x+U
P(x, t) dx {z }
proportion of representative agents located in [x,x+U]
proportion of velocity−active agents acting on k
⇓ Effective Fokker-Planck equation: » „Z x+U «– ff ∂ ∂ 1 ∂2 P(x, t) = − v(t) + γ P(x, t)dx P(x, t) + q2 2 [P(x, t)] , ∂t ∂x 2 ∂x x {z } | non−linear and non−local field equation
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
6 / 15
Stylized Model for Smart Parts Dynamics
Small Influence Region - Burgers’ Equation Dynamics Small values of U ⇒ Taylor expand up to 1st order in U
⇓
R x+U x
P(x, t)dx ≃ U P(x, t)
∂ ∂ 1 ∂2 P(x, t) = − {[v(t) + γ U P(x, t)] P(x, t)} + q2 2 [P(x, t)] ∂t ∂x | {z } 2 ∂x non−linear but local drift field
t 7→ τ = γt
⇓
x 7→ z =
R x− 0t v(s) ds 2U
Burgers’ Equation (to be solved with initial condition P(z, t) = δ(z)Θ(z))
˙ t) = P(z,
Olivier Gallay (EPFL)
1 ∂ 2 ∂z
P(z, t)
2
+
h
q2 8U 2 γ
Centralized Versus Decentralized Control
i
∂2 ∂z2
[P(z, t)]
EURO XXIV, 14/07/2010
7 / 15
Stylized Model for Smart Parts Dynamics
⇐
Burgers’ Eq.
logarithmic transformation (Hopf - Cole )
⇓
Heat Eq.
exact integration
" ` R ´ „ «# e −1 y q2 ∂ √ = − ln 1 + Erfc 4γU 2 ∂y 2 q t 2 3 2 ` R ´ 1 − y2 q t √ e e − 1 R 7 16 πq2 t 6 7 := 1 (e − 1)G(y, t) ” “ 5 (eR −1) R4 R E(y, t) y 1 + 2 Erfc q√ t
=
P(y, t)
⇒
=
# !)*
R +,-,")""! U =>0.0004
!)'
R +,-,")! U =>4
!)%
R U =>100 +,-,")&
R +,-,")"% U =>0.64
2 Typical shape of P(y, t) for various R := 4U 2 γ factors q
R U =>16 +,-,")# R U =>1600 +,-,#
!)#
(viewed from the relative moving frame) P (y, t) =! ")* ")'
Normalization and positivity are visually manifest !!
")% ")# " !!
"
!
#
$
%
&
'
(
y
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
8 / 15
Stylized Model for Smart Parts Dynamics
Benefit of Competition - Noise Induced Transport Enhancement 1.8
t=0 1.6 1.4
t = 20
1.2 1
P (y, t)
t = 40
0.8
t = 60 0.6 0.4 0.2 0
0
10
20
30
40
50
60
70
traveled distance y
Position probability distribution: without interaction, with interactions 4U √ γt, 3
Additional traveled distance when R =
4γU 2 q2
→ ∞: hX(t)it→∞ ≃
Additional traveled distance when R =
4γU 2 q2
→ 0: hX(t)it→∞ ≃ 0.
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
9 / 15
Average Costs Estimation
Optimal Effective Centralized Control Controlled diffusion process: dYt =
c(Y, t) dt | {z }
+ q dBt ,
Y =0 , | 0 {z }
(0 ≤ t ≤ T) ,
initial condition
effective central controller
⇓
(Fokker-Planck equation)
∂ q2 ∂ 2 ∂ Pc (y, t) = − [c(y, t)Pc (y, t)] + Pc (y, t) ∂t ∂y 2 ∂y2
Construct a drift controller c(Y, t) which, for time T, fulfills Pc (y, T) | {z }
=
Prob. density with central controller
Olivier Gallay (EPFL)
P(y, T) | {z }
Prob. density due to agent interactions Burgers’ exact solution
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
10 / 15
Average Costs Estimation
Optimal Effective Centralized Control (continued) Introduce a utility function Jcentral,T [c(y, t; T)] defined as:
Jcentral,T [c(y, t; T)] = h
Z
T 0
c2 (y, s; T) ds i, 2q2 | {z }
cost rate ρ(y,s)
(h·i := average over the realization of underlying stochastic process)
————————————————————————— Optimal Control Problem Construct an
optimal drift | {z }
c∗ (y, t; T)
such that:
i.e. yielding minimal cost
Jcentral,T [c∗ (y, t; T)] ≤ Jcentral,T [c(y, t; T)]
Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
11 / 15
Average Costs Estimation
The Dai Pra Solution of the Optimal Control Problem Optimal drift controller: c∗ (y, t; T) = h(y, t) =
Z
R
∂ ∂y
ln [h(y, t)] ,
G [(z − y), (T − t)]
P(z, T) dz. G(z, t)
Paolo Dai Pra, "A Stochastic Control Approach to Reciprocal Diffusion Processes", Appl. Math. Optim. 23, (1991), 313-329.
———————————————————— Minimal cost:
Jcentral,T [c∗ (y, t; T)] =
N · |{z}
D(P|G) | {z }
♯ population Kullback−Leibler
Olivier Gallay (EPFL)
=
8 > < 0
> : N R + N ln 2
Centralized Versus Decentralized Control
h
(eR −1) R
i
for
t = 0,
for
t > 0.
EURO XXIV, 14/07/2010
12 / 15
Average Costs Estimation
Decentralized Agent Control - Cost Estimation Cost Jagents,T for decentralized evolution during time horizon T: Z T Jagents,T := N · Φ(s) , ·ρ ds |{z} |{z} 0 ♯ population
interacting agents
kinetic energy
•
z }| { γ 2 U 2 /2 ρ= := individual cost rate function, q2 |{z} diffusion rate
•
Φ(t) ∈ [0, 1] := proportion of interacting agents at time t.
******************************************************************************** Cost upper-bound, reached when Φ(t) ≡ 1
⇓ Jagents,T ≤ NρT Olivier Gallay (EPFL)
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
13 / 15
Average Costs Estimation
Costs Comparison - Centralized vs Decentralized
cumulative costs upper-bounded decentralized costs actual decentralized costs
Kullback-Leibler entropy
centralized costs
0
Olivier Gallay (EPFL)
T < Tc time horizons for which agent interactions beat the optimal effective centralized controller
Centralized Versus Decentralized Control
time horizon T
EURO XXIV, 14/07/2010
14 / 15
Conclusion
To Summarize and to Somehow "Philosophically" Conclude
The stylized model cartoons basic and somehow "universal" features:
•
Agents’ mimetic interactions produce an emergent structure - (here a "shock"- like wave),
•
Competition enhances global transport flow -
•
Self-organization via autonomous agents interactions can reduce costs.
Olivier Gallay (EPFL)
(here a
√
t-increase of the traveled distance),
Centralized Versus Decentralized Control
EURO XXIV, 14/07/2010
15 / 15
