infinitesimal perturbation analysis (ipa)
THE PERTURBATION ANALYSIS STORY: 1979 - 2017 C. G. Cassandras Division of Systems Engineering Dept. of Electrical and Computer Engineering Center for Information and Systems Engineering Boston University
https://christosgcassandras.org Christos G. Cassandras
CODES Lab. - Boston University
OUTLINE ▪ 1979: How a “real problem” gives birth to … an academic discipline … a research field … a toolbox for practical problem solutions ▪ 1979-1999: Development chronology in Perturbation Analysis (PA) ▪ 2002: Re-inventing Infinitesimal Perturbation Analysis (IPA) for Hybrid Systems ▪ 2017: The IPA Calculus, Event-Driven control theory, Data-driven methods come of age Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM (circa 1978)
What is the best way to distribute BUFFER capacity in a manufacturing transfer line?
PARTS IN
SERIAL OPERATIONS
Ho, Eyler, Chien, Cassandras, Cao,…
PRODUCTS OUT
BUFFERS Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM (circa 1978)
What is the best way to distribute BUFFER capacity in a manufacturing transfer line?
PARTS IN
PRODUCTS 0UT
SLOW…
PRETTY FAST…
BUFFERS Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM (circa 1978)
Complexity of this buffer allocation process (K buffers, N stages)
K N 1 K
Example: K = 24, N = 6 → 118,755 possible allocations • “Brute Force” trial-and-error: test each allocation for about a week to get statistically meaningful results (that’s if the manager allows you to mess with the system…) → about 2300 years… • Suppose you can reduce to only 1000 “promising” allocations: → about 19 years… • Using a simulated transfer line, about 3 minutes per trial (1980s computing technology…) → about 250 days… Christos G. Cassandras
CODES Lab. - Boston University
WHY IS THIS PROBLEM IMPORTANT ? Manufacturing system with N sequential operations: l(t) …
…
…
…
THROUGHPUT
DELAY
THROUGHPUT increases (GOOD) Christos G. Cassandras
average DELAY increases (BAD)
AV. DELAY
SYSTEM CAPACITY
INCREASE l(t)
SWEET SPOT CONTROLLED BY BUFFER ALLOCATION
THROUGHPUT CODES Lab. - Boston University
TWO KEY OBSERVATIONS 1. This is a dynamic system. But it’s not like the usual TIME-DRIVEN ones, i.e., described by differential equations
dx f ( x, u , t ) dt
Need a NEW modeling framework for these EVENT-DRIVEN systems → DISCRETE EVENT DYNAMIC SYSTEMS 2. You don’t need brute-force trial-and-error for each allocation… Once the system dynamics are understood, you can predict what happens by changing allocations (adding, removing, moving buffers) → PERTURBATION ANALYSIS THEORY Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
SYSTEM DYNAMICS: HOW ONE COMPONENT OF THE SYSTEM AFFECTS OTHER COMPONENTS A
B
C
ARRIVAL 1 ARRIVAL 2 ARRIVAL 3
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 1 FROM A
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 1 FROM B
ARRIVAL 4
Christos G. Cassandras
IDLING
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 2 FROM A
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 3 FROM A DEPARTURE 2 FROM B
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 3 FROM B
ARRIVAL 5
BLOCKING
PERTURBATION ANALYSIS Christos G. Cassandras
WHAT IF THIS HAD BEEN ADDED? RECORD BLOCK START TIME: DB(3) CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 4 FROM A
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 1 FROM C
BLOCKING ENDS
PERTURBATION ANALYSIS Christos G. Cassandras
RECORD BLOCK END TIME: Dc(1) PART SYSTEM DELAY WOULD HAVE BEEN REDUCED BY: Dc(1) - DB(3) CODES Lab. - Boston University
LEARNING BY TRIAL AND ERROR
Designs, Options, Policies, Parameters
SYSTEM
Performance Measures
CONVENTIONAL TRIAL-AND-ERROR ANALYSIS • Repeatedly change parameters/operating policies • Test different conditions
• Answer multiple WHAT IF questions N “What-If” questions N+1 trials ! Christos G. Cassandras
CODES Lab. - Boston University
LEARNING WITH PERTURBATION ANALYSIS
Designs, Options, Policies, Parameters
SYSTEM
Performance Measures
PERTURBATION ANALYSIS
WHAT IF… • Parameter p1 = a were replaced by p1 = b • Design option 1 were replaced by option 2 • •
Performance Measures under all WHAT IF Questions
ANSWERS TO MULTIPLE “WHAT IF” QUESTIONS AUTOMATICALLY PROVIDED FROM A SINGLE TRIAL Christos G. Cassandras
CODES Lab. - Boston University
DES CONTROVERSY IN THE 1980s… Anonymous referee comments for 1983 papers on Supervisory Control: (courtesy W.M. Wonham) • Automatica (reject) “Automata have no place in control engineering” • Math. Systems Theory (reject) “FSM and regular languages are nothing new at best and trivial at worst” • SIAM J. Control and Optimization (accept) “If it’s optimal control we’ll take it”
W.M. Wonham’s conclusion: “Crossing cultural divides can be a chilly business” Christos G. Cassandras
CODES Lab. - Boston University
LEARNING THROUGH PERTURBATION ANALYSIS BUFFER
Part Arrivals
MACHINE
Part Departures
x(t): SYSTEM CONTENT
Observed with K = 2 buffers
2 1 1
2
3
x(t+) = 2 Dx = -1
x(t) 2
x(t+) = 0 Dx = 0
1
1
Dx = 0
Christos G. Cassandras
4
Perturbed with K = 1 buffers
4
2
Dx = -1
Dx = 0 [THOUGHT EXPERIMENT] CODES Lab. - Boston University
DERIVATIVE ESTIMATION: INFINITESIMAL PERTURBATION ANALYSIS (IPA) “Brute Force” Derivative Estimation: q qDq
SYSTEM SYSTEM
Jˆ (q )
J q Dq
]
J ( q Dq ) J( q ) dJ Dq dq est
DRAWBACKS: • Intrusive: actively introduce perturbation Dq • Computational cost: 2 observation processes [ (N+1) for N-dim q ] • Inherently inaccurate: Dq large poor derivative approx. Dq small numerical instability Infinitesimal Perturbation Analysis (IPA):
q
SYSTEM IPA
Christos G. Cassandras
J q dJ dq est
CISE - CODES Lab. - Boston University
SAMPLE BIBLIOGRAPHY Ho, Y.C., M.A. Eyler, and D.T. Chien, “A Gradient Technique for General Buffer Storage Design in a Serial Production Line,” Intl. Journal of Production Research, Vol. 17, pp. 557-580, 1979. Ho, Y.C., and C.G. Cassandras, “A New Approach to the Analysis of Discrete Event Dynamic Systems,” Automatica, Vol. 19, No. 2, pp. 149-167, 1983. Ho, Y.C., X. Cao, and C.G. Cassandras, “Infinitesimal and Finite Perturbation Analysis for Queueing Networks,” Automatica, Vol. 19, pp. 439-445, 1983. Cao, X., “Convergence of Parameter Sensitivity Estimates in a Stochastic Experiment,” IEEE Transactions on Automatic Control, Vol. 30, pp. 834-843, 1985. Cassandras, C.G., Wardi, Y., Panayotou, C.G., and Yao, C., “Perturbation Analysis and Optimization of Stochastic Hybrid Systems”, European Journal of Control, Vol. 16, No. 6, pp. 642-664, 2010 Ramadge, P.J., and W.M. Wonham, "The control of discrete event systems," Proceedings of the IEEE, Vol. 77, No. 1, pp. 81--98,1989. David, R., and H. Alla, Petri Nets & Grafcet: Tools for Modelling Discrete Event Systems, Prentice-Hall, New York, 1992. Moody, J.O., and P.J. Antsaklis, Supervisory Control of Discrete Event Systems Using Petri Nets, Kluwer Academic Publishers, 1998.
SELECTED BOOKS • • • • • • • •
Ho, Y.C., and X. Cao, Perturbation Analysis of Discrete Event Dynamic Systems, Kluwer Academic Publisher, 1991. Glasserman, P., Gradient Estimation via Perturbation Analysis, Kluwer Academic Publishers, Boston, 1991 Cassandras, C.G., Discrete Event Systems: Modeling and Performance Analysis, Irwin Publ. 1993. Cao, X., Realization Probabilities: The Dynamics of Queueing Systems, Springer-Verlag, 1994. Glasserman, P., and D.D. Yao, Monotone Structure in Discrete Event Systems, Wiley, 1994. Fu, M.C., and J.Q. Hu, Conditional Monte Carlo: Gradient Estimation and Optimization Applications, Kluwer Academic Publ., 1997. Cao, X., Stochastic Learning and Optimization – A Sensitivity-Based Approach, Springer, 2007. Cassandras, C.G, and S. Lafortune, Introduction to Discrete Event Systems, 2nd Edition, Springer, 2008. Christos G. Cassandras
CODES Lab. - Boston University
INFINITESIMAL PERTURBATION ANALYSIS (IPA) Christos G. Cassandras
CODES Lab. - Boston University
INFINITESIMAL PERTURBATION ANALYSIS (IPA) 1. INITIALIZATION: If a feasible at x0:
Da :
Else for all other a:
dVa ,1 dq
Da : 0
2. WHENEVER b IS OBSERVED: If a activated with new lifetime Va :
2.1. Compute dVa/dq dVa 2.2. Da : D b dq Christos G. Cassandras
CODES Lab. - Boston University
INFINITESIMAL PERTURBATION ANALYSIS (IPA) QUESTION: Are IPA sensitivity estimators “good” ? ▪ Unbiasedness ▪ Consistency ANSWER: Yes, for a large class of DES that includes
▪ G/G/1 queueing systems ▪ Jackson-like queueing networks (e.g., no blocking allowed) NOTE: IPA applies to REAL-VALUED parameters of some event process distribution (e.g., mean interarrival and service times) Christos G. Cassandras
CODES Lab. - Boston University
IPA UNBIASEDNESS
IPA ESTIMATOR
dJ d d E[ L (q )] E L (q ) dq dq dq T
T
…provided interchange of E and d/dq is possible ! Christos G. Cassandras
CODES Lab. - Boston University
Ho, Y.C., and C.G. Cassandras, “A New Approach to the Analysis of Discrete Event Dynamic Systems,” Automatica, Vol. 19, No. 2, pp. 149-167, 1983. Heidelberger, P., X. Cao, M. Zazanis, and R. Suri, “Convergence Properties of Infinitesimal Perturbation Analysis Estimates,” Management Science, Vol. 34, No. 11, pp. 1281-1302, 1988. Suri, R., and M. Zazanis, “Perturbation Analysis Gives Strongly Consistent Sensitivity Estimates for the M/G/1 Queue,” Management Science, Vol. 34, pp. 39-64, 1988.
C.G. Cassandras
1979 M.A. Eyler
M. Zazanis
1982
1984
1986
X. Cao Ho, Y.C., X. Cao, and C.G. Cassandras, “Infinitesimal and Finite Perturbation Analysis for Queueing Networks,” Automatica, Vol. 19, pp. 439-445, 1983.
Cao, X., “Convergence of Parameter Sensitivity Estimates in a Stochastic Experiment,” IEEE Transactions on Automatic Control, Vol. 30, pp. 834-843, 1985.
Ho, Y.C., M.A. Eyler, and D.T. Chien, “A Gradient Technique for General Buffer Storage Design in a Serial Production Line,” Intl. Journal of Production Research, Vol. 17, pp. 557-580, 1979.
Glasserman, P., and W.B. Gong, “Smoothed Perturbation Analysis for a Class of Discrete Event Systems,” IEEE Transactions on Automatic Control, Vol. AC-35, No. 11, pp. 1218-1230, 1990. Glasserman, P., Gradient Estimation via Perturbation Analysis, Kluwer Academic Publishers, Boston, 1991. Fu, M.C., “Convergence of a Stochastic Approximation Algorithm for the GI/G/1 Queue Using Infinitesimal Perturbation Analysis,” Journal of Optimization Theory and Applications, Vol. 65, pp. 149-160, 1990.
P. Glasserman
1987 W. Gong
M. Fu
1988 P. Vakili
1989
Cassandras, C.G., and S.G. Strickland, “Observable Augmented Systems for Sensitivity Analysis of Markov and Semi-Markov Processes,” IEEE Transactions on Automatic Control, Vol. AC-34, No. 10, pp. 1026-1037, 1989
S. Li Ho, Y.C., and S. Li, “Extensions of Perturbation Analysis of Discrete Event Dynamic Systems,” IEEE Transactions on Automatic Control, Vol. AC-33, pp. 427-438, 1988.
Vakili, P., and Y.C. Ho, “Infinitesimal Perturbation Analysis of a Multiclass Routing Problem,” Proceedings of 25th Allerton Conference on Communication, Control, and Computing, pp. 279-286, 1987. Gong, W.B., and Y.C. Ho, “Smoothed Perturbation Analysis of Discrete Event Systems,” IEEE Transactions on Automatic Control, Vol. AC-32, No. 10, pp. 858-866, 1987.
Ho, Y.C., and X. Cao, Perturbation Analysis of Discrete Event Dynamic Systems, Kluwer Academic Publisher, 1991. Vakili, P., “A Standard Clock Technique for Efficient Simulation,” Operations Research Letters, Vol. 10, pp. 445-452, 1991. Chong, E.K.P., and P.J. Ramadge,, “Convergence of Recursive Optimization Algorithms Using Infinitesimal Perturbation Analysis Estimates,” Journal of Discrete Event Dynamic Systems, Vol. 1, No. 4, pp. 339-372, 1992. Bremaud, P., and F.J. Vazquez-Abad, “On the Pathwise Computation of Derivatives with Respect to the Rate of a Point Process: The Phantom RPA Method,” Queueing Systems, Vol. 10, pp. 249-270, 1992.
1990
1991
J.Q. Hu
Wardi, Y., and J.Q. Hu, “Strong Consistency of Infinitesimal Perturbation Analysis for Tandem Queueing Networks,” Journal of Discrete Event Dynamic Systems, Vol. 1, No. 1, pp. 37-60, 1991. Hu, J.Q. and Strickland, S.G., "Strong Consistency of Sample Path Derivative Estimates," Applied Mathematics Letters, Vol. 3, No. 4, pp. 55-58, 1990.
1992 L. Shi Ho, Y-C, Shi, L., Dai, L., and Gong, W-B. “A New Method for Optimizing Complex Networks,” 30th IEEE conference on Decision and Control, 1991, pp. 105-109.
Shi, L. “Variance Property of Discontinuous Perturbation Analysis,” Winter Simulation Conference, Dec. 1996, pp. 412-417.
Ho, Y.C. and Hu, J.Q., "An Infinitesimal Perturbation Analysis Algorithm for a Multiclass G/G/1 Queue," Operations Research Letters, 9, pp.35-44, 1990.
Cassandras, C.G., Discrete Event Systems: Modeling and Performance Analysis, Irwin Publ. 1993.
Fu, M.C., and J.Q. Hu, Conditional Monte Carlo: Gradient Estimation and Optimization Applications, Kluwer Academic Publishers, Boston, 1997.
1993
1994
1997
1999
L. Dai Cassandras, C.G., and C.G. Panayiotou, “Concurrent Sample Path Analysis of Discrete Event Systems,” Journal of Discrete Event Dynamic Systems, 1999.
Cao, X., Realization Probabilities: The Dynamics of Queueing Systems, Springer-Verlag, 1994. Dai, L., and Y.C. Ho, “Structural Infinitesimal Perturbation Analysis (SIPA) for Derivative Estimation of Discrete Event Dynamic Systems,” IEEE Transactions on Automatic Control, Vol. 40, pp. 1154-1166, 1995.
IPA RE-BORN: HYBRID SYSTEMS Cassandras, C.G., Wardi, Y., Panayotou, C.G., and Yao, C., “Perturbation Analysis and Optimization of Stochastic Hybrid Systems”, European Journal of Control, Vol. 16, No. 6, pp. 642-664, 2010.
Geng, Y., and Cassandras, C.G., “Multi-intersection Traffic Light Control with Blocking”, J. of Discrete Event Dynamic Systems, Vol. 25, 1, pp. 7-30, 2015.
Cassandras, C.G., Lin, X., and Ding X.C. “An Optimal Control Approach to the Multi-agent Persistent Monitoring Problem”, IEEE Trans. on Automatic Control, AC-58, 4, pp. 947-961, 2013.
2002
2010
2017
Yao, C, and Cassandras, C.G., “A Solution to the Optimal Lot Sizing Problem as a Stochastic Resource Contention Game”, IEEE Trans. on Automation Science and Engineering, Vol. 9, No. 2, pp. 250-264, 2012.
Cassandras, C.G., Wardi, Y., Melamed, B., Sun, G., and Panayiotou, C.G., "Perturbation Analysis for On-Line Control and Optimization of Stochastic Fluid Models", IEEE Trans. on Automatic Control, AC-47, 8, pp. 1234-1248, 2002 Christos G. Cassandras
Fleck, J.L., and Cassandras, C.G., “Optimal Design of Personalized Prostate Cancer Therapy using Infinitesimal Perturbation Analysis”, Nonlinear Analysis: Hybrid Systems, Vol. 25, pp. 246-262, 2017. CODES Lab. - Boston University
THE IPA CALCULUS
NOTATION:
x t
q xq , t , k k q q
HYBRID AUTOMATA HYBRID AUTOMATA Gh (Q, X , E,U , f , , Inv, guard , , q0 , x0 ) Q:
set of discrete states (modes)
X:
set of continuous states (normally Rn)
E:
set of events
U:
set of admissible controls
f: :
vector field, f : Q X U X discrete state transition function, : Q X E Q
Inv:
set defining an invariant condition (domain), Inv Q X
guard: set defining a guard condition, guard Q Q X : reset function, : Q Q X E X q0 :
initial discrete state
x0 :
initial continuous state
Christos G. Cassandras
CODES Lab. - Boston University
HYBRID AUTOMATA STOCHASTIC HYBRID AUTOMATA Event at time k(q)
Event at time k+1(q)
x f k ( x,q, t )
kth discrete state (mode)
q : control parameter, q (system design parameter,
parameter of an input process, or parameter that characterizes a control policy)
Christos G. Cassandras
CODES Lab. - Boston University
IPA: THREE FUNDAMENTAL EQUATIONS System dynamics over (k(q), k+1(q)]: x f k ( x, q , t ) NOTATION:
x t
k q xq , t , k q q
1. Continuity at events: x( k ) x( k ) Take d/dq :
x' ( k ) x' ( k ) [ f k 1 ( k ) f k ( k )] 'k
d (q, q, x, , ) If no continuity, use reset condition x' ( ) dq k
Christos G. Cassandras
CISE - CODES Lab. - Boston University
IPA: THREE FUNDAMENTAL EQUATIONS 2. Take d/dq of system dynamics x f k ( x, q , t ) over (k(q), k+1(q)]: f (t ) dx' (t ) f k (t ) x' (t ) k dt x q
Solve
f (t ) dx' (t ) f k (t ) x' (t ) k over (k(q), k+1(q)]: dt x q t
x(t ) e
k
f k ( u ) du x
t f (v) v f k (u ) du k x k e dv x( k ) k q
initial condition from 1 above
NOTE: If there are no events (pure time-driven system), IPA reduces to this equation Christos G. Cassandras
CISE - CODES Lab. - Boston University
IPA: THREE FUNDAMENTAL EQUATIONS 3. Get k depending on the event type: - Exogenous event: By definition, k 0 - Endogenous event: occurs when g k ( x (q , k ),q ) 0 1
g g g k f k ( k ) x( k ) x q x
- Induced events: 1
y ( ) k k k yk ( k ) t Christos G. Cassandras
CISE - CODES Lab. - Boston University
IPA: THREE FUNDAMENTAL EQUATIONS Ignoring resets and induced events: Recall:
1. x' ( ) x' ( ) [ f k 1 ( ) f k ( )] 'k k
k
f k ( u ) du k x t
2. x(t ) e
k
k
x t
t f (v) v f k (u ) du k x k e dv x' ( k ) k q
k
xq , t q
k q q
1
3. k 0 or
x' ( ) k
1 3
Christos G. Cassandras
g g g k f k ( k ) x( k ) x q x 2 Cassandras et al, Europ. J. Control, 2010 CISE - CODES Lab. - Boston University
IPA PROPERTIES 1. ROBUSTNESS TO NOISE: Performance sensitivities can often be obtained from information limited to event times, which is easily observed No need to track system in between events ! 2. DECOMPOSABILITY: Performance sensitivities are often reset to 0 sample path can be conveniently decomposed 3. SCALABILITY: IPA estimators are EVENT-DRIVEN IPA scales with the EVENT SET, not the STATE SPACE !
Christos G. Cassandras
CISE - CODES Lab. - Boston University
TIME-DRIVEN v EVENT-DRIVEN CONTROL REFERENCE
+
ERROR
CONTROLLER
INPUT
MEASURED OUTPUT
PLANT
OUTPUT
SENSOR
EVENT-DRIVEN CONTROL: Act only when needed (or on TIMEOUT) - not based on a clock REFERENCE
+
ERROR
CONTROLLER
INPUT
PLANT
MEASURED OUTPUT
EVENT:
g(STATE) ≤ 0 Christos G. Cassandras
OUTPUT
SENSOR CODES Lab. - Boston University
SELECTED REFERENCES - EVENT-DRIVEN CONTROL - Astrom, K.J., and B. M. Bernhardsson, “Comparison of Riemann and Lebesgue sampling for first order stochastic systems,” Proc. 41st Conf. Decision and Control, pp. 2011–2016, 2002. - T. Shima, S. Rasmussen, and P. Chandler, “UAV Team Decision and Control using Efficient Collaborative Estimation,” ASME J. of Dynamic Systems, Measurement, and Control, vol. 129, no. 5, pp. 609–619, 2007. - Heemels, W. P. M. H., J. H. Sandee, and P. P. J. van den Bosch, “Analysis of event-driven controllers for linear systems,” Intl. J. Control, 81, pp. 571–590, 2008. - P. Tabuada, “Event-triggered real-time scheduling of stabilizing control tasks,” IEEE Trans. Autom. Control, vol. 52, pp. 1680–1685, 2007. - J. H. Sandee, W. P. M. H. Heemels, S. B. F. Hulsenboom, and P. P. J. van den Bosch, “Analysis and experimental validation of a sensor-based event-driven controller,” Proc. American Control Conf., pp. 2867–2874, 2007. - J. Lunze and D. Lehmann, “A state-feedback approach to event-based control,” Automatica, 46, pp. 211–215, 2010. - P. Wan and M. D. Lemmon, “Event triggered distributed optimization in sensor networks,” Proc. of 8th ACM/IEEE Intl. Conf. on Information Processing in Sensor Networks, 2009. - Zhong, M., and Cassandras, C.G., “Asynchronous Distributed Optimization with Event-Driven Communication”, IEEE Trans. on Automatic Control, AC-55, 12, pp. 2735-2750, 2010. Christos G. Cassandras
CODES Lab. - Boston University
CONCLUSIONS 1979: Buffer allocation problem Continuous-parameter optimization
Basic PA technique IPA
UNBIASEDNESS ??? YES
Large-scale system optimization Realization Factors
NO
Alternative PA techniques: SPA, RPA, cut-and-paste, etc COMPLEXITY
IPA Calculus, Stochastic Hybrid Systems Complexity management in optimization and control
https://christosgcassandras.org Christos G. Cassandras
CODES Lab. - Boston University
OUTLINE ▪ 1979: How a “real problem” gives birth to … an academic discipline … a research field … a toolbox for practical problem solutions ▪ 1979-1999: Development chronology in Perturbation Analysis (PA) ▪ 2002: Re-inventing Infinitesimal Perturbation Analysis (IPA) for Hybrid Systems ▪ 2017: The IPA Calculus, Event-Driven control theory, Data-driven methods come of age Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM (circa 1978)
What is the best way to distribute BUFFER capacity in a manufacturing transfer line?
PARTS IN
SERIAL OPERATIONS
Ho, Eyler, Chien, Cassandras, Cao,…
PRODUCTS OUT
BUFFERS Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM (circa 1978)
What is the best way to distribute BUFFER capacity in a manufacturing transfer line?
PARTS IN
PRODUCTS 0UT
SLOW…
PRETTY FAST…
BUFFERS Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM (circa 1978)
Complexity of this buffer allocation process (K buffers, N stages)
K N 1 K
Example: K = 24, N = 6 → 118,755 possible allocations • “Brute Force” trial-and-error: test each allocation for about a week to get statistically meaningful results (that’s if the manager allows you to mess with the system…) → about 2300 years… • Suppose you can reduce to only 1000 “promising” allocations: → about 19 years… • Using a simulated transfer line, about 3 minutes per trial (1980s computing technology…) → about 250 days… Christos G. Cassandras
CODES Lab. - Boston University
WHY IS THIS PROBLEM IMPORTANT ? Manufacturing system with N sequential operations: l(t) …
…
…
…
THROUGHPUT
DELAY
THROUGHPUT increases (GOOD) Christos G. Cassandras
average DELAY increases (BAD)
AV. DELAY
SYSTEM CAPACITY
INCREASE l(t)
SWEET SPOT CONTROLLED BY BUFFER ALLOCATION
THROUGHPUT CODES Lab. - Boston University
TWO KEY OBSERVATIONS 1. This is a dynamic system. But it’s not like the usual TIME-DRIVEN ones, i.e., described by differential equations
dx f ( x, u , t ) dt
Need a NEW modeling framework for these EVENT-DRIVEN systems → DISCRETE EVENT DYNAMIC SYSTEMS 2. You don’t need brute-force trial-and-error for each allocation… Once the system dynamics are understood, you can predict what happens by changing allocations (adding, removing, moving buffers) → PERTURBATION ANALYSIS THEORY Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
SYSTEM DYNAMICS: HOW ONE COMPONENT OF THE SYSTEM AFFECTS OTHER COMPONENTS A
B
C
ARRIVAL 1 ARRIVAL 2 ARRIVAL 3
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 1 FROM A
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 1 FROM B
ARRIVAL 4
Christos G. Cassandras
IDLING
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 2 FROM A
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 3 FROM A DEPARTURE 2 FROM B
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 3 FROM B
ARRIVAL 5
BLOCKING
PERTURBATION ANALYSIS Christos G. Cassandras
WHAT IF THIS HAD BEEN ADDED? RECORD BLOCK START TIME: DB(3) CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 4 FROM A
Christos G. Cassandras
CODES Lab. - Boston University
THE
PROBLEM: SYSTEM DYNAMICS
DEPARTURE 1 FROM C
BLOCKING ENDS
PERTURBATION ANALYSIS Christos G. Cassandras
RECORD BLOCK END TIME: Dc(1) PART SYSTEM DELAY WOULD HAVE BEEN REDUCED BY: Dc(1) - DB(3) CODES Lab. - Boston University
LEARNING BY TRIAL AND ERROR
Designs, Options, Policies, Parameters
SYSTEM
Performance Measures
CONVENTIONAL TRIAL-AND-ERROR ANALYSIS • Repeatedly change parameters/operating policies • Test different conditions
• Answer multiple WHAT IF questions N “What-If” questions N+1 trials ! Christos G. Cassandras
CODES Lab. - Boston University
LEARNING WITH PERTURBATION ANALYSIS
Designs, Options, Policies, Parameters
SYSTEM
Performance Measures
PERTURBATION ANALYSIS
WHAT IF… • Parameter p1 = a were replaced by p1 = b • Design option 1 were replaced by option 2 • •
Performance Measures under all WHAT IF Questions
ANSWERS TO MULTIPLE “WHAT IF” QUESTIONS AUTOMATICALLY PROVIDED FROM A SINGLE TRIAL Christos G. Cassandras
CODES Lab. - Boston University
DES CONTROVERSY IN THE 1980s… Anonymous referee comments for 1983 papers on Supervisory Control: (courtesy W.M. Wonham) • Automatica (reject) “Automata have no place in control engineering” • Math. Systems Theory (reject) “FSM and regular languages are nothing new at best and trivial at worst” • SIAM J. Control and Optimization (accept) “If it’s optimal control we’ll take it”
W.M. Wonham’s conclusion: “Crossing cultural divides can be a chilly business” Christos G. Cassandras
CODES Lab. - Boston University
LEARNING THROUGH PERTURBATION ANALYSIS BUFFER
Part Arrivals
MACHINE
Part Departures
x(t): SYSTEM CONTENT
Observed with K = 2 buffers
2 1 1
2
3
x(t+) = 2 Dx = -1
x(t) 2
x(t+) = 0 Dx = 0
1
1
Dx = 0
Christos G. Cassandras
4
Perturbed with K = 1 buffers
4
2
Dx = -1
Dx = 0 [THOUGHT EXPERIMENT] CODES Lab. - Boston University
DERIVATIVE ESTIMATION: INFINITESIMAL PERTURBATION ANALYSIS (IPA) “Brute Force” Derivative Estimation: q qDq
SYSTEM SYSTEM
Jˆ (q )
J q Dq
]
J ( q Dq ) J( q ) dJ Dq dq est
DRAWBACKS: • Intrusive: actively introduce perturbation Dq • Computational cost: 2 observation processes [ (N+1) for N-dim q ] • Inherently inaccurate: Dq large poor derivative approx. Dq small numerical instability Infinitesimal Perturbation Analysis (IPA):
q
SYSTEM IPA
Christos G. Cassandras
J q dJ dq est
CISE - CODES Lab. - Boston University
SAMPLE BIBLIOGRAPHY Ho, Y.C., M.A. Eyler, and D.T. Chien, “A Gradient Technique for General Buffer Storage Design in a Serial Production Line,” Intl. Journal of Production Research, Vol. 17, pp. 557-580, 1979. Ho, Y.C., and C.G. Cassandras, “A New Approach to the Analysis of Discrete Event Dynamic Systems,” Automatica, Vol. 19, No. 2, pp. 149-167, 1983. Ho, Y.C., X. Cao, and C.G. Cassandras, “Infinitesimal and Finite Perturbation Analysis for Queueing Networks,” Automatica, Vol. 19, pp. 439-445, 1983. Cao, X., “Convergence of Parameter Sensitivity Estimates in a Stochastic Experiment,” IEEE Transactions on Automatic Control, Vol. 30, pp. 834-843, 1985. Cassandras, C.G., Wardi, Y., Panayotou, C.G., and Yao, C., “Perturbation Analysis and Optimization of Stochastic Hybrid Systems”, European Journal of Control, Vol. 16, No. 6, pp. 642-664, 2010 Ramadge, P.J., and W.M. Wonham, "The control of discrete event systems," Proceedings of the IEEE, Vol. 77, No. 1, pp. 81--98,1989. David, R., and H. Alla, Petri Nets & Grafcet: Tools for Modelling Discrete Event Systems, Prentice-Hall, New York, 1992. Moody, J.O., and P.J. Antsaklis, Supervisory Control of Discrete Event Systems Using Petri Nets, Kluwer Academic Publishers, 1998.
SELECTED BOOKS • • • • • • • •
Ho, Y.C., and X. Cao, Perturbation Analysis of Discrete Event Dynamic Systems, Kluwer Academic Publisher, 1991. Glasserman, P., Gradient Estimation via Perturbation Analysis, Kluwer Academic Publishers, Boston, 1991 Cassandras, C.G., Discrete Event Systems: Modeling and Performance Analysis, Irwin Publ. 1993. Cao, X., Realization Probabilities: The Dynamics of Queueing Systems, Springer-Verlag, 1994. Glasserman, P., and D.D. Yao, Monotone Structure in Discrete Event Systems, Wiley, 1994. Fu, M.C., and J.Q. Hu, Conditional Monte Carlo: Gradient Estimation and Optimization Applications, Kluwer Academic Publ., 1997. Cao, X., Stochastic Learning and Optimization – A Sensitivity-Based Approach, Springer, 2007. Cassandras, C.G, and S. Lafortune, Introduction to Discrete Event Systems, 2nd Edition, Springer, 2008. Christos G. Cassandras
CODES Lab. - Boston University
INFINITESIMAL PERTURBATION ANALYSIS (IPA) Christos G. Cassandras
CODES Lab. - Boston University
INFINITESIMAL PERTURBATION ANALYSIS (IPA) 1. INITIALIZATION: If a feasible at x0:
Da :
Else for all other a:
dVa ,1 dq
Da : 0
2. WHENEVER b IS OBSERVED: If a activated with new lifetime Va :
2.1. Compute dVa/dq dVa 2.2. Da : D b dq Christos G. Cassandras
CODES Lab. - Boston University
INFINITESIMAL PERTURBATION ANALYSIS (IPA) QUESTION: Are IPA sensitivity estimators “good” ? ▪ Unbiasedness ▪ Consistency ANSWER: Yes, for a large class of DES that includes
▪ G/G/1 queueing systems ▪ Jackson-like queueing networks (e.g., no blocking allowed) NOTE: IPA applies to REAL-VALUED parameters of some event process distribution (e.g., mean interarrival and service times) Christos G. Cassandras
CODES Lab. - Boston University
IPA UNBIASEDNESS
IPA ESTIMATOR
dJ d d E[ L (q )] E L (q ) dq dq dq T
T
…provided interchange of E and d/dq is possible ! Christos G. Cassandras
CODES Lab. - Boston University
Ho, Y.C., and C.G. Cassandras, “A New Approach to the Analysis of Discrete Event Dynamic Systems,” Automatica, Vol. 19, No. 2, pp. 149-167, 1983. Heidelberger, P., X. Cao, M. Zazanis, and R. Suri, “Convergence Properties of Infinitesimal Perturbation Analysis Estimates,” Management Science, Vol. 34, No. 11, pp. 1281-1302, 1988. Suri, R., and M. Zazanis, “Perturbation Analysis Gives Strongly Consistent Sensitivity Estimates for the M/G/1 Queue,” Management Science, Vol. 34, pp. 39-64, 1988.
C.G. Cassandras
1979 M.A. Eyler
M. Zazanis
1982
1984
1986
X. Cao Ho, Y.C., X. Cao, and C.G. Cassandras, “Infinitesimal and Finite Perturbation Analysis for Queueing Networks,” Automatica, Vol. 19, pp. 439-445, 1983.
Cao, X., “Convergence of Parameter Sensitivity Estimates in a Stochastic Experiment,” IEEE Transactions on Automatic Control, Vol. 30, pp. 834-843, 1985.
Ho, Y.C., M.A. Eyler, and D.T. Chien, “A Gradient Technique for General Buffer Storage Design in a Serial Production Line,” Intl. Journal of Production Research, Vol. 17, pp. 557-580, 1979.
Glasserman, P., and W.B. Gong, “Smoothed Perturbation Analysis for a Class of Discrete Event Systems,” IEEE Transactions on Automatic Control, Vol. AC-35, No. 11, pp. 1218-1230, 1990. Glasserman, P., Gradient Estimation via Perturbation Analysis, Kluwer Academic Publishers, Boston, 1991. Fu, M.C., “Convergence of a Stochastic Approximation Algorithm for the GI/G/1 Queue Using Infinitesimal Perturbation Analysis,” Journal of Optimization Theory and Applications, Vol. 65, pp. 149-160, 1990.
P. Glasserman
1987 W. Gong
M. Fu
1988 P. Vakili
1989
Cassandras, C.G., and S.G. Strickland, “Observable Augmented Systems for Sensitivity Analysis of Markov and Semi-Markov Processes,” IEEE Transactions on Automatic Control, Vol. AC-34, No. 10, pp. 1026-1037, 1989
S. Li Ho, Y.C., and S. Li, “Extensions of Perturbation Analysis of Discrete Event Dynamic Systems,” IEEE Transactions on Automatic Control, Vol. AC-33, pp. 427-438, 1988.
Vakili, P., and Y.C. Ho, “Infinitesimal Perturbation Analysis of a Multiclass Routing Problem,” Proceedings of 25th Allerton Conference on Communication, Control, and Computing, pp. 279-286, 1987. Gong, W.B., and Y.C. Ho, “Smoothed Perturbation Analysis of Discrete Event Systems,” IEEE Transactions on Automatic Control, Vol. AC-32, No. 10, pp. 858-866, 1987.
Ho, Y.C., and X. Cao, Perturbation Analysis of Discrete Event Dynamic Systems, Kluwer Academic Publisher, 1991. Vakili, P., “A Standard Clock Technique for Efficient Simulation,” Operations Research Letters, Vol. 10, pp. 445-452, 1991. Chong, E.K.P., and P.J. Ramadge,, “Convergence of Recursive Optimization Algorithms Using Infinitesimal Perturbation Analysis Estimates,” Journal of Discrete Event Dynamic Systems, Vol. 1, No. 4, pp. 339-372, 1992. Bremaud, P., and F.J. Vazquez-Abad, “On the Pathwise Computation of Derivatives with Respect to the Rate of a Point Process: The Phantom RPA Method,” Queueing Systems, Vol. 10, pp. 249-270, 1992.
1990
1991
J.Q. Hu
Wardi, Y., and J.Q. Hu, “Strong Consistency of Infinitesimal Perturbation Analysis for Tandem Queueing Networks,” Journal of Discrete Event Dynamic Systems, Vol. 1, No. 1, pp. 37-60, 1991. Hu, J.Q. and Strickland, S.G., "Strong Consistency of Sample Path Derivative Estimates," Applied Mathematics Letters, Vol. 3, No. 4, pp. 55-58, 1990.
1992 L. Shi Ho, Y-C, Shi, L., Dai, L., and Gong, W-B. “A New Method for Optimizing Complex Networks,” 30th IEEE conference on Decision and Control, 1991, pp. 105-109.
Shi, L. “Variance Property of Discontinuous Perturbation Analysis,” Winter Simulation Conference, Dec. 1996, pp. 412-417.
Ho, Y.C. and Hu, J.Q., "An Infinitesimal Perturbation Analysis Algorithm for a Multiclass G/G/1 Queue," Operations Research Letters, 9, pp.35-44, 1990.
Cassandras, C.G., Discrete Event Systems: Modeling and Performance Analysis, Irwin Publ. 1993.
Fu, M.C., and J.Q. Hu, Conditional Monte Carlo: Gradient Estimation and Optimization Applications, Kluwer Academic Publishers, Boston, 1997.
1993
1994
1997
1999
L. Dai Cassandras, C.G., and C.G. Panayiotou, “Concurrent Sample Path Analysis of Discrete Event Systems,” Journal of Discrete Event Dynamic Systems, 1999.
Cao, X., Realization Probabilities: The Dynamics of Queueing Systems, Springer-Verlag, 1994. Dai, L., and Y.C. Ho, “Structural Infinitesimal Perturbation Analysis (SIPA) for Derivative Estimation of Discrete Event Dynamic Systems,” IEEE Transactions on Automatic Control, Vol. 40, pp. 1154-1166, 1995.
IPA RE-BORN: HYBRID SYSTEMS Cassandras, C.G., Wardi, Y., Panayotou, C.G., and Yao, C., “Perturbation Analysis and Optimization of Stochastic Hybrid Systems”, European Journal of Control, Vol. 16, No. 6, pp. 642-664, 2010.
Geng, Y., and Cassandras, C.G., “Multi-intersection Traffic Light Control with Blocking”, J. of Discrete Event Dynamic Systems, Vol. 25, 1, pp. 7-30, 2015.
Cassandras, C.G., Lin, X., and Ding X.C. “An Optimal Control Approach to the Multi-agent Persistent Monitoring Problem”, IEEE Trans. on Automatic Control, AC-58, 4, pp. 947-961, 2013.
2002
2010
2017
Yao, C, and Cassandras, C.G., “A Solution to the Optimal Lot Sizing Problem as a Stochastic Resource Contention Game”, IEEE Trans. on Automation Science and Engineering, Vol. 9, No. 2, pp. 250-264, 2012.
Cassandras, C.G., Wardi, Y., Melamed, B., Sun, G., and Panayiotou, C.G., "Perturbation Analysis for On-Line Control and Optimization of Stochastic Fluid Models", IEEE Trans. on Automatic Control, AC-47, 8, pp. 1234-1248, 2002 Christos G. Cassandras
Fleck, J.L., and Cassandras, C.G., “Optimal Design of Personalized Prostate Cancer Therapy using Infinitesimal Perturbation Analysis”, Nonlinear Analysis: Hybrid Systems, Vol. 25, pp. 246-262, 2017. CODES Lab. - Boston University
THE IPA CALCULUS
NOTATION:
x t
q xq , t , k k q q
HYBRID AUTOMATA HYBRID AUTOMATA Gh (Q, X , E,U , f , , Inv, guard , , q0 , x0 ) Q:
set of discrete states (modes)
X:
set of continuous states (normally Rn)
E:
set of events
U:
set of admissible controls
f: :
vector field, f : Q X U X discrete state transition function, : Q X E Q
Inv:
set defining an invariant condition (domain), Inv Q X
guard: set defining a guard condition, guard Q Q X : reset function, : Q Q X E X q0 :
initial discrete state
x0 :
initial continuous state
Christos G. Cassandras
CODES Lab. - Boston University
HYBRID AUTOMATA STOCHASTIC HYBRID AUTOMATA Event at time k(q)
Event at time k+1(q)
x f k ( x,q, t )
kth discrete state (mode)
q : control parameter, q (system design parameter,
parameter of an input process, or parameter that characterizes a control policy)
Christos G. Cassandras
CODES Lab. - Boston University
IPA: THREE FUNDAMENTAL EQUATIONS System dynamics over (k(q), k+1(q)]: x f k ( x, q , t ) NOTATION:
x t
k q xq , t , k q q
1. Continuity at events: x( k ) x( k ) Take d/dq :
x' ( k ) x' ( k ) [ f k 1 ( k ) f k ( k )] 'k
d (q, q, x, , ) If no continuity, use reset condition x' ( ) dq k
Christos G. Cassandras
CISE - CODES Lab. - Boston University
IPA: THREE FUNDAMENTAL EQUATIONS 2. Take d/dq of system dynamics x f k ( x, q , t ) over (k(q), k+1(q)]: f (t ) dx' (t ) f k (t ) x' (t ) k dt x q
Solve
f (t ) dx' (t ) f k (t ) x' (t ) k over (k(q), k+1(q)]: dt x q t
x(t ) e
k
f k ( u ) du x
t f (v) v f k (u ) du k x k e dv x( k ) k q
initial condition from 1 above
NOTE: If there are no events (pure time-driven system), IPA reduces to this equation Christos G. Cassandras
CISE - CODES Lab. - Boston University
IPA: THREE FUNDAMENTAL EQUATIONS 3. Get k depending on the event type: - Exogenous event: By definition, k 0 - Endogenous event: occurs when g k ( x (q , k ),q ) 0 1
g g g k f k ( k ) x( k ) x q x
- Induced events: 1
y ( ) k k k yk ( k ) t Christos G. Cassandras
CISE - CODES Lab. - Boston University
IPA: THREE FUNDAMENTAL EQUATIONS Ignoring resets and induced events: Recall:
1. x' ( ) x' ( ) [ f k 1 ( ) f k ( )] 'k k
k
f k ( u ) du k x t
2. x(t ) e
k
k
x t
t f (v) v f k (u ) du k x k e dv x' ( k ) k q
k
xq , t q
k q q
1
3. k 0 or
x' ( ) k
1 3
Christos G. Cassandras
g g g k f k ( k ) x( k ) x q x 2 Cassandras et al, Europ. J. Control, 2010 CISE - CODES Lab. - Boston University
IPA PROPERTIES 1. ROBUSTNESS TO NOISE: Performance sensitivities can often be obtained from information limited to event times, which is easily observed No need to track system in between events ! 2. DECOMPOSABILITY: Performance sensitivities are often reset to 0 sample path can be conveniently decomposed 3. SCALABILITY: IPA estimators are EVENT-DRIVEN IPA scales with the EVENT SET, not the STATE SPACE !
Christos G. Cassandras
CISE - CODES Lab. - Boston University
TIME-DRIVEN v EVENT-DRIVEN CONTROL REFERENCE
+
ERROR
CONTROLLER
INPUT
MEASURED OUTPUT
PLANT
OUTPUT
SENSOR
EVENT-DRIVEN CONTROL: Act only when needed (or on TIMEOUT) - not based on a clock REFERENCE
+
ERROR
CONTROLLER
INPUT
PLANT
MEASURED OUTPUT
EVENT:
g(STATE) ≤ 0 Christos G. Cassandras
OUTPUT
SENSOR CODES Lab. - Boston University
SELECTED REFERENCES - EVENT-DRIVEN CONTROL - Astrom, K.J., and B. M. Bernhardsson, “Comparison of Riemann and Lebesgue sampling for first order stochastic systems,” Proc. 41st Conf. Decision and Control, pp. 2011–2016, 2002. - T. Shima, S. Rasmussen, and P. Chandler, “UAV Team Decision and Control using Efficient Collaborative Estimation,” ASME J. of Dynamic Systems, Measurement, and Control, vol. 129, no. 5, pp. 609–619, 2007. - Heemels, W. P. M. H., J. H. Sandee, and P. P. J. van den Bosch, “Analysis of event-driven controllers for linear systems,” Intl. J. Control, 81, pp. 571–590, 2008. - P. Tabuada, “Event-triggered real-time scheduling of stabilizing control tasks,” IEEE Trans. Autom. Control, vol. 52, pp. 1680–1685, 2007. - J. H. Sandee, W. P. M. H. Heemels, S. B. F. Hulsenboom, and P. P. J. van den Bosch, “Analysis and experimental validation of a sensor-based event-driven controller,” Proc. American Control Conf., pp. 2867–2874, 2007. - J. Lunze and D. Lehmann, “A state-feedback approach to event-based control,” Automatica, 46, pp. 211–215, 2010. - P. Wan and M. D. Lemmon, “Event triggered distributed optimization in sensor networks,” Proc. of 8th ACM/IEEE Intl. Conf. on Information Processing in Sensor Networks, 2009. - Zhong, M., and Cassandras, C.G., “Asynchronous Distributed Optimization with Event-Driven Communication”, IEEE Trans. on Automatic Control, AC-55, 12, pp. 2735-2750, 2010. Christos G. Cassandras
CODES Lab. - Boston University
CONCLUSIONS 1979: Buffer allocation problem Continuous-parameter optimization
Basic PA technique IPA
UNBIASEDNESS ??? YES
Large-scale system optimization Realization Factors
NO
Alternative PA techniques: SPA, RPA, cut-and-paste, etc COMPLEXITY
IPA Calculus, Stochastic Hybrid Systems Complexity management in optimization and control
