Estimating Simulation Input Uncertainty
Quantification of Simulation Input Uncertainty Henry Lam Department of Industrial Engineering & Operations Research Columbia University Joint work with Huajie Qian (University of Michigan) ISIM 2017, Durham, UK
Background Simulation replication 1 Simulation replication 2
Input Model
Output analysis
Simulation replication 3
Simulation replication 𝑅 Simulation is run by generating random variates specified by distributions
Background Simulation replication 1 Simulation replication 2
Input Data
Output analysis
Simulation replication 3
Simulation replication 𝑅 Input uncertainty/variability/error arises when the input distributions are noisy estimates from data
Background Simulation replication 1 Simulation replication 2
Input Data
Output analysis
Simulation replication 3
Simulation replication 𝑅 The input uncertainty quantification problem (Henderson ‘03, Chick ‘06, Barton ’12, Song & Nelson ‘14…): Estimate confidence interval for a performance measure under both simulation and input noises
Background Total output variability
=
Simulation variability
+
Input variability
• Input data and simulation replication are of comparable sizes • Simulation cost is significant
Background Total output variability
=
Simulation variability
+
Input variability
• Input data and simulation replication are of comparable sizes • Simulation cost is significant Focus of the talk: Balance of statistical performance with simulation effort
Background • Basic framework: • 𝑛𝑖 i.i.d. input data for input model/distribution 𝑖 = 1, … , 𝑚 • Nonparametric (without loss of generality)
• Notations: • • • •
𝑭 = 𝐹1 , … , 𝐹𝑚 : distributions of the 𝑚 input models 𝑭 = 𝐹1 , … , 𝐹𝑚 : empirical distributions of the 𝑚 input models 𝜓(𝑭): performance measure of interest (expectation type) 𝜓(⋅): estimated performance measure from (a number of) simulation runs
• Example: 𝜓(𝑭) = expected workload in a queue driven by 𝑭 = 𝐹1 , 𝐹2 where interarrival times ∼ 𝐹1 and service times ∼ 𝐹2
Background Further notations: For positive sequences 𝑎, 𝑏 → ∞, 𝑎 = 𝑂(𝑏) iff 𝑎 = 𝑜(𝑏) iff 𝑎 = Ω 𝑏 iff 𝑎 = 𝜔(𝑏) iff
𝑎 ≤𝐶< 𝑏 𝑎 →0 𝑏 𝑎 ≥𝐶> 𝑏 𝑎 →∞ 𝑏
𝑎 = Θ 𝑏 iff 0 < 𝐶 ≤
∞ 0 𝑎 𝑏
≤𝐶
Background Simulation replication 1 Simulation replication 2
Input Model
Output analysis
Simulation replication 3
Simulation replication 𝑅 Simulation is run by generating random variates specified by distributions
Background Simulation replication 1 Simulation replication 2
Input Data
Output analysis
Simulation replication 3
Simulation replication 𝑅 Input uncertainty/variability/error arises when the input distributions are noisy estimates from data
Background Simulation replication 1 Simulation replication 2
Input Data
Output analysis
Simulation replication 3
Simulation replication 𝑅 The input uncertainty quantification problem (Henderson ‘03, Chick ‘06, Barton ’12, Song & Nelson ‘14…): Estimate confidence interval for a performance measure under both simulation and input noises
Background Total output variability
=
Simulation variability
+
Input variability
• Input data and simulation replication are of comparable sizes • Simulation cost is significant
Background Total output variability
=
Simulation variability
+
Input variability
• Input data and simulation replication are of comparable sizes • Simulation cost is significant Focus of the talk: Balance of statistical performance with simulation effort
Background • Basic framework: • 𝑛𝑖 i.i.d. input data for input model/distribution 𝑖 = 1, … , 𝑚 • Nonparametric (without loss of generality)
• Notations: • • • •
𝑭 = 𝐹1 , … , 𝐹𝑚 : distributions of the 𝑚 input models 𝑭 = 𝐹1 , … , 𝐹𝑚 : empirical distributions of the 𝑚 input models 𝜓(𝑭): performance measure of interest (expectation type) 𝜓(⋅): estimated performance measure from (a number of) simulation runs
• Example: 𝜓(𝑭) = expected workload in a queue driven by 𝑭 = 𝐹1 , 𝐹2 where interarrival times ∼ 𝐹1 and service times ∼ 𝐹2
Background Further notations: For positive sequences 𝑎, 𝑏 → ∞, 𝑎 = 𝑂(𝑏) iff 𝑎 = 𝑜(𝑏) iff 𝑎 = Ω 𝑏 iff 𝑎 = 𝜔(𝑏) iff
𝑎 ≤𝐶< 𝑏 𝑎 →0 𝑏 𝑎 ≥𝐶> 𝑏 𝑎 →∞ 𝑏
𝑎 = Θ 𝑏 iff 0 < 𝐶 ≤
∞ 0 𝑎 𝑏
≤𝐶
