New Error Bounds for Approximations from ... - Semantic Scholar
Joint Technical Report of U.H. and M.I.T. Technical Report C-2008-43 Dept. Computer Science University of Helsinki
and
LIDS Report 2797 Dept. EECS M.I.T.
July 2008; revised July and Dec 2009 To appear in Mathematics of Operations Research
New Error Bounds for Approximations from Projected Linear Equations Huizhen Yu∗ [email protected]
Dimitri P. Bertsekas† [email protected]
Abstract We consider linear fixed point equations and their approximations by projection on a low dimensional subspace. We derive new bounds on the approximation error of the solution, which are expressed in terms of low dimensional matrices and can be computed by simulation. When the fixed point mapping is a contraction, as is typically the case in Markov decision processes (MDP), one of our bounds is always sharper than the standard contraction-based bounds, and another one is often sharper. The former bound is also tight in a worst-case sense. Our bounds also apply to the non-contraction case, including policy evaluation in MDP with nonstandard projections that enhance exploration. There are no error bounds currently available for this case to our knowledge.
∗ Huizhen † Dimitri
Yu is with HIIT and Dept. Computer Science, University of Helsinki, Finland. Bertsekas is with the Laboratory for Information and Decision Systems (LIDS), M.I.T.
1
Error Bounds for Projected Linear Equations
2
Contents 1 Introduction
3
2 Main Results 2.1 Proofs of Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 8
2.2
Comparison of Error Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 On the Error Bound of Theorem 2.2 . . . . . . . . . . . . . . . . . . . . . . . . . .
11 12
2.2.2
12
On the Error Bound of Theorem 2.1 . . . . . . . . . . . . . . . . . . . . . . . . . .
3 Related Results 3.1 3.2
Error Bounds in Decomposition Forms for Projected Equations . . . . . . . . . . . . . . . Error Bound for an Alternative Approximation Method . . . . . . . . . . . . . . . . . . .
4 Applications 4.1
4.2
15 15 18 20
Cost Function Approximation for MDP . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
4.1.1 4.1.2
Discounted Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Average Cost and Stochastic Shortest Path (SSP) Problems . . . . . . . . . . . . .
20 24
4.1.3 Optimal Stopping Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Large General Systems of Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . .
25 26
5 Estimating the Low-Dimensional Matrices in the Bounds
26
6 Conclusion
28
References
29
Error Bounds for Projected Linear Equations
1
3
Introduction
For a given n × n matrix A and vector b ∈
and
LIDS Report 2797 Dept. EECS M.I.T.
July 2008; revised July and Dec 2009 To appear in Mathematics of Operations Research
New Error Bounds for Approximations from Projected Linear Equations Huizhen Yu∗ [email protected]
Dimitri P. Bertsekas† [email protected]
Abstract We consider linear fixed point equations and their approximations by projection on a low dimensional subspace. We derive new bounds on the approximation error of the solution, which are expressed in terms of low dimensional matrices and can be computed by simulation. When the fixed point mapping is a contraction, as is typically the case in Markov decision processes (MDP), one of our bounds is always sharper than the standard contraction-based bounds, and another one is often sharper. The former bound is also tight in a worst-case sense. Our bounds also apply to the non-contraction case, including policy evaluation in MDP with nonstandard projections that enhance exploration. There are no error bounds currently available for this case to our knowledge.
∗ Huizhen † Dimitri
Yu is with HIIT and Dept. Computer Science, University of Helsinki, Finland. Bertsekas is with the Laboratory for Information and Decision Systems (LIDS), M.I.T.
1
Error Bounds for Projected Linear Equations
2
Contents 1 Introduction
3
2 Main Results 2.1 Proofs of Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 8
2.2
Comparison of Error Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 On the Error Bound of Theorem 2.2 . . . . . . . . . . . . . . . . . . . . . . . . . .
11 12
2.2.2
12
On the Error Bound of Theorem 2.1 . . . . . . . . . . . . . . . . . . . . . . . . . .
3 Related Results 3.1 3.2
Error Bounds in Decomposition Forms for Projected Equations . . . . . . . . . . . . . . . Error Bound for an Alternative Approximation Method . . . . . . . . . . . . . . . . . . .
4 Applications 4.1
4.2
15 15 18 20
Cost Function Approximation for MDP . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
4.1.1 4.1.2
Discounted Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Average Cost and Stochastic Shortest Path (SSP) Problems . . . . . . . . . . . . .
20 24
4.1.3 Optimal Stopping Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Large General Systems of Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . .
25 26
5 Estimating the Low-Dimensional Matrices in the Bounds
26
6 Conclusion
28
References
29
Error Bounds for Projected Linear Equations
1
3
Introduction
For a given n × n matrix A and vector b ∈
