y« p - Semantic Scholar
IEEE COMMUNICATIONS LETTERS, VOL. 13, NO. 11, NOVEMBER 2009
817
Estimation of a Probability with Guaranteed Normalized Mean Absolute Error Luis Mendo
Abstract—The estimation of a probability p from repeated Bernoulli trials is considered in this letter. A sequential approach is followed, using a simple stopping rule. A closed-form expression and an upper bound are obtained for the mean absolute error of the unbiased estimator of p. The results given permit the estimation of an arbitrary probability with a prescribed level of normalized mean absolute error. Index Terms—Monte Carlo methods, sequential mean absolute error, simulation.
estimation,
sensitive to outlying values. Apparently, its lack of use is in large part motivated by the analytical difficulty associated with the absolute value [6] [7] [8]. I I . RESULT
Let
0
and
be defined as N
no I.
S
2(N -
estimation of a probability from a set of observations is considered in this letter. This problem arises, among other fields, in Monte Carlo simulation of communication systems, in which performance is usually measured by a bit error rate (BER) or block error rate (BLER). Contrary to fixed-size Monte Carlo methods, in sequential estimation the sample size is not fixed in advance, but is (randomly) determined by the outcome of the simulation, using a certain stopping rule. In this letter, the observations are assumed to be a sequence of independent Bernoulli trials with probability of success p, which is to be estimated. A simple stopping rule known as inverse binomial (or negative binomial) sampling is considered. This rule consists in observing the sequence until a given number N of successes is reached. The resulting number of trials is denoted as n. (Random variables are displayed in bold type throughout the letter.) The uniformly minimum variance unbiased estimator of p, for N > 2, is [1]
Q-N
EQUENTIAL
N-Í •
(1)
n - 1 For N > 3 the mean square error (MSE) of (1) is known to satisfy E[(p - p)2]/p2 < 1/(N - 2) irrespective of p [1]. Recent works [2] [3] have shown that, for the modified estimator p = (N - l ) / n , the confidence level associated with a relative interval of the form [p//x2,i>/xi] also satisfies a lower bound irrespective ofp, for N > 3 and /xi, /¿2 not smaller than certain values. The same result holds for the estimator (1), albeit for a reduced range of /xi,/x2 values [4]. The referred bound can be improved by allowing estimators of the form p = il/(n + d), where Í1 and d are selectable parameters [5]. This letter analyzes the mean absolute error (MAE) of the estimator (1), for N > 2. Compared to the M S E , the M A E is a more natural error measure, and has several advantages [6] [7]. It is simpler, it has a clearer meaning, and it is less Manuscript received May 25, 2009. The associate editor coordinating the review of this letter and approving it for publication was C. Charalambous. E. T S. Ingenieros de Telecomunicacio´n, Polytechnic University of Madrid, 28040 Madrid, Spain (e-mail: [email protected]). Digital Object Identifier 10.1109/LCOMM.2009.091128
l
(2)
+ 1,
P
INTRODUCTION
, P=
-
l)N~2e~N+1 (N-2)\
(3)
For N > 2, the MAE of (1) satisfies the following. E(|p-p|) p
lim p^0
n0
=2
E(|p-p|)
l
N
P
\n0-N+l
K
(4)
(5)
O.N.
p
Furthermore, E(|p and function of
p\)/p
E(|p-p|)
< aN
is a monotonically decreasing
for all p e (0,1).
(6)
V
Proof: Let E(|p - p\)/p be denoted as e(p). Given N, the probability (mass) function of n, /jv(n) = Pr[n = n], is
-\y«
ÍN{n) =
-N
p)
> N.
(7)
The corresponding distribution function is denoted as f/v(n). Using the identities ÍN(n)
pjW_i(n-l) N-Í
i
E
i
(8)
p
n - l
(9)
N-V
the MAE is computed as no
E(|p-p|)=
fN(n)
i
Í N - Í
\ 1 N - Í
ÍN{n) ( n o
2
no
1
2p[FN_1(n0-l)
1089-7798/09$25.00 c 2009 IEEE
Authorized licensed use limited to: Univ Politecnica de Madrid. Downloaded on November 18, 2009 at 04:42 from IEEE Xplore. Restrictions apply.
N
OO
E ÍN{n) E
ni=N
=
v ^ ÍN(n)
ÍN{n)
n=N
(
1
n
n=no + l
ÍNH
n=N
FN(n0)}.
(10)
818
IEEE COMMUNICATIONS LETTERS, VOL. 13, NO. 11, NOVEMBER 2009
Let bn¡p(i) denote the binomial probability function with parameters n, p evaluated at i. Taking into account that FN-i(no
- 1) = FN(n0) + (1 -p)bno-ltP(N
10 N-2
•
'
- 1), (11) N
from (10) it is seen that (12)
:(p) = 2 ( l - i j ) 6 n o _ i , p ( J V - l ) ,
which establishes (4). The limit result (5) follows from Poisson’s theorem [9, p. 113]. Let S = {(N-l)/k, k = N,N+l,...}andT = (0, l)\S. For p e T, no does not change if p is altered by a sufficiently small amount, which implies that e is continuous and differentiable, with dg(p) AvN-2(l = 2 no -p)^-N(N-l-n0p). (13) dp l)P N Substituting (2), this expression is seen to be negative. Let p e S, i.e. p = (N - l)/k for some k = N, N + 1,.... Although no has a jump discontinuity at every point of S, the function e is continuous, because N-Í V A:
lim e /i->-0-
=2
A; TV
lim e /i->-0+ )
(JV
l)"-i(k-N+l)k
k-N+l
kk
1
(14)
In addition, e has left and right derivatives at p = (N l)/k; these are given by (13) replacing no by k +1 or k respectively, with the result that the left derivative is negative and the right derivative is 0. The function e is thus continuous with negative derivative on T and nonnegative one-sided derivatives on S. This implies that e is monotonically decreasing, and (6) follows. It is interesting to note that the monotonicity of e(p) and the result (6) for p e S can also be established using a similar procedure to that in [3]. For these values of p, (2) simplifies to no = (N - l)/p + 1. Defining x = - In ——,
(15)
the inequality (6) is equivalent to x > 0. It follows that N-2
,
x= — V l n ( 1
ip )
pfy V
(16)
_ I K—1 _ N +2 ln(l -p)- ——-. P P P x The variable x can be written [3] as ^3=0 o jP^ with
N-20
\
x.
- •
-¿
10
-£
0-e
Fig. 1.
-A
0
10 -3 p
10
Normalized MAE as a function of
0
-i
0-
10
.
Let yj denote the right-hand side of (19). Computing dyj/dj as if j were a continuous variable, it is seen that dyj/dj > 0. Thus yj > yo = 1/(N — 1) > 0 for any j > 1, which implies that all the coefficients Xj are positive. Therefore x > 0 for N > 2, and xp = J2"L0 Xjpi+1 is increasing on S, from which e(p) = ajs¡e^xp is decreasing on S. • III.
DISCUSSION
The result above allows the estimation of a probability p with a prescribed value of the normalized M A E , E( \p —p\)/p. This value is guaranteed irrespective of the unknown p. For example, if a normalized M A E not exceeding 10% is desired, N = 65 suffices, according to (6). The behaviour of the normalized M A E as a function of p is depicted in Fig. 1, with solid lines. The curves show the decreasing character of the normalized M A E . Its nondifferentiability at the points p = (N — l)/k, k ef$ (see proof of the result in Section II) can also be clearly observed, specially for low N and large p. Figure 2 shows the bound ajv as a function of N. The bound for the root mean square error (RMSE) normalized by P, i.e. E[(p-ri 2 ] — 1 (20) < A /N 2 p
is also shown for comparison. Both error measures are seen to have the same type of behaviour, with MAE lower than ^V 2^ -1+1 N-l N-2 1 iv — 1 ÍV — ¿ RMSE. Xj J . (17) It is interesting to compare (4) with the normalized MAE (J —— 1 —-—- > t -\ l)(N-l)i+ ^i j +2 j +l resulting from a fixed sample size n. In this case, denoting by For N = 2, Xj reduces to l/(j + 2), and is thus positive. For TV the random number of successes, the unbiased estimator N > 3, substituting the inequality p = N/n has a normalized MAE given by [10, eq. (1.1)] N 2
~
i AT
^
oy'+2
(18)
j +2
¿=1
P
into (17) gives /
(j + l)(j + 2)xj > (N-2)
E [\p-p\]
-.
11
_
J+l
AT-l)
+ J-N + 3. (19)
=2 (
n
1 »Ar°~1(l
V^o - 1
•sn-No + l
(21)
with /Vo = \np\ +1. Since the average sample size in inverse binomial sampling is N/p, the comparison is restricted to probabilities p such that N/p is an integer value, and the sample size n in the fixed case is taken equal to this value. The
Authorized licensed use limited to: Univ Politecnica de Madrid. Downloaded on November 18, 2009 at 04:42 from IEEE Xplore. Restrictions apply.
MENDO: ESTIMATION OF A PROBABILITY WITH GUARANTEED NORMALIZED MEAN ABSOLUTE ERROR
I IV s 7 MAE /p
I V . CONCLUSION i
The M A E in the estimation of a probability by means of inverse binomial sampling has been characterized. It has been shown that the estimator guarantees a certain value of the normalized M A E irrespective of the unknown . This allows to a priori select a value of the parameter that meets a prescribed level of normalized error. The result is quite general, and has many potential applications. In particular, it can be used in simulation and analysis of communication systems, where the performance metric is typically the probability of a certain event.
>
10
V. -£
10
819
10
0
10
ACKNOWLEDGMENT
The author wishes to thank J. M . Hernando for his valuable help, and the anonymous reviewers for their useful comments.
N
Fig. 2.
Bounds on normalized MAE and RMSE as a function of
.
REFERENCES [1] G . Prasad and A . Sahai, “Sharper variance upper bound for unbiased estimation in inverse sampling,” Biometrika, vol. 69, no. 1, p. 286, 1982.
resulting fixed-size normalized MAE is shown in Fig. 1 with dashed lines. Dividing (4) by (21) with = / , it is easily seen that, for 0, the MAE with inverse binomial sampling +1 is asymptotically ( 1 + 1/( -1))times larger than the MAE with fixed sample size. This value is close to 1 except for very small values of . This is observed in Fig. 1, which also shows that the MAE ratio is approximately maintained for all values of . It is thus concluded that, in order to guarantee a given normalized MAE, inverse binomial sampling gives an average sample size that is only slightly larger than the sample size that would be necessary in the fixed case (the latter being a function of the unknown ). Possible extensions to this study are: analyzing the effect of imposing a deterministic bound on the number of observations; considering other stopping rules that may be less conservative for not close to 0; and replacing the assumption of independent Bernoulli trials by other distributions for the observed variables.
[2] L . Mendo and J. M . Hernando, “A simple sequential stopping rule for Monte Carlo simulation,” IEEE Trans. Commun., vol. 54, no. 2, pp. 231–241, Feb. 2006. [3] ——, “Improved sequential stopping rule for Monte Carlo simulation,” IEEE Trans. Commun., vol. 56, no. 11, pp. 1761–1764, Nov. 2008. [4] ——, “Unbiased Monte Carlo estimator with guaranteed confidence,” in IEEE International Workshop on Signal Processing Advances in Wireless Communications, 2008. [5] ——, “Estimation of a probability with optimum guaranteed confidence in inverse binomial sampling,” to appear in Bernoulli J. [6] C . R . Blyth, “Expected absolute error of the usual estimator of the binomial parameter,” The American Statistician, vol. 34, no. 3, pp. 155– 157, Aug. 1980. [7] S . Gorard, “Revisiting a 90-year old debate: the advantages of the mean deviation,” British J. Educational Studies, vol. 53, no. 4, pp. 417–430, 2005. [8] S . K . Bar-Lev, B . Boukai, and P. Enis, “On the mean squared error, the mean absolute error and the like,” Commun. in Statistics - Theory and Methods, vol. 28, no. 8, pp. 1813–1822, 1999. [9] A. Papoulis and S. U. Pillai, Probability, Random Variables and Stochastic Processes, 4th ed. McGraw-Hill, 2002. [10] P. Diaconis and S . Zabell, “Closed form summation for classical distributions: variations on a theme of De Moivre,” Statistical Science, vol. 6, no. 3, pp. 284–302, 1991.
Authorized licensed use limited to: Univ Politecnica de Madrid. Downloaded on November 18, 2009 at 04:42 from IEEE Xplore. Restrictions apply.
817
Estimation of a Probability with Guaranteed Normalized Mean Absolute Error Luis Mendo
Abstract—The estimation of a probability p from repeated Bernoulli trials is considered in this letter. A sequential approach is followed, using a simple stopping rule. A closed-form expression and an upper bound are obtained for the mean absolute error of the unbiased estimator of p. The results given permit the estimation of an arbitrary probability with a prescribed level of normalized mean absolute error. Index Terms—Monte Carlo methods, sequential mean absolute error, simulation.
estimation,
sensitive to outlying values. Apparently, its lack of use is in large part motivated by the analytical difficulty associated with the absolute value [6] [7] [8]. I I . RESULT
Let
0
and
be defined as N
no I.
S
2(N -
estimation of a probability from a set of observations is considered in this letter. This problem arises, among other fields, in Monte Carlo simulation of communication systems, in which performance is usually measured by a bit error rate (BER) or block error rate (BLER). Contrary to fixed-size Monte Carlo methods, in sequential estimation the sample size is not fixed in advance, but is (randomly) determined by the outcome of the simulation, using a certain stopping rule. In this letter, the observations are assumed to be a sequence of independent Bernoulli trials with probability of success p, which is to be estimated. A simple stopping rule known as inverse binomial (or negative binomial) sampling is considered. This rule consists in observing the sequence until a given number N of successes is reached. The resulting number of trials is denoted as n. (Random variables are displayed in bold type throughout the letter.) The uniformly minimum variance unbiased estimator of p, for N > 2, is [1]
Q-N
EQUENTIAL
N-Í •
(1)
n - 1 For N > 3 the mean square error (MSE) of (1) is known to satisfy E[(p - p)2]/p2 < 1/(N - 2) irrespective of p [1]. Recent works [2] [3] have shown that, for the modified estimator p = (N - l ) / n , the confidence level associated with a relative interval of the form [p//x2,i>/xi] also satisfies a lower bound irrespective ofp, for N > 3 and /xi, /¿2 not smaller than certain values. The same result holds for the estimator (1), albeit for a reduced range of /xi,/x2 values [4]. The referred bound can be improved by allowing estimators of the form p = il/(n + d), where Í1 and d are selectable parameters [5]. This letter analyzes the mean absolute error (MAE) of the estimator (1), for N > 2. Compared to the M S E , the M A E is a more natural error measure, and has several advantages [6] [7]. It is simpler, it has a clearer meaning, and it is less Manuscript received May 25, 2009. The associate editor coordinating the review of this letter and approving it for publication was C. Charalambous. E. T S. Ingenieros de Telecomunicacio´n, Polytechnic University of Madrid, 28040 Madrid, Spain (e-mail: [email protected]). Digital Object Identifier 10.1109/LCOMM.2009.091128
l
(2)
+ 1,
P
INTRODUCTION
, P=
-
l)N~2e~N+1 (N-2)\
(3)
For N > 2, the MAE of (1) satisfies the following. E(|p-p|) p
lim p^0
n0
=2
E(|p-p|)
l
N
P
\n0-N+l
K
(4)
(5)
O.N.
p
Furthermore, E(|p and function of
p\)/p
E(|p-p|)
< aN
is a monotonically decreasing
for all p e (0,1).
(6)
V
Proof: Let E(|p - p\)/p be denoted as e(p). Given N, the probability (mass) function of n, /jv(n) = Pr[n = n], is
-\y«
ÍN{n) =
-N
p)
> N.
(7)
The corresponding distribution function is denoted as f/v(n). Using the identities ÍN(n)
pjW_i(n-l) N-Í
i
E
i
(8)
p
n - l
(9)
N-V
the MAE is computed as no
E(|p-p|)=
fN(n)
i
Í N - Í
\ 1 N - Í
ÍN{n) ( n o
2
no
1
2p[FN_1(n0-l)
1089-7798/09$25.00 c 2009 IEEE
Authorized licensed use limited to: Univ Politecnica de Madrid. Downloaded on November 18, 2009 at 04:42 from IEEE Xplore. Restrictions apply.
N
OO
E ÍN{n) E
ni=N
=
v ^ ÍN(n)
ÍN{n)
n=N
(
1
n
n=no + l
ÍNH
n=N
FN(n0)}.
(10)
818
IEEE COMMUNICATIONS LETTERS, VOL. 13, NO. 11, NOVEMBER 2009
Let bn¡p(i) denote the binomial probability function with parameters n, p evaluated at i. Taking into account that FN-i(no
- 1) = FN(n0) + (1 -p)bno-ltP(N
10 N-2
•
'
- 1), (11) N
from (10) it is seen that (12)
:(p) = 2 ( l - i j ) 6 n o _ i , p ( J V - l ) ,
which establishes (4). The limit result (5) follows from Poisson’s theorem [9, p. 113]. Let S = {(N-l)/k, k = N,N+l,...}andT = (0, l)\S. For p e T, no does not change if p is altered by a sufficiently small amount, which implies that e is continuous and differentiable, with dg(p) AvN-2(l = 2 no -p)^-N(N-l-n0p). (13) dp l)P N Substituting (2), this expression is seen to be negative. Let p e S, i.e. p = (N - l)/k for some k = N, N + 1,.... Although no has a jump discontinuity at every point of S, the function e is continuous, because N-Í V A:
lim e /i->-0-
=2
A; TV
lim e /i->-0+ )
(JV
l)"-i(k-N+l)k
k-N+l
kk
1
(14)
In addition, e has left and right derivatives at p = (N l)/k; these are given by (13) replacing no by k +1 or k respectively, with the result that the left derivative is negative and the right derivative is 0. The function e is thus continuous with negative derivative on T and nonnegative one-sided derivatives on S. This implies that e is monotonically decreasing, and (6) follows. It is interesting to note that the monotonicity of e(p) and the result (6) for p e S can also be established using a similar procedure to that in [3]. For these values of p, (2) simplifies to no = (N - l)/p + 1. Defining x = - In ——,
(15)
the inequality (6) is equivalent to x > 0. It follows that N-2
,
x= — V l n ( 1
ip )
pfy V
(16)
_ I K—1 _ N +2 ln(l -p)- ——-. P P P x The variable x can be written [3] as ^3=0 o jP^ with
N-20
\
x.
- •
-¿
10
-£
0-e
Fig. 1.
-A
0
10 -3 p
10
Normalized MAE as a function of
0
-i
0-
10
.
Let yj denote the right-hand side of (19). Computing dyj/dj as if j were a continuous variable, it is seen that dyj/dj > 0. Thus yj > yo = 1/(N — 1) > 0 for any j > 1, which implies that all the coefficients Xj are positive. Therefore x > 0 for N > 2, and xp = J2"L0 Xjpi+1 is increasing on S, from which e(p) = ajs¡e^xp is decreasing on S. • III.
DISCUSSION
The result above allows the estimation of a probability p with a prescribed value of the normalized M A E , E( \p —p\)/p. This value is guaranteed irrespective of the unknown p. For example, if a normalized M A E not exceeding 10% is desired, N = 65 suffices, according to (6). The behaviour of the normalized M A E as a function of p is depicted in Fig. 1, with solid lines. The curves show the decreasing character of the normalized M A E . Its nondifferentiability at the points p = (N — l)/k, k ef$ (see proof of the result in Section II) can also be clearly observed, specially for low N and large p. Figure 2 shows the bound ajv as a function of N. The bound for the root mean square error (RMSE) normalized by P, i.e. E[(p-ri 2 ] — 1 (20) < A /N 2 p
is also shown for comparison. Both error measures are seen to have the same type of behaviour, with MAE lower than ^V 2^ -1+1 N-l N-2 1 iv — 1 ÍV — ¿ RMSE. Xj J . (17) It is interesting to compare (4) with the normalized MAE (J —— 1 —-—- > t -\ l)(N-l)i+ ^i j +2 j +l resulting from a fixed sample size n. In this case, denoting by For N = 2, Xj reduces to l/(j + 2), and is thus positive. For TV the random number of successes, the unbiased estimator N > 3, substituting the inequality p = N/n has a normalized MAE given by [10, eq. (1.1)] N 2
~
i AT
^
oy'+2
(18)
j +2
¿=1
P
into (17) gives /
(j + l)(j + 2)xj > (N-2)
E [\p-p\]
-.
11
_
J+l
AT-l)
+ J-N + 3. (19)
=2 (
n
1 »Ar°~1(l
V^o - 1
•sn-No + l
(21)
with /Vo = \np\ +1. Since the average sample size in inverse binomial sampling is N/p, the comparison is restricted to probabilities p such that N/p is an integer value, and the sample size n in the fixed case is taken equal to this value. The
Authorized licensed use limited to: Univ Politecnica de Madrid. Downloaded on November 18, 2009 at 04:42 from IEEE Xplore. Restrictions apply.
MENDO: ESTIMATION OF A PROBABILITY WITH GUARANTEED NORMALIZED MEAN ABSOLUTE ERROR
I IV s 7 MAE /p
I V . CONCLUSION i
The M A E in the estimation of a probability by means of inverse binomial sampling has been characterized. It has been shown that the estimator guarantees a certain value of the normalized M A E irrespective of the unknown . This allows to a priori select a value of the parameter that meets a prescribed level of normalized error. The result is quite general, and has many potential applications. In particular, it can be used in simulation and analysis of communication systems, where the performance metric is typically the probability of a certain event.
>
10
V. -£
10
819
10
0
10
ACKNOWLEDGMENT
The author wishes to thank J. M . Hernando for his valuable help, and the anonymous reviewers for their useful comments.
N
Fig. 2.
Bounds on normalized MAE and RMSE as a function of
.
REFERENCES [1] G . Prasad and A . Sahai, “Sharper variance upper bound for unbiased estimation in inverse sampling,” Biometrika, vol. 69, no. 1, p. 286, 1982.
resulting fixed-size normalized MAE is shown in Fig. 1 with dashed lines. Dividing (4) by (21) with = / , it is easily seen that, for 0, the MAE with inverse binomial sampling +1 is asymptotically ( 1 + 1/( -1))times larger than the MAE with fixed sample size. This value is close to 1 except for very small values of . This is observed in Fig. 1, which also shows that the MAE ratio is approximately maintained for all values of . It is thus concluded that, in order to guarantee a given normalized MAE, inverse binomial sampling gives an average sample size that is only slightly larger than the sample size that would be necessary in the fixed case (the latter being a function of the unknown ). Possible extensions to this study are: analyzing the effect of imposing a deterministic bound on the number of observations; considering other stopping rules that may be less conservative for not close to 0; and replacing the assumption of independent Bernoulli trials by other distributions for the observed variables.
[2] L . Mendo and J. M . Hernando, “A simple sequential stopping rule for Monte Carlo simulation,” IEEE Trans. Commun., vol. 54, no. 2, pp. 231–241, Feb. 2006. [3] ——, “Improved sequential stopping rule for Monte Carlo simulation,” IEEE Trans. Commun., vol. 56, no. 11, pp. 1761–1764, Nov. 2008. [4] ——, “Unbiased Monte Carlo estimator with guaranteed confidence,” in IEEE International Workshop on Signal Processing Advances in Wireless Communications, 2008. [5] ——, “Estimation of a probability with optimum guaranteed confidence in inverse binomial sampling,” to appear in Bernoulli J. [6] C . R . Blyth, “Expected absolute error of the usual estimator of the binomial parameter,” The American Statistician, vol. 34, no. 3, pp. 155– 157, Aug. 1980. [7] S . Gorard, “Revisiting a 90-year old debate: the advantages of the mean deviation,” British J. Educational Studies, vol. 53, no. 4, pp. 417–430, 2005. [8] S . K . Bar-Lev, B . Boukai, and P. Enis, “On the mean squared error, the mean absolute error and the like,” Commun. in Statistics - Theory and Methods, vol. 28, no. 8, pp. 1813–1822, 1999. [9] A. Papoulis and S. U. Pillai, Probability, Random Variables and Stochastic Processes, 4th ed. McGraw-Hill, 2002. [10] P. Diaconis and S . Zabell, “Closed form summation for classical distributions: variations on a theme of De Moivre,” Statistical Science, vol. 6, no. 3, pp. 284–302, 1991.
Authorized licensed use limited to: Univ Politecnica de Madrid. Downloaded on November 18, 2009 at 04:42 from IEEE Xplore. Restrictions apply.
