RATES OF GROWTH AND SAMPLE MODULI FOR WEIGHTED ...
by
Kenneth S. Alexander
TECHNICAL REPORT No. 86 Revised June 1986
Department of Statistics, GN-22 University ()fWashiI1gtrin Seattle, W;lshington 98195 USA
Rates of Growth and Sample :Moduli for Weighted Empirical Processes Indexed by Sets Kenneth S. Alexander"
University of Washington**
Revised June 1986
ABSTRACT
Probability inequalities are. obtained for the supremum of a weighted empirical process indexed. by a Vapnik-Cervonenkis class C of sets. These inequalities are particularly useful under the assumption P(ufC E: C : P(C) < t D -? 0 as t -? o. They are used to obtain almost sure bounds on the rate of growth of the process as the sample size approaches infinity, to find an asymptotic sample modulus for the unweighted empirical process, and to study the ratio Pn/P of the empirical measure to the actual measure.
Key words and phrases: Cervonenkis class,
process, law of the modulus.
it
""~,,,t..rl
Icgarfthm, Vaonik-
60FI0
AM'S 1980 subject classifications:
Abbreviated title: Growth rates for
~mnlrl(>al processes
1 Research 5l1l!)nlllrte~d under an NSF Postdoctoral Fel]o~rship 11686, and in NSF No. DMS~301807. DelJartm.ent
No. MeS8S-
of Maithemsltics, UrJ.versitv of Southern
L Introduction
ure n s: Pn'. -- n- 1",Lli=lUX.. ,
lin;
=n
-P).
may be as a stochastic process C. C 1)d: = H-"".x] : x E $. 1171. becomes a normalized distribution tion; we call the d.f. case. Properties of 1171. in case, in related case C class ld of of ]Rd. been extensively studied. Recently, attention has been given to more general classes of sets or functions, both in the theory (Dudley (1978), Gine and Zirin (1984), Cam 1, 198 1111.
To detailed information about the behavior of 1171. on small sets. it is often helpful to weight 1171. at each set C by a function of c;2(C): = P(C)(l - P(C)), the variance of IIn (C ). Since c;2(C) '" P(C) as P(C) -7 0, this is often equivalent to weighting by the same function of P(C); when convenient we do the latter. In particular, given a nonnegative nondecreasing function q E C[O,l] and sequences 771. .... and (an). we may ask for a finite R and a sequence (on) such that
°
$ =R
P[lvn(C)1 > onq(c;2(C))
(1.2)
for someC
E
C with c;2(C) ~ 711.] .... 0,
a particular rate. ask for a functioIl gsuch that (1.1) or (1.2) some < 0 < 00, i.e.
:C -
co
E
0 define fJT to be the solution B.> 1 of fJ(logfJ - 1) fJ",,: 1. ,-1)-1 as T -l> 0, fJT . . .
=
(1.
fJT -
1 ......
- 1~
T-l>
as
for all T> 0,
=
(1.1
=
+
00,
+
=(1 -
T)/T, and
-3in the one-cnmenstonai d.f. case with P
(1.13)
): C
E:
D 1,a2(C) ~
I,.J
=R
a.s.
where t
t
R = {2(a + 1»2 and bn = (LLn)2 if n- 1LLn = o{,n) and LL,:;:;l/LLn t
t
R = max(2,T2 {P-r - 1» and bn = (LLnf2i f ,71. =
Tn-
-+
a
1LLn for all n
t
R = 1 and bn = LLn/«n,n)2L(LLn/n,n» if
(1.14)
In
=o (n-
1LLn)
Wellner (1978) showed that,
(1.15)
and LLn/L(LLn/n , n)
t eo
P uniform on [0,1), if n- 1 ~
E: D
-+
= 0(,71.) then
0 in probability,
while if
(1.16)
SU7JJll-'_(C)/P{C) - 11: C
E: D l,P{C)
In~ -eDa.s.
Note that the growth rates (bn ) in (1.13) differ from those in (1.3)-(1.6), and the cutoff levels (,n) in (1.15) and (1.16) differ from those in (1.9); we would like to understand such differences from a general point of view. In this paper we present an approach which unifies all the results (1.3)-(1.9), (1.13), and (1.15)-(1.16). We will extend them to all VC classes of sets,including extension to higher dimensions of the d.f. and interval cases, and show how to choose optimal q ,(bn ), and (,n) in general. aSjrm.1Pto,tic mqdulus for 1171. ,call Gaus:sian process Gpwhich is it'ljll 1
the weak limit of
(1.17)
lin
on C. That is, ql(t):= 1/Jl(t 2 ) should satisfy
0 < lirnsuPt ->0 sup Hcp(C)I/q 1{U 2(C» : C
E:
C, u 2 (C ) ~ t ~ < co
a. s.
The problem of findingsuchaqtwas considered in Alexander (1986). The main result can be summarized as follows. t ~ 0:
c,
:=
lC
E:
) ~ t,P(C) ~ 2 U lce : C E: C,u 2 {C) ~ t,P{C) > ~ ~
C:
Ct,s:= lC\D: C
aCt) := ) :=
t
E:
Ct , u 2(C\D) ~ s ]
-4-
give a lower bound on the number needed. This is quantified section 3 using the concept of a C. Note that since IIn (C ) = - IIn(CC) , one can often simplify matters, especially the definition of Ct , by the concntion that P(C) ;;;; 2 for all C E C, considering separately C EC with P(C) ;;;; ~
and the complement of those with p(C) > 2' This means
a 2(C) is of the order of P(C) for those sets of principal interest, i.e. those with small a 2 (C ) . For a reasonably regular VC class C. 1
ql(t):= (2t(Lg(t)+LLt- I
(1.18)
»"2
satisfies (1. 17). Suppose q ;;;; qi is given, and we wish to find growth constants (bn ) such that (1.1) or (1.2) holds. Since lin is less well-behaved on smaller sets, we might expect that roughly sup lilln (C)I/q (a 2(C» : C 2(C»
llIn (C)I
q l(a q(a 2C»
~ sup qI(a 2(C»
!
for some 0 < R
bg(t) + u for some ;:i P
+
J';:i
t
;:i
a and G E C(t)J
+ (bg(r)/n
and
- 7-
Remark 2.2. the function 9 0.-"; I.Uctn Hence for a
nonincreasing. lion. I
,or", 111""".' we assume h",'''r>C'""f"\lr1" possibility of modifying ?f makes (2.7) a
,-,,.,,,,"'''' to fol9 (or ?f) is mild condi-
-8HI. The Square Root Weight Function section may compared to lies
to obtamtng eenavior of Nz(u, C. P):
lYnn"'",
bounds
=minlk ;;;; 1: there
13).
lim sup
existCt..... Ck
miTl.;.iO:k P(C 6.C;.)
E
C such that
< u Z for all C
E C~.
The function log N z( •• C.P) is called the entropy of C in LZ(P). (Note P(C aD) /lle - 1n lIi2(p)') It measures the size of C, telling us "how totally Cis. < t &1/4)byPt(A peA () ef}/p(Et ) . Hence usingLernrna2. 7 of1\1exancier of Dudley (1978), we have for any
=
1.
(3.1)
1._1.
N Z(u t 2.Dt .P ) = N Z(u t 2P (E t ) 2,D toP t ) & 2(16g(t)u- 2L(8g(t)u-2» V(C)- t
& K(g (t )/u Z ) V(C)-
for some constant K
sets C 1.' " ,C"
E
=K(o,V(C»
1+6
for each 0 > O. the following concepiwillhe use{::Jv,'r~T sufficiently small A>· o there sumcierrtty t > a there are k ;;;; E:>.11 (t )1->..
C with (T2(C,;)
if
=t
and p(ein (Uj",,';Cj
»& A peed. of nearly-disjoint sets of any Ci's be cnosen
Theorem 3.1. Cbe a
Won:= Yn :=
c 1 := lim
SUPnWn
Cg:= lim SUPnWn
for some
-9-
A =A(6)
o and
U
E
(0,1).
(A) Then limsuPnsuPfllln(C)VbnO"(C): C E C,0"2(C) ~ I'n~
(3.4)
=R
a.s,
where
(I)
if
n -l wn
1
=0(/71.)
.1
n- 1w;
and
decreases,
then
bn
=w;J.
and
R ~ (2(PC1+C2+Cs)f2;
(li) if 771. '" Tn- 1w,; for some 1
T>
1
R ;;;; max«2(pc 1+ C2+ ca)) 2", T-Z(fia-r-
(iii) if 771. =
o(n-lwn ) , n-1Yn
1», where
J.
w; decreases, B = (pc 1+ ca)-l;
0 and n- 1
J.
then bn = W; and
decreases, and
(3.5)
=
then bn Yn and R ~ pc 1 + Cll' (B) If p ~ 1, C is full, the lim sups in (3.2) are actually limits, and (for (i) and (li) only) Lg(t )/Lr 1 is nondecreasing, then the upper bounds for R in (i)(iii) above are also lower bounds, so (i)-(m) give the true values of R in (3.4). (C) If the assumptions in (B) hold, C is spatially full, and Cll 0, the "lim sup" may be replaced by "lim" in (3.4) for each of (i)-(ui). I By (3.1), (3.3) is always valid with 7]/2 = p = V(C)- L Later however, willshow< thatthis need notb~ optimal. In fact, we will with V(C) arbitrarilylarge.
=
Remark 3.2. The condition (3.3) is related to the "relative metric entropy" condition (1.25) in Alexander (1985). In fact, is easy to verify that 1
N 2(ut 2 ' C t \C(1-u 2/4)t 'P ) ;;;; NU(u/2,C,P)
for all 0 < U < 2 and t > 0,
Wh,F'T"p
NU is defined
Alexander (1985).
aO-l
in (3.4)
of R (3.4) are as follows. C. c 1 comes .,..,.1'1,,,,1'1 must be considered; this numner of values cr 2( C) must be C3 comes requirement that sums of certain probabilities, for geometrically increasina subsequences of values of n, must be finite, as some proofs of the If C is full, it is clear that pmust be at 1 if g is unbounded, hence in T'lRrtlI""1l1Fll'" if C 1 > there Theorem 3.1 (B) arising
When (3.6)
LL-,:;;.1 = o (LLn) and Lg(-'r,J = o (LLn), 1
we see from Theorem 3.1 (i) that the value of R in (3.4) is 2'2. This is true, for
ing to the tails in the d.f. case, is the sole determiner of the rate of growth of the weighted empirical process when (3.6) holds . .Kxa.mple 3.3 p= J.
Theorem
Y1I.
=LLn/((n-'1I.)2·L(LLn/n-'rzJ), we have (3.5) 3.1. I
Th'''rll'''PTn
.c!
=
2
= aca.
and
- 11 -
In (ii}, if 71'1. . . . An- ILLn, we get
T
= "A(d -1)-1, pc 1 + c2 + Cs = (d + 1)/(d -1). and 0 = (d -1)/d.
In (iii). if (3.5) holds. i.e. if n- 1(Ln)-t = 0(71'1.) for all e > 0, the values are W n ......
(d -1)LLn.cl = l.cs = (d -1)-1. and (d -1)(PCl +cs) = d.
To see that Ud is full. :fix
(3.7)
F:
a < "A < 1 and a < t < I, and let J1.:
="Ad- 1 and
= ~[a.x]: x =(f (t)J1.-it.J1.i 2•... ,J1.id ) E [a.l]d for some integers ji G a with
it = El;'2ji~
where f (t) ~ ~ is given by ! (t )(1-! (t» = t. Then
(3.8)
IFI
=lHh.... ,jd) E ints~-1
: Et=2it ~
e« {(log r 1)/ (log J1.- 1»
d- l
(lo,g! (t)-I)/(log J1.- 1HI
1
G l:d),.g (t )
for some constants l:d and eo: where ints ... denotes the nonnegative integers. Since P{C n (VD€F.D"cCD» ;;i dp.?{C) for C E F. this shows thatD d is full. This establishes (up to the proof of (3.3» the
for d
> 1.
•
Coronary 3.5.
P be LL7;:;1/LLn
-+
on [0,
(d G 1) and let 7n.j,
a for some aGO. ~ 7n~
R=
R
+
=
R=dand
and
=
=0
)
+ =
=R
11'1. . . . 71'1.
=
a.s,
a
so that
- 12-
I Example 3.6. Let P be a nondegenerat on ]Rd, Cbv : E : X 1 they are
By Theorem 3.1 (i), if L< a
.
h t (>.) ~ 2(1-11.) for all A> O.
(6.3) and
ht(X)
·.·I·v4
if X ~ 4 >. ~ 4.
~(LX)/2.if
Bennett's ill equality (Bennett, 1962) tells us that 1
1
FElvn (C)! >M] ~ 2exp(-Mn 2ht(M/n 2u2(C»)
(6.5)
for all M ~ 0 and all C. Hence (6.4) and Theorem 2.8 of Alexander (1984) give us the following. Proposition 6.l. Let C be a VC class of sets. n ~ 1. M > O. and a ~ supc u 2(C). There exists a constant Ko (V(C» s uc h that if ....... ".,.. ""' . . 1
1
M 2 ~Koet.L(n/a) and M ~ KoL (n / a )/ n 2L (M/ n 2 a}
(6.5) or (li)
(6.7) then 1
1
(5.8) F[supclvn(C)!>M] ~·16exp(-M2/6a}+ 16exp(-lMn2L (M/n 2 a}}. (6.8) corresponds to a 1.
If M/n 2a Poisson
(2.8) hold. q{a»" r for all j ~ 1. ~ 0:
.
>
tj+t
~
)+u
approximation. the
the Gaussian approximation is domdominates.
Proof of Theorem 2.1.
.N ,,-
I
d. and r:' := f or some ., ~
t
~
a and C €
- 21 -
> bq(t)
for some 7 ~ t < r' and C E
>
+
for
E
+
:= pCC}
pCO).
We
~U'pplJ8e 7 ~
.!
t < t:', C
.!
C(t),
E .!
Ivn(C)1 = n 2 p (C ) ~ 2n 2 0'a(C ) ~ 2n 2 q (t )/ z (t) ~ bq(t).
It follows that (6.10) p(o) ~ P[Pn (C) > 0 for some C nP (Er ·)
;?;
E:
C(t) and
"I;?;
t < r']
;?; p.
Turning next to the
pP}, we have
(6.11) pp} ~
+ P[lvn(.E't;)1 > ~ bq ._ p{2}
.-
j
+ p{3) j'
Fixj and let k h k a be nonnegative integers such that
IVn (.E'dl J
(6.12)
1
~ a1 bq (t j )fj(tj) a if and only if k 1 ~ nPn (JJ: t . ) ~ k a. J
Then
(6.13) De:fin.e a new prooamnta measure pti(.)
=P(-IEt)
and let P tj := (pti)OO be the corresponding product measure on(X OO , (tOO). (C(ti ) is easily shown to be deviation-measurable for ptj , since it is deviation(6.14)
ptj(C)
=P(C)/p(Et.) and J
O'~(C) := pti(C)(l- ptj(C» ~ O'a(C)/p(E\) Fix k , kl;?;, k
~ ka,
and define 14: := k
C E 1
ap
k 1
_ kn -aptj (C
= --=-~...,-'-IJ-'
k
n
;?;
fj(t j )-1 for C E: C(tj
).
Now P« to e(tj}'
k]
P n.lC(t . )
= Ft,[kPk:. E.], 1
(6.15) F[suPc(tj)lvn (C}l > ~ o.q (t i }lnPn (E\)
the restriction of
l
=k]
1 =Fti[suPc(tj) n £2( kPk:.(C) n -P(C)}I > Z-bq(ti } ]
J
=Ft.[suPc(t.)!(k/n)
1
s
> ; (n/k)Zoq(ti ) ]·
1
1
2
__1,. t
,uk:. (C) + len 2p i(C) -n 2P(C)!
1
> -2,bq (t i
)]
1
1
We now wish to apply Proposition6.! with M:= ;(n/kyabq(tj), and g(tj)-t in the role of the
€X
there (see (6.14».
Case 1: 1
z(t j
)
~ 2n z/ b , i.e.
(6.16)
= Hence by (2.2),
M2 ·.~b2q2(tj }/32~(tj)g(tj )3~.Kg(tj )-tL'fj(tj)
(6.18)
by
.In 4
(na(tj
)}
~ KLk .
Proposition 6.1 (Ii), and use (6.17) and
If K (6.18) to get for ~
(6.19)
so
(6.
of (6.11), )/za~(tj»
(6.20)
+ 1
- 23-
.
:i 2exp (- b 2 g2(t j )/28t(tj)}
since, by (6. the (6.11) and (6.20), we see
FP>:i
(6.22)
of hi above is at most for the Fi O of (6.9),
Combining
1
1
18exp (-b 2g 2(t j)/zSt(tj» + 18exp(-2-8n 2bg (t j )L(bg(tj )/n2t(t j») .1
+ 34exp(-2-8n 2bg(tj )L'i/(tj ». (The last term in (6.22) is superfluous now but will be used Iater.) Case 2; 1
z(t j ) > 2n 2/b. Then r ~ t j < 5, so (2.4) holds, and .1
( 6 . 2 3 ) b g ( t j) >211. 2t(t j).
We wish to use Propositioni~.l(i). To prove (6.6) it issuificient to show that, for the constant Ko of that proposition, .
1
1
M2'i/(tj) ~ ~Mk-iL(M?/(tj)/k2)~KoL(k'i/(tj»'
(6.24)
We need two subcases, according to which of the two terms added in (6.17) is the larger. Case 2a;
(6.25)
bg(t j)
~
.1
.1
8n2t(tj)'i/(tj) 2 .
again valid. By (6.17), (6.23),.an.d (2.4), 1
M'fj(tj)/k2
1
=n 2bg(t")'i/(tj)/4k .1
(6.26)
~ bg(tj)/Bn2t(tj)
(6.27)
~ (1/4)
v(K/Bnt(tj».
1
BY(6.2'7),.M9(ti)7lc-a~~L(M?/
and the .flrst inequality in (6.24) follows. The second follows, if K is large enough, from the following inequalities, which are consequences of (6.17), of (2.4) and (6.27), and of (6.26) and (2.5), respectively. L(k'i/(tj»~ L(2na(t j)'i/(tj
»
» ~ L (2/nt(t j + 2L (na(tj » ~ 4 1
(6.28)
1
)/k 2) ~ ~ n 2bg(tj)L(bg
»
- 24argument of hl in (6.21) is at most 1, so by (6.5) Thereforeso does (6.22).
(6.4), (6.21) again holds.
Case 2b: 1
1
bq{t j) > Bn2t(tj)g{tjf2.
(6.29)
Here, by (6.12) and (6.29), (6.30)
lc 1 n ~ a{t j) + an
.1
bq{t j)g{tj)2 ~
1
4n
-.1
.1
2bq(t j)g(tj)2.
It follows that 1
Mg{tj)/lc 2
(6.31)
1
1
=n 2bq(tj)'g{tj)/4lc ~ g(tj)2 ~ 1,
so the :first inequality in (6.24) holds. in (6.31), if large>enoughthen
(6.30), (2.4), and the :first inequality
1
KoL(lcg(tj »
KoL(~n-ibq(tj»+ ~KoLg(tj) 1
~ 3~n2bq(tj)Lg(tj) 1
1
~ ~ Mk 2L(Mg(tj )/lc 2).
Thus we have (6.24). From Proposition 6.1 (I), (6.24), and (6.31) we conclude that 1
1
P1~4) ~ 16exp(-~M2g(tj» + 16exp(-~Mlc2L(Mg{tj)/lc2» 1 .••
1
;i
3Zexp(- 3~Mlc2L(Mg(tj)/lc2»
;i
3Zexp (-z-8n 2bq (t j )Lf/(tj »
.1
so by (6.13), 1
pF') ti3Zexp(-Z-,an 2bg{tj)Lg(tj
(6.32)
».
To bound pP) we use (6.5) to conclude that (6.33) P}3);i Zexp (- ~ bq (tj )f/(tj f~n~hl(bq (tj )/Bn ~t(tj )g(tj yli» ;i
nct\I1.IUI
Zexp
--:--00
(6.22) holds. it to sum
estabnsned (6.22) (t (t
- 25monctoructty of g{t)/r;{t) and q{t)fJ- l9'i{t) (which follows t i + 1 ~ t,/2 and 9'i(t j + 1) ~ 2-(1-P>!Pi(t j
(6.34)
)
(2.7», we
(j
0 small enough so 262 +.2r- 16 < e. Fix rn , (1-6 2 )n < m ~ n. If (Xit ... ,x.m) EAm • then there exists C = C(Xl' ... ,x.m)EC with
11lm(C}! >bmq(u2(C» + Urn ~ (1- ( 2 )(bn q {u2(C»
(7.9)
Cand
here we have
(7.10) by the dejjnition ~
(1-
~
(1-
1, ••.
1
» +Un)
,x.m)
=
Un)
- 28For n(k):= [(1+0 2/2)k] (the integer part), we have by (7.8) that 2:k ; O. Then there exists TJn ~ 0 such that
=
whenever (7.12)
k
= {1+A)np, kn;:;i k:
;:;i
In, 0
0 such that P[lIn(C) >.lvf] ~ exp(-(1+e).lvf2/2a2(C»
whenever n
~
1-
1,.lvf2 ~ Ka2(C), and M/n 2a2(C)
;:i
I
'-0.
Proof of Theorem 3.1 (A).
It is easily
that
(i)..(iii). It follows that _1
1'nl~(nb~n) ,2
=
0
(1).
From the Kolmogorov 0-1 law we then conclude that the lim sup in (3.4) is some constant a.s, By Lemma 7.2, to prove (A) it suffices to show that for each 0 >0,
P-[llIn(c)1 > (1+8o)R obnl7(C) for some C
(7.17)
€
Cwith a2(C) ~ 1'11.]
where R 0 J. R o aso ~ 0 and R(J is Ro (i), (ii), or (iii) we are 0 (1+4o)R obnt//2 for
some C
FU))
E
i=O
Ne",
:=
l:.Pj+Pn *'
i=O
We now consider separately the cases (i)-(iii) of (3.4). Proof of (A) (I). Here we take an == 1/4. Un. == u for some 0 < u < 0 to be specified later, and R o := (2«P + o)(c 1 + 0) +
J. c2
+
Ca
+ 20»2. 1
n
-+
Now a2(C) ;;;; t i for C E F(j), and maxiW", wJ/2/n 2ti 00 since n- 1wn = 0 (-Yn). so by (6.5). (6.2), and (7.18).
-+
0
as
2IF(j)le~(-{1+40)RSwn/2)
;;;; exp (-[(1+4o)RS/2 - (p+o)(c 1+0)]Wn) ;;;; e~(-(1+o)(c2+ca+2o)wn)' SinceNn ;;;; (log (1-u 2/4}-1)-1 £-y:;;1, it follows that N".
(7.21)
l: Pi s Nne~(-(1+o)(C2+C3+2o)wn)
1==0
;;;;
e~{-(l+o)LL7l.).
'rneerem 2.1, (7. g(t) w n / 4u 2n
"'nT~"'T",nT
from
-y := -Yn' of Th~:>nl"prn 2.1, aCt) u- 2g (t) (see Remark 2.2), 0 (-Yn). so r > s (2.5) 1; if we u ;;;; 6RoK- 1/2 b := "'... (J.....",
=
for =0
771.
t
1
(')''11)
~
for all t
=o «nWn Y2)
and
Yn, so _J.
Kt 2L{na{t»
~Kt
_.1
_.1
2L{nt) +Kt 2Lg(t)
for all t ~ On, and (2.3) follows. u 2 < 0 2/512,
~
.1
6R 6(nwn)2
Theorem 2.1 now tells us that, if we take
1/4
JPn •
s
36
J
t-1exp(-02R~n/512u2)dt
"''''/''2
.1
+ 6Bexp(-oR 6{nYn wn)2/256)
~
140exp (-{l+o)(ca+o)Wn)
~
140exp{-{1+6)LLn).
With (7.20) and (7.21) this proves (7.17), and (A) (i) follows.
Proof of (A) (li). Here we take Un
o
for
.1
with
R 6 := max(R 61,Roz)
to bevspeeified later, and
1
R 62 := T2(Pe(o)T-1), where B(6) := {(P+ 6)(Cl + 6) + Ca+ 0)-1. We take (7.22)
CXn
J. 0
= o(an) and U{y;l cx·nJ = o(w,J.
7'11
.-
. Since
=
Lg(an} ~Lg(7n) ~ L(cx.;;:ta(cxn » + L(y;ICXn) Lg(an) + o(wn ), we have W n "" wn and n- 1wn 0 (cxn ) . It is then VPl"ift,,'rl stants C1. Cz. Ca are unchanced if Yn is replaced by an in (3.2).
=
above proof of
(7.23)
(A) (i) (C)! >
) for some C
E C with
= sumces to barilla
>
and
R o1 := (2«P+O)(Cl+0)+Cz+ca+26»2
for some C
E
a2(C) ~
CXn
1
with
- 32-
~ 2IF(j)!exp( -(1+40)2R12wnAi';hl(A.,J).
Set
Tn := n7nw~1
and
t := fJa{o)'T' -
Tn
1.
-l>
T,
(1+40)R o2 T n ~ (1+30)t. From (1.11) we know that thl(t)
we
have
=
=(8{0)T)-1.
It follow
that
A.,,,h 1(A.,J ~ >..rn2{1+30 )t h l(t ) = (1 +30)Tn/( 1+40)2R l2 B(O)T ~ (1+Zo)/(1+40)2Rl2 B(o)
so by (7.24) and (7.18), N",
2J ZIF(j)Jexp(- (1+ZO)8{0)-1U/n)
1=0
~
exp(-(1+6)LLn)
»
since Llln = O{LL(7,;l exn = O. In combination with (7.20) and (7.25) this proves (7.17), and (A) (ii) is proved.
Proof of (A) (iii). This time we later, and take Cln
Un
=n- 1wn
=(n7n/wn)stfor
some (large) p.. > 0 to be specified
and
R o := (p + 6)(c 1 + 6) + C3 + O. Set wn := ) w'"n1/2 -- 0 (Yn, for some C
E C
(ii) it SUIIlCI~S to bound
of
= since Nn
=
(6.2). Now LU;;1 = Q (tlJn) by (3.5), so (7.25), (7.29) (7.18) by
h I(An) '" O(LU;;2+LL
~
and
e::cp(-(l+o)LLn).
Once again Theorem 2.1 will provide the needed bound on P n ". As before we take (t):= u1t, b := oR 6Yn, 1':= 1'11" a:= an, g(t}:= t 1/2, and Crt) as (7.19), ),g(t)
so
=
(nt )-l/2L(nt) ~ (nl'nJ
z(t)=U;; 2t
_l.
_l.
".»
=Ynw;;lL(wn/n'YrJL (ni'n'>
2 L(nl'n)
~ Ynw;;I(4/-L}-lL(U;;4}Lwn ~ (4/-L}-l ynL(wn/nu,Jt).
For the second term,
(ii). LL (27;;,lcx n )
now
apply
=LL (Z'Y;;,ln -lwn) =
esraonsned as in the and use the fact that 1 to obtain, if /-L is large enough,
l.
l.
+ 36e::cp(-Z-8oR 6Yn(n'Yn )2L (oR 6Yn/(n '111, )~u1»
~
140exp
(A) by
(3.5),
- 35The result now follows as the proof of (A) (ii). I We introduce now some notation and preliminary calculations for use in the proofs of the next three propositions. Let R > 0 and 6, A,J,L E (0,1) be constants to be specified later. C be a full VC class. For each t E (0,1/4] let D t c C with eA!l(t}l-A ~ cr(C)
=
lD t
~ eA!l(t}l-A
t, P(C} ~ 1/2, and P(Cfi(UJ)€Dt,D"oCD»
+
1, and
s AP(C)
for all C EDt,
where ex ~ 1/8 is the constant in the definition of "full". Let (i'71.)' (an), and (on) be nonnegative sequences with 7'11. ~ an ~ 1/4 and n V 2bnQ(7n) .....
(7.31)
00,
Let (n(k ),k ;;;: 0) be a strictly increasing sequence of integers with nCO) set
= 0,
and
m(k} :=n(k} - n(k -1}
n~)
Y.I:{C) :=
lc(Xi )
i=n{.I:-l)+l 1
S.I:(C)
:=
Y.I:(C) - m(k)P(C)
1
= n(kfivn(.I:)(C)
- n(k
-lf2 Vn (.I: _ l )( C )
t.l:j := an (.I: )p/, N.I:
= minU ;;;: 0
: tk.j+l
N'1e := minU ;;;: 0 : t.l: j ~
< 7n(k)L
7J,f(16j,
1.1: := Hi,i): N'.I: ~ i ~ Nk> 1 r(k,i)
si
~
IDtkilL
:=IIltkil,
and observe that D tki
= ~Ckji : 1 s i
;;;; r(k,i)~
for some sets C.l:ji. Note the k indexes the number of sample points, j indexes the sizes of the sets, and i indexes the collection of sets correspondto each and i. The Ckji are nearly di:.;joint for fixed k and i : we wish to replace them with fully disjoint sets Dk ji • Define G'.l:ji :=
C.l:ji
D'kji :=
C.l:ji \G'leji'
G
fi (Um"oiCkjm),
:= D 'kji fi (Ul:>j Um:lir(.I:.l)Cktm),
while P(UiC"kfd ~
2tkLr(k,l)
~ EL>f(2e>..a(tkL)1-.A.t,Q
+ 2tkd
~ EL>f (2e>..a(tkj)1- Att,p!'(L-f)
+ 2t kjp!'(L-j»
~ 2(e.A. + 1)p!'(1- f.LA)-l a(tkj )l-Atk~'
If,
as
we
henceforth
assume, f.L is chosen small enough so 2(e.A.+1)f.L.A.(l-~tl;;i>..e>..(l->")/2, it follows that P(UiC"kji);;i>"P(UiD'kjd/2. Hence for fixed k,j, for at least half the values of i we have P(C"kfi) ;;i AP(D'kfi)' By reducing E:.A. and ID t,l;jl by half if necessary, we may assume this is valid for all i; it then follows from C P(Dkji)·~ (1-
(1- 2>,,)P(Ckjd, so
(7.32)CT~(Dkji).~(1 ...
2>..P(Ckji).
Observe also that (7.33)
Ckji
=Dkji U Gkji
as a disjoint union, and
Gkji () (UL~j Um~r{k.j,.nkLm.)
= ¢.
".'
We now define events 1
A kji := [Sk(Ckji) ~ (l-26)Rn (k)2b n{k)q (a2(C kji»] A'kji :=
1
Ekji := [V n(k-l)(Ckji) ~ -oR (n(k )/n(k -1»2b n{k)q(a2(Ckji »]
r,
:= u{j,i)€h"Ei.ji
B k ..-
B' k
U{j.i)e:Ik A 'kji'
Note that A and A "kfi imply Akj i, that Akji and E kji together imply that Vn{k)(Ckji) ~ (1-3o)Rb n{k)q(cr(Ckji». Thus (7.34)
lim SUPnSUPlVn(C}/bn{k)q(cr(C» : C
C, 7n
E
a.s.
i.o. } Sk are mdependent,
EP
so we
)
= 1,
(7.35)
P(Fk i.o.) .LVl.'LU"
)=00, EP
= O.
if
< 00,
;;i cr(C} ~
Cln
$
Ukj := [Yk (Hkj) ~ Zkj] Zkj
by
=6Rn(k)l/2on{k)Q(tkj)/2.
16Atkj (zJcj - m ( k )P (H kj »
(7.37) k and
""
B 'k by
E
T I (",, ) := minU : "" EA'kji for some
is.
T e(",,):= minfi: "" EA'kTt{t..l)iS,
and let T I = T e = CX) off B'k. If 5k is large on Dkji, it is probably also large on because D kji is most of Ckji by (7.32). To make this precise, we will show that
Ckji ,
P(BkIB'k,Ukj,(TitTe) = (j,i»
(7.38)
~ P(A"kjiIB
T z)
Once (7.38) is establishe(f, we have 2P(BJ.., then continue the present calculation. Proposition 7.9. Let C be a full VC class and q
E
Q, and suppose
1
q(t)/t 2
(7.43)
,l.,
q(t}/(tLt-l}l/2 t, Lg(t}/Lr 1 t.
Let (7n), (an), (b n) be nonnegative sequences satisfying and 1
(7.44)
bnq(7n}/n27n
-+
O.
Suppose the following limits exist and are finite: (7.45)
c 1 := lilTl.n 7nLg(7n}/b.fq2(7n) c2
:
=lilTl.n YnLL (y;;: 1ex'll. }/b.fq 2( Yn)
Ca:= lilTl.n YnLLn/b.fq2(7n}. Then (7.46)
Proof.
o < 0 < 1 and
V:=
R=
assume R > O.
, we ccntinue our
, >..=
-.;aJLI,.; U.LaLJLUU
takmg
].
Set u(k):= e>..g(7n{k»1-0/8,N(k) :=Nk -N'k
Since tg(t) and Lg(t)/Lr 1 increase, for j
+ I.
we have
;E N'k
r(k,j);E eA!J(tkj)l->";E eA!J()'~(f{16)1-0/16 G u(k).
Hence by (7.42) and (7.47),
IfW'k
> o then
so while similarly
;E (-1 + (o/16log J.l-l) log (7;;(k)(Xn{k») v L
Since 7;;(k)CXn{k) ~
00
0, it follows that c2(1-o/4)b,r{k)q2(7n{k»/7n{k)
i::51Illi1arJ.y since logpk-2 •
For i ;EN'k Hoetfdulg (1963)
"rlttf>/2.
J
rtence by (7.50),
we see
Using this and Theorem 1 of
- 408P(Bk ) ~ N(k)u(k}Pk A 1.
(7.52) (7.49) we obtain
(7.53)
N(k luCk )Pk = £).N(k )g(7n(k)P-6/B exp( -(1-0/4)R2b~(k)q2(1n(k»/27n(k» ~ £).exp( -(1-o/4)csb~(A:)q2(7n(k»/7n(k» ~
e).exp(-(1-o/8)LLn(k})
~
£).k-(1-6/16).
=
With (7.52) this shows EP(B k ) 00. To establish (7.36) it remains 7n(k) ~ t ~ £k := 7~(f(16,
to
bound P(Fd.
By
(7.43),
for
and ~:2: q2(7n(k» t ,n(k)
Lei 1 :2: q2(7n(k» L';;(k) - 2,n(k) .
Also N(k) ~ N k + 1 ~ (log j.l-1)-11 0 !1. (' ;;(k )(Xn(k » + L
Combining these facts with (6.5), (6.4), (7.44), and (7.43) we obtain P(Fk.) . .~.E(j.i)~h.2er[J(--o2VIi2~~(k)q2(a2{C~ji»/tk;2·(Ck#}}
~ Ef:N'1; 4g (t kj )exp (-4R2b~(k)q2(tkj)/tkj) ~ E:'~N'I;4exp (- (B 2+2c 2+2c s)b~(k )q2(t~j )/tkj ) ~ 4N{k)exp(-(R 2/2+C2+ Cs)b;(k)q2(,n{k»/,n(k»
~
4exp(-(5/4)LLn(k})
and (7.36), and (7.34), follow. Since 0 may be arbitrarily small, this TI1"'I"1V"""Z the proposition. •
Remark 7.10.
C4:=
lim
- 41 one, for us to obtain some lower bound on ccrresponding lim
I Proposition 7.11. Let C be a full VC class. Let 7T'" w n , Cit and Ca be as in Theorem 3.1 and 0:= (Cl+Ca)-l, and suppose 771. "" Tn- 1W n for some T> O. Suppose the lim sups in (3.2) are actually limits. .1
lim sUPnsupHvn(C)I/w~(T(C): C E C, a2(C)
(7.54)
= 771.5
1
;;;; T
2 ({JeT - 1)
a.s,
Proof. q (t)
A :::
-w 1/'2", _ .... 11.
take
0 and set Y« * := 1n v aJ+6. We wish to apply Proposition 7.9. Consider the sequences in (7.45): since Lg(t)/Lt- 1 increases, Lg l(1n *) ~ (l+o)Lg l(an) so li'mn.1~Lg(1~)/qr(1:)=
Cl
lim infn1:LL«1~)-lan)/qf(1:) ~ (1+0)-lc2
Since 0 is arbitrary, Proposition and Remark 7.10 prove (u). For (iii), by increasing 1n we may assume 1n an' The proof is then just like that of Theorem 3.1 (C), since C2 = 0 and C 1 = 1 whenever Cs O. I
=
=
Proof of Theorem 4.1. localasym.ptotic modulus at ¢for (v n ) . To show 1/10 is an asymptotic modulus of let 1n,an .!. 0 with nan t, 1n ~ an, n-1Ln 0 (1n), LLn O(La~l). It suffices in (4.5) to consider C,D ~P(CAD Let D:= : C,D E C, 1n/2;:;i cf4(C\D) ~ an! : C.IJ EC, 1n ;:;i ) ;:;i We g(t) := t- l ) ,we
=
=
EC
» :C
E
E.1n ~
: C ED
- 46-
I
so the result follows from Theorem 4.2.
Proof of Theorem 4.4. We use 2.1, with C{t) := Ct. ((t):= t , q{t):= '¢'1{t 1/2 ) , 1:= 711.' and 0::= we use p:= 0.) r = 711. > S (2.1), n- 1Lg{7n) e (7n). If b is large enough then (2.2) is clear, (2.3) follows easily from the observation that
=
L{na{t)) ~ L{nt) + Lg{t),
(7.65)
and (2.4) and (2.5) are vacuous. Hence (2.6) holds. If b is large then exp{-bZqz(t )/512t) ~ b- 1{Lt - 1)Z, so the second term on the right side of (2.6) 1
can be made small. Since q{1n)n 2 ~ (n1n)VZ small for large n,and the theorem fellows.
-'> 00
as n
-'> 00,
the third term is
I
Proof()fTheorems 5.1 arid 5.2. Observe that for M > 0, Pn{C) . P[supfl P{C) -
(7.66)
11 : C
E
C,p{C) ~ 111. ~ > M]
1
~ P[lvn{C)1 > Mn 2a2{C) for some C
E
Cwith a2{C) ~1n/2].
Thus to prove the desired results we use Theorem 2.1 with C{t):= Ct> 1
and b := Mn 2". If (2.2)-(2.5) hold then
({t) := t , 1:= 111./2, 0::= 1/4, (2.6) bounds the of
68exp ( -u« 1n/256)
(7.67)
= 0(exp(-M2n1n/1024» + 0(exp{-.Mn1n/256».
In (2.1) the values
(7.68)
if
s=
r=
M
~
2
To prove (5.2), we take M fixed but arbitrarily small in (7.66). Then (5.1), that n1n -'> 00 and (2.3). (2.4) and (2.5) are (7.65), and the vacuous by (7.68), so (5.2) (7.67). If (5.3) .u."'.1."'-'", same
O({Ln)-Z), and a.s. C011verlH:::I1c:e ~ n- 1
~ 2,
(7.69)
(C) for some C E C with P n S1J.'lI)l-~":-
-
1/ : C
E
C,P{C)
~ R1~j
Pn{C) F 0, ) F
>
are vacuous.
(7.66), vacuous. If 9
is
t, then all e > 0 by
bounded,
supfPn(C)/P(C) : C € C, P(C)
;S e/~S
(5.7), while for R > 0, P[sup~Pn.(C)/P{C) : C € C,P(C) ~ P[supfPn (C)
:C
€
< e/~S > R]
CP(C) < e/~S > 0]
~ naky~) ~ nAe/~ ~ AeLA
-l'
0 as e
-l'
0,
and the last statement in Theorem 5.2 follows. It remains to prove (5.5) when C is full and (5A) holds. By (5.4) we have Un) for all n some so if wedefinet'n to be the solu7=
then -''11. for some 0
;l;;
(22d)d-1 U-2(d-l)g(t),
I
=
6
- 49-
(8.1) Ct \C(I-'Ul3/4)t
=iCatl : r
~
b < r\
v E sa-IS u
iCCu : -r- < b ~
-r, v E sa-IS
Let
two vectors
=
(8.2)
~
MO-(d-l)
for some o,i! depending only on d. We now prove (3.3). specify 0 := u Z/ 16r . Let C E C t \C(I-u 2/4)t ' It is clear that there is awE Sd-l for which C C Crw and
(8.3) P(C ACrw }
~ P(Crw ) - P(Cr ' w )
=f (t) -
f
«1-u z/ 4)t ) ~ u Zt / 2.
Since V is maximal, there is a v E V making an angle ex can show that
~
0 with w. Suppose we
(8.4) With (8.3) and (8.2) I
M un-(d-l) s; 16d-IMu-Z(d-l)rd-1 N Z(ut 2 ,Ct\C (1-u2/4)t' P) s; .
Since tfJ-l(l-f(t» "" (2Lf(t)-l)VZ as t
-)0
0, there exists K= K(d) such that
K-1g (t) ~ r
(8.5)
d- 1
and (3.3) follows. The equality in (8.4) is clear. Since
H
(j
~
Kg (t)
etc; \Crw )
depends only on r and the
T closed half .plane bounded by lz and disjoint from Crw , and W the wedge Crv with vertex at TV. Then
(8.6) (3.3) follows
=tfJ(r) -
tfJ(r cos ex)
~
r(1-cosex)exp{-rZ(cos2ex)/2) ~ r 02 exp (r 2 0 z/2) exp (-r z/2)/2 ~
~ u Z( 1- tfJ(r» / 8 ~ u Zt / 4. TV
we obtam
(8.8)
peW)
=
Combining (8.6). (8.7). and (8.8) proves (8.4). and (3.3) follows. To show C is full we use similar ideas. but change 8 to (16dLr )V2/r . Fix A E (0.1) and take b = b(d) large enough so
v.w since the angle between If xECrv nCrw for v and w is at least 8. we have IIxl1 2 ~ r 2 + r 2tan2( 8/ 2). so IIx!l ~ r(1+8 2/16). It follows using (8.9) and (8.10) that
etc;
n
(UW€v.w,",vCrw)
~
POx:
~
b (2r )d-2 exp (-r 282/16)exp (-r 2/2)
Ilxll ~ r(l+02/16)D
Since by (8.2) and (8.5).
IVI ~ 0 8-{d-i) ~ or tt - 1/(H3dLr ){tt-i)/2 .~ Eg(t )1-],. for some constant
thatC is f u l l . ;
E
P 1. u E (0.1) (0,1/4] and set r := and r := . Let Zl+ denote nonnegative For each j ,k E Zl~ ~ r 131;/" for alIi ~ d and E}i, ~ rr, define ,bik E 1Jd by ,
numoer of rectangles
b 'tjk
...--
J --.............. 13 ·
1.
+13
] )
(t
}'i.
:=
.- max)!,
r
for
i ~ d.
Now 2r- 1(Vi -Wi) ~ a?7t: ~ Vi and [aik.b ik] and Vi -
so [v.w]
C
Wi
~ bi7c ~
P([aik .b i k ] ) ~ (1+5r- 1)dp ([v ,w])
(3.3) now follows.
I
Wi
+ 3r- 1(Wi -Vi).
P([v ,w]) + u 2t.
KS.: ineeualtties for empizieal processes and a law of the iterated l,..,r'::I7'iH'rrl Ann. Probability. 12 Alexander, KS.: Rates of growth weighted Proceedings of the Berkeley Conference in honor of Jerzy Neyman and Jack Kiefer, Vol. 2 (L. LeCam and R. Olshen, eds.) Belmont, CA: Wadsworth 1985. Alexander, KS.: Sample moduli for set-indexed Gaussian processes. Ann. Probability 14, 598-611 (1986).
Breiman,L., Friedman, J.E., and Stone, C.J.: Regression Trees. Belmont, CA: Wadsworth 1984.
Classification and
Csaki, E.: The law of the iterated logarithrn for normalized empirical distribution function. Z. Wahrsch. verui. Cebieie 38, 147-167 (1977). Diaconis, P. and Freedman, D.: Asymptotics of graphical projection pursuit. Ann. Statistics 12, 793-815 (1984).
Probability 1, 66-103 (1973). Dudley, RM.: Central limit theorems for empirical measures. Ann. Probability 6,899....929 (1978).
P.: Empirical Processes. IMS Lecture Notes - Monogra:ph Series 3 (1983).
Hoertdmg, W.: J.
sums
- 53Jogdeo, ~an:lue>Ls, S.M.: MOJ:lotlone convergence of bin,orrual prclbalolli1:1es and a Ann. Math. 39, 1 -1195 (1968).
J.: Pn. ,J. O. Proc. 6th Berkeley Sym/pos. Math. Statist. Probab. Vol. 1, 227-244. Berkeley: Univ. of (1972). LeCam, A remark on measures. A Festschrift for Erich L. Lehmann Honor of 65th Birthday Bickel, K. Doksum, and J. Hodges, editors). Belmont, CA: Wadsworth (1983). Mason, D.M., Shorack, G.R., and Wellner, J.A.: Strong limit theorems for oscillation moduli of the uniform empirical process. Z. Wahrsch. ueru: Gebiete 65, 83-97 (1983). Orey, S. and Pruitt, W.E.: Sample functions of the N-parameter Wiener process. Ann. Probability 1, 138-163 (1973). Pollard, D.: A central limit theorem for empirical processes. J. Austral. Math. Soc. (Ser. A) 33, 235-248 (1982). Pollard, D.: Convergence of Stochastic Processes, New York: Springer Verlag (1984). Shorack, G.R. and Wellner, J.A.: Limit theorems and inequalities for the uniform empirical process indexed by intervals. Ann. Probability 10, 639-652 (1982). Stout, W.: Almost Sure Convergence. New York: Academic (1974). Stute, W.: oscillation behavior of empirical processes. Ann. Probability 10, 86-107 (1982a). Stute, W.: bility 10,
IrHT"",.,l>n'lTl
for 1r""1"',.,,,,, 1 deIlsit.y estsmatcrs Ann. Proba-
Wellner, J.A.: Limit theorems for the the empirical distrfbution tion to the distribution function. Z. Wahrsch. verui. Gebiete 45, (1978). Yukich, J.:
Laws of large numbers for classes of functions.
Analysis 17, 245-260 (1985).
Department of Statistics University of Washington Seattle, WA 98195
J. Multivar.
