Nonparametric Independence Testing for Small Sample Sizes
Nonparametric Independence Testing for Small Sample Sizes Aaditya Ramdas, Leila Wehbe (IJCAI-2015)
Zolt´ an Szab´ o
Machine Learning Journal Club, Gatsby Unit April 4, 2016
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing. Idea: 1
large cov (X , Y ) ⇒ declare dependence.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing. Ideas: 1 2
large cov (X , Y ) ⇒ declare dependence. large supf ∈F,g ∈G cov (f (X ), g (Y )) ⇒ dependence nice asymptotic results.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing. Ideas: 1 2
large cov (X , Y ) ⇒ declare dependence. large supf ∈F,g ∈G cov (f (X ), g (Y )) ⇒ dependence nice asymptotic results.
Focus: small sample size, small false positive regime: ’avoid’ false dependence detection.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing. Ideas: 1 2
large cov (X , Y ) ⇒ declare dependence. large supf ∈F,g ∈G cov (f (X ), g (Y )) ⇒ dependence nice asymptotic results.
Focus: small sample size, small false positive regime: ’avoid’ false dependence detection.
Trick: introduce some bias to reduce variance - Stein. large shrunk[ supf ∈F,g ∈G cov (f (X ), g (Y ))] ⇒ dependence
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: independence testing problem
i .i .d.
Given: {(xi , yi )}ni=1 ∼ PXY . Marginals of PXY : PX , PY . Hypotheses: H0 : PXY = PX × PY ,
Zolt´ an Szab´ o
H1 : PXY 6= PX × PY .
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: independence testing problem
i .i .d.
Given: {(xi , yi )}ni=1 ∼ PXY . Marginals of PXY : PX , PY . Hypotheses: H0 : PXY = PX × PY ,
H1 : PXY 6= PX × PY .
Aim: 1
Low type-I error = P(detect dependence, when there isn’t any) ≤ α, {z } | false positive
2
High power = P(detect dependence, when there is).
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: cross-covariance X ∈ (X, k), Y ∈ (Y, ℓ), k, ℓ: kernels. RKHSs: Hk , Hℓ .
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: cross-covariance X ∈ (X, k), Y ∈ (Y, ℓ), k, ℓ: kernels. RKHSs: Hk , Hℓ . Mean embedding and its empirical counterpart: µX = Ex∼PX k(·, x), | {z }
µY = Ey ∼PY ℓ(·, y ), | {z }
i =1
i =1
=:φ(x)
µ ˆX =
1 n
n X
φ(xi ),
Zolt´ an Szab´ o
=:ψ(y )
µ ˆY =
1 n
n X
ψ(yi ).
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: cross-covariance X ∈ (X, k), Y ∈ (Y, ℓ), k, ℓ: kernels. RKHSs: Hk , Hℓ . Mean embedding and its empirical counterpart: µX = Ex∼PX k(·, x), | {z }
µY = Ey ∼PY ℓ(·, y ), | {z }
i =1
i =1
=:φ(x)
µ ˆX =
n X
1 n
φ(xi ),
=:ψ(y )
µ ˆY =
1 n
n X
ψ(yi ).
Cross-covariance: ΣXY = E(x,y )∼PXY [φ(x) − µX ] ⊗ [ψ(y ) − µY ] : Hℓ → Hk , | {z } | {z } ˜ =:φ(x)
SXY =
1 n
˜ ) =:ψ(y
n X
[φ(xi ) − µ ˆX ] ⊗ [ψ(yi ) − µ ˆY ].
i =1
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ?
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ? Cb (X) and Cb (Y) would be sufficient: Jacod and Protter 2000.
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ? Cb (X) and Cb (Y) would be sufficient: Jacod and Protter 2000. Trick [Gretton et al. ’05]: guarantee the denseness of Hk in Cb (X), Hℓ in Cb (Y).
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ? Cb (X) and Cb (Y) would be sufficient: Jacod and Protter 2000. Trick [Gretton et al. ’05]: guarantee the denseness of Hk in Cb (X), Hℓ in Cb (Y). Space: compact metric, kernel: universal X
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ? Cb (X) and Cb (Y) would be sufficient: Jacod and Protter 2000. Trick [Gretton et al. ’05]: guarantee the denseness of Hk in Cb (X), Hℓ in Cb (Y). Space: compact metric, kernel: universal X Examples: ′ 2
k(x, x′ ) = e −γkx−x k2 ,
′
k(x, x′ ) = e −γkx−x k1 .
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Side-note
ΣXY ∈ HS(Hℓ , Hk ) =: HS(G, F). What does this mean? Extension of Frobenious norm.
kC k2F =
X
Cij 2
i ,j
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Side-note
ΣXY ∈ HS(Hℓ , Hk ) =: HS(G, F). What does this mean? Extension of Frobenious norm.
kC k2F =
X
Cij 2 ,
i ,j
kC k2HS
=
X
hCgj , fi i2F < ∞,
i ,j
where C : G → F bounded linear operator. G, F are separable Hilbert spaces with ONBs {gj }j , {fi }i .
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HS operator example: f ⊗ g
Intuition: fgT . (fgT )u = f (gT u). | {z } =hg,ui
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HS operator example: f ⊗ g
Intuition: fgT . (fgT )u = f (gT u). | {z } =hg,ui
Outer product: f ⊗ g (f ∈ F, g ∈ G) (f ⊗ g )(u) = f hg , uiG , ∀u ∈ G.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HS operator example: f ⊗ g
Intuition: fgT . (fgT )u = f (gT u). | {z } =hg,ui
Outer product: f ⊗ g (f ∈ F, g ∈ G) (f ⊗ g )(u) = f hg , uiG , ∀u ∈ G. HS norm of f ⊗ g : kf ⊗ g k2HS = hf , f iF hg , g iG . Cross-covariance: made of f ⊗ g -type quantities.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HSIC 2 It is easy to compute kΣXY kHS =: HSIC.
HSIC =
kΣXY k2HS
=
*
+ n n X 1X˜ 1 ˜ i ), ˜ j ) ⊗ ψ(y ˜ j) φ(xi ) ⊗ ψ(y φ(x n n i =1
Zolt´ an Szab´ o
j=1
HS
Nonparametric Independence Testing for Small Sample Sizes
HSIC 2 It is easy to compute kΣXY kHS =: HSIC.
HSIC =
kΣXY k2HS
=
=
*
+ n n X 1X˜ 1 ˜ i ), ˜ j ) ⊗ ψ(y ˜ j) φ(xi ) ⊗ ψ(y φ(x n n
1 n2
i =1 n D X
j=1
HS
E
˜ i ) ⊗ ψ(y ˜ i ), φ(x ˜ j ) ⊗ ψ(y ˜ j) φ(x HS {z } | i ,j=1 ˜ ˜ ˜ ˜ ˜ ˜ φ(x ), φ(x ) ψ(y ), ψ(y ) K L = h i j iH h i j iH ij ij k ℓ
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HSIC 2 It is easy to compute kΣXY kHS =: HSIC.
HSIC =
kΣXY k2HS
=
*
+ n n X 1X˜ 1 ˜ i ), ˜ j ) ⊗ ψ(y ˜ j) φ(xi ) ⊗ ψ(y φ(x n n
1 n2
i =1 n D X
j=1
HS
E
˜ i ) ⊗ ψ(y ˜ i ), φ(x ˜ j ) ⊗ ψ(y ˜ j) φ(x HS {z } | i ,j=1 ˜ ˜ ˜ ˜ ˜ ˜ φ(x ), φ(x ) ψ(y ), ψ(y ) K L = h i j iH h i j iH ij ij k ℓ E D 1 ˜ ˜ = 2 K, L . n F
=
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HSIC 2 It is easy to compute kΣXY kHS =: HSIC.
HSIC =
kΣXY k2HS
=
*
+ n n X 1X˜ 1 ˜ i ), ˜ j ) ⊗ ψ(y ˜ j) φ(xi ) ⊗ ψ(y φ(x n n
1 n2
i =1 n D X
j=1
HS
E
˜ i ) ⊗ ψ(y ˜ i ), φ(x ˜ j ) ⊗ ψ(y ˜ j) φ(x HS {z } | i ,j=1 ˜ ˜ ˜ ˜ ˜ ˜ φ(x ), φ(x ) ψ(y ), ψ(y ) K L = h i j iH h i j iH ij ij k ℓ E D 1 ˜ ˜ = 2 K, L . n F
=
˜ = HKH, H = In − 1 11T , L ˜ = HLH. K n Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Independence test using HSIC [Gretton et al. 2005]
Given: samples and α ∈ (0, 1). Test statistics: T = HSIC = kΣXY k2HS . Simulated null distribution of T : via {y1 , . . . , yn } permutations ⇒ tα . Decision: reject H0 if tα < T .
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Observation
SXY is unbiased estimator of ΣXY : E[SXY ] = ΣXY . Issue: SXY can have high variance for small sample numbers.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Observation
SXY is unbiased estimator of ΣXY : E[SXY ] = ΣXY . Issue: SXY can have high variance for small sample numbers. Idea [Stein, 1956]: decrease the variance by adding some bias.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Observation
SXY is unbiased estimator of ΣXY : E[SXY ] = ΣXY . Issue: SXY can have high variance for small sample numbers. Idea [Stein, 1956]: decrease the variance by adding some bias. [Maundet et al. 2014]: 2 shrinkage based estimators.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Observation
SXY is unbiased estimator of ΣXY : E[SXY ] = ΣXY . Issue: SXY can have high variance for small sample numbers. Idea [Stein, 1956]: decrease the variance by adding some bias. [Maundet et al. 2014]: 2 shrinkage based estimators. Questions 1 How do they perform in independence testing? 2
Optimality?
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Variations: shrinking towards zero Recall: SXY =
1 n
Pn
SXY =
˜
i =1 φ(xi ) ⊗
˜ i) ⇒ ψ(y
n
2 1 X
˜ ˜ i) − Z
.
φ(xi ) ⊗ ψ(y HS Z ∈HS(Hℓ ,Hk ) n
arg min
i =1
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Variations: shrinking towards zero Recall: SXY =
1 n
Pn
SXY =
˜
i =1 φ(xi ) ⊗
˜ i) ⇒ ψ(y
n
2 1 X
˜ ˜ i) − Z
.
φ(xi ) ⊗ ψ(y HS Z ∈HS(Hℓ ,Hk ) n
arg min
i =1
SCOSE (simple covariance shrinkage estimator, λ > 0): n
2 1 X
˜ 2 S ˜ i) − Z SXY = arg min
φ(xi ) ⊗ ψ(y
+ λ kZ kHS . HS Z ∈HS(Hℓ ,Hk ) n i =1
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Variations: shrinking towards zero Recall: SXY =
1 n
Pn
SXY =
˜
i =1 φ(xi ) ⊗
˜ i) ⇒ ψ(y
n
2 1 X
˜ ˜ i) − Z
.
φ(xi ) ⊗ ψ(y HS Z ∈HS(Hℓ ,Hk ) n
arg min
i =1
SCOSE (simple covariance shrinkage estimator, λ > 0): n
2 1 X
˜ 2 S ˜ i) − Z SXY = arg min
φ(xi ) ⊗ ψ(y
+ λ kZ kHS . HS Z ∈HS(Hℓ ,Hk ) n i =1
FCOSE (flexible covariance shrinkage estimator): F = SXY
n X βj j=1
n
˜ j ) ⊗ ψ(y ˜ j ), φ(x
2
n n X X
β 1 j ˜ i ) ⊗ ψ(y ˜ i) − ˜ j ) ⊗ ψ(y ˜ j ) + λ kβk2 .
φ(x φ(x β = arg min 2
n n β
j=1 i =1 HS
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
SCOSE vs FCOSE
In both cases: λ is chosen via leave-one-out CV. SCOSE: analytical formula for λ∗
2
S HSIC S = SXY
= 1 − HS
Zolt´ an Szab´ o
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC
(n − 2)HSIC +
1 n
Pn
i =1
n
˜ ii L ˜ii K
2
HSIC . +
Nonparametric Independence Testing for Small Sample Sizes
SCOSE vs FCOSE
In both cases: λ is chosen via leave-one-out CV. SCOSE: analytical formula for λ∗
2
S HSIC S = SXY
= 1 − HS
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC
(n − 2)HSIC +
1 n
Pn
i =1
n
˜ ◦L ˜ [O(n3 )], ’/λ’: O(n2 ). FCOSE: after SVD of K
Zolt´ an Szab´ o
˜ ii L ˜ii K
2
HSIC . +
Nonparametric Independence Testing for Small Sample Sizes
SCOSE vs FCOSE
In both cases: λ is chosen via leave-one-out CV. SCOSE: analytical formula for λ∗
2
S HSIC S = SXY
= 1 − HS
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC
(n − 2)HSIC +
1 n
Pn
i =1
n
˜ ◦L ˜ [O(n3 )], ’/λ’: O(n2 ). FCOSE: after SVD of K
˜ ii L ˜ii K
2
HSIC . +
Statement SCOSE is (essentially) the oracle linear shrinkage estimator w.r.t. the quadratic loss.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Oracle estimator: linear shrinkage, quadratic loss
Proposition (S ∗ , ρ∗ ) :=
arg min Z ∈HS(Hℓ ,Hk ),Z =(1−ρ)SXY ,ρ∈[0,1]
E kZ − ΣXY k2HS .
∗
S = (1 − ρ∗ )SXY , ρ∗ =
E kSXY − ΣXY k2HS E kSXY k2HS
.
Intuition: we shrink SXY towards zero, optimally in quadratic sense.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Indeed Using E[SXY ] = ΣXY : E kZ − ΣXY k2HS = E k(1 − ρ)SXY − ΣXY k2HS = = E k−ρSXY + (SXY − ΣXY )k2HS
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Indeed Using E[SXY ] = ΣXY : E kZ − ΣXY k2HS = E k(1 − ρ)SXY − ΣXY k2HS = = E k−ρSXY + (SXY − ΣXY )k2HS = ρ2 E kSXY k2HS + E kSXY − ΣXY k2HS −2ρ E hSXY , SXY − ΣXY iHS | | {z } {z } EkSXY k2HS −kΣXY k2HS
Zolt´ an Szab´ o
EkSXY k2HS −kΣXY k2HS
Nonparametric Independence Testing for Small Sample Sizes
Indeed Using E[SXY ] = ΣXY : E kZ − ΣXY k2HS = E k(1 − ρ)SXY − ΣXY k2HS = = E k−ρSXY + (SXY − ΣXY )k2HS = ρ2 E kSXY k2HS + E kSXY − ΣXY k2HS −2ρ E hSXY , SXY − ΣXY iHS | | {z } {z } EkSXY k2HS −kΣXY k2HS
EkSXY k2HS −kΣXY k2HS
= ρ2 E kSXY k2HS + (1 − 2ρ)E kSXY − ΣXY k2HS =: J(ρ).
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Indeed Using E[SXY ] = ΣXY : E kZ − ΣXY k2HS = E k(1 − ρ)SXY − ΣXY k2HS = = E k−ρSXY + (SXY − ΣXY )k2HS = ρ2 E kSXY k2HS + E kSXY − ΣXY k2HS −2ρ E hSXY , SXY − ΣXY iHS | | {z } {z } EkSXY k2HS −kΣXY k2HS
EkSXY k2HS −kΣXY k2HS
= ρ2 E kSXY k2HS + (1 − 2ρ)E kSXY − ΣXY k2HS =: J(ρ).
Optimizing in ρ: 0= J ′ (ρ) = 2ρE kSXY k2HS − 2E kSXY − ΣXY k2HS ⇒ ρ∗ =
E kSXY − ΣXY k2HS E kSXY k2HS
Zolt´ an Szab´ o
.
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS E kSXY k2HS
=
β , δ
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS E kSXY k2HS
=
β b , δ = kSXY k2HS = HSIC , δ
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS
β b , δ = kSXY k2HS = HSIC , δ
2 i 2 1
˜
˜ i ) ⊗ φ(y ˜ i ) − ΣXY ˜ φ(x
= E φ(x i ) ⊗ φ(yi ) − ΣXY
n HS
E kSXY k2HS n h X
1
β = E
n
i =1
=
HS
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS
β b , δ = kSXY k2HS = HSIC , δ
2 i 2 1
˜
˜ i ) ⊗ φ(y ˜ i ) − ΣXY ˜ φ(x
= E φ(x i ) ⊗ φ(yi ) − ΣXY
n HS
E kSXY k2HS n h X
1
β = E
n
i =1 n X
=
HS
2 1
˜ ˜ i ) − SXY ≈ 2
φ(xi ) ⊗ ψ(y n HS | {z } i =1 2 ˜ ii L ˜ii +kSXY k −2hφ(x ˜ i )⊗ψ(y ˜ i ),SXY i K HS HS
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS
β b , δ = kSXY k2HS = HSIC , δ
2 i 2 1
˜
˜ i ) ⊗ φ(y ˜ i ) − ΣXY ˜ φ(x
= E φ(x i ) ⊗ φ(yi ) − ΣXY
n HS
E kSXY k2HS n h X
1
β = E
n
i =1 n X
=
HS
2 1
˜ ˜ i ) − SXY ≈ 2
φ(xi ) ⊗ ψ(y n HS | {z } i =1 2 ˜ ii L ˜ii +kSXY k −2hφ(x ˜ i )⊗ψ(y ˜ i ),SXY i K HS HS n h i 1 1X˜ ˜ b Kii Lii + kSXY k2HS − 2 kSXY k2HS =: β, = n n | {z } i =1
−kSXY k2HS =−HSIC
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS
β b , δ = kSXY k2HS = HSIC , δ
2 i 2 1
˜
˜ i ) ⊗ φ(y ˜ i ) − ΣXY ˜ φ(x
= E φ(x i ) ⊗ φ(yi ) − ΣXY
n HS
E kSXY k2HS n h X
1
β = E
n
i =1 n X
=
HS
2 1
˜ ˜ i ) − SXY ≈ 2
φ(xi ) ⊗ ψ(y n HS | {z } i =1 2 ˜ ii L ˜ii +kSXY k −2hφ(x ˜ i )⊗ψ(y ˜ i ),SXY i K HS HS n h i 1 1X˜ ˜ b Kii Lii + kSXY k2HS − 2 kSXY k2HS =: β, = n n | {z } i =1
\∗ = ⇒HSIC
−kSXY k2HS =−HSIC
1−
1 n
Pn
˜ ˜ − HSIC nHSIC
i =1 Kii Lii
Zolt´ an Szab´ o
!2
HSIC .
Nonparametric Independence Testing for Small Sample Sizes
Comparison
SCOSE:
HSIC S = 1 −
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC
(n − 2)HSIC +
1 n
Pn
i =1
˜ ii L ˜ii K
n
Oracle estimator with plug-in: \∗ = HSIC
1−
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC nHSIC
2
HSIC . +
!2
HSIC .
SCOSE ≈ oracle with perturbed plug-in.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Numerical experiments Shrinkage usually improves power.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Numerical experiments Shrinkage usually improves power. FCOSE: often achieves better power → non-linear shrinkage?, non-quadratic loss?
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Numerical experiments Shrinkage usually improves power. FCOSE: often achieves better power → non-linear shrinkage?, non-quadratic loss? Soft HSIC shrinkage:
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Thank you for the attention!
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Zolt´ an Szab´ o
Machine Learning Journal Club, Gatsby Unit April 4, 2016
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing. Idea: 1
large cov (X , Y ) ⇒ declare dependence.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing. Ideas: 1 2
large cov (X , Y ) ⇒ declare dependence. large supf ∈F,g ∈G cov (f (X ), g (Y )) ⇒ dependence nice asymptotic results.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing. Ideas: 1 2
large cov (X , Y ) ⇒ declare dependence. large supf ∈F,g ∈G cov (f (X ), g (Y )) ⇒ dependence nice asymptotic results.
Focus: small sample size, small false positive regime: ’avoid’ false dependence detection.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
One-page summary
Goal: nonparametric independence testing. Ideas: 1 2
large cov (X , Y ) ⇒ declare dependence. large supf ∈F,g ∈G cov (f (X ), g (Y )) ⇒ dependence nice asymptotic results.
Focus: small sample size, small false positive regime: ’avoid’ false dependence detection.
Trick: introduce some bias to reduce variance - Stein. large shrunk[ supf ∈F,g ∈G cov (f (X ), g (Y ))] ⇒ dependence
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: independence testing problem
i .i .d.
Given: {(xi , yi )}ni=1 ∼ PXY . Marginals of PXY : PX , PY . Hypotheses: H0 : PXY = PX × PY ,
Zolt´ an Szab´ o
H1 : PXY 6= PX × PY .
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: independence testing problem
i .i .d.
Given: {(xi , yi )}ni=1 ∼ PXY . Marginals of PXY : PX , PY . Hypotheses: H0 : PXY = PX × PY ,
H1 : PXY 6= PX × PY .
Aim: 1
Low type-I error = P(detect dependence, when there isn’t any) ≤ α, {z } | false positive
2
High power = P(detect dependence, when there is).
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: cross-covariance X ∈ (X, k), Y ∈ (Y, ℓ), k, ℓ: kernels. RKHSs: Hk , Hℓ .
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: cross-covariance X ∈ (X, k), Y ∈ (Y, ℓ), k, ℓ: kernels. RKHSs: Hk , Hℓ . Mean embedding and its empirical counterpart: µX = Ex∼PX k(·, x), | {z }
µY = Ey ∼PY ℓ(·, y ), | {z }
i =1
i =1
=:φ(x)
µ ˆX =
1 n
n X
φ(xi ),
Zolt´ an Szab´ o
=:ψ(y )
µ ˆY =
1 n
n X
ψ(yi ).
Nonparametric Independence Testing for Small Sample Sizes
Ingredients: cross-covariance X ∈ (X, k), Y ∈ (Y, ℓ), k, ℓ: kernels. RKHSs: Hk , Hℓ . Mean embedding and its empirical counterpart: µX = Ex∼PX k(·, x), | {z }
µY = Ey ∼PY ℓ(·, y ), | {z }
i =1
i =1
=:φ(x)
µ ˆX =
n X
1 n
φ(xi ),
=:ψ(y )
µ ˆY =
1 n
n X
ψ(yi ).
Cross-covariance: ΣXY = E(x,y )∼PXY [φ(x) − µX ] ⊗ [ψ(y ) − µY ] : Hℓ → Hk , | {z } | {z } ˜ =:φ(x)
SXY =
1 n
˜ ) =:ψ(y
n X
[φ(xi ) − µ ˆX ] ⊗ [ψ(yi ) − µ ˆY ].
i =1
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ?
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ? Cb (X) and Cb (Y) would be sufficient: Jacod and Protter 2000.
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ? Cb (X) and Cb (Y) would be sufficient: Jacod and Protter 2000. Trick [Gretton et al. ’05]: guarantee the denseness of Hk in Cb (X), Hℓ in Cb (Y).
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ? Cb (X) and Cb (Y) would be sufficient: Jacod and Protter 2000. Trick [Gretton et al. ’05]: guarantee the denseness of Hk in Cb (X), Hℓ in Cb (Y). Space: compact metric, kernel: universal X
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Cross-covariance as an independence measure Known: hf , ΣXY g iHk = cov (f (X ), g (Y )), ∀g ∈ Hℓ , f ∈ Hk . Are Hℓ and Hk enough for the independence testing of X and Y ? Cb (X) and Cb (Y) would be sufficient: Jacod and Protter 2000. Trick [Gretton et al. ’05]: guarantee the denseness of Hk in Cb (X), Hℓ in Cb (Y). Space: compact metric, kernel: universal X Examples: ′ 2
k(x, x′ ) = e −γkx−x k2 ,
′
k(x, x′ ) = e −γkx−x k1 .
Yes ⇒ Test: ΣXY = 0. Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Side-note
ΣXY ∈ HS(Hℓ , Hk ) =: HS(G, F). What does this mean? Extension of Frobenious norm.
kC k2F =
X
Cij 2
i ,j
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Side-note
ΣXY ∈ HS(Hℓ , Hk ) =: HS(G, F). What does this mean? Extension of Frobenious norm.
kC k2F =
X
Cij 2 ,
i ,j
kC k2HS
=
X
hCgj , fi i2F < ∞,
i ,j
where C : G → F bounded linear operator. G, F are separable Hilbert spaces with ONBs {gj }j , {fi }i .
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HS operator example: f ⊗ g
Intuition: fgT . (fgT )u = f (gT u). | {z } =hg,ui
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HS operator example: f ⊗ g
Intuition: fgT . (fgT )u = f (gT u). | {z } =hg,ui
Outer product: f ⊗ g (f ∈ F, g ∈ G) (f ⊗ g )(u) = f hg , uiG , ∀u ∈ G.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HS operator example: f ⊗ g
Intuition: fgT . (fgT )u = f (gT u). | {z } =hg,ui
Outer product: f ⊗ g (f ∈ F, g ∈ G) (f ⊗ g )(u) = f hg , uiG , ∀u ∈ G. HS norm of f ⊗ g : kf ⊗ g k2HS = hf , f iF hg , g iG . Cross-covariance: made of f ⊗ g -type quantities.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HSIC 2 It is easy to compute kΣXY kHS =: HSIC.
HSIC =
kΣXY k2HS
=
*
+ n n X 1X˜ 1 ˜ i ), ˜ j ) ⊗ ψ(y ˜ j) φ(xi ) ⊗ ψ(y φ(x n n i =1
Zolt´ an Szab´ o
j=1
HS
Nonparametric Independence Testing for Small Sample Sizes
HSIC 2 It is easy to compute kΣXY kHS =: HSIC.
HSIC =
kΣXY k2HS
=
=
*
+ n n X 1X˜ 1 ˜ i ), ˜ j ) ⊗ ψ(y ˜ j) φ(xi ) ⊗ ψ(y φ(x n n
1 n2
i =1 n D X
j=1
HS
E
˜ i ) ⊗ ψ(y ˜ i ), φ(x ˜ j ) ⊗ ψ(y ˜ j) φ(x HS {z } | i ,j=1 ˜ ˜ ˜ ˜ ˜ ˜ φ(x ), φ(x ) ψ(y ), ψ(y ) K L = h i j iH h i j iH ij ij k ℓ
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HSIC 2 It is easy to compute kΣXY kHS =: HSIC.
HSIC =
kΣXY k2HS
=
*
+ n n X 1X˜ 1 ˜ i ), ˜ j ) ⊗ ψ(y ˜ j) φ(xi ) ⊗ ψ(y φ(x n n
1 n2
i =1 n D X
j=1
HS
E
˜ i ) ⊗ ψ(y ˜ i ), φ(x ˜ j ) ⊗ ψ(y ˜ j) φ(x HS {z } | i ,j=1 ˜ ˜ ˜ ˜ ˜ ˜ φ(x ), φ(x ) ψ(y ), ψ(y ) K L = h i j iH h i j iH ij ij k ℓ E D 1 ˜ ˜ = 2 K, L . n F
=
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
HSIC 2 It is easy to compute kΣXY kHS =: HSIC.
HSIC =
kΣXY k2HS
=
*
+ n n X 1X˜ 1 ˜ i ), ˜ j ) ⊗ ψ(y ˜ j) φ(xi ) ⊗ ψ(y φ(x n n
1 n2
i =1 n D X
j=1
HS
E
˜ i ) ⊗ ψ(y ˜ i ), φ(x ˜ j ) ⊗ ψ(y ˜ j) φ(x HS {z } | i ,j=1 ˜ ˜ ˜ ˜ ˜ ˜ φ(x ), φ(x ) ψ(y ), ψ(y ) K L = h i j iH h i j iH ij ij k ℓ E D 1 ˜ ˜ = 2 K, L . n F
=
˜ = HKH, H = In − 1 11T , L ˜ = HLH. K n Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Independence test using HSIC [Gretton et al. 2005]
Given: samples and α ∈ (0, 1). Test statistics: T = HSIC = kΣXY k2HS . Simulated null distribution of T : via {y1 , . . . , yn } permutations ⇒ tα . Decision: reject H0 if tα < T .
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Observation
SXY is unbiased estimator of ΣXY : E[SXY ] = ΣXY . Issue: SXY can have high variance for small sample numbers.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Observation
SXY is unbiased estimator of ΣXY : E[SXY ] = ΣXY . Issue: SXY can have high variance for small sample numbers. Idea [Stein, 1956]: decrease the variance by adding some bias.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Observation
SXY is unbiased estimator of ΣXY : E[SXY ] = ΣXY . Issue: SXY can have high variance for small sample numbers. Idea [Stein, 1956]: decrease the variance by adding some bias. [Maundet et al. 2014]: 2 shrinkage based estimators.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Observation
SXY is unbiased estimator of ΣXY : E[SXY ] = ΣXY . Issue: SXY can have high variance for small sample numbers. Idea [Stein, 1956]: decrease the variance by adding some bias. [Maundet et al. 2014]: 2 shrinkage based estimators. Questions 1 How do they perform in independence testing? 2
Optimality?
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Variations: shrinking towards zero Recall: SXY =
1 n
Pn
SXY =
˜
i =1 φ(xi ) ⊗
˜ i) ⇒ ψ(y
n
2 1 X
˜ ˜ i) − Z
.
φ(xi ) ⊗ ψ(y HS Z ∈HS(Hℓ ,Hk ) n
arg min
i =1
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Variations: shrinking towards zero Recall: SXY =
1 n
Pn
SXY =
˜
i =1 φ(xi ) ⊗
˜ i) ⇒ ψ(y
n
2 1 X
˜ ˜ i) − Z
.
φ(xi ) ⊗ ψ(y HS Z ∈HS(Hℓ ,Hk ) n
arg min
i =1
SCOSE (simple covariance shrinkage estimator, λ > 0): n
2 1 X
˜ 2 S ˜ i) − Z SXY = arg min
φ(xi ) ⊗ ψ(y
+ λ kZ kHS . HS Z ∈HS(Hℓ ,Hk ) n i =1
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Variations: shrinking towards zero Recall: SXY =
1 n
Pn
SXY =
˜
i =1 φ(xi ) ⊗
˜ i) ⇒ ψ(y
n
2 1 X
˜ ˜ i) − Z
.
φ(xi ) ⊗ ψ(y HS Z ∈HS(Hℓ ,Hk ) n
arg min
i =1
SCOSE (simple covariance shrinkage estimator, λ > 0): n
2 1 X
˜ 2 S ˜ i) − Z SXY = arg min
φ(xi ) ⊗ ψ(y
+ λ kZ kHS . HS Z ∈HS(Hℓ ,Hk ) n i =1
FCOSE (flexible covariance shrinkage estimator): F = SXY
n X βj j=1
n
˜ j ) ⊗ ψ(y ˜ j ), φ(x
2
n n X X
β 1 j ˜ i ) ⊗ ψ(y ˜ i) − ˜ j ) ⊗ ψ(y ˜ j ) + λ kβk2 .
φ(x φ(x β = arg min 2
n n β
j=1 i =1 HS
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
SCOSE vs FCOSE
In both cases: λ is chosen via leave-one-out CV. SCOSE: analytical formula for λ∗
2
S HSIC S = SXY
= 1 − HS
Zolt´ an Szab´ o
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC
(n − 2)HSIC +
1 n
Pn
i =1
n
˜ ii L ˜ii K
2
HSIC . +
Nonparametric Independence Testing for Small Sample Sizes
SCOSE vs FCOSE
In both cases: λ is chosen via leave-one-out CV. SCOSE: analytical formula for λ∗
2
S HSIC S = SXY
= 1 − HS
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC
(n − 2)HSIC +
1 n
Pn
i =1
n
˜ ◦L ˜ [O(n3 )], ’/λ’: O(n2 ). FCOSE: after SVD of K
Zolt´ an Szab´ o
˜ ii L ˜ii K
2
HSIC . +
Nonparametric Independence Testing for Small Sample Sizes
SCOSE vs FCOSE
In both cases: λ is chosen via leave-one-out CV. SCOSE: analytical formula for λ∗
2
S HSIC S = SXY
= 1 − HS
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC
(n − 2)HSIC +
1 n
Pn
i =1
n
˜ ◦L ˜ [O(n3 )], ’/λ’: O(n2 ). FCOSE: after SVD of K
˜ ii L ˜ii K
2
HSIC . +
Statement SCOSE is (essentially) the oracle linear shrinkage estimator w.r.t. the quadratic loss.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Oracle estimator: linear shrinkage, quadratic loss
Proposition (S ∗ , ρ∗ ) :=
arg min Z ∈HS(Hℓ ,Hk ),Z =(1−ρ)SXY ,ρ∈[0,1]
E kZ − ΣXY k2HS .
∗
S = (1 − ρ∗ )SXY , ρ∗ =
E kSXY − ΣXY k2HS E kSXY k2HS
.
Intuition: we shrink SXY towards zero, optimally in quadratic sense.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Indeed Using E[SXY ] = ΣXY : E kZ − ΣXY k2HS = E k(1 − ρ)SXY − ΣXY k2HS = = E k−ρSXY + (SXY − ΣXY )k2HS
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Indeed Using E[SXY ] = ΣXY : E kZ − ΣXY k2HS = E k(1 − ρ)SXY − ΣXY k2HS = = E k−ρSXY + (SXY − ΣXY )k2HS = ρ2 E kSXY k2HS + E kSXY − ΣXY k2HS −2ρ E hSXY , SXY − ΣXY iHS | | {z } {z } EkSXY k2HS −kΣXY k2HS
Zolt´ an Szab´ o
EkSXY k2HS −kΣXY k2HS
Nonparametric Independence Testing for Small Sample Sizes
Indeed Using E[SXY ] = ΣXY : E kZ − ΣXY k2HS = E k(1 − ρ)SXY − ΣXY k2HS = = E k−ρSXY + (SXY − ΣXY )k2HS = ρ2 E kSXY k2HS + E kSXY − ΣXY k2HS −2ρ E hSXY , SXY − ΣXY iHS | | {z } {z } EkSXY k2HS −kΣXY k2HS
EkSXY k2HS −kΣXY k2HS
= ρ2 E kSXY k2HS + (1 − 2ρ)E kSXY − ΣXY k2HS =: J(ρ).
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Indeed Using E[SXY ] = ΣXY : E kZ − ΣXY k2HS = E k(1 − ρ)SXY − ΣXY k2HS = = E k−ρSXY + (SXY − ΣXY )k2HS = ρ2 E kSXY k2HS + E kSXY − ΣXY k2HS −2ρ E hSXY , SXY − ΣXY iHS | | {z } {z } EkSXY k2HS −kΣXY k2HS
EkSXY k2HS −kΣXY k2HS
= ρ2 E kSXY k2HS + (1 − 2ρ)E kSXY − ΣXY k2HS =: J(ρ).
Optimizing in ρ: 0= J ′ (ρ) = 2ρE kSXY k2HS − 2E kSXY − ΣXY k2HS ⇒ ρ∗ =
E kSXY − ΣXY k2HS E kSXY k2HS
Zolt´ an Szab´ o
.
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS E kSXY k2HS
=
β , δ
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS E kSXY k2HS
=
β b , δ = kSXY k2HS = HSIC , δ
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS
β b , δ = kSXY k2HS = HSIC , δ
2 i 2 1
˜
˜ i ) ⊗ φ(y ˜ i ) − ΣXY ˜ φ(x
= E φ(x i ) ⊗ φ(yi ) − ΣXY
n HS
E kSXY k2HS n h X
1
β = E
n
i =1
=
HS
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS
β b , δ = kSXY k2HS = HSIC , δ
2 i 2 1
˜
˜ i ) ⊗ φ(y ˜ i ) − ΣXY ˜ φ(x
= E φ(x i ) ⊗ φ(yi ) − ΣXY
n HS
E kSXY k2HS n h X
1
β = E
n
i =1 n X
=
HS
2 1
˜ ˜ i ) − SXY ≈ 2
φ(xi ) ⊗ ψ(y n HS | {z } i =1 2 ˜ ii L ˜ii +kSXY k −2hφ(x ˜ i )⊗ψ(y ˜ i ),SXY i K HS HS
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS
β b , δ = kSXY k2HS = HSIC , δ
2 i 2 1
˜
˜ i ) ⊗ φ(y ˜ i ) − ΣXY ˜ φ(x
= E φ(x i ) ⊗ φ(yi ) − ΣXY
n HS
E kSXY k2HS n h X
1
β = E
n
i =1 n X
=
HS
2 1
˜ ˜ i ) − SXY ≈ 2
φ(xi ) ⊗ ψ(y n HS | {z } i =1 2 ˜ ii L ˜ii +kSXY k −2hφ(x ˜ i )⊗ψ(y ˜ i ),SXY i K HS HS n h i 1 1X˜ ˜ b Kii Lii + kSXY k2HS − 2 kSXY k2HS =: β, = n n | {z } i =1
−kSXY k2HS =−HSIC
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Plug-in estimator ρ∗ =
E kSXY − ΣXY k2HS
β b , δ = kSXY k2HS = HSIC , δ
2 i 2 1
˜
˜ i ) ⊗ φ(y ˜ i ) − ΣXY ˜ φ(x
= E φ(x i ) ⊗ φ(yi ) − ΣXY
n HS
E kSXY k2HS n h X
1
β = E
n
i =1 n X
=
HS
2 1
˜ ˜ i ) − SXY ≈ 2
φ(xi ) ⊗ ψ(y n HS | {z } i =1 2 ˜ ii L ˜ii +kSXY k −2hφ(x ˜ i )⊗ψ(y ˜ i ),SXY i K HS HS n h i 1 1X˜ ˜ b Kii Lii + kSXY k2HS − 2 kSXY k2HS =: β, = n n | {z } i =1
\∗ = ⇒HSIC
−kSXY k2HS =−HSIC
1−
1 n
Pn
˜ ˜ − HSIC nHSIC
i =1 Kii Lii
Zolt´ an Szab´ o
!2
HSIC .
Nonparametric Independence Testing for Small Sample Sizes
Comparison
SCOSE:
HSIC S = 1 −
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC
(n − 2)HSIC +
1 n
Pn
i =1
˜ ii L ˜ii K
n
Oracle estimator with plug-in: \∗ = HSIC
1−
1 n
Pn
˜ ˜ i =1 Kii Lii − HSIC nHSIC
2
HSIC . +
!2
HSIC .
SCOSE ≈ oracle with perturbed plug-in.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Numerical experiments Shrinkage usually improves power.
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Numerical experiments Shrinkage usually improves power. FCOSE: often achieves better power → non-linear shrinkage?, non-quadratic loss?
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Numerical experiments Shrinkage usually improves power. FCOSE: often achieves better power → non-linear shrinkage?, non-quadratic loss? Soft HSIC shrinkage:
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
Thank you for the attention!
Zolt´ an Szab´ o
Nonparametric Independence Testing for Small Sample Sizes
