Linear Contextual Bandits with Knapsacks
Linear Contextual Bandits with Global Constraints and Objective Shipra Agrawal∗
Nikhil R. Devanur†
arXiv:1507.06738v1 [cs.LG] 24 Jul 2015
July 27, 2015
Abstract We consider the linear contextual bandit problem with global convex constraints and a concave objective function. In each round, the outcome of pulling an arm is a vector, that depends linearly on the context of that arm. The global constraints require the average of these vectors to lie in a certain convex set. The objective is a concave function of this average vector. This problem turns out to be a common generalization of classic linear contextual bandits (linContextual) [8, 17, 1], bandits with concave rewards and convex knapsacks (BwCR) [4], and the online stochastic convex programming (OSCP) problem [5]. We present algorithms with near-optimal regret bounds for this problem. Our bounds compare favorably to results on the unstructured version of the problem [6, 12] where the relation between the contexts and the outcomes could be arbitrary, but the algorithm only competes against a fixed set of policies. We combine techniques from the work on linContextual, BwCR and OSCP in a nontrivial manner while also tackling new difficulties that are not present in any of these special cases.
∗ †
Microsoft Research. [email protected]. Microsoft Research. [email protected].
1
Introduction
In the contextual bandit problem [8, 14, 22, 3], the decision maker observes a sequence of contexts (or features). In every round she needs to pull one out of K arms, after observing the context for that round. The outcome of pulling an arm may be used along with the contexts to decide future arms. Contextual bandit problems have found many useful applications such as online recommendation systems, online advertising, and clinical trials, where the decision in every round needs to be customized to the features of the user being served. The linear contextual bandit problem [1, 8, 17] is a special case of the contextual bandit problem, where the outcome is linear in the feature vector encoding the context. As pointed by [3], contextual bandit problems represent a natural half-way point between supervised learning and reinforcement learning: the use of features to encode contexts and the models for the relation between these feature vectors and the outcome are often inherited from supervised learning, while managing the exploration-exploitation tradeoff is necessary to ensure good performance in reinforcement learning. The linear contextual bandit problem can thus be thought of as a midway between the linear regression model of supervised learning, and reinforcement learning. Recently, there has been a significant interest in introducing multiple “global constraints” in the standard bandit setting [11, 4, 12, 6]. Such constraints are crucial for many important real-world applications. For example, in clinical trials, the treatment plans may be constrained by the total availability of medical facilities, drugs and other resources. In online advertising, there are budget constraints that restrict the number of times an ad is shown. Other applications include dynamic pricing, dynamic procurement, crowdsourcing, etc.; see [11, 4] for many such examples. In this paper, we consider a very general version of the linear contextual bandit problem with global constraints and objective (henceforth, linCBwCR). In this problem, the context vectors are generated i.i.d. in every round from some unknown distribution, and the expected outcome on picking an arm is in the form of a vector that depends linearly on the context vector. The aim of the decision maker is to maximize a given concave objective function of the average of these outcome vectors while ensuring that the average lies in a given convex set. Below, we give a more precise definition of this problem. We use the following notational convention throughout: vectors are denoted by bold face lower case letters, while matrices are denoted by regular face upper case letters. Other quantities such as sets, scalars, etc. may be of either case, but never bold faced. All vectors are column vectors, i.e., a vector in n dimensions is treated as an n × 1 matrix. The transpose of matrix A is A⊤ . Definition 1 (linCBwCR). There are K “arms”, which we identify with the set [K]. The algorithm is initially given as input a convex set S ⊆ [0, 1]d , and an L-Lipschitz concave function f with domain [0, 1]d . We then proceed in rounds t = 1, 2, . . . , T, where in every round t, a pair of a context matrix and an outcome matrix (Xt , Vt ) ∈ [0, 1]m×K × [0, 1]d×K are drawn from an unknown distribution D, independent of everything in previous rounds. The column of Xt corresponding to arm a ∈ [K] is called the context vector for arm a and is denoted by xt (a) ∈ [0, 1]m . The column of Vt corresponding to an arm a is called the “outcome” vector for arm a and is denoted by vt (a) ∈ [0, 1]d . There exist an (unknown) weight matrix W∗ ∈ [0, 1]m×d such that E[Vt |Xt ] = W∗⊤ Xt . In each round, the algorithm first observes context Xt , and then chooses an arm at ∈ A, P and finally 1 observes vt (at ). The goal of the algorithm is to pick arms such that the average vector T Tt=1 vt (at ) P maximizes f ( T1 Tt=1 vt (at )) and is inside S.
We compare the performance of an algorithm to that of the optimal adaptive policy that knows the distribution D and the weight matrix W∗ , and can take into account the history upto that point as well as the current context to decide (possibly with randomization) which arm to pull at time t. However, it is easier to work with an upper bound on this, which is the optimal expected reward of a static policy 1
that is required to satisfy the constraints only in expectation. This technique has been used in several related problems and is standard by now [20, 11]. Definition 2 (Optimal Static Policy). We consider policies that are context dependent but are nonadaptive: for a policy q, let q(X) ∈ ∆K (the unit simplex) denote the probability distribution over arms played when the context is X ∈ X . Define v(q) to be the expected outcome vector of policy q, i.e. v(q) := E(X,V )∼D [V q(X)] = EX∼D [W∗⊤ Xq(X)]. Let
q ∗ :=
arg maxq f (v(q)) such that v(q) ∈ S
(1) (2)
be the optimal static policy. We assume that there exists a feasible policy, i.e. v(q ∗ ) ∈ S, and we denote the optimal value by OPT := f (v(q ∗ )). Definition 3 (Regret). We define two regrets: regret in objective and regret in constraints. P Let at be the ¯ 1:T := T1 Tt=1 vt (at ), arm played at time t by the algorithm. For convenience of notation let us define v • Regret in objective. avg-regret1 (T ) = OPT − f (¯ v1:T ).
• Regret in constraints is defined as the distance of the average of played vectors from S; we let d(·, ·) denote a distance function. avg-regret2 (T ) := d(¯ v1:T , S). We consider following useful special cases. Feasibility problem (linCBwC): In this special case of linCBwCR, there is no objective function ¯ 1:T be in the set S. The f , and the aim of the algorithm is to have the average outcome vector v ¯ 1:T to S, i.e., by avg-regret2 (T ). We will performance of the algorithm is measured by the distance of v first illustrate our algorithm and proof techniques for this special yet nontrivial case. Budget constraints (linCBwK): This is a well studied special case with linear objectives and constraints; a generalization of the Bandits with Knapsacks (BwK) problem of [11]. Here, the outcome vector can be broken down into two components: a scalar reward rt (at )P ∈ [0, 1] and a d − 1-dimensional consumption vector vt (at ) ∈ [0, 1]d−1 . The objective is to maximize Tt=1 rt (at ) while ensuring that PT t=1 vt (at ) ≤ B1, where 1 is the vector of all 1s and B > 0 is some scalar. Such budget constraints are equivalent to having the constraint set S be equal to {v : 0 ≤ v ≤ B T 1}. Also, it is possible to simply stop once the budget is consumed; equivalently we assume that the algorithm always has the “do nothing” option of getting 0 reward and 0 cost. For this problem, we provide an algorithm that always satisfies the budget constraints, i.e., avg-regret2 (T ) = 0, while bounding avg-regret 1 (T ).
1.1
Main results
Our main result is an algorithm with near-optimal regret bound for linCBwCR. The algorithm needs to know a certain problem dependent parameter Z, defined by Assumption 1 in Section 4. Such knowledge is not required for linCBwC, however. For the special case of budget constraints (linCBwK), any 2OPT Z ≥ (B/T ) can be used, and we show how one such Z can be obtained from the first few rounds. Below we state the main result when k · k denotes either ℓ2 or ℓ∞ norm. In the text, we will provide a more detailed theorem statement that is applicable to any norm. Theorem 1. Let k · k denote either ℓ2 or ℓ∞ norm, and be used to define the distance d(·, ·). Assume f is L-Lipschitz with respect to k · k. Then, for linCBwCR, there is an algorithm that takes as input a problem-dependent parameter Z ≥ L (satisfying Assumption 1), and with probability at least 1 − δ has p p avg-regret1 (T ) = O Zmk1d k ln(dT /δ) ln(T )/T , avg-regret2 (T ) = O mk1d k ln(dT /δ) ln(T )/T . 2
For linCBwC, no knowledge of Z is needed, and the algorithm has the same avg-regret2 (T ) as above. Remark 1. (Near-optimality of regret bounds) In [18], it was shown that for the√linear contextual bandits problem, no online algorithm can achieve a regret bound better than Ω(m T ). In fact, they prove this lower bound for linear contextual bandits with static contexts. Since that problem is a special case of our linCBwCR problem with d = 1, this shows that the dependence on m and T in the above (average) regret bounds is optimal upto log factors. The dependence on Zk1d k is common in special cases of this problem [4, 5]. Moreover, we don’t need to know D, as is assumed in related work [6, 12, 38]. Remark 2. (Budget constraints) For the special case of linCBwK, we show how to estimate Z using ˜ ((OPT/(B/T ) + 1) m/√T ), the observations from the first few rounds. We get an overall regret bound of O which compares favorably to the regret bounds for the general contextual version obtained in [12, 6]. 3/4 (Details in Section 1.2.) The only difference is that √ we need the budget B to be Ω(mT ). Getting similar bounds for budgets as small as B = Θ(m T ) is an interesting open problem. Due to space constraints, we have eliminated many proofs from the main text. All the missing proofs are in the appendix.
1.2
Challenges and related work
Relation to BwCR [4] and OSCP [5]. It is easy to see that the linCBwCR problem is a generalization of the linear contextual bandits problem. There, the outcome is scalar and the goal is to simply maximize the sum of these. Remarkably, the linCBwCR problem also turns out to be a common generalization of bandits with concave rewards and convex knapsacks (BwCR) problem considered in [4], and the online stochastic convex programming (OSCP) problem of [5]. In both BwCR and OSCP problems, P the outcome of every round t is a vector vt and the goal of the algorithm is to maximize f ( T1 Tt=1 vt ) P while ensuring that T1 Tt=1 vt ∈ S. The problems differ in how this vector is picked. In the OSCP problem, in every round t, the algorithm may pick any vector from a given set At of d-dimensional vectors. The set At is drawn i.i.d. from an unknown distribution over sets of vectors. This corresponds to the special case of linCBwCR, where m = d and the outcome Vt is always equal to the context Xt . In the BwCR problem, there is a fixed set of arms, and for each arm there is an unknown distribution over vectors. The algorithm picks an arm and a vector is drawn from the corresponding distribution for that arm. This corresponds to the special case of linCBwCR, where m = K and the context Xt = I, the identity matrix, for all t. We use techniques from all three special cases: our algorithms follow the primal-dual paradigm using Fenchel duality, that was established in [4]. In order to deal with linear contexts, we use techniques from [1, 8, 17] to estimate the weight matrix W∗ , and define “optimistic estimates” of W∗ . We also use the technique of combining the objective and the constraints using a certain tradeoff parameter that was introduced in [5]. Further new difficulties arise, such as in estimating the optimum value from the first few rounds, a task that follows from standard techniques in each of the special cases but is very challenging here. One can see that the problem is indeed more than the sum of its parts, from the fact 3/4 ), unlike either special case for ˜ that we get the optimal bound for linCBwK only when B ≥ Ω(mT √ ˜ which the optimal bound holds for all B (but is meaningful only for B = Ω(m T )). The approach in [4] (for BwCR) extends to the case of “static” contexts,1 where each arm has a context that doesn’t change over time. The OSCP problem of [5] is not a special case of linCBwCR with static contexts; this is one indication of the additional difficulty of dynamic over static contexts. Relation to general contextual bandits. There have been recent papers [6, 12] that solve problems similar to linCBwCR but for general contextual bandits. Here the relation between contexts and 1
It was incorrectly claimed in [4] that the approach can be extended to dynamic contexts without much modifications.
3
outcome vectors is arbitrary and the algorithms p of context dependent pcompete with an arbitrary fixed set policies Π, with the regret bounds scaling as K log(|Π|). (In our results, this K log(|Π|) is replaced by m.) Since linCBwCR is a special case of this problem, we may compare the regret bounds obtained by applying these results. Firstly, we have no dependence on K, which enables us to consider problems with large action spaces. Further, with global constraints and objective, the effective policy space for linear bandits is doubly exponential in its dimensions, thus giving exponentially large regret bounds for linCBwCR using this approach. The key is to determine the size of the smallest policy set that is guaranteed to contain a near optimal policy. Trivially, the number of possible context dependent policies K |X | , where X denotes the space of contexts. In linCBwCR, X = [0, 1]m×K , giving |X | = O(1/ǫ(mK) ) using a discretization of the space into multiples of some small enough ǫ. We could try to reduce the policy space using linearity. In linear contextual bandits, the objective which is the sum of rewards, is decomposable across time steps. Given the weight vector W∗ , the optimal policy is to chose the arm a that maximizes the expected reward W∗⊤ xt (a) at time t, which is independent of the distribution D of the contexts. It thus suffices to have Π be the set consisting of one policy of this type for every possible W∗ , giving |Π| = ǫ1m (since d = 1). However, in the 1 P linCBwCR problem, due to nondecomposable objective f ( T t W∗⊤ Xt q(Xt )), and/or distance function P d( T1 t W∗⊤ Xt q(Xt ), S), the optimal policy cannot locally maximize the reward at every step. Even for a fixed W∗ , the optimal policy may depend on D. We can have Π consist of one policy for every possible 1 pair of D and W∗ , but that gives |Π| = |∆X | · ǫd×m which is again doubly exponential in m × K since |X | = O(1/ǫ(mK) ). Other related work. Budget constraints in a bandit setting has recieved considerable attention, but most of the early work focussed on special cases such as a single budget constraint in the regular √ (non-contextual) setting [21, 24, 26, 29, 35, 36]. Recently, [38] showed an O( T ) regret in the linear contextual setting with a single budget constraint, when costs depend only on contexts and not arms. Budget constraints that arise in particular applications such as online advertising [15, 31], dynamic pricing [9, 13] and crowdsourcing [10, 33, 34] have also been considered. There has also been a long line of work studying special cases of the OSCP problem [19, 20, 23, 7, 28, 25, 37, 30, 27, 16].
2 2.1
Preliminaries Confidence Ellipsoid
Consider a stochastic process which in each round t, generates a pair of observations (rt , y t ), such that rt is an unknown linear function of y t plus some 0-mean bounded noise, i.e., rt = µ⊤ ∗ y t + ηt , where m y t , µ∗ ∈ R , |ηt | ≤ 2R, and E[ηt |y 1 , r1 , . . . , y t−1 , rt−1 , y t ] = 0. At any time t, a high confidence estimate of the unknown vector µ∗ can be obtained by building ˆ t constructed from the oba “Confidence Ellipsoid” around the ℓ2 -regularized least-square estimate µ servations made so far. This technique is common in prior work on linear contextual bandits (e.g., in [8, 17, 1]). For any regularization parameter λ > 0, let P P ⊤ ˆ t := Mt−1 t−1 Mt := λI + t−1 i=1 y i y i , and µ i=1 y i ri .
ˆ t . For with center µ The following result from [1] shows that µ∗ lies with high probability in an ellipsoid p any positive semi-definite (PSD) matrix M, define the M -norm as kµkM := µ⊤ M µ. The confidence ellipsoid at time t is defined as n √ o p ˆ t kMt ≤ R m ln ((1+tm/λ)/δ ) + λm . Ct := µ ∈ Rm : kµ − µ Lemma 1 (Theorem 2 of [1]). If ∀ t, kµ∗ k2 ≤
√
m and ky t k2 ≤ 4
√
m, then with prob. 1 − δ, µ∗ ∈ Ct .
Another useful observation about this construction is stated below. It first appeared as Lemma 11 of [8], and was also proved as Lemma 3 in [17]. p PT Lemma 2 (Lemma 11 of [8]). ky k mT ln(T ). −1 ≤ t t=1 M t
As a corollary of the above two lemmas, we obtain a bound on the total error in the estimate provided by “any point” from the confidence ellipsoid. (Proof is in Appendix B.)
˜ t ∈ Ct be a point in the confidence ellipsoid, with λ = 1, 2R = 1. Corollary 1. For t = 1, . . . , T , let µ Then, with probability 1 − δ, p PT ⊤ (1+T m)/δ ) ln(T ). µ⊤ t y t − µ∗ y t | ≤ 2m T ln ( t=1 |˜
2.2
Fenchel duality
As mentioned earlier, our algorithms are primal-dual algorithms, that use Fenchel duality. Below we provide the basic background on this useful concept. Let h be a convex function defined on [0, 1]d . We define h∗ as the Fenchel conjugate of h, h∗ (θ) := max{y · θ − h(y) : y ∈ [0, 1]d }. Similarly for a concave function f on [0, 1]d , define f ∗ (θ) := maxy∈[0,1]d {y · θ + f (y)}. Note that the Fenchel conjugates h∗ and f ∗ are both convex functions of θ. Suppose that at every point y, every supergradient gy of h (and f ) have bounded dual norm ||g y ||∗ ≤ L. Then, the following dual relationship is known between h and h∗ (f and f ∗ ).
Lemma 3. h(z) = max||θ||∗ ≤L {θ · z − h∗ (θ)}, f (z) = min||θ||∗ ≤L {f ∗ (θ) − θ · z}
A special case is when h(y) = d(y, S) for some convex set S. This function is 1-Lipschitz with respect to norm || · || used in the definition of distance. In this case, h∗ (θ) = hS (θ) := maxy∈S θ · y, and Lemma 3 specializes to the following relation (which also appears in [2]). d(y, S) = max{θ · y − hS (θ) : ||θ||∗ ≤ 1}.
2.3
(3)
Online Learning
The online convex optimization (OCO) problem considers a T round game played between a learner and an adversary, where in round t, the learner chooses a θ t ∈ Ω, and then the adversary picks a concave function gt (θ t ) : Ω → R. The learner’s choice θ t may only depend on learner’s and adversary’s choices in previous rounds, and we denote this by θ t := OL-Predict(θ 1 , g1 , . . . , θ t−1 , gt−1 ). The goal of the learner is to minimize regret defined as the difference between the learner’s objective value and the value of the best single choice on hindsight: P P R(T ) := maxθ∈Ω Tt=1 gt (θ) − Tt=1 gt (θ t ). Some popular algorithms for OCO are online mirror descent (OMD) and online gradient descent, which have very fast per step update rules, and provide the following regret guarantees. Lemma 4. [32] The online mirror-descent algorithm for the OCO problem that achieves regret √ R(T ) = O(G DT ), where G is an upper bound on the norm of gradient of gt at all time t, and D is the diameter of an appropriate regularizer (in the dual norm). The value of these parameters are problem specific. In particular, for gt (θ) of form gt (θ) = θ · z − h∗ (θ) and Ω = {θ : ||θ||∗ ≤√L}, where h is an L∗ Lipschitz p function, h is Fenchel dual of h, regret is bounded as R(T ) ≤ O(L dT ) for ℓ2 norm, and O(L log(d)T ) for ℓ∞ . 5
3
Algorithm for the Feasibility problem (linCBwC)
We start with the case when there are only constraints, i.e., we are only given the convex set S and our goal is to minimize only avg-regret2 (T ). In every round t, our algorithm uses online learning to generate a vector θ t that acts as the dual price to weigh different components of the outcome vector estimate. Using this weighted sum of outcome vector components, for every arm we generate an optimistic estimate of the weight matrix W∗ , using techniques similar to the UCB based algorithms for linear contextual bandits. We then pick the best arm according to these optimistic estimates.
3.1
Optimistic estimates of weight matrix
Let at denote the arm played by the algorithm at time t. In the beginning of every round, we use the outcome and contexts from previous rounds to construct a confidence ellipsoid for every column of W∗ . For this, we use the techniques in Section P 2.1 with y t =⊤xt (at ), and rt = vt (at )j for every j. As in Section 2.1, let Mt := I + t−1 i=1 xi (ai )xi (ai ) , and construct the regularized least squares estimate for matrix W∗ as ˆ t := M −1 Pt−1 xi (ai )vi (ai )⊤ . (4) W t i=1
Let wj denote the j th column of matrix W . We define a confidence ellipsoid for each column j, as n p √ o Ct,j := µ ∈ Rm : kµ − w ˆ tj kMt ≤ m ln ((d+tmd)/δ ) + m ,
and denote by Gt , the Cartesian product of all these ellipsoids: Gt := {W ∈ Rm×d : wj ∈ Ct,j }. Note that Lemma 1 implies W∗ ∈ Gt with probability 1 − δ. Now, given a vector θ t ∈ Rd , we define the optimistic estimate of weight matrix at time t w.r.t. θ t , for every arm a ∈ [K], as : ˜ t (a) := arg minW ∈Gt xt (a)⊤ W θ t . W
(5)
˜ t , along with the results in Lemma 1 and Corollary 1 about confidence ellipsoids, Using the definition of W the following can be derived. Corollary 2. With probability 1 − δ,
˜ t (a)θ t ≤ xt (a)⊤ W∗ θ t , for all arms a ∈ [K], for all time t. 1. xt (a)⊤ W p P ˜ t (at ) − W∗ )⊤ xt (at )k ≤ k1d k 2m T ln ((d+tmd)/δ ) ln(T ) . 2. k Tt=1 (W
3.2
Algorithm and Regret Analysis
At every step t, the algorithm constructs optimistic estimates of the unknown weight matrix, as described in the previous section, for vector θ t which is itself generated by an online learning subroutine. The algorithm then picks the arm which appears to be the best according to these optimistic estimates. Algorithm 1 Algorithm for Feasibility problem ˆ 1 , G1 , OL-Predict with domain Ω = {kθk∗ ≤ 1}, and θ 1 . Initialize W for all t = 1 . . . T do Observe Xt . ˜ t (a) as defined in (5), for all a. Compute W ˜ t (a)θ t . Play arm at := arg mina∈[K] xt (a)⊤ W Observe vt (at ). ˆ t+1 and Gt+1 . Use xt (at ) and vt (at ) to obtain W ˜ t (at )θ − hS (θ). θ t+1 := OL-Predict(θ 1 , g1 , . . . , θ t , gt ) with gt (θ) = xt (at )⊤ W end for 6
Theorem 2. Let R(T ) be the regret for OCO with {gt , θ t }, aspgiven by Lemma 4. Specifically, R(T ) = √ O( dT ) when distance is given by ℓ2 norm and R(T ) = O( log(d)T ) for ℓ∞ . Algorithm 1 achieves the following regret bound, with probability 1 − δ. p P d( T1 t vt (at ), S) ≤ R(T )/T + O mk1d k ln(mdT /δ) ln(T )/T .
Proof. Let Ht−1 be the history of plays and observations before time t, i.e. Ht−1 := {θ τ , Xτ , aτ , vτ (aτ ), τ = ˆ t , Gt , but it does not determine Xt , at , W ˜ t (since at and 1, . . . , t − 1}. Note that Ht−1 determines θ t , W ˜ t (a) depend on the context Xt at time t). The proof is in 3 steps: W Step 1: Since E[vt (at )|Xt , at , Ht−1 ] = W∗⊤ xt (at ), it is sufficient to bound the distance to S of the average of the vectors W∗⊤ xt (at ) and then apply Azuma-Hoeffding to get the required result. Step 2:
From Corollary 2, with probability 1 − δ,
P p
1 T ˜ t (at ))⊤ xt (at )
≤ O mk1d k ln(mdT /δ) ln(T )/T .
T t=1 (W∗ − W
(6)
˜ t (at )⊤ xt (at ). It is therefore sufficient to bound the distance to S of the average of the vectors W ˜ t (at )⊤ xt (at ) and v ˜ t := W ˜ avg := Step 3: We use the shorthand notation of v this proof. The proof is completed by showing that with probability 1 − δ, p d(˜ vavg , S) ≤ R(T )/T + k1d k 2 ln(dT /δ)/T .
1 T
PT
˜t t=1 v
for the rest of (7)
˜ t⊤ θ − hS (θ) to predict θ t , The proof of (7) is as follows. Since we used online learning with gt (θ) = v therefore, from (3) and the definition of R(T ), we have that P ˜ avg − hS (θ)} ≤ 1/T Tt=1 θ ⊤ ˜ t − hS (θ t ) + R(T )/T . d(˜ vavg , S) = max||θ||∗ ≤1 {θ ⊤ v (8) t v
Now, for all t,
⊤ ˜ ˜ t |Ht−1 ] = EXt ∼D [θ ⊤ EXt ∼D [θ ⊤ t Wt (at ) xt (at )|Ht−1 ] t v ⊤ ˜ ≤ EX ∼D,a∼q∗ (X ) [θ ⊤ t Wt (a) xt (a)|Ht−1 ] t
t
⊤ ≤ EXt ∼D,a∼q∗ (Xt ) [θ ⊤ t W∗ xt (a)|Ht−1 ] (with probability 1 − δ) h i ⊤ ∗ ⊤ ∗ = θ⊤ t EX∼D W∗ Xq (X) = θ t v(q ).
(9)
The first inequality above is due to the choice of at in the algorithm. The second inequality is from the ∗ ∗ ), S) = 0. The ˜ t |Ht−1 ] − hS (θ t ) ≤ θ ⊤ first part of Corollary 2. This gives E[θ ⊤ t v PTt v(q⊤ ) − hS (θ t ) ≤ d(v(q p ˜ t − hS (θ t ) ≤ k1d k 2T ln(dT /δ), proof is now completed by using Azuma-Hoeffding to bound t=1 θt v with probability 1 − δ, and using this in (8).
4
The general case (linCBwCR)
In this section, we consider the linCBwCR problem (Definition 1). If we know the value of OPT := f (v(q ∗ )), we can reduce this problem to the Feasibility Problem (linCBwC) by defining the constraint set to be S ′ = {v : f (v) ≥ OPT, v ∈ S}. 7
A common approach [20, 7] to eliminate the assumption that OPT is known is to get increasingly accurate estimates of OPT computed at geometric time intervals as the algorithm progresses. However this approach does not seem to be directly employable here. As we shall see later, due to bandit feedback in this problem, estimating OPT seems to require “pure exploration” for some rounds where the arms picked are tailored to getting such an estimate. Further, the error in the estimate thus obtained using √ ∗ / T ), where Z ∗ is a parameter (defined later) that ˜ T0 rounds turns out to be of form O(mk1 kZ 0 d captures the tradeoff between the objective and constraints. Thus, the knowledge or estimation of Z ∗ is required for this approach in addition to estimating OPT. Instead, following [5], we use an approach that requires only the knowledge or even an O(1) approximation of Z ∗ .2 Let OPTγ denote the value of the optimal policy when the feasibility constraint is relaxed from v(q) ∈ S to d(v(q), S) ≤ γ for any γ ≥ 0. OPTγ :=
maxq f (v(q)) such that d(v(q), S) ≤ γ
Lemma 5. There exists Z ≥ 0 such that that for all γ ≥ 0, OPTγ ≤ OPT + minimum value of such Z.
(10) Z 2 γ.
Let Z ∗ denote the
Assumption 1. Assume we are given Z ≥ L satisfying the condition in Lemma 5.
Below, we present an algorithm under Assumption 1. The algorithm is similar to that in the previous section: it constructs estimates of the weight matrix exactly as in Section 3.1. Here, we run two instances of the online learning algorithm: we search over the domain of Fenchel dual variables for the objective in addition to that for the constraints as in the previous section. The optimistic estimates are constructed using the vector obtained on combining the dual variables for the objective and those for the constraints using Z. The algorithm then chooses the arm which appears to be the best according to optimistic estimates, as before. Algorithm 2 Algorithm for linCBwCR ˆ 1 , G1 , two instances of OL-Predict with domain Ω1 = {θ : kθk∗ ≤ 1}, and Ω2 = {φ : Initialize W ||φ||∗ ≤ L}, θ 1 and φ1 . for all t = 1, ..., T do Observe Xt . ˜ t (a) := arg maxW ∈Gt xt (a)⊤ W (φt + Zθ t ). For every a ∈ [K], compute W ˜ t (a)(φt + Zθ t ). Play the arm at := arg mina∈[K] xt (a)⊤ W Observe vt (at ). ˆ t+1 and Gt+1 . Use xt (at ) and vt (at ) to obtain W ˜ t (at )θ − hS (θ). θ t+1 := OL-Predict1 (θ 1 , g1 , . . . , θ t , gt ) with gt (θ) = xt (at )⊤ W ⊤ ˜ t (at )φ − f ∗ (φ). φt+1 := OL-Predict2 (φ1 , ψ1 , . . . , φt , ψt ) with ψt (θ) = xt (at ) W end for Theorem 3. Given a Z ≥ L as per Assumption 1, Algorithm 2 achieves the following bounds with probability 1 − δ: p avg-regret1 (T ) ≤ Z · O R(T )/T + mk1d k ln(dT /δ) ln(T )/T , p avg-regret2 (T ) ≤ O R(T )/T + mk1d k ln(dT /δ) ln(T )/T .
Here R(T ) = R1 (T ) + R2Z(T ) , with R1 (T ) being the regret for OCO with {gt , θ t }, and R2 (T ) being the √ regret for OCO withp{ψt , φt }, as given by Lemma 4. Specifically, R(T ) = O( dT ) when ℓ2 norm is used and R(T ) = O( log(d)T ) for ℓ∞ norm. 2 As an application of this approach, for the special case of budget constraints (linCBwK), we show how to get an approximation of Z ∗ using the initial few rounds, giving a sub-linear regret overall; see Section 5 for details.
8
5
Budget Constraints (linCBwK)
For the special case of linCBwK, knowing or estimating OPT is sufficient to estimate Z. OPT Lemma 6. Z2 = (B/T ) satisfies the condition in Lemma 5, when S is given by budget constraints and ℓ∞ norm is used for distance.
Therefore, if we know OPT, we can use Algorithm 2 with above value of Z plus 1, and distance ˜ OPT + 1) √m ) regret in objective (as given by Theorem 3). We defined by ℓ∞ norm, to achieve an O(( (B/T ) T show how to estimate Z as part of the algorithm for the special case of linCBwK, using the first few rounds. Also, we are not allowed to violate the budgets at all for this problem, so the algorithm needs to be modified to handle this. Given parameters T0 and B ′ , the algorithm is as follows. In the first T0 rounds do a “pure exploration” and estimate a Z satisfying Lemma 5. Set aside a B ′ amount from the remaining budget for each dimension. Run Algorithm 2 for the remaining time steps with the remaining budget. Stop if the full actual budget B is consumed for any dimension. p √ Theorem 4. Using the above algorithm with T0 such that B > max{T0 , mT / T0 }, B ′ = m T ln(dT /δ) ln(T ) and an appropriate estimation procedure for Z, we get a high probability regret bound of ˜ ((OPT/(B/T ) + 1) (T0/T + m/√T )) . O In particular, using T0 =
√
T , and assuming B > mT 3/4 gives a regret bound of ˜ ((OPT/(B/T ) + 1) m/√T ) . O
The proof of Theorem 4 uses the following estimate for Z obtained as described in Section 5.1. p Lemma 7. Let γ = 2m ln(T0 ) ln(T0 d/δ)/T0 . Using the first T0 rounds for exploration, we can obtain Z such that with probability 1 − O(δ), OPT (B/T )
7γ OPT (B/T ) )( (B/T )
+ 1).
mT ˜ √ ≥ γ, i.e., B ≥ Ω( ), Z is a constant factor approximation of T0 m OPT ˜ ( √ + 1 ≥ Z ∗ , therefore Theorem 3 provides an O + 1) regret bound if the algorithm is (B/T ) T
Lemma 7 implies that as long as OPT (B/T )
+ 1 ≤ Z ≤ (1 + B T
not stopped due to budget violation. As implied by the bound on avg-regret 2 (T ) in Theorem 3, setting aside B ′ budget ensures that budget is not violated with high probability. However, the reduction of ′ the budget by B ′ incurs additional BB OPT regret. Also, using the first T0 rounds for pure exploration can consume (at most) T0 amount from the budget, incurring an additional regret of TB0 OPT. Together, these observations give Theorem 4.
5.1
Estimating OPT
Here we sketch the high level idea behind obtaining the result in Lemma 7 by estimating OPT. Due to bandit feedback, estimating OPT in our problem has significant challenges compared to the full information settings like those in [5, 20, 7]. The approach used in those papers is to use the value of the policy that performs the best on the observations seen so far. However, in case of bandit feedback, observations are made only for the arms picked by the algorithm, therefore, it is not possible to directly evaluate any policy other than the one that was followed in the history. A fix for this, inspired by [22, 3], is to play every arm with some minimum probability in every round, so that using importance sampling we can re-weight the observations and construct an unbiased estimator of the outcome vector 9
for any arm. A crude version of this that plays every arm √ with probability 1/K in every round for T0 ∗ / T ). Instead, we provide a method that takes ˜ rounds would obtain an estimation error of O(KmZ 0 advantage of the linear structure of the problem, and explores in the m-dimensional space of contexts and weight vectors to obtain bounds independent of K. We use the following procedure. In every round t = 1, . . . , T0 , after observing Xt , let pt ∈ ∆[K] be pt := arg max kXt pkM −1 , where Mt := I
(11)
t p∈∆[K] Pt−1 + i=1 (Xi pi )(Xi pi )⊤ .
(12)
Select arm at = a with probability pt (a). In fact, since Mt is a PSD matrix, due to convexity of the ˆt function kXt pk2M −1 , it is the same as playing at = arg maxa∈[K] kxt (a)kM −1 . Construct an estimate W t t of W∗ at time t as ˆ t := M −1 Pt−1 (Xi pi )vi (ai )⊤ . W t i=1 ˆ γ of OPT as: And, for some value of γ defined later, obtain an estimate OPT ˆ γ OPT
:=
P 0 ˆ⊤ maxq f ( T10 Ti=1 Wi Xi q(Xi )]) 1 PT0 ˆ ⊤ such that d( T0 i=1 Wi Xi q(Xi )], S) ≤ γ.
(13)
For an intuition about the choice of arm in (11), observe from the discussion in Section 2.1 that every column w∗j of W∗ is guaranteed to lie inside the confidence ellipsoid centered at column w ˆ tj 2 ˆ of Wt , namely the ellipsoid, kµ − w ˆ tj kMt ≤ 4m ln(T m/δ). Note that this ellipsoid has principle axes as eigenvectors of Mt , and the length of semi-principle axes is given by inverse eigenvalues of Mt . Therefore, by maximizing kXt pkM −1 we are choosing the context closest to the direction of the longest t principal axes of the confidence ellipsoid, i.e. in the direction of maximum uncertainty. Intuitively, this corresponds to pure exploration: by making an observation in the direction where uncertainty is large we can reduce the uncertainty in our estimate most effectively. ˆ γ , we want the A more algebraic explanation is as follows. For aP good estimation of OPT by OPT T0 ˆ ˆ ⊤ ˆ t and W∗ to be close enough so that | 1 estimates W t=1 (Wt − W∗ ) Xt q(Xt )| is small for all policies T0 q, and in particular for optimal policies. Now, using Cauchy-Schwartz this is bounded by 1 PT0 ˆ t=1 k1d kkWt − W∗ kMt kXt q(Xt ))kM −1 , T0 t
where we define kW kM , the M -norm of matrix p W to be the max of column-wise M -norms. Using Lemma ˆ t − W∗ kMt is bounded by 2 m ln(T0 md/δ) with probability 1 − δ. Lemma 2 bounds the 1, the term kW P 0 second term Tt=1 kXt q(Xt )kM −1 but only when q is the played policy. This is where we use that the t P 0 P 0 played policy pt was chosen to maximize kXt pt kM −1 , so that Tt=1 kXt q(Xt )kM −1 ≤ Tt=1 kXt pt kM −1 t t t p PT0 PT0 and the bound t=1 kXt pt kM −1 ≤ mT0 ln(T0 ) given by Lemma 2 actually bounds t=1 kXt q(Xt )kM −1 t t for all q. We prove the following lemma. p Lemma 8. For γ = 2k1d km ln(T0 ) ln(T0 d/δ)/T0 , with probability 1 − O(δ), ˆ 2γ ≤ OPT + 6γ(Z ∗ + L). OPT − Lγ ≤ OPT
For linCBwK, Z ∗ ≤ bounds of Lemma 7.
OPT (B/T ) , L
2γ
= 1, k1d k = k1d k∞ = 1. Therefore, Z =
10
ˆ (OPT +Lγ) (B/T )
+ 1 satisfies the
References [1] Y. Abbasi-yadkori, D. P´ al, and C. Szepesv´ari. Improved algorithms for linear stochastic bandits. In NIPS, 2012. [2] J. Abernethy, P. L. Bartlett, and E. Hazan. Blackwell approachability and low-regret learning are equivalent. In COLT, 2011. [3] A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. E. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In ICML 2014, June 2014. [4] S. Agrawal and N. R. Devanur. Bandits with concave rewards and convex knapsacks. In Proceedings of the Fifteenth ACM Conference on Economics and Computation, EC ’14, 2014. [5] S. Agrawal and N. R. Devanur. Fast algorithms for online stochastic convex programming. In SODA, pages 1405–1424, 2015. [6] S. Agrawal, N. R. Devanur, and L. Li. Contextual Bandits with Global Constraints and Objective. ArXiv e-prints, June 2015. [7] S. Agrawal, Z. Wang, and Y. Ye. A dynamic near-optimal algorithm for online linear programming. Operations Research, 62:876 – 890, 2014. [8] P. Auer. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res., 3, Mar. 2003. [9] M. Babaioff, S. Dughmi, R. D. Kleinberg, and A. Slivkins. Dynamic pricing with limited supply. ACM Trans. Economics and Comput., 3(1):4, 2015. [10] A. Badanidiyuru, R. Kleinberg, and Y. Singer. Learning on a budget: posted price mechanisms for online procurement. In Proc. of the 13th ACM EC, pages 128–145. ACM, 2012. [11] A. Badanidiyuru, R. Kleinberg, and A. Slivkins. Bandits with knapsacks. In FOCS, pages 207–216, 2013. [12] A. Badanidiyuru, J. Langford, and A. Slivkins. Resourceful contextual bandits. In Proceedings of The Twenty-Seventh Conference on Learning Theory (COLT-14), pages 1109–1134, 2014. [13] O. Besbes and A. Zeevi. Dynamic pricing without knowing the demand function: Risk bounds and near-optimal algorithms. Operations Research, 57(6):1407–1420, 2009. [14] A. Beygelzimer, J. Langford, L. Li, L. Reyzin, and R. E. Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proc. of the 14th AIStats, pages 19–26, 2011. [15] D. Chakrabarti and E. Vee. Traffic shaping to optimize ad delivery. In Proceedings of the 13th ACM Conference on Electronic Commerce, EC ’12, 2012. [16] X. Chen and Z. Wang. A near-optimal dynamic learning algorithm for online matching problems with concave returns. http://arxiv.org/abs/1307.5934, 2013. [17] W. Chu, L. Li, L. Reyzin, and R. E. Schapire. Contextual Bandits with Linear Payoff Functions. Journal of Machine Learning Research - Proceedings Track, 15:208–214, 2011. [18] V. Dani, T. P. Hayes, and S. M. Kakade. Stochastic Linear Optimization under Bandit Feedback. In COLT, 2008. 11
[19] N. R. Devanur and T. P. Hayes. The adwords problem: online keyword matching with budgeted bidders under random permutations. In EC, 2009. [20] N. R. Devanur, K. Jain, B. Sivan, and C. A. Wilkens. Near optimal online algorithms and fast approximation algorithms for resource allocation problems. In EC, 2011. [21] W. Ding, T. Qin, X.-D. Zhang, and T.-Y. Liu. Multi-armed bandit with budget constraint and variable costs. In Proc. of the 27th AAAI, pages 232–238, 2013. [22] M. Dud´ık, D. Hsu, S. Kale, N. Karampatziakis, J. Langford, L. Reyzin, and T. Zhang. Efficient optimal learning for contextual bandits. In Proc. of the 27th UAI, pages 169–178, 2011. [23] J. Feldman, M. Henzinger, N. Korula, V. S. Mirrokni, and C. Stein. Online stochastic packing applied to display ad allocation. In Proceedings of the 18th Annual European Conference on Algorithms: Part I, ESA’10, 2010. [24] S. Guha and K. Munagala. Approximation algorithms for budgeted learning problems. In STOC, pages 104–113, 2007. [25] A. Gupta and M. Molinaro. How the Experts Algorithm Can Help Solve LPs Online. Algorithms - ESA 2014, Lecture Notes in Computer Science, 8737:517–529, 2014. [26] A. Gy¨orgy, L. Kocsis, I. Szab´ o, and C. Szepesv´ari. Continuous time associative bandit problems. In Proc. of the 20th IJCAI, pages 830–835, 2007. [27] C. Karande, A. Mehta, and P. Tripathi. Online bipartite matching with unknown distributions. In STOC, 2011. [28] T. Kesselheim, A. T¨ onnis, K. Radke, and B. V¨ocking. Primal beats dual on online packing LPs in the random-order model. In STOC, 2014. [29] O. Madani, D. J. Lizotte, and R. Greiner. The budgeted multi-armed bandit problem. In Learning Theory, pages 643–645. Springer, 2004. [30] M. Mahdian and Q. Yan. Online bipartite matching with random arrivals: an approach based on strongly factor-revealing LPs. In STOC, 2011. [31] S. Pandey and C. Olston. Handling advertisements of unknown quality in search advertising. In Advances in Neural Information Processing Systems, pages 1065–1072, 2006. [32] S. Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2):107–194, 2012. [33] A. Singla and A. Krause. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In Proc. of the 22nd WWW, pages 1167–1178, 2013. [34] A. Slivkins and J. W. Vaughan. Online decision making in crowdsourcing markets: Theoretical challenges (position paper). CoRR, abs/1308.1746, 2013. [35] L. Tran-Thanh, A. C. Chapman, E. M. de Cote, A. Rogers, and N. R. Jennings. Epsilon-first policies for budget-limited multi-armed bandits. In Proc. of the 24th AAAI, 2010. [36] L. Tran-Thanh, A. C. Chapman, A. Rogers, and N. R. Jennings. Knapsack based optimal policies for budget-limited multi-armed bandits. In AAAI, 2012.
12
[37] E. Vee, S. Vassilvitskii, and J. Shanmugasundaram. Optimal online assignment with forecasts. In EC ’10: Proceedings of the 11th ACM conference on Electronic commerce, 2010. [38] H. Wu, R. Srikant, X. Liu, and C. Jiang. Algorithms with logarithmic or sublinear regret for constrained contextual bandits. CoRR, abs/1504.06937, 2015.
13
Appendix
A
Concentration Inequalities
Lemma 9 (Azuma-Hoeffding inequality). If a super-martingale (Yt ; t ≥ 0), corresponding to filtration Ft , satisfies |Yt − Yt−1 | ≤ ct for some constant ct , for all t = 1, . . . , T , then for any a ≥ 0, −
Pr(YT − Y0 ≥ a) ≤ e
B
2
a2 PT 2 t=1 ct
.
Confidence ellipsoids
Proof of Corollary 1. The following holds with probability 1 − δ. T X t=1
⊤ |˜ µ⊤ t xt − µ∗ xt | ≤
T X t=1
≤
k˜ µt − µ∗ kMt kxt kM −1
s
t
m ln
1 + tm δ
! p √ mT ln(T ). + m
The inequality in the first line is a matrix-norm version of Cauchy-Schwartz (Lemma 10). The inequality in the second line is due to Lemmas 1 and 2. The lemma follows from multiplying out the two factors in the second line. Lemma 10. For any positive definite matrix M ∈ Rn×n and any two vectors a, b ∈ Rn , |a⊤ b| ≤ kakM kbkM −1 . ⊤ . Further, Proof. Since M is positive definite, there exists a matrix M1/2 such that M = M1/2 M1/2 −1 ⊤ . M −1 = M−1/2 M−1/2 where M−1/2 = M1/2 ⊤ ka⊤ M1/2 k2 = a⊤ M1/2 M1/2 a = a⊤ M a = kak2M .
Similarly, kM−1/2 bk2 = kbk2M −1 . Now applying Cauchy-Schwartz, we get that |a⊤ b| = |a⊤ M1/2 M−1/2 b| ≤ ka⊤ M1/2 kkM−1/2 bk = kakM kbkM −1 . ˜ t (a) and the observation Proof of Corollary 2. Here, the first claim follows simply from definition of W ∗ that with probability 1 − δ, W ∈ Gt . To obtain the second claim, P apply Corollary 1 with µ∗ = th column of W ⊤ ˜ ˜ ˜ ˜ w , y = x (a ), µ = [ W (a )] (the j (a )), to bound | t t t t j t t t t ([Wt (at )]j − w∗j ) xt (at )| ≤ P∗j t˜ ⊤ t |([Wt (at )]j − w∗j ) xt (at )| for every j, and then take the norm.
C
Appendix for Section 4 (linCBwCR)
Proof of Lemma 5. The proof of this lemma is similar to the proof of Lemma 5.1 in [5]. Let Ω := {y : ∃q, y = EX∼D [W∗⊤ Xq(X)]}. Then, Ω is a convex set. And, OPTγ can be written as the following convex optimization problem over y OPTγ := max f (y) such that d(y, S) ≤ γ. y∈Ω
14
Then, applying Lagrangian duality for convex programs OPTγ = min max f (y) + λ(γ − d(y, S)) λ≥0 y∈Ω
By Fenchel duality, for any L-Lipschitz concave function f , f (y) =
min
φ:kφk∗ ≤L
f ∗ (φ) − φ⊤ y,
where f ∗ is the Fenchel dual of f . Specifically, for distance function, d(y, S) =
max θ ⊤ y − hS (θ),
θ:kθk∗ ≤1
where for any convex set S, hS is defined as: hS (θ) := maxv∈S θ ⊤ v. Substituting in the expression for OPTγ , we get OPTγ
= min max
(f ∗ (φ) − φ⊤ y) + λγ − λ(θ ⊤ y − hS (θ))
min
λ≥0 y∈Ω |φk∗ ≤L,kθk∗ ≤1
= = ≤
min
max f ∗ (φ) − φ⊤ y − λθ ⊤ y + λhS (θ) + λγ
min
f (φ) + hΩ (−φ − λθ) + λhS (θ) + λγ
λ≥0,|φk∗ ≤L,kθk∗ ≤1 y∈Ω ∗
λ≥0,|φk∗ ≤L,kθk∗ ≤1 f ∗ (φ∗ ) + hΩ (−φ∗ ∗
= OPT + λ γ
− λ∗ θ ∗ ) + λ∗ hS (θ ∗ ) + λ∗ γ
Here, φ∗ , θ ∗ , λ∗ denote the optimal value of these variables for the corresponding optimization problem for OPT. Therefore, Z = λ∗ satisfies the condition in Lemma 5. Proof of Theorem 3. The proof is an extension of proof of Theorem 2, and uses the properties of Z to handle the objective in addition to constraints. P ˜ t (at )⊤ xt (at ), v ˜ t in this section. Similar ˜ t := W ˜ avg = T1 Tt=1 v We use the shorthand notation of v to the observation in Equation (8) in the proof of Theorem 2, using online learning guarantees we can obtain that for the choice of θ t , φt made by this algorithm, d(˜ vavg , S) ≤ f (˜ vavg ) ≥
T R (T ) 1 X ⊤ 1 ˜ t − hS (θ t ) + θt v . T t=1 T
T R (T ) 1 X ∗ 2 ˜ v f (φt ) − φ⊤ . t t + T T
(14)
(15)
t=1
˜ t (a)(φt + Zθ t ), and because For all t, since the algorithm selected at ∈ [K] to minimize xt (a)⊤ W ˜ t (a) is an optimistic estimate for every a, therefore using similar line of arguments as in Equation (9), W we get that with probability 1 − δ, ˜ t |Ht−1 ] ≥ f ∗ (φt ) + ZhS (θ t ) − (φt + Zθt )⊤ v(q ∗ ). E[f ∗ (φt ) + ZhS (θ t ) − (φt + Zθ t )⊤ v Below are the details of this argument. ˜ t (at )⊤ xt (at )|Ht−1 ] ˜ t |Ht−1 ] = E[(φt + Zθ t )⊤ W E[(φt + Zθ t )⊤ v P ˜ t (a)⊤ xt (a)q ∗ (Xt , a) |Ht−1 ] ≤ E[(φt + Zθ t )⊤ W a
≤ E[(φt + Zθ t )⊤ W∗⊤ Xt q ∗ (Xt )|Ht−1 ] (with probability 1 − δ) h i = (φt + Zθ t )⊤ Ex∼D W∗⊤ Xq ∗ (X) = (φt + Zθ t )⊤ v(q ∗ ). 15
(16)
Now, the term on the right hand side in Equation (16) is greater than or equal to ∗ ⊤ ∗ ⊤ ∗ min f (φt ) − φ v(q ) − Z max θ t v(q ) − hS (θt ) = f (v(q ∗ )) − Zd(v(q ∗ ), S) = OPT kφk∗ ≤L
kθk∗ ≤1
where the second last equality uses Fenchel duality, and the last equality uses d(v(q ∗ ), S) = 0. Therefore, for every t, the conditional expectation on the left hand side of (16) is lower bounded by OPT. Summing over t, and using Azuma-Hoeffding, we obtain that probability 1 − δ, T X t=1
p ˜ t − Z(θ ⊤ ˜ t − hS (θ t )) ≥ T OPT − k1d k 2T ln(dT /δ) f ∗ (φt ) − φ⊤ t v t v
(17)
Combining Equation (14) and (15) using Z, and substituting the bound from (17), we get f (˜ vavg ) − Zd(˜ vavg , S) ≥ OPT −
k1d k p R(T ) 2T ln(dT /δ) − Z T T
P ˜ avg := T1 Tt=1 v ˜ t is close Now, Step 1 and Step 2 in the proof of Theorem 2 hold as it is, so that v 1 PT ˜ avg by vavg in above to get with probability 1 − δ to vavg := T t=1 vt (at ). Therefore, we can replace v mk1d k p R(T ) f (vavg ) − Zd(vavg , S) ≥ OPT − Z O T ln(mdT /δ) ln(T ) + . (18) T T
Since d(vavg , S) ≥ 0, this proves the bound on avg-regret1 (T ). To obtain the bound on distance from the constraint set, first note that E[f (vavg )] ≤ OPTE[d(vavg ,S)]
To see this, let the random variable q˜ denote the empirical played policy, i.e. the policy implied by the contexts observed and the arms played X1 , a1 , . . . , XT , aT , so that by definition v(E[˜ q ]) = E[vavg ], and d(v(E[˜ q ]), S) = d(E[vavg ], S) ≤ E[d(vavg , S)]. Therefore, E[˜ q ] is a feasible solution for (10) with γ = E[d(vavg , S)], implying f (v(E[˜ q ])) ≤ OPTγ . Also, by concavity of f and definition of policy q˜, E[f (vavg )] ≤ f (E[vavg ]) = f (v(E[˜ q ])). Therefore, the above observation follows. Now, using the assumption about the parameter Z, E[f (vavg )] −
Z Z E[d(vavg , S)] ≤ OPTE[d(vavg ,S)] − E[d(vavg , S)] ≤ OPT 2 2
Using (18), this implies E[f (vavg )] −
Z E[d(vavg , S)] ≤ OPT ≤ E[f (vavg )] − ZE[d(vavg , S)] 2 mk1d k p R(T ) . +Z O T ln(mdT /δ) ln(T ) + T T
Rearranging the terms, we get
Z E[d(vavg , S)] ≤ Z O 2
mk1d k p R(T ) T ln(mdT /δ) ln(T ) + T T
.
Then, using d(E[vavg ], S) ≤ E[d(vavg , S)] and applying Azuma-Hoeffding, we get the desired bound on avg-regret2 (T ) with high probability.
16
D
Appendix for Section 5 (linCBwK)
Proof of Lemma 6. The proof of this lemma is similar to the proof of Lemma 4 in [6]. In linCBwK problem, OPTγ represents the value of optimal policy when the budget constraints are relaxed by γ, i.e. for γ OPT feasibility set S γ = {v : v ≤ B T 1 + γ}. Now, suppose for contradiction that OPT > OPT + (B/T ) γ. Let qˆ be the optimal policy for OPTγ . Then, one could construct a feasible policy q ′ for the original B problem with budget B by scaling down the probabilities given by qˆ by B+γT . (And do nothing with the ′ remaining probability). Since qˆ consumed at most B + γT budget, q will consume at most B budget. B OPT B > (OPT + (B/T Also, the value of policy q ′ will be at least OPTγ B+γT ) γ) B+γT = OPT. Thus we arrive at a contradiction.
E
Appendix for Section 5.1 (Estimating OPT)
Proof of Lemma 8. Let us define an intermediate ”sample optimal” as: P 0 W∗⊤ Xi q(Xi )]) maxq f ( T10 Ti=1 γ P OPT := T 0 such that d( T10 i=1 W∗⊤ Xi q(Xi )], S) ≤ γ
(19)
Above sample optimal knows the weight matrix W∗ , the error comes only from approximating the expected value over context distribution by average over the observed contexts. We do not actually γ compute OPT , but will use it for the convenience of proof exposition. The proof involves two steps. γ
Step 1: Bound |OPT − OPT|. 2γ
ˆ Step 2: Bound |OPT
γ
− OPT | by bounding k
ˆ − W∗ )⊤ Xi q(Xi )k for all q.
PT0
i=1 (Wi
Step 1 bound can be borrowed from the work on Online Stochastic Convex Programming in [5]: since W ∗ is known so there is effectively full information before making the decision, i.e., consider the vectors W∗⊤ xt (a) as reward vectors which can be observed for all arms a before choosing the distribution over ˆ γ as defined by Equation arms to be played at time t, therefore, the setting in [5] applies. In fact, OPT γ (F.10) in [5] when At = {W∗⊤ xt (a), a ∈ [K]}, is same as thepOPT defined here. And using Lemma F.4 and Lemma F.6 in [5], we obtain that for any γ ≥ 2k1d km ln(T0 ) ln(T0 d/δ)/T0 , with probability 1 − O(δ), γ
OPT − Lγ ≤ OPT ≤ OPT + 2γ(Z ∗ + L).
For Step 2, we show that with probability 1 − δ, for all q, γ ≥ 2k1d km T0 1 X ˆ i − W∗ )⊤ Xi q(Xi )k ≤ γ (W k T0
(20)
p
ln(T0 ) ln(T0 d/δ)/T0
(21)
i=1
p ˆ 2γ for γ ≥ 2k1d km ln(T0 ) ln(T0 d/δ)/T0 . This is sufficient to prove both lower and upper bound on OPT γ For lower bound, we can simply use (21) for optimal policy for OPT , denoted by q¯. This implies ˆ 2γ , and that (because of relaxation of distance constraint by γ) q¯ is a feasible primal solution for OPT therefore using (20), ˆ 2γ ≥ OPTγ ≥ OPT − Lγ. OPT ˆ 2γ . Then, using (20), For the upper bound, we can use (21) for the optimal policy qˆ for OPT ˆ 2γ ≤ OPT3γ ≤ OPT + 6γ(Z ∗ + L). OPT 17
Combining, this proves the desired lemma statement: ˆ 2γ ≤ OPT + 6γ(Z ∗ + L) OPT − Lγ ≤ OPT
(22)
What remains is to proof the claim in (21). Observe that for any q, k
T0 T0 X X ˆ t − W∗ )⊤ Xt q(Xt )k ˆ t − W∗ )⊤ Xt q(Xt )k ≤ k(W (W t=1 T0 X
t=1
≤
t=1
ˆ t − W∗ kMt kXt q(Xt )k −1 k1d kkW M t
ˆ t − W∗ kMt = maxj kw where kW ˆ tj − w∗j kMt . ˆ t , we have that with probability 1 − δ for all t, Now, applying Lemma 1 to every column w ˆ tj of W p p ˆ t − W∗ kMt ≤ 2 m log(td/δ) ≤ 2 m log(T0 d/δ) kW
And, by choice of pt
kXt q(Xt )kM −1 ≤ kXt pt kM −1 . t
t
Also, by Lemma 2, T0 X t=1
Therefore, substituting, k
kXt pt kM −1 ≤ t
p mT0 ln(T0 )
T0 T0 X X p ˆ t − W∗ )⊤ Xt q(Xt )k ≤ k1d k(2 m log(T0 d/δ)) kXt pt kM −1 (W t=1
t=1
t
p p ≤ k1d k(2 m log(T0 d/δ)) mT0 ln(T0 )
≤ Tγ
18
Nikhil R. Devanur†
arXiv:1507.06738v1 [cs.LG] 24 Jul 2015
July 27, 2015
Abstract We consider the linear contextual bandit problem with global convex constraints and a concave objective function. In each round, the outcome of pulling an arm is a vector, that depends linearly on the context of that arm. The global constraints require the average of these vectors to lie in a certain convex set. The objective is a concave function of this average vector. This problem turns out to be a common generalization of classic linear contextual bandits (linContextual) [8, 17, 1], bandits with concave rewards and convex knapsacks (BwCR) [4], and the online stochastic convex programming (OSCP) problem [5]. We present algorithms with near-optimal regret bounds for this problem. Our bounds compare favorably to results on the unstructured version of the problem [6, 12] where the relation between the contexts and the outcomes could be arbitrary, but the algorithm only competes against a fixed set of policies. We combine techniques from the work on linContextual, BwCR and OSCP in a nontrivial manner while also tackling new difficulties that are not present in any of these special cases.
∗ †
Microsoft Research. [email protected]. Microsoft Research. [email protected].
1
Introduction
In the contextual bandit problem [8, 14, 22, 3], the decision maker observes a sequence of contexts (or features). In every round she needs to pull one out of K arms, after observing the context for that round. The outcome of pulling an arm may be used along with the contexts to decide future arms. Contextual bandit problems have found many useful applications such as online recommendation systems, online advertising, and clinical trials, where the decision in every round needs to be customized to the features of the user being served. The linear contextual bandit problem [1, 8, 17] is a special case of the contextual bandit problem, where the outcome is linear in the feature vector encoding the context. As pointed by [3], contextual bandit problems represent a natural half-way point between supervised learning and reinforcement learning: the use of features to encode contexts and the models for the relation between these feature vectors and the outcome are often inherited from supervised learning, while managing the exploration-exploitation tradeoff is necessary to ensure good performance in reinforcement learning. The linear contextual bandit problem can thus be thought of as a midway between the linear regression model of supervised learning, and reinforcement learning. Recently, there has been a significant interest in introducing multiple “global constraints” in the standard bandit setting [11, 4, 12, 6]. Such constraints are crucial for many important real-world applications. For example, in clinical trials, the treatment plans may be constrained by the total availability of medical facilities, drugs and other resources. In online advertising, there are budget constraints that restrict the number of times an ad is shown. Other applications include dynamic pricing, dynamic procurement, crowdsourcing, etc.; see [11, 4] for many such examples. In this paper, we consider a very general version of the linear contextual bandit problem with global constraints and objective (henceforth, linCBwCR). In this problem, the context vectors are generated i.i.d. in every round from some unknown distribution, and the expected outcome on picking an arm is in the form of a vector that depends linearly on the context vector. The aim of the decision maker is to maximize a given concave objective function of the average of these outcome vectors while ensuring that the average lies in a given convex set. Below, we give a more precise definition of this problem. We use the following notational convention throughout: vectors are denoted by bold face lower case letters, while matrices are denoted by regular face upper case letters. Other quantities such as sets, scalars, etc. may be of either case, but never bold faced. All vectors are column vectors, i.e., a vector in n dimensions is treated as an n × 1 matrix. The transpose of matrix A is A⊤ . Definition 1 (linCBwCR). There are K “arms”, which we identify with the set [K]. The algorithm is initially given as input a convex set S ⊆ [0, 1]d , and an L-Lipschitz concave function f with domain [0, 1]d . We then proceed in rounds t = 1, 2, . . . , T, where in every round t, a pair of a context matrix and an outcome matrix (Xt , Vt ) ∈ [0, 1]m×K × [0, 1]d×K are drawn from an unknown distribution D, independent of everything in previous rounds. The column of Xt corresponding to arm a ∈ [K] is called the context vector for arm a and is denoted by xt (a) ∈ [0, 1]m . The column of Vt corresponding to an arm a is called the “outcome” vector for arm a and is denoted by vt (a) ∈ [0, 1]d . There exist an (unknown) weight matrix W∗ ∈ [0, 1]m×d such that E[Vt |Xt ] = W∗⊤ Xt . In each round, the algorithm first observes context Xt , and then chooses an arm at ∈ A, P and finally 1 observes vt (at ). The goal of the algorithm is to pick arms such that the average vector T Tt=1 vt (at ) P maximizes f ( T1 Tt=1 vt (at )) and is inside S.
We compare the performance of an algorithm to that of the optimal adaptive policy that knows the distribution D and the weight matrix W∗ , and can take into account the history upto that point as well as the current context to decide (possibly with randomization) which arm to pull at time t. However, it is easier to work with an upper bound on this, which is the optimal expected reward of a static policy 1
that is required to satisfy the constraints only in expectation. This technique has been used in several related problems and is standard by now [20, 11]. Definition 2 (Optimal Static Policy). We consider policies that are context dependent but are nonadaptive: for a policy q, let q(X) ∈ ∆K (the unit simplex) denote the probability distribution over arms played when the context is X ∈ X . Define v(q) to be the expected outcome vector of policy q, i.e. v(q) := E(X,V )∼D [V q(X)] = EX∼D [W∗⊤ Xq(X)]. Let
q ∗ :=
arg maxq f (v(q)) such that v(q) ∈ S
(1) (2)
be the optimal static policy. We assume that there exists a feasible policy, i.e. v(q ∗ ) ∈ S, and we denote the optimal value by OPT := f (v(q ∗ )). Definition 3 (Regret). We define two regrets: regret in objective and regret in constraints. P Let at be the ¯ 1:T := T1 Tt=1 vt (at ), arm played at time t by the algorithm. For convenience of notation let us define v • Regret in objective. avg-regret1 (T ) = OPT − f (¯ v1:T ).
• Regret in constraints is defined as the distance of the average of played vectors from S; we let d(·, ·) denote a distance function. avg-regret2 (T ) := d(¯ v1:T , S). We consider following useful special cases. Feasibility problem (linCBwC): In this special case of linCBwCR, there is no objective function ¯ 1:T be in the set S. The f , and the aim of the algorithm is to have the average outcome vector v ¯ 1:T to S, i.e., by avg-regret2 (T ). We will performance of the algorithm is measured by the distance of v first illustrate our algorithm and proof techniques for this special yet nontrivial case. Budget constraints (linCBwK): This is a well studied special case with linear objectives and constraints; a generalization of the Bandits with Knapsacks (BwK) problem of [11]. Here, the outcome vector can be broken down into two components: a scalar reward rt (at )P ∈ [0, 1] and a d − 1-dimensional consumption vector vt (at ) ∈ [0, 1]d−1 . The objective is to maximize Tt=1 rt (at ) while ensuring that PT t=1 vt (at ) ≤ B1, where 1 is the vector of all 1s and B > 0 is some scalar. Such budget constraints are equivalent to having the constraint set S be equal to {v : 0 ≤ v ≤ B T 1}. Also, it is possible to simply stop once the budget is consumed; equivalently we assume that the algorithm always has the “do nothing” option of getting 0 reward and 0 cost. For this problem, we provide an algorithm that always satisfies the budget constraints, i.e., avg-regret2 (T ) = 0, while bounding avg-regret 1 (T ).
1.1
Main results
Our main result is an algorithm with near-optimal regret bound for linCBwCR. The algorithm needs to know a certain problem dependent parameter Z, defined by Assumption 1 in Section 4. Such knowledge is not required for linCBwC, however. For the special case of budget constraints (linCBwK), any 2OPT Z ≥ (B/T ) can be used, and we show how one such Z can be obtained from the first few rounds. Below we state the main result when k · k denotes either ℓ2 or ℓ∞ norm. In the text, we will provide a more detailed theorem statement that is applicable to any norm. Theorem 1. Let k · k denote either ℓ2 or ℓ∞ norm, and be used to define the distance d(·, ·). Assume f is L-Lipschitz with respect to k · k. Then, for linCBwCR, there is an algorithm that takes as input a problem-dependent parameter Z ≥ L (satisfying Assumption 1), and with probability at least 1 − δ has p p avg-regret1 (T ) = O Zmk1d k ln(dT /δ) ln(T )/T , avg-regret2 (T ) = O mk1d k ln(dT /δ) ln(T )/T . 2
For linCBwC, no knowledge of Z is needed, and the algorithm has the same avg-regret2 (T ) as above. Remark 1. (Near-optimality of regret bounds) In [18], it was shown that for the√linear contextual bandits problem, no online algorithm can achieve a regret bound better than Ω(m T ). In fact, they prove this lower bound for linear contextual bandits with static contexts. Since that problem is a special case of our linCBwCR problem with d = 1, this shows that the dependence on m and T in the above (average) regret bounds is optimal upto log factors. The dependence on Zk1d k is common in special cases of this problem [4, 5]. Moreover, we don’t need to know D, as is assumed in related work [6, 12, 38]. Remark 2. (Budget constraints) For the special case of linCBwK, we show how to estimate Z using ˜ ((OPT/(B/T ) + 1) m/√T ), the observations from the first few rounds. We get an overall regret bound of O which compares favorably to the regret bounds for the general contextual version obtained in [12, 6]. 3/4 (Details in Section 1.2.) The only difference is that √ we need the budget B to be Ω(mT ). Getting similar bounds for budgets as small as B = Θ(m T ) is an interesting open problem. Due to space constraints, we have eliminated many proofs from the main text. All the missing proofs are in the appendix.
1.2
Challenges and related work
Relation to BwCR [4] and OSCP [5]. It is easy to see that the linCBwCR problem is a generalization of the linear contextual bandits problem. There, the outcome is scalar and the goal is to simply maximize the sum of these. Remarkably, the linCBwCR problem also turns out to be a common generalization of bandits with concave rewards and convex knapsacks (BwCR) problem considered in [4], and the online stochastic convex programming (OSCP) problem of [5]. In both BwCR and OSCP problems, P the outcome of every round t is a vector vt and the goal of the algorithm is to maximize f ( T1 Tt=1 vt ) P while ensuring that T1 Tt=1 vt ∈ S. The problems differ in how this vector is picked. In the OSCP problem, in every round t, the algorithm may pick any vector from a given set At of d-dimensional vectors. The set At is drawn i.i.d. from an unknown distribution over sets of vectors. This corresponds to the special case of linCBwCR, where m = d and the outcome Vt is always equal to the context Xt . In the BwCR problem, there is a fixed set of arms, and for each arm there is an unknown distribution over vectors. The algorithm picks an arm and a vector is drawn from the corresponding distribution for that arm. This corresponds to the special case of linCBwCR, where m = K and the context Xt = I, the identity matrix, for all t. We use techniques from all three special cases: our algorithms follow the primal-dual paradigm using Fenchel duality, that was established in [4]. In order to deal with linear contexts, we use techniques from [1, 8, 17] to estimate the weight matrix W∗ , and define “optimistic estimates” of W∗ . We also use the technique of combining the objective and the constraints using a certain tradeoff parameter that was introduced in [5]. Further new difficulties arise, such as in estimating the optimum value from the first few rounds, a task that follows from standard techniques in each of the special cases but is very challenging here. One can see that the problem is indeed more than the sum of its parts, from the fact 3/4 ), unlike either special case for ˜ that we get the optimal bound for linCBwK only when B ≥ Ω(mT √ ˜ which the optimal bound holds for all B (but is meaningful only for B = Ω(m T )). The approach in [4] (for BwCR) extends to the case of “static” contexts,1 where each arm has a context that doesn’t change over time. The OSCP problem of [5] is not a special case of linCBwCR with static contexts; this is one indication of the additional difficulty of dynamic over static contexts. Relation to general contextual bandits. There have been recent papers [6, 12] that solve problems similar to linCBwCR but for general contextual bandits. Here the relation between contexts and 1
It was incorrectly claimed in [4] that the approach can be extended to dynamic contexts without much modifications.
3
outcome vectors is arbitrary and the algorithms p of context dependent pcompete with an arbitrary fixed set policies Π, with the regret bounds scaling as K log(|Π|). (In our results, this K log(|Π|) is replaced by m.) Since linCBwCR is a special case of this problem, we may compare the regret bounds obtained by applying these results. Firstly, we have no dependence on K, which enables us to consider problems with large action spaces. Further, with global constraints and objective, the effective policy space for linear bandits is doubly exponential in its dimensions, thus giving exponentially large regret bounds for linCBwCR using this approach. The key is to determine the size of the smallest policy set that is guaranteed to contain a near optimal policy. Trivially, the number of possible context dependent policies K |X | , where X denotes the space of contexts. In linCBwCR, X = [0, 1]m×K , giving |X | = O(1/ǫ(mK) ) using a discretization of the space into multiples of some small enough ǫ. We could try to reduce the policy space using linearity. In linear contextual bandits, the objective which is the sum of rewards, is decomposable across time steps. Given the weight vector W∗ , the optimal policy is to chose the arm a that maximizes the expected reward W∗⊤ xt (a) at time t, which is independent of the distribution D of the contexts. It thus suffices to have Π be the set consisting of one policy of this type for every possible W∗ , giving |Π| = ǫ1m (since d = 1). However, in the 1 P linCBwCR problem, due to nondecomposable objective f ( T t W∗⊤ Xt q(Xt )), and/or distance function P d( T1 t W∗⊤ Xt q(Xt ), S), the optimal policy cannot locally maximize the reward at every step. Even for a fixed W∗ , the optimal policy may depend on D. We can have Π consist of one policy for every possible 1 pair of D and W∗ , but that gives |Π| = |∆X | · ǫd×m which is again doubly exponential in m × K since |X | = O(1/ǫ(mK) ). Other related work. Budget constraints in a bandit setting has recieved considerable attention, but most of the early work focussed on special cases such as a single budget constraint in the regular √ (non-contextual) setting [21, 24, 26, 29, 35, 36]. Recently, [38] showed an O( T ) regret in the linear contextual setting with a single budget constraint, when costs depend only on contexts and not arms. Budget constraints that arise in particular applications such as online advertising [15, 31], dynamic pricing [9, 13] and crowdsourcing [10, 33, 34] have also been considered. There has also been a long line of work studying special cases of the OSCP problem [19, 20, 23, 7, 28, 25, 37, 30, 27, 16].
2 2.1
Preliminaries Confidence Ellipsoid
Consider a stochastic process which in each round t, generates a pair of observations (rt , y t ), such that rt is an unknown linear function of y t plus some 0-mean bounded noise, i.e., rt = µ⊤ ∗ y t + ηt , where m y t , µ∗ ∈ R , |ηt | ≤ 2R, and E[ηt |y 1 , r1 , . . . , y t−1 , rt−1 , y t ] = 0. At any time t, a high confidence estimate of the unknown vector µ∗ can be obtained by building ˆ t constructed from the oba “Confidence Ellipsoid” around the ℓ2 -regularized least-square estimate µ servations made so far. This technique is common in prior work on linear contextual bandits (e.g., in [8, 17, 1]). For any regularization parameter λ > 0, let P P ⊤ ˆ t := Mt−1 t−1 Mt := λI + t−1 i=1 y i y i , and µ i=1 y i ri .
ˆ t . For with center µ The following result from [1] shows that µ∗ lies with high probability in an ellipsoid p any positive semi-definite (PSD) matrix M, define the M -norm as kµkM := µ⊤ M µ. The confidence ellipsoid at time t is defined as n √ o p ˆ t kMt ≤ R m ln ((1+tm/λ)/δ ) + λm . Ct := µ ∈ Rm : kµ − µ Lemma 1 (Theorem 2 of [1]). If ∀ t, kµ∗ k2 ≤
√
m and ky t k2 ≤ 4
√
m, then with prob. 1 − δ, µ∗ ∈ Ct .
Another useful observation about this construction is stated below. It first appeared as Lemma 11 of [8], and was also proved as Lemma 3 in [17]. p PT Lemma 2 (Lemma 11 of [8]). ky k mT ln(T ). −1 ≤ t t=1 M t
As a corollary of the above two lemmas, we obtain a bound on the total error in the estimate provided by “any point” from the confidence ellipsoid. (Proof is in Appendix B.)
˜ t ∈ Ct be a point in the confidence ellipsoid, with λ = 1, 2R = 1. Corollary 1. For t = 1, . . . , T , let µ Then, with probability 1 − δ, p PT ⊤ (1+T m)/δ ) ln(T ). µ⊤ t y t − µ∗ y t | ≤ 2m T ln ( t=1 |˜
2.2
Fenchel duality
As mentioned earlier, our algorithms are primal-dual algorithms, that use Fenchel duality. Below we provide the basic background on this useful concept. Let h be a convex function defined on [0, 1]d . We define h∗ as the Fenchel conjugate of h, h∗ (θ) := max{y · θ − h(y) : y ∈ [0, 1]d }. Similarly for a concave function f on [0, 1]d , define f ∗ (θ) := maxy∈[0,1]d {y · θ + f (y)}. Note that the Fenchel conjugates h∗ and f ∗ are both convex functions of θ. Suppose that at every point y, every supergradient gy of h (and f ) have bounded dual norm ||g y ||∗ ≤ L. Then, the following dual relationship is known between h and h∗ (f and f ∗ ).
Lemma 3. h(z) = max||θ||∗ ≤L {θ · z − h∗ (θ)}, f (z) = min||θ||∗ ≤L {f ∗ (θ) − θ · z}
A special case is when h(y) = d(y, S) for some convex set S. This function is 1-Lipschitz with respect to norm || · || used in the definition of distance. In this case, h∗ (θ) = hS (θ) := maxy∈S θ · y, and Lemma 3 specializes to the following relation (which also appears in [2]). d(y, S) = max{θ · y − hS (θ) : ||θ||∗ ≤ 1}.
2.3
(3)
Online Learning
The online convex optimization (OCO) problem considers a T round game played between a learner and an adversary, where in round t, the learner chooses a θ t ∈ Ω, and then the adversary picks a concave function gt (θ t ) : Ω → R. The learner’s choice θ t may only depend on learner’s and adversary’s choices in previous rounds, and we denote this by θ t := OL-Predict(θ 1 , g1 , . . . , θ t−1 , gt−1 ). The goal of the learner is to minimize regret defined as the difference between the learner’s objective value and the value of the best single choice on hindsight: P P R(T ) := maxθ∈Ω Tt=1 gt (θ) − Tt=1 gt (θ t ). Some popular algorithms for OCO are online mirror descent (OMD) and online gradient descent, which have very fast per step update rules, and provide the following regret guarantees. Lemma 4. [32] The online mirror-descent algorithm for the OCO problem that achieves regret √ R(T ) = O(G DT ), where G is an upper bound on the norm of gradient of gt at all time t, and D is the diameter of an appropriate regularizer (in the dual norm). The value of these parameters are problem specific. In particular, for gt (θ) of form gt (θ) = θ · z − h∗ (θ) and Ω = {θ : ||θ||∗ ≤√L}, where h is an L∗ Lipschitz p function, h is Fenchel dual of h, regret is bounded as R(T ) ≤ O(L dT ) for ℓ2 norm, and O(L log(d)T ) for ℓ∞ . 5
3
Algorithm for the Feasibility problem (linCBwC)
We start with the case when there are only constraints, i.e., we are only given the convex set S and our goal is to minimize only avg-regret2 (T ). In every round t, our algorithm uses online learning to generate a vector θ t that acts as the dual price to weigh different components of the outcome vector estimate. Using this weighted sum of outcome vector components, for every arm we generate an optimistic estimate of the weight matrix W∗ , using techniques similar to the UCB based algorithms for linear contextual bandits. We then pick the best arm according to these optimistic estimates.
3.1
Optimistic estimates of weight matrix
Let at denote the arm played by the algorithm at time t. In the beginning of every round, we use the outcome and contexts from previous rounds to construct a confidence ellipsoid for every column of W∗ . For this, we use the techniques in Section P 2.1 with y t =⊤xt (at ), and rt = vt (at )j for every j. As in Section 2.1, let Mt := I + t−1 i=1 xi (ai )xi (ai ) , and construct the regularized least squares estimate for matrix W∗ as ˆ t := M −1 Pt−1 xi (ai )vi (ai )⊤ . (4) W t i=1
Let wj denote the j th column of matrix W . We define a confidence ellipsoid for each column j, as n p √ o Ct,j := µ ∈ Rm : kµ − w ˆ tj kMt ≤ m ln ((d+tmd)/δ ) + m ,
and denote by Gt , the Cartesian product of all these ellipsoids: Gt := {W ∈ Rm×d : wj ∈ Ct,j }. Note that Lemma 1 implies W∗ ∈ Gt with probability 1 − δ. Now, given a vector θ t ∈ Rd , we define the optimistic estimate of weight matrix at time t w.r.t. θ t , for every arm a ∈ [K], as : ˜ t (a) := arg minW ∈Gt xt (a)⊤ W θ t . W
(5)
˜ t , along with the results in Lemma 1 and Corollary 1 about confidence ellipsoids, Using the definition of W the following can be derived. Corollary 2. With probability 1 − δ,
˜ t (a)θ t ≤ xt (a)⊤ W∗ θ t , for all arms a ∈ [K], for all time t. 1. xt (a)⊤ W p P ˜ t (at ) − W∗ )⊤ xt (at )k ≤ k1d k 2m T ln ((d+tmd)/δ ) ln(T ) . 2. k Tt=1 (W
3.2
Algorithm and Regret Analysis
At every step t, the algorithm constructs optimistic estimates of the unknown weight matrix, as described in the previous section, for vector θ t which is itself generated by an online learning subroutine. The algorithm then picks the arm which appears to be the best according to these optimistic estimates. Algorithm 1 Algorithm for Feasibility problem ˆ 1 , G1 , OL-Predict with domain Ω = {kθk∗ ≤ 1}, and θ 1 . Initialize W for all t = 1 . . . T do Observe Xt . ˜ t (a) as defined in (5), for all a. Compute W ˜ t (a)θ t . Play arm at := arg mina∈[K] xt (a)⊤ W Observe vt (at ). ˆ t+1 and Gt+1 . Use xt (at ) and vt (at ) to obtain W ˜ t (at )θ − hS (θ). θ t+1 := OL-Predict(θ 1 , g1 , . . . , θ t , gt ) with gt (θ) = xt (at )⊤ W end for 6
Theorem 2. Let R(T ) be the regret for OCO with {gt , θ t }, aspgiven by Lemma 4. Specifically, R(T ) = √ O( dT ) when distance is given by ℓ2 norm and R(T ) = O( log(d)T ) for ℓ∞ . Algorithm 1 achieves the following regret bound, with probability 1 − δ. p P d( T1 t vt (at ), S) ≤ R(T )/T + O mk1d k ln(mdT /δ) ln(T )/T .
Proof. Let Ht−1 be the history of plays and observations before time t, i.e. Ht−1 := {θ τ , Xτ , aτ , vτ (aτ ), τ = ˆ t , Gt , but it does not determine Xt , at , W ˜ t (since at and 1, . . . , t − 1}. Note that Ht−1 determines θ t , W ˜ t (a) depend on the context Xt at time t). The proof is in 3 steps: W Step 1: Since E[vt (at )|Xt , at , Ht−1 ] = W∗⊤ xt (at ), it is sufficient to bound the distance to S of the average of the vectors W∗⊤ xt (at ) and then apply Azuma-Hoeffding to get the required result. Step 2:
From Corollary 2, with probability 1 − δ,
P p
1 T ˜ t (at ))⊤ xt (at )
≤ O mk1d k ln(mdT /δ) ln(T )/T .
T t=1 (W∗ − W
(6)
˜ t (at )⊤ xt (at ). It is therefore sufficient to bound the distance to S of the average of the vectors W ˜ t (at )⊤ xt (at ) and v ˜ t := W ˜ avg := Step 3: We use the shorthand notation of v this proof. The proof is completed by showing that with probability 1 − δ, p d(˜ vavg , S) ≤ R(T )/T + k1d k 2 ln(dT /δ)/T .
1 T
PT
˜t t=1 v
for the rest of (7)
˜ t⊤ θ − hS (θ) to predict θ t , The proof of (7) is as follows. Since we used online learning with gt (θ) = v therefore, from (3) and the definition of R(T ), we have that P ˜ avg − hS (θ)} ≤ 1/T Tt=1 θ ⊤ ˜ t − hS (θ t ) + R(T )/T . d(˜ vavg , S) = max||θ||∗ ≤1 {θ ⊤ v (8) t v
Now, for all t,
⊤ ˜ ˜ t |Ht−1 ] = EXt ∼D [θ ⊤ EXt ∼D [θ ⊤ t Wt (at ) xt (at )|Ht−1 ] t v ⊤ ˜ ≤ EX ∼D,a∼q∗ (X ) [θ ⊤ t Wt (a) xt (a)|Ht−1 ] t
t
⊤ ≤ EXt ∼D,a∼q∗ (Xt ) [θ ⊤ t W∗ xt (a)|Ht−1 ] (with probability 1 − δ) h i ⊤ ∗ ⊤ ∗ = θ⊤ t EX∼D W∗ Xq (X) = θ t v(q ).
(9)
The first inequality above is due to the choice of at in the algorithm. The second inequality is from the ∗ ∗ ), S) = 0. The ˜ t |Ht−1 ] − hS (θ t ) ≤ θ ⊤ first part of Corollary 2. This gives E[θ ⊤ t v PTt v(q⊤ ) − hS (θ t ) ≤ d(v(q p ˜ t − hS (θ t ) ≤ k1d k 2T ln(dT /δ), proof is now completed by using Azuma-Hoeffding to bound t=1 θt v with probability 1 − δ, and using this in (8).
4
The general case (linCBwCR)
In this section, we consider the linCBwCR problem (Definition 1). If we know the value of OPT := f (v(q ∗ )), we can reduce this problem to the Feasibility Problem (linCBwC) by defining the constraint set to be S ′ = {v : f (v) ≥ OPT, v ∈ S}. 7
A common approach [20, 7] to eliminate the assumption that OPT is known is to get increasingly accurate estimates of OPT computed at geometric time intervals as the algorithm progresses. However this approach does not seem to be directly employable here. As we shall see later, due to bandit feedback in this problem, estimating OPT seems to require “pure exploration” for some rounds where the arms picked are tailored to getting such an estimate. Further, the error in the estimate thus obtained using √ ∗ / T ), where Z ∗ is a parameter (defined later) that ˜ T0 rounds turns out to be of form O(mk1 kZ 0 d captures the tradeoff between the objective and constraints. Thus, the knowledge or estimation of Z ∗ is required for this approach in addition to estimating OPT. Instead, following [5], we use an approach that requires only the knowledge or even an O(1) approximation of Z ∗ .2 Let OPTγ denote the value of the optimal policy when the feasibility constraint is relaxed from v(q) ∈ S to d(v(q), S) ≤ γ for any γ ≥ 0. OPTγ :=
maxq f (v(q)) such that d(v(q), S) ≤ γ
Lemma 5. There exists Z ≥ 0 such that that for all γ ≥ 0, OPTγ ≤ OPT + minimum value of such Z.
(10) Z 2 γ.
Let Z ∗ denote the
Assumption 1. Assume we are given Z ≥ L satisfying the condition in Lemma 5.
Below, we present an algorithm under Assumption 1. The algorithm is similar to that in the previous section: it constructs estimates of the weight matrix exactly as in Section 3.1. Here, we run two instances of the online learning algorithm: we search over the domain of Fenchel dual variables for the objective in addition to that for the constraints as in the previous section. The optimistic estimates are constructed using the vector obtained on combining the dual variables for the objective and those for the constraints using Z. The algorithm then chooses the arm which appears to be the best according to optimistic estimates, as before. Algorithm 2 Algorithm for linCBwCR ˆ 1 , G1 , two instances of OL-Predict with domain Ω1 = {θ : kθk∗ ≤ 1}, and Ω2 = {φ : Initialize W ||φ||∗ ≤ L}, θ 1 and φ1 . for all t = 1, ..., T do Observe Xt . ˜ t (a) := arg maxW ∈Gt xt (a)⊤ W (φt + Zθ t ). For every a ∈ [K], compute W ˜ t (a)(φt + Zθ t ). Play the arm at := arg mina∈[K] xt (a)⊤ W Observe vt (at ). ˆ t+1 and Gt+1 . Use xt (at ) and vt (at ) to obtain W ˜ t (at )θ − hS (θ). θ t+1 := OL-Predict1 (θ 1 , g1 , . . . , θ t , gt ) with gt (θ) = xt (at )⊤ W ⊤ ˜ t (at )φ − f ∗ (φ). φt+1 := OL-Predict2 (φ1 , ψ1 , . . . , φt , ψt ) with ψt (θ) = xt (at ) W end for Theorem 3. Given a Z ≥ L as per Assumption 1, Algorithm 2 achieves the following bounds with probability 1 − δ: p avg-regret1 (T ) ≤ Z · O R(T )/T + mk1d k ln(dT /δ) ln(T )/T , p avg-regret2 (T ) ≤ O R(T )/T + mk1d k ln(dT /δ) ln(T )/T .
Here R(T ) = R1 (T ) + R2Z(T ) , with R1 (T ) being the regret for OCO with {gt , θ t }, and R2 (T ) being the √ regret for OCO withp{ψt , φt }, as given by Lemma 4. Specifically, R(T ) = O( dT ) when ℓ2 norm is used and R(T ) = O( log(d)T ) for ℓ∞ norm. 2 As an application of this approach, for the special case of budget constraints (linCBwK), we show how to get an approximation of Z ∗ using the initial few rounds, giving a sub-linear regret overall; see Section 5 for details.
8
5
Budget Constraints (linCBwK)
For the special case of linCBwK, knowing or estimating OPT is sufficient to estimate Z. OPT Lemma 6. Z2 = (B/T ) satisfies the condition in Lemma 5, when S is given by budget constraints and ℓ∞ norm is used for distance.
Therefore, if we know OPT, we can use Algorithm 2 with above value of Z plus 1, and distance ˜ OPT + 1) √m ) regret in objective (as given by Theorem 3). We defined by ℓ∞ norm, to achieve an O(( (B/T ) T show how to estimate Z as part of the algorithm for the special case of linCBwK, using the first few rounds. Also, we are not allowed to violate the budgets at all for this problem, so the algorithm needs to be modified to handle this. Given parameters T0 and B ′ , the algorithm is as follows. In the first T0 rounds do a “pure exploration” and estimate a Z satisfying Lemma 5. Set aside a B ′ amount from the remaining budget for each dimension. Run Algorithm 2 for the remaining time steps with the remaining budget. Stop if the full actual budget B is consumed for any dimension. p √ Theorem 4. Using the above algorithm with T0 such that B > max{T0 , mT / T0 }, B ′ = m T ln(dT /δ) ln(T ) and an appropriate estimation procedure for Z, we get a high probability regret bound of ˜ ((OPT/(B/T ) + 1) (T0/T + m/√T )) . O In particular, using T0 =
√
T , and assuming B > mT 3/4 gives a regret bound of ˜ ((OPT/(B/T ) + 1) m/√T ) . O
The proof of Theorem 4 uses the following estimate for Z obtained as described in Section 5.1. p Lemma 7. Let γ = 2m ln(T0 ) ln(T0 d/δ)/T0 . Using the first T0 rounds for exploration, we can obtain Z such that with probability 1 − O(δ), OPT (B/T )
7γ OPT (B/T ) )( (B/T )
+ 1).
mT ˜ √ ≥ γ, i.e., B ≥ Ω( ), Z is a constant factor approximation of T0 m OPT ˜ ( √ + 1 ≥ Z ∗ , therefore Theorem 3 provides an O + 1) regret bound if the algorithm is (B/T ) T
Lemma 7 implies that as long as OPT (B/T )
+ 1 ≤ Z ≤ (1 + B T
not stopped due to budget violation. As implied by the bound on avg-regret 2 (T ) in Theorem 3, setting aside B ′ budget ensures that budget is not violated with high probability. However, the reduction of ′ the budget by B ′ incurs additional BB OPT regret. Also, using the first T0 rounds for pure exploration can consume (at most) T0 amount from the budget, incurring an additional regret of TB0 OPT. Together, these observations give Theorem 4.
5.1
Estimating OPT
Here we sketch the high level idea behind obtaining the result in Lemma 7 by estimating OPT. Due to bandit feedback, estimating OPT in our problem has significant challenges compared to the full information settings like those in [5, 20, 7]. The approach used in those papers is to use the value of the policy that performs the best on the observations seen so far. However, in case of bandit feedback, observations are made only for the arms picked by the algorithm, therefore, it is not possible to directly evaluate any policy other than the one that was followed in the history. A fix for this, inspired by [22, 3], is to play every arm with some minimum probability in every round, so that using importance sampling we can re-weight the observations and construct an unbiased estimator of the outcome vector 9
for any arm. A crude version of this that plays every arm √ with probability 1/K in every round for T0 ∗ / T ). Instead, we provide a method that takes ˜ rounds would obtain an estimation error of O(KmZ 0 advantage of the linear structure of the problem, and explores in the m-dimensional space of contexts and weight vectors to obtain bounds independent of K. We use the following procedure. In every round t = 1, . . . , T0 , after observing Xt , let pt ∈ ∆[K] be pt := arg max kXt pkM −1 , where Mt := I
(11)
t p∈∆[K] Pt−1 + i=1 (Xi pi )(Xi pi )⊤ .
(12)
Select arm at = a with probability pt (a). In fact, since Mt is a PSD matrix, due to convexity of the ˆt function kXt pk2M −1 , it is the same as playing at = arg maxa∈[K] kxt (a)kM −1 . Construct an estimate W t t of W∗ at time t as ˆ t := M −1 Pt−1 (Xi pi )vi (ai )⊤ . W t i=1 ˆ γ of OPT as: And, for some value of γ defined later, obtain an estimate OPT ˆ γ OPT
:=
P 0 ˆ⊤ maxq f ( T10 Ti=1 Wi Xi q(Xi )]) 1 PT0 ˆ ⊤ such that d( T0 i=1 Wi Xi q(Xi )], S) ≤ γ.
(13)
For an intuition about the choice of arm in (11), observe from the discussion in Section 2.1 that every column w∗j of W∗ is guaranteed to lie inside the confidence ellipsoid centered at column w ˆ tj 2 ˆ of Wt , namely the ellipsoid, kµ − w ˆ tj kMt ≤ 4m ln(T m/δ). Note that this ellipsoid has principle axes as eigenvectors of Mt , and the length of semi-principle axes is given by inverse eigenvalues of Mt . Therefore, by maximizing kXt pkM −1 we are choosing the context closest to the direction of the longest t principal axes of the confidence ellipsoid, i.e. in the direction of maximum uncertainty. Intuitively, this corresponds to pure exploration: by making an observation in the direction where uncertainty is large we can reduce the uncertainty in our estimate most effectively. ˆ γ , we want the A more algebraic explanation is as follows. For aP good estimation of OPT by OPT T0 ˆ ˆ ⊤ ˆ t and W∗ to be close enough so that | 1 estimates W t=1 (Wt − W∗ ) Xt q(Xt )| is small for all policies T0 q, and in particular for optimal policies. Now, using Cauchy-Schwartz this is bounded by 1 PT0 ˆ t=1 k1d kkWt − W∗ kMt kXt q(Xt ))kM −1 , T0 t
where we define kW kM , the M -norm of matrix p W to be the max of column-wise M -norms. Using Lemma ˆ t − W∗ kMt is bounded by 2 m ln(T0 md/δ) with probability 1 − δ. Lemma 2 bounds the 1, the term kW P 0 second term Tt=1 kXt q(Xt )kM −1 but only when q is the played policy. This is where we use that the t P 0 P 0 played policy pt was chosen to maximize kXt pt kM −1 , so that Tt=1 kXt q(Xt )kM −1 ≤ Tt=1 kXt pt kM −1 t t t p PT0 PT0 and the bound t=1 kXt pt kM −1 ≤ mT0 ln(T0 ) given by Lemma 2 actually bounds t=1 kXt q(Xt )kM −1 t t for all q. We prove the following lemma. p Lemma 8. For γ = 2k1d km ln(T0 ) ln(T0 d/δ)/T0 , with probability 1 − O(δ), ˆ 2γ ≤ OPT + 6γ(Z ∗ + L). OPT − Lγ ≤ OPT
For linCBwK, Z ∗ ≤ bounds of Lemma 7.
OPT (B/T ) , L
2γ
= 1, k1d k = k1d k∞ = 1. Therefore, Z =
10
ˆ (OPT +Lγ) (B/T )
+ 1 satisfies the
References [1] Y. Abbasi-yadkori, D. P´ al, and C. Szepesv´ari. Improved algorithms for linear stochastic bandits. In NIPS, 2012. [2] J. Abernethy, P. L. Bartlett, and E. Hazan. Blackwell approachability and low-regret learning are equivalent. In COLT, 2011. [3] A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. E. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In ICML 2014, June 2014. [4] S. Agrawal and N. R. Devanur. Bandits with concave rewards and convex knapsacks. In Proceedings of the Fifteenth ACM Conference on Economics and Computation, EC ’14, 2014. [5] S. Agrawal and N. R. Devanur. Fast algorithms for online stochastic convex programming. In SODA, pages 1405–1424, 2015. [6] S. Agrawal, N. R. Devanur, and L. Li. Contextual Bandits with Global Constraints and Objective. ArXiv e-prints, June 2015. [7] S. Agrawal, Z. Wang, and Y. Ye. A dynamic near-optimal algorithm for online linear programming. Operations Research, 62:876 – 890, 2014. [8] P. Auer. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn. Res., 3, Mar. 2003. [9] M. Babaioff, S. Dughmi, R. D. Kleinberg, and A. Slivkins. Dynamic pricing with limited supply. ACM Trans. Economics and Comput., 3(1):4, 2015. [10] A. Badanidiyuru, R. Kleinberg, and Y. Singer. Learning on a budget: posted price mechanisms for online procurement. In Proc. of the 13th ACM EC, pages 128–145. ACM, 2012. [11] A. Badanidiyuru, R. Kleinberg, and A. Slivkins. Bandits with knapsacks. In FOCS, pages 207–216, 2013. [12] A. Badanidiyuru, J. Langford, and A. Slivkins. Resourceful contextual bandits. In Proceedings of The Twenty-Seventh Conference on Learning Theory (COLT-14), pages 1109–1134, 2014. [13] O. Besbes and A. Zeevi. Dynamic pricing without knowing the demand function: Risk bounds and near-optimal algorithms. Operations Research, 57(6):1407–1420, 2009. [14] A. Beygelzimer, J. Langford, L. Li, L. Reyzin, and R. E. Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proc. of the 14th AIStats, pages 19–26, 2011. [15] D. Chakrabarti and E. Vee. Traffic shaping to optimize ad delivery. In Proceedings of the 13th ACM Conference on Electronic Commerce, EC ’12, 2012. [16] X. Chen and Z. Wang. A near-optimal dynamic learning algorithm for online matching problems with concave returns. http://arxiv.org/abs/1307.5934, 2013. [17] W. Chu, L. Li, L. Reyzin, and R. E. Schapire. Contextual Bandits with Linear Payoff Functions. Journal of Machine Learning Research - Proceedings Track, 15:208–214, 2011. [18] V. Dani, T. P. Hayes, and S. M. Kakade. Stochastic Linear Optimization under Bandit Feedback. In COLT, 2008. 11
[19] N. R. Devanur and T. P. Hayes. The adwords problem: online keyword matching with budgeted bidders under random permutations. In EC, 2009. [20] N. R. Devanur, K. Jain, B. Sivan, and C. A. Wilkens. Near optimal online algorithms and fast approximation algorithms for resource allocation problems. In EC, 2011. [21] W. Ding, T. Qin, X.-D. Zhang, and T.-Y. Liu. Multi-armed bandit with budget constraint and variable costs. In Proc. of the 27th AAAI, pages 232–238, 2013. [22] M. Dud´ık, D. Hsu, S. Kale, N. Karampatziakis, J. Langford, L. Reyzin, and T. Zhang. Efficient optimal learning for contextual bandits. In Proc. of the 27th UAI, pages 169–178, 2011. [23] J. Feldman, M. Henzinger, N. Korula, V. S. Mirrokni, and C. Stein. Online stochastic packing applied to display ad allocation. In Proceedings of the 18th Annual European Conference on Algorithms: Part I, ESA’10, 2010. [24] S. Guha and K. Munagala. Approximation algorithms for budgeted learning problems. In STOC, pages 104–113, 2007. [25] A. Gupta and M. Molinaro. How the Experts Algorithm Can Help Solve LPs Online. Algorithms - ESA 2014, Lecture Notes in Computer Science, 8737:517–529, 2014. [26] A. Gy¨orgy, L. Kocsis, I. Szab´ o, and C. Szepesv´ari. Continuous time associative bandit problems. In Proc. of the 20th IJCAI, pages 830–835, 2007. [27] C. Karande, A. Mehta, and P. Tripathi. Online bipartite matching with unknown distributions. In STOC, 2011. [28] T. Kesselheim, A. T¨ onnis, K. Radke, and B. V¨ocking. Primal beats dual on online packing LPs in the random-order model. In STOC, 2014. [29] O. Madani, D. J. Lizotte, and R. Greiner. The budgeted multi-armed bandit problem. In Learning Theory, pages 643–645. Springer, 2004. [30] M. Mahdian and Q. Yan. Online bipartite matching with random arrivals: an approach based on strongly factor-revealing LPs. In STOC, 2011. [31] S. Pandey and C. Olston. Handling advertisements of unknown quality in search advertising. In Advances in Neural Information Processing Systems, pages 1065–1072, 2006. [32] S. Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2):107–194, 2012. [33] A. Singla and A. Krause. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In Proc. of the 22nd WWW, pages 1167–1178, 2013. [34] A. Slivkins and J. W. Vaughan. Online decision making in crowdsourcing markets: Theoretical challenges (position paper). CoRR, abs/1308.1746, 2013. [35] L. Tran-Thanh, A. C. Chapman, E. M. de Cote, A. Rogers, and N. R. Jennings. Epsilon-first policies for budget-limited multi-armed bandits. In Proc. of the 24th AAAI, 2010. [36] L. Tran-Thanh, A. C. Chapman, A. Rogers, and N. R. Jennings. Knapsack based optimal policies for budget-limited multi-armed bandits. In AAAI, 2012.
12
[37] E. Vee, S. Vassilvitskii, and J. Shanmugasundaram. Optimal online assignment with forecasts. In EC ’10: Proceedings of the 11th ACM conference on Electronic commerce, 2010. [38] H. Wu, R. Srikant, X. Liu, and C. Jiang. Algorithms with logarithmic or sublinear regret for constrained contextual bandits. CoRR, abs/1504.06937, 2015.
13
Appendix
A
Concentration Inequalities
Lemma 9 (Azuma-Hoeffding inequality). If a super-martingale (Yt ; t ≥ 0), corresponding to filtration Ft , satisfies |Yt − Yt−1 | ≤ ct for some constant ct , for all t = 1, . . . , T , then for any a ≥ 0, −
Pr(YT − Y0 ≥ a) ≤ e
B
2
a2 PT 2 t=1 ct
.
Confidence ellipsoids
Proof of Corollary 1. The following holds with probability 1 − δ. T X t=1
⊤ |˜ µ⊤ t xt − µ∗ xt | ≤
T X t=1
≤
k˜ µt − µ∗ kMt kxt kM −1
s
t
m ln
1 + tm δ
! p √ mT ln(T ). + m
The inequality in the first line is a matrix-norm version of Cauchy-Schwartz (Lemma 10). The inequality in the second line is due to Lemmas 1 and 2. The lemma follows from multiplying out the two factors in the second line. Lemma 10. For any positive definite matrix M ∈ Rn×n and any two vectors a, b ∈ Rn , |a⊤ b| ≤ kakM kbkM −1 . ⊤ . Further, Proof. Since M is positive definite, there exists a matrix M1/2 such that M = M1/2 M1/2 −1 ⊤ . M −1 = M−1/2 M−1/2 where M−1/2 = M1/2 ⊤ ka⊤ M1/2 k2 = a⊤ M1/2 M1/2 a = a⊤ M a = kak2M .
Similarly, kM−1/2 bk2 = kbk2M −1 . Now applying Cauchy-Schwartz, we get that |a⊤ b| = |a⊤ M1/2 M−1/2 b| ≤ ka⊤ M1/2 kkM−1/2 bk = kakM kbkM −1 . ˜ t (a) and the observation Proof of Corollary 2. Here, the first claim follows simply from definition of W ∗ that with probability 1 − δ, W ∈ Gt . To obtain the second claim, P apply Corollary 1 with µ∗ = th column of W ⊤ ˜ ˜ ˜ ˜ w , y = x (a ), µ = [ W (a )] (the j (a )), to bound | t t t t j t t t t ([Wt (at )]j − w∗j ) xt (at )| ≤ P∗j t˜ ⊤ t |([Wt (at )]j − w∗j ) xt (at )| for every j, and then take the norm.
C
Appendix for Section 4 (linCBwCR)
Proof of Lemma 5. The proof of this lemma is similar to the proof of Lemma 5.1 in [5]. Let Ω := {y : ∃q, y = EX∼D [W∗⊤ Xq(X)]}. Then, Ω is a convex set. And, OPTγ can be written as the following convex optimization problem over y OPTγ := max f (y) such that d(y, S) ≤ γ. y∈Ω
14
Then, applying Lagrangian duality for convex programs OPTγ = min max f (y) + λ(γ − d(y, S)) λ≥0 y∈Ω
By Fenchel duality, for any L-Lipschitz concave function f , f (y) =
min
φ:kφk∗ ≤L
f ∗ (φ) − φ⊤ y,
where f ∗ is the Fenchel dual of f . Specifically, for distance function, d(y, S) =
max θ ⊤ y − hS (θ),
θ:kθk∗ ≤1
where for any convex set S, hS is defined as: hS (θ) := maxv∈S θ ⊤ v. Substituting in the expression for OPTγ , we get OPTγ
= min max
(f ∗ (φ) − φ⊤ y) + λγ − λ(θ ⊤ y − hS (θ))
min
λ≥0 y∈Ω |φk∗ ≤L,kθk∗ ≤1
= = ≤
min
max f ∗ (φ) − φ⊤ y − λθ ⊤ y + λhS (θ) + λγ
min
f (φ) + hΩ (−φ − λθ) + λhS (θ) + λγ
λ≥0,|φk∗ ≤L,kθk∗ ≤1 y∈Ω ∗
λ≥0,|φk∗ ≤L,kθk∗ ≤1 f ∗ (φ∗ ) + hΩ (−φ∗ ∗
= OPT + λ γ
− λ∗ θ ∗ ) + λ∗ hS (θ ∗ ) + λ∗ γ
Here, φ∗ , θ ∗ , λ∗ denote the optimal value of these variables for the corresponding optimization problem for OPT. Therefore, Z = λ∗ satisfies the condition in Lemma 5. Proof of Theorem 3. The proof is an extension of proof of Theorem 2, and uses the properties of Z to handle the objective in addition to constraints. P ˜ t (at )⊤ xt (at ), v ˜ t in this section. Similar ˜ t := W ˜ avg = T1 Tt=1 v We use the shorthand notation of v to the observation in Equation (8) in the proof of Theorem 2, using online learning guarantees we can obtain that for the choice of θ t , φt made by this algorithm, d(˜ vavg , S) ≤ f (˜ vavg ) ≥
T R (T ) 1 X ⊤ 1 ˜ t − hS (θ t ) + θt v . T t=1 T
T R (T ) 1 X ∗ 2 ˜ v f (φt ) − φ⊤ . t t + T T
(14)
(15)
t=1
˜ t (a)(φt + Zθ t ), and because For all t, since the algorithm selected at ∈ [K] to minimize xt (a)⊤ W ˜ t (a) is an optimistic estimate for every a, therefore using similar line of arguments as in Equation (9), W we get that with probability 1 − δ, ˜ t |Ht−1 ] ≥ f ∗ (φt ) + ZhS (θ t ) − (φt + Zθt )⊤ v(q ∗ ). E[f ∗ (φt ) + ZhS (θ t ) − (φt + Zθ t )⊤ v Below are the details of this argument. ˜ t (at )⊤ xt (at )|Ht−1 ] ˜ t |Ht−1 ] = E[(φt + Zθ t )⊤ W E[(φt + Zθ t )⊤ v P ˜ t (a)⊤ xt (a)q ∗ (Xt , a) |Ht−1 ] ≤ E[(φt + Zθ t )⊤ W a
≤ E[(φt + Zθ t )⊤ W∗⊤ Xt q ∗ (Xt )|Ht−1 ] (with probability 1 − δ) h i = (φt + Zθ t )⊤ Ex∼D W∗⊤ Xq ∗ (X) = (φt + Zθ t )⊤ v(q ∗ ). 15
(16)
Now, the term on the right hand side in Equation (16) is greater than or equal to ∗ ⊤ ∗ ⊤ ∗ min f (φt ) − φ v(q ) − Z max θ t v(q ) − hS (θt ) = f (v(q ∗ )) − Zd(v(q ∗ ), S) = OPT kφk∗ ≤L
kθk∗ ≤1
where the second last equality uses Fenchel duality, and the last equality uses d(v(q ∗ ), S) = 0. Therefore, for every t, the conditional expectation on the left hand side of (16) is lower bounded by OPT. Summing over t, and using Azuma-Hoeffding, we obtain that probability 1 − δ, T X t=1
p ˜ t − Z(θ ⊤ ˜ t − hS (θ t )) ≥ T OPT − k1d k 2T ln(dT /δ) f ∗ (φt ) − φ⊤ t v t v
(17)
Combining Equation (14) and (15) using Z, and substituting the bound from (17), we get f (˜ vavg ) − Zd(˜ vavg , S) ≥ OPT −
k1d k p R(T ) 2T ln(dT /δ) − Z T T
P ˜ avg := T1 Tt=1 v ˜ t is close Now, Step 1 and Step 2 in the proof of Theorem 2 hold as it is, so that v 1 PT ˜ avg by vavg in above to get with probability 1 − δ to vavg := T t=1 vt (at ). Therefore, we can replace v mk1d k p R(T ) f (vavg ) − Zd(vavg , S) ≥ OPT − Z O T ln(mdT /δ) ln(T ) + . (18) T T
Since d(vavg , S) ≥ 0, this proves the bound on avg-regret1 (T ). To obtain the bound on distance from the constraint set, first note that E[f (vavg )] ≤ OPTE[d(vavg ,S)]
To see this, let the random variable q˜ denote the empirical played policy, i.e. the policy implied by the contexts observed and the arms played X1 , a1 , . . . , XT , aT , so that by definition v(E[˜ q ]) = E[vavg ], and d(v(E[˜ q ]), S) = d(E[vavg ], S) ≤ E[d(vavg , S)]. Therefore, E[˜ q ] is a feasible solution for (10) with γ = E[d(vavg , S)], implying f (v(E[˜ q ])) ≤ OPTγ . Also, by concavity of f and definition of policy q˜, E[f (vavg )] ≤ f (E[vavg ]) = f (v(E[˜ q ])). Therefore, the above observation follows. Now, using the assumption about the parameter Z, E[f (vavg )] −
Z Z E[d(vavg , S)] ≤ OPTE[d(vavg ,S)] − E[d(vavg , S)] ≤ OPT 2 2
Using (18), this implies E[f (vavg )] −
Z E[d(vavg , S)] ≤ OPT ≤ E[f (vavg )] − ZE[d(vavg , S)] 2 mk1d k p R(T ) . +Z O T ln(mdT /δ) ln(T ) + T T
Rearranging the terms, we get
Z E[d(vavg , S)] ≤ Z O 2
mk1d k p R(T ) T ln(mdT /δ) ln(T ) + T T
.
Then, using d(E[vavg ], S) ≤ E[d(vavg , S)] and applying Azuma-Hoeffding, we get the desired bound on avg-regret2 (T ) with high probability.
16
D
Appendix for Section 5 (linCBwK)
Proof of Lemma 6. The proof of this lemma is similar to the proof of Lemma 4 in [6]. In linCBwK problem, OPTγ represents the value of optimal policy when the budget constraints are relaxed by γ, i.e. for γ OPT feasibility set S γ = {v : v ≤ B T 1 + γ}. Now, suppose for contradiction that OPT > OPT + (B/T ) γ. Let qˆ be the optimal policy for OPTγ . Then, one could construct a feasible policy q ′ for the original B problem with budget B by scaling down the probabilities given by qˆ by B+γT . (And do nothing with the ′ remaining probability). Since qˆ consumed at most B + γT budget, q will consume at most B budget. B OPT B > (OPT + (B/T Also, the value of policy q ′ will be at least OPTγ B+γT ) γ) B+γT = OPT. Thus we arrive at a contradiction.
E
Appendix for Section 5.1 (Estimating OPT)
Proof of Lemma 8. Let us define an intermediate ”sample optimal” as: P 0 W∗⊤ Xi q(Xi )]) maxq f ( T10 Ti=1 γ P OPT := T 0 such that d( T10 i=1 W∗⊤ Xi q(Xi )], S) ≤ γ
(19)
Above sample optimal knows the weight matrix W∗ , the error comes only from approximating the expected value over context distribution by average over the observed contexts. We do not actually γ compute OPT , but will use it for the convenience of proof exposition. The proof involves two steps. γ
Step 1: Bound |OPT − OPT|. 2γ
ˆ Step 2: Bound |OPT
γ
− OPT | by bounding k
ˆ − W∗ )⊤ Xi q(Xi )k for all q.
PT0
i=1 (Wi
Step 1 bound can be borrowed from the work on Online Stochastic Convex Programming in [5]: since W ∗ is known so there is effectively full information before making the decision, i.e., consider the vectors W∗⊤ xt (a) as reward vectors which can be observed for all arms a before choosing the distribution over ˆ γ as defined by Equation arms to be played at time t, therefore, the setting in [5] applies. In fact, OPT γ (F.10) in [5] when At = {W∗⊤ xt (a), a ∈ [K]}, is same as thepOPT defined here. And using Lemma F.4 and Lemma F.6 in [5], we obtain that for any γ ≥ 2k1d km ln(T0 ) ln(T0 d/δ)/T0 , with probability 1 − O(δ), γ
OPT − Lγ ≤ OPT ≤ OPT + 2γ(Z ∗ + L).
For Step 2, we show that with probability 1 − δ, for all q, γ ≥ 2k1d km T0 1 X ˆ i − W∗ )⊤ Xi q(Xi )k ≤ γ (W k T0
(20)
p
ln(T0 ) ln(T0 d/δ)/T0
(21)
i=1
p ˆ 2γ for γ ≥ 2k1d km ln(T0 ) ln(T0 d/δ)/T0 . This is sufficient to prove both lower and upper bound on OPT γ For lower bound, we can simply use (21) for optimal policy for OPT , denoted by q¯. This implies ˆ 2γ , and that (because of relaxation of distance constraint by γ) q¯ is a feasible primal solution for OPT therefore using (20), ˆ 2γ ≥ OPTγ ≥ OPT − Lγ. OPT ˆ 2γ . Then, using (20), For the upper bound, we can use (21) for the optimal policy qˆ for OPT ˆ 2γ ≤ OPT3γ ≤ OPT + 6γ(Z ∗ + L). OPT 17
Combining, this proves the desired lemma statement: ˆ 2γ ≤ OPT + 6γ(Z ∗ + L) OPT − Lγ ≤ OPT
(22)
What remains is to proof the claim in (21). Observe that for any q, k
T0 T0 X X ˆ t − W∗ )⊤ Xt q(Xt )k ˆ t − W∗ )⊤ Xt q(Xt )k ≤ k(W (W t=1 T0 X
t=1
≤
t=1
ˆ t − W∗ kMt kXt q(Xt )k −1 k1d kkW M t
ˆ t − W∗ kMt = maxj kw where kW ˆ tj − w∗j kMt . ˆ t , we have that with probability 1 − δ for all t, Now, applying Lemma 1 to every column w ˆ tj of W p p ˆ t − W∗ kMt ≤ 2 m log(td/δ) ≤ 2 m log(T0 d/δ) kW
And, by choice of pt
kXt q(Xt )kM −1 ≤ kXt pt kM −1 . t
t
Also, by Lemma 2, T0 X t=1
Therefore, substituting, k
kXt pt kM −1 ≤ t
p mT0 ln(T0 )
T0 T0 X X p ˆ t − W∗ )⊤ Xt q(Xt )k ≤ k1d k(2 m log(T0 d/δ)) kXt pt kM −1 (W t=1
t=1
t
p p ≤ k1d k(2 m log(T0 d/δ)) mT0 ln(T0 )
≤ Tγ
18
