Bernoulli bandits with covariates - Department of Statistics ...

More documents

Recommendations

Info

SUMMARY Sequential selections are to be made from two stochastic processes, or "arms", each yielding Bernoulli responses. A t each stage the arm selected depends on previous observations. The objectfve fs to maximize the expected number of successes In the first n selections. The probability of success for a given selection depends on a covariate through a logistic transformation, For one arm, this transformation is completely known; for the other, it depends on an unknown parameter, Optimal strategies are developed In terms of a break-even value for the cavariate: it is optimal to observe the arm w-l th unknown parameter if the covarfate is less than the break-even value, Other properties of optimal strategies are related to those for non-covariate model s.
1, Introduction A bandit problem involves sequential selections or "pulls" from a number of stochastic processes (or "arms", machines, treatments, etc. ) . The available processes have unknown characteristics, so learning can take place as the processes are observed. As Sn Bradt, Johnson, and Kaslin (19561, we shall restrict our attention to the class of finite horizon Bernoulli bandits, in which the responses are Bernoulli random variables and the goal is to maximize the expected sum of the first n observations. Berry and Frjstedt (1985) discuss this and many other forms of bandi t model s. The bandit model has been proposed as a model for a clfnlcal trial. The arms represent treatments, and the goal is to a1 1 ocate treatments sequentially so as to mximire the expected total number of successes. However, in a typical bandit problem, the observations made on a given arm are assumed to be exchangeable (see, for example, Berry and Frlstedt, 1979). In a clinical trial, this imp1 ies that all subjects receiving the same treatment have the same (marginal) probabtl i ty of success. In thi s paper we extend thf s notion by supposing that, for a given arm, the probabflity of success for a given subject depends both on the treatment being used and on a covariate that can encode relevant characteristics affecting the chance of success. this might include such things as the general health of the subject, age and sex of the subject, and so on. To describe our model formally, we begin by assuming that there are two arms. Let Xi and Yi denote the results from arms 1 and 2 respectively, at stage i : for i C n exactly one of the pair (Xi ,yi) is actually observed. We also assume that prior to makfng the ith observation we can observe a covariate Si.
Page 1: . DEPARTMENT OF STATISTICS --I-----
Page 5 and 6: observed val ues of the covaria te.
Page 7 and 8: t, then we are faced with a (t,n-l,
Page 9 and 10: In words, W,(R,c) is bounded above
Page 11 and 12: Proof: Suppose to the contrary, tha
Page 13 and 14: Since R' is to the rlght of R, E(p(
Page 15 and 16: 1 2 (v) l i m Wn(t.R,c) = Tim W,(t,
Page 17 and 18: Remark: If R is degenerate at c, th
Page 19 and 20: If Wl( t,R,c) = Ep(t), then we note
Page 21 and 22: 2 2 1 if Hz(-1,R) = W2(-l,R), then
Page 23 and 24: Remark: The Rypothesfs of the propo
Page 25 and 26: REFERENCES Berry, D.A. (19721, A be

Bernoulli bandits with covariates - Department of Statistics ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?