Minimax fixed-design linear regression

Peter L. Bartlett, Wouter M. Koolen, Alan Malek, Eiji Takimoto, Manfred K. Warmuth

Research output: Contribution to journalConference article

1 Citation (Scopus)

Abstract

We consider a linear regression game in which the covariates are known in advance: at each round, the learner predicts a real-value, the adversary reveals a label, and the learner incurs a squared error loss. The aim is to minimize the regret with respect to linear predictions. For a variety of constraints on the adversary's labels, we show that the minimax optimal strategy is linear, with a parameter choice that is reminiscent of ordinary least squares (and as easy to compute). The predictions depend on all covariates, past and future, with a particular weighting assigned to future covariates corresponding to the role that they play in the minimax regret. We study two families of label sequences: box constraints (under a covariate compatibility condition), and a weighted 2- norm constraint that emerges naturally from the analysis. The strategy is adaptive in the sense that it requires no knowledge of the constraint set. We obtain an explicit expression for the minimax regret for these games. For the case of uniform box constraints, we show that, with worst case covariate sequences, the regret is O(d log T), with no dependence on the scaling of the covariates.

Original languageEnglish
JournalJournal of Machine Learning Research
Volume40
Issue number2015
Publication statusPublished - Jan 1 2015
Event28th Conference on Learning Theory, COLT 2015 - Paris, France
Duration: Jul 2 2015Jul 6 2015

Fingerprint

Fixed Design
Linear regression
Minimax
Covariates
Labels
Regret
Box Constraints
Game
Squared Error Loss
Linear Prediction
Ordinary Least Squares
Compatibility Conditions
Optimal Strategy
Weighting
Scaling
Minimise
Norm
Predict
Prediction

All Science Journal Classification (ASJC) codes

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence

Cite this

Bartlett, P. L., Koolen, W. M., Malek, A., Takimoto, E., & Warmuth, M. K. (2015). Minimax fixed-design linear regression. Journal of Machine Learning Research, 40(2015).

Minimax fixed-design linear regression. / Bartlett, Peter L.; Koolen, Wouter M.; Malek, Alan; Takimoto, Eiji; Warmuth, Manfred K.

In: Journal of Machine Learning Research, Vol. 40, No. 2015, 01.01.2015.

Research output: Contribution to journalConference article

Bartlett, PL, Koolen, WM, Malek, A, Takimoto, E & Warmuth, MK 2015, 'Minimax fixed-design linear regression', Journal of Machine Learning Research, vol. 40, no. 2015.
Bartlett PL, Koolen WM, Malek A, Takimoto E, Warmuth MK. Minimax fixed-design linear regression. Journal of Machine Learning Research. 2015 Jan 1;40(2015).
Bartlett, Peter L. ; Koolen, Wouter M. ; Malek, Alan ; Takimoto, Eiji ; Warmuth, Manfred K. / Minimax fixed-design linear regression. In: Journal of Machine Learning Research. 2015 ; Vol. 40, No. 2015.
@article{e15da3385bc8412983ffb2463aff7c75,
title = "Minimax fixed-design linear regression",
abstract = "We consider a linear regression game in which the covariates are known in advance: at each round, the learner predicts a real-value, the adversary reveals a label, and the learner incurs a squared error loss. The aim is to minimize the regret with respect to linear predictions. For a variety of constraints on the adversary's labels, we show that the minimax optimal strategy is linear, with a parameter choice that is reminiscent of ordinary least squares (and as easy to compute). The predictions depend on all covariates, past and future, with a particular weighting assigned to future covariates corresponding to the role that they play in the minimax regret. We study two families of label sequences: box constraints (under a covariate compatibility condition), and a weighted 2- norm constraint that emerges naturally from the analysis. The strategy is adaptive in the sense that it requires no knowledge of the constraint set. We obtain an explicit expression for the minimax regret for these games. For the case of uniform box constraints, we show that, with worst case covariate sequences, the regret is O(d log T), with no dependence on the scaling of the covariates.",
author = "Bartlett, {Peter L.} and Koolen, {Wouter M.} and Alan Malek and Eiji Takimoto and Warmuth, {Manfred K.}",
year = "2015",
month = "1",
day = "1",
language = "English",
volume = "40",
journal = "Journal of Machine Learning Research",
issn = "1532-4435",
publisher = "Microtome Publishing",
number = "2015",

}

TY - JOUR

T1 - Minimax fixed-design linear regression

AU - Bartlett, Peter L.

AU - Koolen, Wouter M.

AU - Malek, Alan

AU - Takimoto, Eiji

AU - Warmuth, Manfred K.

PY - 2015/1/1

Y1 - 2015/1/1

N2 - We consider a linear regression game in which the covariates are known in advance: at each round, the learner predicts a real-value, the adversary reveals a label, and the learner incurs a squared error loss. The aim is to minimize the regret with respect to linear predictions. For a variety of constraints on the adversary's labels, we show that the minimax optimal strategy is linear, with a parameter choice that is reminiscent of ordinary least squares (and as easy to compute). The predictions depend on all covariates, past and future, with a particular weighting assigned to future covariates corresponding to the role that they play in the minimax regret. We study two families of label sequences: box constraints (under a covariate compatibility condition), and a weighted 2- norm constraint that emerges naturally from the analysis. The strategy is adaptive in the sense that it requires no knowledge of the constraint set. We obtain an explicit expression for the minimax regret for these games. For the case of uniform box constraints, we show that, with worst case covariate sequences, the regret is O(d log T), with no dependence on the scaling of the covariates.

AB - We consider a linear regression game in which the covariates are known in advance: at each round, the learner predicts a real-value, the adversary reveals a label, and the learner incurs a squared error loss. The aim is to minimize the regret with respect to linear predictions. For a variety of constraints on the adversary's labels, we show that the minimax optimal strategy is linear, with a parameter choice that is reminiscent of ordinary least squares (and as easy to compute). The predictions depend on all covariates, past and future, with a particular weighting assigned to future covariates corresponding to the role that they play in the minimax regret. We study two families of label sequences: box constraints (under a covariate compatibility condition), and a weighted 2- norm constraint that emerges naturally from the analysis. The strategy is adaptive in the sense that it requires no knowledge of the constraint set. We obtain an explicit expression for the minimax regret for these games. For the case of uniform box constraints, we show that, with worst case covariate sequences, the regret is O(d log T), with no dependence on the scaling of the covariates.

UR - http://www.scopus.com/inward/record.url?scp=84984693648&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84984693648&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:84984693648

VL - 40

JO - Journal of Machine Learning Research

JF - Journal of Machine Learning Research

SN - 1532-4435

IS - 2015

ER -