Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization

TY - JOUR

T1 - Transfer Meta-Learning

T2 - Information-Theoretic Bounds and Information Meta-Risk Minimization

AU - Jose, Sharu Theresa

AU - Simeone, Osvaldo

AU - Durisi, Giuseppe

N1 - Publisher Copyright: © 1963-2012 IEEE.

PY - 2022/1/1

Y1 - 2022/1/1

N2 - Meta-learning automatically infers an inductive bias by observing data from a number of related tasks. The inductive bias is encoded by hyperparameters that determine aspects of the model class or training algorithm, such as initialization or learning rate. Meta-learning assumes that the learning tasks belong to a task environment, and that tasks are drawn from the same task environment both during meta-training and meta-testing. This, however, may not hold true in practice. In this paper, we introduce the problem of transfer meta-learning, in which tasks are drawn from a target task environment during meta-testing that may differ from the source task environment observed during meta-training. Novel information-theoretic upper bounds are obtained on the transfer meta-generalization gap, which measures the difference between the meta-training loss, available at the meta-learner, and the average loss on meta-test data from a new, randomly selected, task in the target task environment. The first bound, on the average transfer meta-generalization gap, captures the meta-environment shift between source and target task environments via the KL divergence between source and target data distributions. The second, PAC-Bayesian bound, and the third, single-draw bound, account for this shift via the log-likelihood ratio between source and target task distributions. Furthermore, two transfer meta-learning solutions are introduced. For the first, termed Empirical Meta-Risk Minimization (EMRM), we derive bounds on the average optimality gap. The second, referred to as Information Meta-Risk Minimization (IMRM), is obtained by minimizing the PAC-Bayesian bound. IMRM is shown via experiments to potentially outperform EMRM.

AB - Meta-learning automatically infers an inductive bias by observing data from a number of related tasks. The inductive bias is encoded by hyperparameters that determine aspects of the model class or training algorithm, such as initialization or learning rate. Meta-learning assumes that the learning tasks belong to a task environment, and that tasks are drawn from the same task environment both during meta-training and meta-testing. This, however, may not hold true in practice. In this paper, we introduce the problem of transfer meta-learning, in which tasks are drawn from a target task environment during meta-testing that may differ from the source task environment observed during meta-training. Novel information-theoretic upper bounds are obtained on the transfer meta-generalization gap, which measures the difference between the meta-training loss, available at the meta-learner, and the average loss on meta-test data from a new, randomly selected, task in the target task environment. The first bound, on the average transfer meta-generalization gap, captures the meta-environment shift between source and target task environments via the KL divergence between source and target data distributions. The second, PAC-Bayesian bound, and the third, single-draw bound, account for this shift via the log-likelihood ratio between source and target task distributions. Furthermore, two transfer meta-learning solutions are introduced. For the first, termed Empirical Meta-Risk Minimization (EMRM), we derive bounds on the average optimality gap. The second, referred to as Information Meta-Risk Minimization (IMRM), is obtained by minimizing the PAC-Bayesian bound. IMRM is shown via experiments to potentially outperform EMRM.

KW - Hospitals

KW - information risk minimization

KW - information-theoretic generalization bounds

KW - Loss measurement

KW - PAC-Bayesian bounds

KW - Risk management

KW - single-draw bounds

KW - Task analysis

KW - Training

KW - Transfer learning

KW - Transfer meta-learning

KW - Upper bound

UR - http://www.scopus.com/inward/record.url?scp=85117321978&partnerID=8YFLogxK

U2 - 10.1109/TIT.2021.3119605

DO - 10.1109/TIT.2021.3119605

M3 - Article

AN - SCOPUS:85117321978

SN - 0018-9448

VL - 68

SP - 474

EP - 501

JO - IEEE TRANSACTIONS ON INFORMATION THEORY

JF - IEEE TRANSACTIONS ON INFORMATION THEORY

IS - 1

ER -