Exploring Memory-Accuracy Trade-off in Incremental Learning

Student thesis: Doctoral ThesisDoctor of Philosophy

Abstract

Deep neural networks have demonstrated impressive capabilities but are prone to catastrophic forgetting when learning incrementally. Catastrophic forgetting refers to the tendency of these models to overwrite previously acquired knowledge with new information, leading to a degradation in performance on earlier learned tasks.

To address this issue, considerable efforts have been made, resulting in two main categories of incremental learning: class incremental and task incremental learning.

In class incremental settings, replay-based methods are emphasised, where the challenge is to manage the memory footprint while storing exemplars from each encountered class. This leads to a complex trade-off between model performance and the number of stored exemplars. In task incremental settings, the emphasis shifts to parameter-isolationbased methods, where the main issue revolves around controlling the addition of extra network modules or structures, as this tends to increase the overall memory usage. Both paradigms grapple with the fundamental trade-off between memory efficiency and model accuracy.

To address these challenges, this research introduces novel techniques for both settings. In the class incremental context, we present Information-Back Discrete Representation Replay, a two-step compression approach that efficiently stores and utilises past samples to minimise memory overhead. For task incremental learning, Neural Weight Search is proposed, which automatically finds optimal combinations of frozen weights to construct new models for new tasks, thereby controlling memory growth.

Additionally, this thesis explores the intersection of model fairness and incremental learning. The Multi-Group Parity (MGP) method is introduced to incrementally reduce biases for multiple sensitive attributes in personality computing. Unlike previous works that require separate models for each task, MGP utilises a unified model to handle multiple tasks, thus optimising memory usage.

In summary, this thesis offers solutions to the pivotal problem of balancing memory and performance in incremental learning. The proposed techniques serve to make incremental learning more efficient, scalable, and ethically responsible.


Date of Award1 Jun 2024
Original languageEnglish
Awarding Institution
  • King's College London
SupervisorOya Celiktutan Dikici (Supervisor) & Ernest Kamavuako (Supervisor)

Cite this

'