A Unified Federated Deep Q Learning Caching Scheme for Scalable Collaborative Edge Networks

TY - JOUR

T1 - A Unified Federated Deep Q Learning Caching Scheme for Scalable Collaborative Edge Networks

AU - Zhao, Ming

AU - Nakhai, Mohammad

N1 - Publisher Copyright: © 2002-2012 IEEE.

PY - 2024/12

Y1 - 2024/12

N2 - Edge caching-enabled networks can efficiently alleviate data traffic and improve quality of service. However, effectively adapting to users' heterogeneous requests and coordinating among multiple edge servers remains a challenge. In this paper, we address the collaborative cache update and request delivery problem in an edge caching system, aiming to minimize the long-term average system cost under uncertainties of users' heterogeneous demands and dynamic content popularity. To overcome the curse of dimensionality, we decompose the formulated problem into two subproblems: the coordinated proactive cache updating and local request processing. Next, we propose a unified federated deep Q learning (DQL) caching scheme to tackle and coordinate these two subproblems. Particularly, our scheme features a scalable DQL approach with a two-phase action selection procedure to learn the heterogeneous user requests across distributed servers in an online manner. Furthermore, we develop a federated learning (FL)-empowered training process to improve coordination among multiple servers, in which a Thompson sampling (TS)-based algorithm is introduced for smart server selection. We evaluate the performance of our proposed caching scheme in both small-scale and large-scale scenarios through comprehensive experiments, which highlights the advantages of the proposed scheme in terms of caching performance, scalability and robustness.

AB - Edge caching-enabled networks can efficiently alleviate data traffic and improve quality of service. However, effectively adapting to users' heterogeneous requests and coordinating among multiple edge servers remains a challenge. In this paper, we address the collaborative cache update and request delivery problem in an edge caching system, aiming to minimize the long-term average system cost under uncertainties of users' heterogeneous demands and dynamic content popularity. To overcome the curse of dimensionality, we decompose the formulated problem into two subproblems: the coordinated proactive cache updating and local request processing. Next, we propose a unified federated deep Q learning (DQL) caching scheme to tackle and coordinate these two subproblems. Particularly, our scheme features a scalable DQL approach with a two-phase action selection procedure to learn the heterogeneous user requests across distributed servers in an online manner. Furthermore, we develop a federated learning (FL)-empowered training process to improve coordination among multiple servers, in which a Thompson sampling (TS)-based algorithm is introduced for smart server selection. We evaluate the performance of our proposed caching scheme in both small-scale and large-scale scenarios through comprehensive experiments, which highlights the advantages of the proposed scheme in terms of caching performance, scalability and robustness.

KW - Collaborative edge caching

KW - deep reinforcement learning (DRL)

KW - federated learning

KW - smart server selection

UR - http://www.scopus.com/inward/record.url?scp=85189293906&partnerID=8YFLogxK

U2 - 10.1109/TMC.2024.3382824

DO - 10.1109/TMC.2024.3382824

M3 - Article

SN - 1536-1233

VL - 23

SP - 10855

EP - 10866

JO - IEEE Transactions on Mobile Computing

JF - IEEE Transactions on Mobile Computing

IS - 12

M1 - TMC-2023-06-0636

ER -