Hierarchial-DQN Position-Aided Beamforming for Uplink mmWave Cellular-Connected UAVs

Praneeth Susarla; Yansha Deng; Markku Juntti; Olli Silven

doi:10.1109/GLOBECOM48099.2022.10001044

Hierarchial-DQN Position-Aided Beamforming for Uplink mmWave Cellular-Connected UAVs

Praneeth Susarla^*, Yansha Deng, Markku Juntti^*, Olli Silven^*

^*Corresponding author for this work

Research output: Contribution to conference types › Paper › peer-review

3 Citations (Scopus)

Abstract

Unmanned aerial vehicles (UAVs) are the vital components of sixth generation (6G) millimeter wave (mmWave) wireless networks. Fast and reliable beam alignment is essential for efficient beam-based mmWave communications between UAVs and the base stations (BSs). Learning-based approaches may greatly reduce the overhead by leveraging UAV data, such as position, to identify the optimal beam directions. In this paper, we propose a deep reinforcement learning (DRL)-based framework for UAV-BS beam alignment using the hierarchical deep Q-Network (hDQN) in a mmWave radio setting. We consider uplink communications where the UAV hovers around 5G new radio (NR) BS coverage area, with three dimensional (3D) beams under diverse channel conditions. A BS serves with learnt beam-pairs in an uplink manner upon every communication request from UAV inside the multi-location environment. Compared to our prior DQN-based method, the proposed hDQN framework uses the location information and the fixed spatial arrangement of the antenna elements to reduce the beam search complexity and maximize the data rates efficiently. The results show that our proposed hDQN-based framework converges faster than the DQN-based approach with an average overall training reduction of 43% and, is generic to multi-location environments across different uniform planar array (UPA) configurations and diverse channel conditions.

Original language	English
Pages	1308-1313
Number of pages	6
DOIs	https://doi.org/10.1109/GLOBECOM48099.2022.10001044
Publication status	Published - 2022
Event	2022 IEEE Global Communications Conference, GLOBECOM 2022 - Virtual, Online, Brazil Duration: 4 Dec 2022 → 8 Dec 2022

Conference

Conference	2022 IEEE Global Communications Conference, GLOBECOM 2022
Country/Territory	Brazil
City	Virtual, Online
Period	4/12/2022 → 8/12/2022

Keywords

5G and beyond
6G
Deep Q-Network
hierarchical Beam alignment
mmWave

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/GLOBECOM48099.2022.10001044

Cite this

@conference{c538aa47a12d4dac824ae1a65d4dd182,

title = "Hierarchial-DQN Position-Aided Beamforming for Uplink mmWave Cellular-Connected UAVs",

abstract = "Unmanned aerial vehicles (UAVs) are the vital components of sixth generation (6G) millimeter wave (mmWave) wireless networks. Fast and reliable beam alignment is essential for efficient beam-based mmWave communications between UAVs and the base stations (BSs). Learning-based approaches may greatly reduce the overhead by leveraging UAV data, such as position, to identify the optimal beam directions. In this paper, we propose a deep reinforcement learning (DRL)-based framework for UAV-BS beam alignment using the hierarchical deep Q-Network (hDQN) in a mmWave radio setting. We consider uplink communications where the UAV hovers around 5G new radio (NR) BS coverage area, with three dimensional (3D) beams under diverse channel conditions. A BS serves with learnt beam-pairs in an uplink manner upon every communication request from UAV inside the multi-location environment. Compared to our prior DQN-based method, the proposed hDQN framework uses the location information and the fixed spatial arrangement of the antenna elements to reduce the beam search complexity and maximize the data rates efficiently. The results show that our proposed hDQN-based framework converges faster than the DQN-based approach with an average overall training reduction of 43% and, is generic to multi-location environments across different uniform planar array (UPA) configurations and diverse channel conditions.",

keywords = "5G and beyond, 6G, Deep Q-Network, hierarchical Beam alignment, mmWave",

author = "Praneeth Susarla and Yansha Deng and Markku Juntti and Olli Silven",

note = "Funding Information: V. CONCLUSION AND FUTURE WORK In this paper, we proposed a hDQN-based position-aided beam alignment framework for cellular-connected mmWave UAVs and maximize their beamforming gain within the BS coverage area in an online manner. We also analyzed the hDQN approach over state-of-the-art DQN-based method under different UPA antenna configurations and diverse channel conditions. Our results shown that, the proposed hDQN approach converges faster than the DQN method with an average overall training reduction of 43% for UPA configurations. Having shown some promising results, we will address the hDQN architecture under different UAV radial distances from BS, large number of beam-directional pairs, interference mitigation etc. as future works. VI. ACKNOWLEDGEMENTS The research was supported by 6G Flagship (Grant No. 346208), Finland and the Engineering and Physical Research Council (EPSRC), U.K., under Grant EP/W004348/1. REFERENCES Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE Global Communications Conference, GLOBECOM 2022 ; Conference date: 04-12-2022 Through 08-12-2022",

year = "2022",

doi = "10.1109/GLOBECOM48099.2022.10001044",

language = "English",

pages = "1308--1313",

}

TY - CONF

T1 - Hierarchial-DQN Position-Aided Beamforming for Uplink mmWave Cellular-Connected UAVs

AU - Susarla, Praneeth

AU - Deng, Yansha

AU - Juntti, Markku

AU - Silven, Olli

N1 - Funding Information: V. CONCLUSION AND FUTURE WORK In this paper, we proposed a hDQN-based position-aided beam alignment framework for cellular-connected mmWave UAVs and maximize their beamforming gain within the BS coverage area in an online manner. We also analyzed the hDQN approach over state-of-the-art DQN-based method under different UPA antenna configurations and diverse channel conditions. Our results shown that, the proposed hDQN approach converges faster than the DQN method with an average overall training reduction of 43% for UPA configurations. Having shown some promising results, we will address the hDQN architecture under different UAV radial distances from BS, large number of beam-directional pairs, interference mitigation etc. as future works. VI. ACKNOWLEDGEMENTS The research was supported by 6G Flagship (Grant No. 346208), Finland and the Engineering and Physical Research Council (EPSRC), U.K., under Grant EP/W004348/1. REFERENCES Publisher Copyright: © 2022 IEEE.

PY - 2022

Y1 - 2022

N2 - Unmanned aerial vehicles (UAVs) are the vital components of sixth generation (6G) millimeter wave (mmWave) wireless networks. Fast and reliable beam alignment is essential for efficient beam-based mmWave communications between UAVs and the base stations (BSs). Learning-based approaches may greatly reduce the overhead by leveraging UAV data, such as position, to identify the optimal beam directions. In this paper, we propose a deep reinforcement learning (DRL)-based framework for UAV-BS beam alignment using the hierarchical deep Q-Network (hDQN) in a mmWave radio setting. We consider uplink communications where the UAV hovers around 5G new radio (NR) BS coverage area, with three dimensional (3D) beams under diverse channel conditions. A BS serves with learnt beam-pairs in an uplink manner upon every communication request from UAV inside the multi-location environment. Compared to our prior DQN-based method, the proposed hDQN framework uses the location information and the fixed spatial arrangement of the antenna elements to reduce the beam search complexity and maximize the data rates efficiently. The results show that our proposed hDQN-based framework converges faster than the DQN-based approach with an average overall training reduction of 43% and, is generic to multi-location environments across different uniform planar array (UPA) configurations and diverse channel conditions.

AB - Unmanned aerial vehicles (UAVs) are the vital components of sixth generation (6G) millimeter wave (mmWave) wireless networks. Fast and reliable beam alignment is essential for efficient beam-based mmWave communications between UAVs and the base stations (BSs). Learning-based approaches may greatly reduce the overhead by leveraging UAV data, such as position, to identify the optimal beam directions. In this paper, we propose a deep reinforcement learning (DRL)-based framework for UAV-BS beam alignment using the hierarchical deep Q-Network (hDQN) in a mmWave radio setting. We consider uplink communications where the UAV hovers around 5G new radio (NR) BS coverage area, with three dimensional (3D) beams under diverse channel conditions. A BS serves with learnt beam-pairs in an uplink manner upon every communication request from UAV inside the multi-location environment. Compared to our prior DQN-based method, the proposed hDQN framework uses the location information and the fixed spatial arrangement of the antenna elements to reduce the beam search complexity and maximize the data rates efficiently. The results show that our proposed hDQN-based framework converges faster than the DQN-based approach with an average overall training reduction of 43% and, is generic to multi-location environments across different uniform planar array (UPA) configurations and diverse channel conditions.

KW - 5G and beyond

KW - 6G

KW - Deep Q-Network

KW - hierarchical Beam alignment

KW - mmWave

UR - http://www.scopus.com/inward/record.url?scp=85146924524&partnerID=8YFLogxK

U2 - 10.1109/GLOBECOM48099.2022.10001044

DO - 10.1109/GLOBECOM48099.2022.10001044

M3 - Paper

AN - SCOPUS:85146924524

SP - 1308

EP - 1313

T2 - 2022 IEEE Global Communications Conference, GLOBECOM 2022

Y2 - 4 December 2022 through 8 December 2022

ER -

Hierarchial-DQN Position-Aided Beamforming for Uplink mmWave Cellular-Connected UAVs

Abstract

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this