Memory-guided Network with Uncertainty-based Feature Augmentation for Few-shot Semantic Segmentation

TY - CONF

T1 - Memory-guided Network with Uncertainty-based Feature Augmentation for Few-shot Semantic Segmentation

AU - Chen, Xinyue

AU - Shi, Miaojing

N1 - Publisher Copyright: © 2024 IEEE.

PY - 2024

Y1 - 2024

N2 - The performance of supervised semantic segmentation methods highly relies on the availability of large-scale training data. To alleviate this dependence, few-shot semantic segmentation (FSS) is introduced to leverage the model trained on base classes with sufficient data into the segmentation of novel classes with few data. FSS methods face the challenge of model generalization on novel classes due to the distribution shift between base and novel classes. To overcome this issue, we propose a class-shared memory (CSM) module consisting of a set of learnable memory vectors. These memory vectors learn elemental object patterns from base classes during training whilst re-encoding query features during both training and inference, thereby improving the distribution alignment between base and novel classes. Furthermore, to cope with the performance degradation resulting from the intra-class variance across images, we introduce an uncertainty-based feature augmentation (UFA) module to produce diverse query features during training for improving the model's robustness. We integrate CSM and UFA into representative FSS works, with experimental results on the widely-used PASCAL-5i and COCO-20i datasets demonstrating the superior performance of ours over state of the art.

AB - The performance of supervised semantic segmentation methods highly relies on the availability of large-scale training data. To alleviate this dependence, few-shot semantic segmentation (FSS) is introduced to leverage the model trained on base classes with sufficient data into the segmentation of novel classes with few data. FSS methods face the challenge of model generalization on novel classes due to the distribution shift between base and novel classes. To overcome this issue, we propose a class-shared memory (CSM) module consisting of a set of learnable memory vectors. These memory vectors learn elemental object patterns from base classes during training whilst re-encoding query features during both training and inference, thereby improving the distribution alignment between base and novel classes. Furthermore, to cope with the performance degradation resulting from the intra-class variance across images, we introduce an uncertainty-based feature augmentation (UFA) module to produce diverse query features during training for improving the model's robustness. We integrate CSM and UFA into representative FSS works, with experimental results on the widely-used PASCAL-5i and COCO-20i datasets demonstrating the superior performance of ours over state of the art.

KW - class-shared memory

KW - feature augmentation

KW - Few-shot semantic segmentation

UR - http://www.scopus.com/inward/record.url?scp=85206577213&partnerID=8YFLogxK

U2 - 10.1109/ICME57554.2024.10688161

DO - 10.1109/ICME57554.2024.10688161

M3 - Paper

AN - SCOPUS:85206577213

T2 - 2024 IEEE International Conference on Multimedia and Expo, ICME 2024

Y2 - 15 July 2024 through 19 July 2024

ER -