Abstract

Info

Title:: Multimodal fusion of PET-CT for semantic image segmentation: a review

Author(s):: ZHOU Xinhong1; HUANG Gang2; 1. School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai 200093, China 2. Shanghai Key Laboratory of Molecular Imaging, Jiading District Central Hospital, Shanghai University of Medicine and Health Sciences, Shanghai 201318, China

Keywords:: Keywords: PET-CT convolutional neural network semantic segmentation multimodal review

Abstract:: Abstract: In recent years, deep learning has exhibited excellent performance in various semantic image segmentation tasks. However, the semantic segmentation based on monomodal image still has limitations in the segmentations of cancer with lymph node metastasis and some small lesions. PET-CT which is a multimodal imaging mode, providing anatomical and functional information about structure, physiology and pathology, is considered to be the preferred imaging choice for cancer diagnosis, staging and treatment response evaluation. The multimodal complementary information carried by PET-CT is introduced into the semantic segmentation model of deep convolutional neural network to assist the segmentation of lesions. The study focuses on the application of PET-CT multimodal image feature fusion in semantic image segmentation using convolutional neural network, and introduces the characteristics of multimodal imaging and commonly-used medical image segmentation networks. Three ideas for multimodal fusion based on deep learning at this stage are summarized and divided into early fusion, late fusion and hybrid fusion. The pros and cons of multimodal fusion based segmentation methods are analyzed in terms of segmentation performance, parameter size, method highlights and shortcomings. Finally, the current issues existing in multimodal medical image segmentation are discussed.