Share：
Share this content in WeChat
X
[Chinese][PDF]3342
• Technical Article •
Research on multi-source cardiac image segmentation method based on modal interaction learning
ZHONG Qiaoxin  ZHAO Yizhong  ZHANG Feiyan  LU Xuesong 

Cite this article as: ZHONG Q X, ZHAO Y Z, ZHANG F Y, et al. Research on multi-source cardiac image segmentation method based on modal interaction learning[J]. Chin J Magn Reson Imaging, 2024, 15(4): 145-152. DOI:10.12015/issn.1674-8034.2024.04.023.


[Abstract] Objective To establish an artificial intelligence (AI) deep learning network for multimodal cardiac magnetic resonance (CMR) image segmentation and improve the Dice coefficient.Materials and Methods A retrospective analysis was performed on a publicly available dataset from the 2019 multi-sequence cardiac CMR segmentation challenge, which contains CMR image data of 45 patients including balanced steady-state free precession (bSSFP) modality, late gadolinium enhancement (LGE) modality, and T2-weighted imaging (T2WI) modality. A new dual-stream U-shaped network framework was constructed to achieve segmentation of cardiac MR images in both bSSFP and LGE modalities, as well as bSSFP and T2WI modalities. During the encoding phase, unregistered images of each modality were alternately fed into their respective branches for feature learning. The obtained feature maps were then fed into a shared layer for the interaction and supplementation of multi-modal information, and the shared features were finally separated and fed into their respective branches for decoding and output. Validation experiments were conducted on the 2019 multi-sequence CMR segmentation challenge dataset using five-fold cross-validation. The proposed model's performance was evaluated using the Dice coefficient, and the Wilcoxon signed-rank test was used to test the differences between the models.Results In the segmentation experiments of bSSFP and LGE modalities, the proposed method showed a significant improvement in average Dice coefficient compared to the traditional UNet model and the latest Swin-Unet model for the bSSFP modality (P<0.001). For the LGE modality, the average Dice coefficient was significantly improved compared to the traditional UNet model (P<0.001), and there was some improvement compared to the Swin-Unet model (P=0.001) and the dual-stream UNet model (P=0.021). In the segmentation experiments of bSSFP and T2WI modalities, the proposed method demonstrates a significant improvement in average Dice coefficient for the bSSFP modality compared to the UNet model, Swin-Unet model, and dual-stream UNet model (P<0.001). For the T2WI modality, the average Dice coefficient was significantly improved compared to the UNet model (P<0.001) and showed improvement compared to the Swin-Unet model (P=0.025).Conclusions The proposed dual-stream U-shaped network framework provides an effective method for multi-modal segmentation of CMR images and improves the Dice coefficient for bSSFP and LGE modalities, as well as bSSFP and T2WI modalities. It effectively addresses the large anatomical differences and grayscale inconsistencies between multi-modal cardiac MR images, thereby enhancing the model's generalization ability.
[Keywords] myocardial infarction；cardiomyopathy；cardiovascular disease；multi-source cardiac image segmentation；deep neural network；modality interaction learning；magnetic resonance imaging

ZHONG Qiaoxin   ZHAO Yizhong   ZHANG Feiyan   LU Xuesong*  

School of Biomedical Engineering, South-Central Minzu University, Wuhan 430074, China

Corresponding author: LU X S, E-mail: 365103248@qq.com

Conflicts of interest   None.

Received  2023-08-25
Accepted  2024-03-22
DOI: 10.12015/issn.1674-8034.2024.04.023
Cite this article as: ZHONG Q X, ZHAO Y Z, ZHANG F Y, et al. Research on multi-source cardiac image segmentation method based on modal interaction learning[J]. Chin J Magn Reson Imaging, 2024, 15(4): 145-152. DOI:10.12015/issn.1674-8034.2024.04.023.

[1]
FLETT A S, HASLETON J, COOK C, et al. Evaluation of techniques for the quantification of myocardial scar of differing etiology using cardiac magnetic resonance[J]. JACC Cardiovasc Imaging, 2011, 4(2): 150-156. DOI: 10.1016/j.jcmg.2010.11.015.
[2]
USLU F, VARELA M, BONIFACE G, et al. LA-net: a multi-task deep network for the segmentation of the left atrium[J]. IEEE Trans Med Imaging, 2022, 41(2): 456-464. DOI: 10.1109/TMI.2021.3117495.
[3]
LIN M Q, JIANG M J, ZHAO M B, et al. Cascaded triplanar autoencoder M-net for fully automatic segmentation of left ventricle myocardial scar from three-dimensional late gadolinium-enhanced MR images[J]. IEEE J Biomed Health Inform, 2022, 26(6): 2582-2593. DOI: 10.1109/JBHI.2022.3146013.
[4]
JACOBS M, BENOVOY M, CHANG L C, et al. Automated segmental analysis of fully quantitative myocardial blood flow maps by first-pass perfusion cardiovascular magnetic resonance[J/OL]. IEEE Access, 2021, 9: 52796-52811 [2023-08-24]. https://pubmed.ncbi.nlm.nih.gov/33996344/. DOI: 10.1109/access.2021.3070320.
[5]
HASSAN B, HASSAN T, AHMED R, et al. Automated Segmentation and Extraction of Posterior Eye Segment using OCT Scans[C]//2021 International Conference on Robotics and Automation in Industry (ICRAI). Rawalpindi, Pakistan. IEEE, 2021: 1-5. DOI: 10.1109/ICRAI54018.2021.9651403.
[6]
ZHUANG X H. Multivariate mixture model for myocardial segmentation combining multi-source images[J]. IEEE Trans Pattern Anal Mach Intell, 2019, 41(12): 2933-2946. DOI: 10.1109/TPAMI.2018.2869576.
[7]
FOUZIA E A, AZIZ D, AZIZ O. Images Segmentation using Deep Learning Algorithms and Metaheuristics[C]//2022 8th International Conference on Optimization and Applications (ICOA). Genoa, Italy. IEEE, 2022: 1-6. DOI: 10.1109/ICOA55659.2022.9934130.
[8]
ZHOU Z W, SIDDIQUEE M M R, TAJBAKHSH N, et al. UNet++: redesigning skip connections to exploit multiscale features in image segmentation[J]. IEEE Trans Med Imaging, 2020, 39(6): 1856-1867. DOI: 10.1109/TMI.2019.2959609.
[9]
ISENSEE F, JAEGER P F, KOHL S A A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation[J]. Nat Methods, 2021, 18(2): 203-211. DOI: 10.1038/s41592-020-01008-z.
[10]
DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. 2020: arXiv: 2010.11929. http://arxiv.org/abs/2010.11929
[11]
CHEN J N, LU Y Y, YU Q H, et al. TransUNet: transformers make strong encoders for medical image segmentation[EB/OL]. 2021: arXiv: 2102.04306. http://arxiv.org/abs/2102.04306
[12]
LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: hierarchical Vision Transformer using Shifted Windows[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada. IEEE, 2021: 9992-10002. DOI: 10.1109/ICCV48922.2021.00986.
[13]
CAO H, WANG Y Y, CHEN J, et al. Swin-unet: unet-like pure transformer for medical image segmentation[EB/OL]. 2021: arXiv: 2105.05537. http://arxiv.org/abs/2105.05537
[14]
HUANG X H, DENG Z F, LI D D, et al. MISSFormer: an effective transformer for 2D medical image segmentation[J]. IEEE Trans Med Imag, 2023, 42(5): 1484-1494. DOI: 10.1109/TMI.2022.3230943.
[15]
CHEN H, DOU Q, YU L Q, et al. VoxResNet: deep voxelwise residual networks for brain segmentation from 3D MR images[J/OL]. Neuroimage, 2018, 170: 446-455 [2023-08-24]. https://pubmed.ncbi.nlm.nih.gov/28445774/. DOI: 10.1016/j.neuroimage.2017.04.041.
[16]
LI X M, DOU Q, CHEN H, et al. 3D multi-scale FCN with random modality voxel dropout learning for Intervertebral Disc Localization and Segmentation from Multi-modality MR Images[J/OL]. Med Image Anal, 2018, 45: 41-54 [2023-08-24]. https://pubmed.ncbi.nlm.nih.gov/29414435/. DOI: 10.1016/j.media.2018.01.004.
[17]
MORRIS E D, GHANEM A I, DONG M, et al. Cardiac substructure segmentation with deep learning for improved cardiac sparing[J]. Med Phys, 2020, 47(2): 576-586. DOI: 10.1002/mp.13940.
[18]
DOLZ J, GOPINATH K, YUAN J, et al. HyperDense-net: a hyper-densely connected CNN for multi-modal image segmentation[J]. IEEE Trans Med Imaging, 2019, 38(5): 1116-1126. DOI: 10.1109/TMI.2018.2878669.
[19]
WANG R Z, CAO S L, MA K, et al. Pairwise learning for medical image segmentation[J/OL]. Med Image Anal, 2021, 67: 101876 [2023-08-24]. https://pubmed.ncbi.nlm.nih.gov/33197863/. DOI: 10.1016/j.media.2020.101876.
[20]
VALINDRIA V V, PAWLOWSKI N, RAJCHL M, et al. Multi-modal learning from unpaired images: application to multi-organ segmentation in CT and MRI[C]//2018 IEEE Winter Conference on Applications of Computer Vision (WACV). Lake Tahoe, NV, USA. IEEE, 2018: 547-556. DOI: 10.1109/WACV.2018.00066.
[21]
DOU Q, LIU Q D, HENG P A, et al. Unpaired multi-modal segmentation via knowledge distillation[J]. IEEE Trans Med Imaging, 2020, 39(7): 2415-2425. DOI: 10.1109/TMI.2019.2963882.
[22]
ZHOU Z Q, QI L, YANG X, et al. Generalizable cross-modality medical image segmentation via style augmentation and dual normalization[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA. IEEE, 2022: 20824-20833. DOI: 10.1109/CVPR52688.2022.02019.
[23]
YANG J, ZHU Y, WANG C Q, et al. Toward unpaired multi-modal medical image segmentation via learning structured semantic consistency[EB/OL]. 2022: arXiv: 2206.10571. http://arxiv.org/abs/2206.10571
[24]
CHEN X Y, ZHOU H Y, LIU F, et al. MASS: Modality-collaborative semi-supervised segmentation by exploiting cross-modal consistency from unpaired CT and MRI images[J/OL]. Med Image Anal, 2022, 80: 102506 [2023-08-24]. https://pubmed.ncbi.nlm.nih.gov/35696875/. DOI: 10.1016/j.media.2022.102506.
[25]
KAMNITSAS K, LEDIG C, NEWCOMBE V F J, et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation[J/OL]. Med Image Anal, 2017, 36: 61-78 [2023-08-24]. https://pubmed.ncbi.nlm.nih.gov/27865153/. DOI: 10.1016/j.media.2016.10.004.
[26]
LIU Y S, WANG W, WANG K Q, et al. An Automatic Cardiac Segmentation Framework based on Multi-sequence MR Image[EB/OL]. 2019: arXiv: 1909.05488. http://arxiv.org/abs/1909.05488
[27]
RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[EB/OL]. 2015: arXiv: 1505.04597. http://arxiv.org/abs/1505.04597
[28]
YI Z T, WU G, PAN X L, et al. The research of anime character portrait generation based on optimized generative adversarial networks[C]//2021 33rd Chinese Control and Decision Conference (CCDC). Kunming, China. IEEE, 2021: 7361-7366. DOI: 10.1109/CCDC52312.2021.9602217.
[29]
PIAO Z G, GU Y H, YOO S J, et al. Segmentation of Cerebral Hemorrhage CT Images using Swin Transformer and HarDNet[C]//2023 International Conference on Information Networking (ICOIN). Bangkok, Thailand. IEEE, 2023: 522-525. DOI: 10.1109/ICOIN56518.2023.10049057.
[30]
JOARDAR B K, DESHWAL A, DOPPA J R, et al. High-throughput training of deep CNNs on ReRAM-based heterogeneous architectures via optimized normalization layers[J]. IEEE Trans Comput Aided Des Integr Circuits Syst, 2022, 41(5): 1537-1549. DOI: 10.1109/TCAD.2021.3083684.
[31]
LIU Y, HU C H, XU L T, et al. Swin Transformer based Unsupervised Network for Low-Light Image Enhancement[C]//2022 China Automation Congress (CAC). Xiamen, China. IEEE, 2022: 1838-1843. DOI: 10.1109/CAC57257.2022.10055781.
[32]
YE S Q, ZENG P C, LI P F, et al. MLP-stereo: heterogeneous feature fusion in MLP for stereo matching[C]//2022 IEEE International Conference on Image Processing (ICIP). Bordeaux, France. IEEE, 2022: 101-105. DOI: 10.1109/ICIP46576.2022.9897348.
[33]
ZENG C B, SONG C L. Swin transformer with feature pyramid networks for scene text detection of the secondary circuit cabinet wiring[C]//2022 IEEE 4th International Conference on Power, Intelligent Computing and Systems (ICPICS). Shenyang, China. IEEE, 2022: 255-258. DOI: 10.1109/ICPICS55264.2022.9873542.
[34]
TAN Y F, YANG L N, LI X C, et al. A fully convolutional neural network based on 2D-unet in cardiac MR image segmentation[C]//2021 International Conference on Computational Science and Computational Intelligence (CSCI). Las Vegas, NV, USA. IEEE, 2021: 1697-1701. DOI: 10.1109/CSCI54926.2021.00322.
[35]
REDDY T R, BALAJI S, RAMYA R, et al. Analyzing data compression techniques for biomedical signals and images using downsampling and upsampling[C]//2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS). Coimbatore, India. IEEE, 2023: 71-76. DOI: 10.1109/ICACCS57279.2023.10112725.
[36]
ZHENG H, WANG L L, CHEN Y C, et al. Cross U-net: reconstructing cardiac MR image for segmentation[C]//2022 IEEE International Conference on Multimedia and Expo (ICME). Taipei, China. IEEE, 2022: 1-6. DOI: 10.1109/ICME52920.2022.9859940.
[37]
S R, BHARADWAJ A S, S K D, et al. Digital implementation of the softmax activation function and the inverse softmax function[C]//2022 4th International Conference on Circuits, Control, Communication and Computing (I4C). Bangalore, India. IEEE, 2022: 64-67. DOI: 10.1109/I4C57141.2022.10057747.
[38]
GAO S Q, ZHOU H Q, GAO Y B, et al. BayeSeg: Bayesian modeling for medical image segmentation with interpretable generalizability[J/OL]. Med Image Anal, 2023, 89: 102889 [2023-08-24]. https://pubmed.ncbi.nlm.nih.gov/37467643/. DOI: 10.1016/j.media.2023.102889.
[39]
SATRIA A, SITOMPUL O S, MAWENGKANG H. 5-fold cross validation on supporting K-nearest neighbour accuration of making consimilar symptoms disease classification[C]//2021 International Conference on Computer Science and Engineering (IC2SE). Padang, Indonesia. IEEE, 2021: 1-5. DOI: 10.1109/IC2SE52832.2021.9792094.
[40]
HUANG Z Y, GAN Y, LYE T, et al. Segmentation and uncertainty measures of cardiac substrates within optical coherence tomography images via convolutional neural networks[C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa City, IA, USA. IEEE, 2020: 1-4. DOI: 10.1109/ISBI45749.2020.9098495.
[41]
CAMPELLO V M, GKONTRA P, IZQUIERDO C, et al. Multi-centre, multi-vendor and multi-disease cardiac segmentation: the M&Ms challenge[J]. IEEE Trans Med Imaging, 2021, 40(12): 3543-3554. DOI: 10.1109/TMI.2021.3090082.
[42]
GHOSH S, RAY N, BOULANGER P, et al. Automated left atrial segmentation from magnetic resonance image sequences using deep convolutional neural network with autoencoder[C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa City, IA, USA. IEEE, 2020: 1756-1760. DOI: 10.1109/ISBI45749.2020.9098646.
[43]
SHEIKH M A A, MAITY T, KOLE A. IRU-net: an efficient end-to-end network for automatic building extraction from remote sensing images[J/OL]. IEEE Access, 2022, 10: 37811-37828 [2023-08-24]. https://ieeexplore.ieee.org/document/9748124. DOI: 10.1109/ACCESS.2022.3164401.
[44]
MU Y C, SUN J W, HE J. The Combined Focal Cross Entropy and Dice Loss Function for Segmentation of Protein Secondary Structures from Cryo-EM 3D Density maps[C]//2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Las Vegas, NV, USA. IEEE, 2022: 3454-3461. DOI: 10.1109/BIBM55620.2022.9995469.
[45]
LU Y H, ZHOU J H, GUAN C T. Minimizing hybrid dice loss for highly imbalanced 3D neuroimage segmentation[C]//2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Montreal, QC, Canada. IEEE, 2020: 1059-1062. DOI: 10.1109/EMBC44109.2020.9176663.
[46]
REGEHR M, VOLK A, NOGA M, et al. Machine learning and graph based approach to automatic right atrial segmentation from magnetic resonance imaging[C]//2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa City, IA, USA. IEEE, 2020: 826-829. DOI: 10.1109/ISBI45749.2020.9098437.
[47]
KUMAR A, SINGH T. U-NET Architecture for Liver Segmentation using Multi Model Scans[C]//2022 IEEE 19th India Council International Conference (INDICON). Kochi, India. IEEE, 2022: 1-6. DOI: 10.1109/INDICON56171.2022.10039786.
[48]
VIERRA A, RAZZAQ A, ANDREADIS A. Continuous variable analyses: T-test, Mann-Whitney U, Wilcoxon sign rank[M]. Translational Surgery, Academic Press, 2023: 165-170. DOI: 10.1016/B978-0-323-90300-4.00045-8.
[49]
AO Y Y, WU H. Swin transformer combined with convolutional encoder for cephalometric landmarks detection[C]//2021 18th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP). Chengdu, China. IEEE, 2021: 184-187. DOI: 10.1109/ICCWAMTIP53232.2021.9674147.
[50]
LI F Y, LI W S, GAO X B, et al. A novel framework with weighted decision map based on convolutional neural network for cardiac MR segmentation[J]. IEEE J Biomed Health Inform, 2022, 26(5): 2228-2239. DOI: 10.1109/JBHI.2021.3131758.
[51]
YUAN X H, ZHU Y S, WANG Y G. Attention based encoder-decoder network for cardiac semantic segmentation[C]//2020 Chinese Automation Congress (CAC). Shanghai, China. IEEE, 2020: 4578-4582. DOI: 10.1109/CAC51589.2020.9326844.
[52]
QIAO G X, SONG J H. Cardiac image segmentation based on improved U-net[C]//2022 International Conference on Image Processing, Computer Vision and Machine Learning (ICICML). Xi'an, China. IEEE, 2022: 133-137. DOI: 10.1109/ICICML57342.2022.10009706.

PREV Optimization of artificial intelligence-assisted compressed sensing sequences for cerebral and carotid 3D-TOF-MRA
NEXT Application of implantable medical devices in magnetic resonance imaging examination and clinical recommendations
  


LINK


Tel & Fax: +8610-67113815    E-mail: editor@cjmri.cn