Video reconstruction refers to generate videos through the high-level representations (edge map, labels and so on), while the reconstruction quality is always unsatisfactory due to sparse high-level representations, especially on video data. In order to improve the video reconstruction quality, we proposed a novel approach that generates realistic video from its multimodal information including structure features and color features. To extract color features, we mainly apply the k-means algorithm to segment labels and the structure features are extracted by an edge detection network. Video generation is regarded as learning the mapping from multimodal representations to the original videos. So, a conditional GAN is applied with a learning objective that models the temporal video dynamics. We use a spatio-temporal generator with attention to model the inter-frame dynamics and video consistency is improved in this way. Moreover, we use a multiscale discriminator to improve the improve the intra-frame quality of the video. Experimental results on Cityscapes, Apolloscape datasets demonstrate that our proposed approach performs better in both traditional and generative evaluating indicators.


    Zugriff

    Zugriff prüfen

    Verfügbarkeit in meiner Bibliothek prüfen

    Bestellung bei Subito €


    Exportieren, teilen und zitieren



    Titel :

    Video Reconstruction with Multimodal Information


    Beteiligte:
    Xie, Zhipeng (Autor:in) / Duan, Yiping (Autor:in) / Du, Qiyuan (Autor:in) / Tao, Xiaoming (Autor:in) / Yu, Jiazhong (Autor:in)


    Erscheinungsdatum :

    10.10.2023


    Format / Umfang :

    2991586 byte





    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch



    Hierarchical & multimodal video captioning: Discovering and transferring multimodal knowledge for vision to language

    Liu, An-An / Xu, Ning / Wong, Yongkang et al. | British Library Online Contents | 2017


    Optimal Reconstruction of Human Motion From Scarce Multimodal Data

    Giuseppe Averta / Matilde Iuculano / Paolo Salaris et al. | BASE | 2022

    Freier Zugriff

    Information enquiry kiosk with multimodal user interface

    Karpov, A. A. / Ronzhin, A. L. | British Library Online Contents | 2009



    Construction of Multimodal Transport Information Platform

    Wang Ya / Cheng Yu / Zhao Zhi | DOAJ | 2018

    Freier Zugriff