Multimodal variational autoencoders (VAEs) have shown promise as efficient generative models for weakly-supervised data. Yet, despite their advantage of weak supervision, they exhibit a gap in generative quality compared to unimodal VAEs, which are completely unsupervised. In an attempt to explain this gap, we uncover a fundamental limitation that applies to a large family of mixture-based multimodal VAEs. We prove that the sub-sampling of modalities enforces an undesirable upper bound on the multimodal ELBO and thereby limits the generative quality of the respective models. Empirically, we showcase the generative quality gap on both synthetic and real data and present the tradeoffs between different variants of multimodal VAEs. We find that none of the existing approaches fulfills all desired criteria of an effective multimodal generative model when applied on more complex datasets than those used in previous benchmarks. In summary, we identify, formalize, and validate fundamental limitations of VAE-based approaches for modeling weakly-supervised data and discuss implications for real-world applications.


    Access

    Download


    Export, share and cite



    Title :

    On the Limitations of Multimodal VAEs


    Contributors:

    Publication date :

    2021


    Remarks:

    ICLR 2022 camera-ready version


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Mitigating Modality Collapse in Multimodal VAEs Via Impartial Optimization

    Javaloy, Adrian / Meghdadi, Maryam / Valera, Isabel | TIBKAT | 2023


    f-VAEs: Improve VAEs with Conditional Flows

    Su, Jianlin / Wu, Guang | ArXiv | 2018

    Free access

    Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization

    Javaloy, Adrián / Meghdadi, Maryam / Valera, Isabel | ArXiv | 2022

    Free access

    LEARNING HIERARCHICAL PRIORS IN VAES

    Klushyn, Alexej / Chen, Nutan / Kurle, Richard et al. | TIBKAT | 2020


    Relaxed-Responsibility Hierarchical Discrete VAEs

    Willetts, Matthew / Miscouridou, Xenia / Roberts, Stephen et al. | ArXiv | 2020

    Free access