Variational autoencoders (VAEs) hold great potential for modelling text, as they could in theory separate high-level semantic and syntactic properties from local regularities of natural language. Practically, however, VAEs with autoregressive decoders often suffer from posterior collapse, a phenomenon where the model learns to ignore the latent variables, causing the sequence VAE to degenerate into a language model. In this paper, we argue that posterior collapse is in part caused by the lack of dispersion in encoder features. We provide empirical evidence to verify this hypothesis, and propose a straightforward fix using pooling. This simple technique effectively prevents posterior collapse, allowing model to achieve significantly better data log-likelihood than standard sequence VAEs. Comparing to existing work, our proposed method is able to achieve comparable or superior performances while being more computationally efficient.


    Access

    Download


    Export, share and cite



    Title :

    On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs


    Contributors:

    Publication date :

    2019


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Mitigating Modality Collapse in Multimodal VAEs Via Impartial Optimization

    Javaloy, Adrian / Meghdadi, Maryam / Valera, Isabel | TIBKAT | 2023


    f-VAEs: Improve VAEs with Conditional Flows

    Su, Jianlin / Wu, Guang | ArXiv | 2018

    Free access


    Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization

    Javaloy, Adrián / Meghdadi, Maryam / Valera, Isabel | ArXiv | 2022

    Free access

    Beyond Diagonal Covariance: Flexible Posterior VAEs via Free-Form Injective Flows

    Sorrenson, Peter / Lührs, Lukas / Olischläger, Hans et al. | ArXiv | 2025

    Free access