Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We explain how posterior collapse may occur in pPCA due to local maxima in the log marginal likelihood. Unexpectedly, we prove that the ELBO objective for the linear VAE does not introduce additional spurious local maxima relative to log marginal likelihood. We show further that training a linear VAE with exact variational inference recovers an identifiable global maximum corresponding to the principal component directions. Empirically, we find that our linear analysis is predictive even for high-capacity, non-linear VAEs and helps explain the relationship between the observation noise, local maxima, and posterior collapse in deep Gaussian VAEs.


    Access

    Download


    Export, share and cite



    Title :

    Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse


    Contributors:

    Publication date :

    2019


    Remarks:

    11 main pages, 10 appendix pages. 13 figures total. Accepted at 33rd Conference on Neural Information Processing Systems (NeurIPS 2019)


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Resetting a fixed broken ELBO

    Cukier, Robert I. | ArXiv | 2023

    Free access

    THE USUAL SUSPECTS? REASSESSING BLAME FOR VAE POSTERIOR COLLAPSE

    Dai, Bin / Wang, Ziyu / Wipf, David | TIBKAT | 2021


    Don't blame bacteria

    Calvelli, E.A. | Engineering Index Backfile | 1963


    Don’t Blame the Bacteria

    Calvelli, E. A. | SAE Technical Papers | 1963


    Don't blame the drivers

    British Library Online Contents | 2004