Variational Auto-Encoders (VAEs) have emerged as powerful probabilistic models for generative tasks. However, their convergence properties have not been rigorously proven. The challenge of proving convergence is inherently difficult due to the highly non-convex nature of the training objective and the implementation of a Stochastic Neural Network (SNN) within VAE architectures. This paper addresses these challenges by characterizing the optimization trajectory of SNNs utilized in VAEs through the lens of Neural Tangent Kernel (NTK) techniques. These techniques govern the optimization and generalization behaviors of ultra-wide neural networks. We provide a mathematical proof of VAE convergence under mild assumptions, thus advancing the theoretical understanding of VAE optimization dynamics. Furthermore, we establish a novel connection between the optimization problem faced by over-parameterized SNNs and the Kernel Ridge Regression (KRR) problem. Our findings not only contribute to the theoretical foundation of VAEs but also open new avenues for investigating the optimization of generative models using advanced kernel methods. Our theoretical claims are verified by experimental simulations.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspective


    Beteiligte:
    Wang, Li (Autor:in) / Huang, Wei (Autor:in)

    Erscheinungsdatum :

    2024


    Anmerkungen:

    Accepted by Machine Learning journal


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Theoretical Convergence Guarantees for Variational Autoencoders

    Surendran, Sobihan / Godichon-Baggioni, Antoine / Corff, Sylvain Le | ArXiv | 2024

    Freier Zugriff

    Variational Autoencoders

    Ghojogh, Benyamin / Crowley, Mark / Karray, Fakhri et al. | Springer Verlag | 2022


    Over-Parameterized Variational Optical Flow

    Nir, T. / Bruckstein, A. M. / Kimmel, R. | British Library Online Contents | 2008



    ENHANCING NEURAL NETWORK EXPLAINABILITY WITH VARIATIONAL AUTOENCODERS

    Tran, Loc / Dolph, Chester / Zhao, Derek | TIBKAT | 2021