Dataset distillation is an effective technique for reducing the cost and complexity of model training while maintaining performance by compressing large datasets into smaller, more efficient versions. In this paper, we present a novel generative dataset distillation method that can improve the accuracy of aligning prediction logits. Our approach integrates self-knowledge distillation to achieve more precise distribution matching between the synthetic and original data, thereby capturing the overall structure and relationships within the data. To further improve the accuracy of alignment, we introduce a standardization step on the logits before performing distribution matching, ensuring consistency in the range of logits. Through extensive experiments, we demonstrate that our method outperforms existing state-of-the-art methods, resulting in superior distillation performance.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Generative Dataset Distillation Based on Self-knowledge Distillation


    Beteiligte:
    Li, Longzhen (Autor:in) / Li, Guang (Autor:in) / Togo, Ren (Autor:in) / Maeda, Keisuke (Autor:in) / Ogawa, Takahiro (Autor:in) / Haseyama, Miki (Autor:in)

    Erscheinungsdatum :

    2025


    Anmerkungen:

    Accepted by ICASSP 2025


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    KNOWLEDGE DISTILLATION TECHNIQUES

    MALACH ERAN / KAPLUN GAL / SHALEV-SHWARTZ SHAI | Europäisches Patentamt | 2023

    Freier Zugriff

    Decoupled Knowledge Distillation

    Zhao, Borui / Cui, Quan / Song, Renjie et al. | ArXiv | 2022

    Freier Zugriff


    Logit Standardization in Knowledge Distillation

    Sun, Shangquan / Ren, Wenqi / Li, Jingzhi et al. | ArXiv | 2024

    Freier Zugriff

    KNOWLEDGE DISTILLATION FOR AUTONOMOUS VEHICLES

    WANG MINFA / DING KAI / CHEN HAOYU et al. | Europäisches Patentamt | 2022

    Freier Zugriff