Logit based knowledge distillation gets less attention in recent years since feature based methods perform better in most cases. Nevertheless, we find it still has untapped potential when we re-investigate the temperature, which is a crucial hyper-parameter to soften the logit outputs. For most of the previous works, it was set as a fixed value for the entire distillation procedure. However, as the logits from different samples are distributed quite variously, it is not feasible to soften all of them to an equal degree by just a single temperature, which may make the previous work transfer the knowledge of each sample inadequately. In this paper, we restudy the hyper-parameter temperature and figure out its incapability to distill the knowledge from each sample sufficiently when it is a single value. To address this issue, we propose Normalized Knowledge Distillation (NormKD), with the purpose of customizing the temperature for each sample according to the characteristic of the sample's logit distribution. Compared to the vanilla KD, NormKD barely has extra computation or storage cost but performs significantly better on CIRAR-100 and ImageNet for image classification. Furthermore, NormKD can be easily applied to the other logit based methods and achieve better performance which can be closer to or even better than the feature based method.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    NormKD: Normalized Logits for Knowledge Distillation


    Beteiligte:
    Chi, Zhihao (Autor:in) / Zheng, Tu (Autor:in) / Li, Hengjia (Autor:in) / Yang, Zheng (Autor:in) / Wu, Boxi (Autor:in) / Lin, Binbin (Autor:in) / Cai, Deng (Autor:in)

    Erscheinungsdatum :

    2023


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Knowledge Distillation with Refined Logits

    Sun, Wujie / Chen, Defang / Lyu, Siwei et al. | ArXiv | 2024

    Freier Zugriff

    Adaptive Temperature Based on Logits Correlation in Knowledge Distillation

    Matsuyama, Kazuhiro / Anjum, Usman / Matsuyama, Satoko et al. | ArXiv | 2025

    Freier Zugriff

    Aligning Logits Generatively for Principled Black-Box Knowledge Distillation

    Ma, Jing / Xiang, Xiang / Wang, Ke et al. | ArXiv | 2022

    Freier Zugriff

    Peak-Controlled Logits Poisoning Attack in Federated Distillation

    Tang, Yuhan / Zhang, Aoxu / Wu, Zhiyuan et al. | ArXiv | 2024

    Freier Zugriff

    BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation

    Li, Minchong / Zhou, Feng / Song, Xiaohui | ArXiv | 2024

    Freier Zugriff