Recent research on knowledge distillation has increasingly focused on logit distillation because of its simplicity, effectiveness, and versatility in model compression. In this paper, we introduce Refined Logit Distillation (RLD) to address the limitations of current logit distillation methods. Our approach is motivated by the observation that even high-performing teacher models can make incorrect predictions, creating a conflict between the standard distillation loss and the cross-entropy loss. This conflict can undermine the consistency of the student model's learning objectives. Previous attempts to use labels to empirically correct teacher predictions may undermine the class correlation. In contrast, our RLD employs labeling information to dynamically refine teacher logits. In this way, our method can effectively eliminate misleading information from the teacher while preserving crucial class correlations, thus enhancing the value and efficiency of distilled knowledge. Experimental results on CIFAR-100 and ImageNet demonstrate its superiority over existing methods. The code is provided at \text{https://github.com/zju-SWJ/RLD}.


    Access

    Download


    Export, share and cite



    Title :

    Knowledge Distillation with Refined Logits


    Contributors:
    Sun, Wujie (author) / Chen, Defang (author) / Lyu, Siwei (author) / Chen, Genlang (author) / Chen, Chun (author) / Wang, Can (author)

    Publication date :

    2024


    Remarks:

    11 pages, 7 figures


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    NormKD: Normalized Logits for Knowledge Distillation

    Chi, Zhihao / Zheng, Tu / Li, Hengjia et al. | ArXiv | 2023

    Free access

    Adaptive Temperature Based on Logits Correlation in Knowledge Distillation

    Matsuyama, Kazuhiro / Anjum, Usman / Matsuyama, Satoko et al. | ArXiv | 2025

    Free access

    Aligning Logits Generatively for Principled Black-Box Knowledge Distillation

    Ma, Jing / Xiang, Xiang / Wang, Ke et al. | ArXiv | 2022

    Free access

    Peak-Controlled Logits Poisoning Attack in Federated Distillation

    Tang, Yuhan / Zhang, Aoxu / Wu, Zhiyuan et al. | ArXiv | 2024

    Free access

    BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation

    Li, Minchong / Zhou, Feng / Song, Xiaohui | ArXiv | 2024

    Free access