Knowledge Distillation (KD), aiming to train a better student model by mimicking the teacher model, plays an important role in model compression. One typical way is to align the output logits. However, we find a common issue named mis-instruction, that the student would be misled when the predictions based on teacher logits do not follow the labels. Meanwhile, there is other useful dark knowledge in the logits such as the class discriminability, which is vital for distillation. In this paper, we propose a simple yet effective Logit Calibration (LoCa) method, which calibrates the logits from the teacher model based on the ground-truth labels. The key insight is to correct the prediction (to address the mis-instruction issue) and maintain useful dark knowledge simultaneously. Our proposed LoCa does not require any additional parameters. Empirical results on image classification and text generation tasks demonstrate that LoCa can effectively improve the performance of baselines.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    LoCa: Logit Calibration for Knowledge Distillation


    Beteiligte:
    Yang, Runming (Autor:in) / Wu, Taiqiang (Autor:in) / Yang, Yujiu (Autor:in)

    Erscheinungsdatum :

    2024


    Anmerkungen:

    Accepted by ECAI 2024


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Logit Standardization in Knowledge Distillation

    Sun, Shangquan / Ren, Wenqi / Li, Jingzhi et al. | ArXiv | 2024

    Freier Zugriff

    Multi-perspective Contrastive Logit Distillation

    Wang, Qi / Zhou, Jinjia | ArXiv | 2024

    Freier Zugriff

    Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment

    Yu, Chengting / Zhang, Fengzhao / Chen, Ruizhe et al. | ArXiv | 2024

    Freier Zugriff

    Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

    Boizard, Nicolas / Haddad, Kevin El / Hudelot, Céline et al. | ArXiv | 2024

    Freier Zugriff

    Logit Poisoning Attack in Distillation-based Federated Learning and its Countermeasures

    Yu, Yonghao / Zhu, Shunan / Hu, Jinglu | ArXiv | 2024

    Freier Zugriff