Knowledge Distillation (KD), a learning manner with a larger teacher network guiding a smaller student network, transfers dark knowledge from the teacher to the student via logits or intermediate features, with the aim of producing a well-performed lightweight model. Notably, many subsequent feature-based KD methods outperformed the earliest logit-based KD method and iteratively generated numerous state-of-the-art distillation methods. Nevertheless, recent work has uncovered the potential of the logit-based method, bringing the simple KD form based on logits back into the limelight. Features or logits? They partially implement the KD with entirely distinct perspectives; therefore, choosing between logits and features is not straightforward. This paper provides a unified perspective of feature alignment in order to obtain a better comprehension of their fundamental distinction. Inheriting the design philosophy and insights of feature-based and logit-based methods, we introduce a block-wise logit distillation framework to apply implicit logit-based feature alignment by gradually replacing teacher's blocks as intermediate stepping-stone models to bridge the gap between the student and the teacher. Our method obtains comparable or superior results to state-of-the-art distillation methods. This paper demonstrates the great potential of combining logit and features, and we hope it will inspire future research to revisit KD from a higher vantage point.


    Access

    Download


    Export, share and cite



    Title :

    Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment


    Contributors:
    Yu, Chengting (author) / Zhang, Fengzhao (author) / Chen, Ruizhe (author) / Wang, Aili (author) / Liu, Zuozhu (author) / Tan, Shurun (author) / Li, Er-Ping (author)

    Publication date :

    2024


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Logit Standardization in Knowledge Distillation

    Sun, Shangquan / Ren, Wenqi / Li, Jingzhi et al. | ArXiv | 2024

    Free access

    LoCa: Logit Calibration for Knowledge Distillation

    Yang, Runming / Wu, Taiqiang / Yang, Yujiu | ArXiv | 2024

    Free access

    Multi-perspective Contrastive Logit Distillation

    Wang, Qi / Zhou, Jinjia | ArXiv | 2024

    Free access

    Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

    Boizard, Nicolas / Haddad, Kevin El / Hudelot, Céline et al. | ArXiv | 2024

    Free access

    Logit Poisoning Attack in Distillation-based Federated Learning and its Countermeasures

    Yu, Yonghao / Zhu, Shunan / Hu, Jinglu | ArXiv | 2024

    Free access