In recent years, large language models (LLMs) have shown exceptional capabilities across various natural language processing (NLP) tasks. However, such impressive performance often comes with the trade-off of an increased parameter size, posing significant challenges for widespread deployment. Knowledge distillation (KD) provides a solution by transferring knowledge from a large teacher model to a smaller student model. In this paper, we explore the task-specific distillation of LLMs at the logit level. Our investigation reveals that the logits of fine-tuned LLMs exhibit a more extreme long-tail distribution than those from vision models, with hidden "noise" in the long tail affecting distillation performance. Furthermore, existing logits distillation methods often struggle to effectively utilize the internal ranking information from the logits. To address these, we propose the Bi-directional Logits Difference (BiLD) loss. The BiLD loss filters out the long-tail noise by utilizing only top-$k$ teacher and student logits, and leverages the internal logits ranking information by constructing logits differences. To evaluate BiLD loss, we conduct comprehensive experiments on 13 datasets using two types of LLMs. Our results show that the BiLD loss, with only the top-8 logits, outperforms supervised fine-tuning (SFT), vanilla KL loss, and five other distillation methods from both NLP and CV fields.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation


    Beteiligte:
    Li, Minchong (Autor:in) / Zhou, Feng (Autor:in) / Song, Xiaohui (Autor:in)

    Erscheinungsdatum :

    2024


    Anmerkungen:

    COLING 2025


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Knowledge Distillation with Refined Logits

    Sun, Wujie / Chen, Defang / Lyu, Siwei et al. | ArXiv | 2024

    Freier Zugriff

    NormKD: Normalized Logits for Knowledge Distillation

    Chi, Zhihao / Zheng, Tu / Li, Hengjia et al. | ArXiv | 2023

    Freier Zugriff

    Peak-Controlled Logits Poisoning Attack in Federated Distillation

    Tang, Yuhan / Zhang, Aoxu / Wu, Zhiyuan et al. | ArXiv | 2024

    Freier Zugriff

    Adaptive Temperature Based on Logits Correlation in Knowledge Distillation

    Matsuyama, Kazuhiro / Anjum, Usman / Matsuyama, Satoko et al. | ArXiv | 2025

    Freier Zugriff

    Aligning Logits Generatively for Principled Black-Box Knowledge Distillation

    Ma, Jing / Xiang, Xiang / Wang, Ke et al. | ArXiv | 2022

    Freier Zugriff