Deploying large language models (LLMs) of several billion parameters can be impractical in most industrial use cases due to constraints such as cost, latency limitations, and hardware accessibility. Knowledge distillation (KD) offers a solution by compressing knowledge from resource-intensive large models to smaller ones. Various strategies exist, some relying on the text generated by the teacher model and optionally utilizing his logits to enhance learning. However, these methods based on logits often require both teacher and student models to share the same tokenizer, limiting their applicability across different LLM families. In this paper, we introduce Universal Logit Distillation (ULD) loss, grounded in optimal transport, to address this limitation. Our experimental results demonstrate the effectiveness of ULD loss in enabling distillation across models with different architectures and tokenizers, paving the way to a more widespread use of distillation techniques.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs


    Beteiligte:

    Erscheinungsdatum :

    2024


    Anmerkungen:

    10 pages, 5 figures
    Transactions on Machine Learning Research, January 2025


    Medientyp :

    Preprint


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Multi-perspective Contrastive Logit Distillation

    Wang, Qi / Zhou, Jinjia | ArXiv | 2024

    Freier Zugriff

    Logit Standardization in Knowledge Distillation

    Sun, Shangquan / Ren, Wenqi / Li, Jingzhi et al. | ArXiv | 2024

    Freier Zugriff

    LoCa: Logit Calibration for Knowledge Distillation

    Yang, Runming / Wu, Taiqiang / Yang, Yujiu | ArXiv | 2024

    Freier Zugriff

    Logit Poisoning Attack in Distillation-based Federated Learning and its Countermeasures

    Yu, Yonghao / Zhu, Shunan / Hu, Jinglu | ArXiv | 2024

    Freier Zugriff

    Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment

    Yu, Chengting / Zhang, Fengzhao / Chen, Ruizhe et al. | ArXiv | 2024

    Freier Zugriff