Black-Box Knowledge Distillation (B2KD) is a formulated problem for cloud-to-edge model compression with invisible data and models hosted on the server. B2KD faces challenges such as limited Internet exchange and edge-cloud disparity of data distributions. In this paper, we formalize a two-step workflow consisting of deprivatization and distillation, and theoretically provide a new optimization direction from logits to cell boundary different from direct logits alignment. With its guidance, we propose a new method Mapping-Emulation KD (MEKD) that distills a black-box cumbersome model into a lightweight one. Our method does not differentiate between treating soft or hard responses, and consists of: 1) deprivatization: emulating the inverse mapping of the teacher function with a generator, and 2) distillation: aligning low-dimensional logits of the teacher and student models by reducing the distance of high-dimensional image points. For different teacher-student pairs, our method yields inspiring distillation performance on various benchmarks, and outperforms the previous state-of-the-art approaches.


    Access

    Download


    Export, share and cite



    Title :

    Aligning Logits Generatively for Principled Black-Box Knowledge Distillation


    Contributors:
    Ma, Jing (author) / Xiang, Xiang (author) / Wang, Ke (author) / Wu, Yuchuan (author) / Li, Yongbin (author)

    Publication date :

    2022


    Remarks:

    To appear at CVPR 2024; significantly rewritten with extra experiments since the preliminary report


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Knowledge Distillation with Refined Logits

    Sun, Wujie / Chen, Defang / Lyu, Siwei et al. | ArXiv | 2024

    Free access

    NormKD: Normalized Logits for Knowledge Distillation

    Chi, Zhihao / Zheng, Tu / Li, Hengjia et al. | ArXiv | 2023

    Free access

    Adaptive Temperature Based on Logits Correlation in Knowledge Distillation

    Matsuyama, Kazuhiro / Anjum, Usman / Matsuyama, Satoko et al. | ArXiv | 2025

    Free access

    Peak-Controlled Logits Poisoning Attack in Federated Distillation

    Tang, Yuhan / Zhang, Aoxu / Wu, Zhiyuan et al. | ArXiv | 2024

    Free access

    BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation

    Li, Minchong / Zhou, Feng / Song, Xiaohui | ArXiv | 2024

    Free access