Assessing the aesthetics of an image is challenging, as it is influenced by multiple factors including composition, color, style, and high-level semantics. Existing image aesthetic assessment (IAA) methods primarily rely on human-labeled rating scores, which oversimplify the visual aesthetic information that humans perceive. Conversely, user comments offer more comprehensive information and are a more natural way to express human opinions and preferences regarding image aesthetics. In light of this, we propose learning image aesthetics from user comments, and exploring vision-language pretraining methods to learn multimodal aesthetic representations. Specifically, we pretrain an image-text encoder-decoder model with image-comment pairs, using contrastive and generative objectives to learn rich and generic aesthetic semantics without human labels. To efficiently adapt the pretrained model for downstream IAA tasks, we further propose a lightweight rank-based adapter that employs text as an anchor to learn the aesthetic ranking concept. Our results show that our pretrained aesthetic vision-language model outperforms prior works on image aesthetic captioning over the AVA-Captions dataset, and it has powerful zero-shot capability for aesthetic tasks such as zero-shot style classification and zero-shot IAA, surpassing many supervised baselines. With only minimal finetuning parameters using the proposed adapter module, our model achieves state-of-the-art IAA performance over the AVA dataset.


    Access

    Download


    Export, share and cite



    Title :

    VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining


    Contributors:
    Ke, Junjie (author) / Ye, Keren (author) / Yu, Jiahui (author) / Wu, Yonghui (author) / Milanfar, Peyman (author) / Yang, Feng (author)

    Publication date :

    2023


    Remarks:

    CVPR 2023, https://github.com/google-research/google-research/tree/master/vila


    Type of media :

    Preprint


    Type of material :

    Electronic Resource


    Language :

    English




    Transmission Line Defect Detection Based on UAV Patrol Images and Vision-language Pretraining

    Zhang, Ke / Zheng, Zhaoye / Guo, Yurong et al. | ArXiv | 2024

    Free access

    APS: ACTIVE PRETRAINING WITH SUCCESSOR FEATURES

    LIU, HAO / ABBEEL, PIETER | TIBKAT | 2022


    Rethinking Masked-Autoencoder-Based 3D Point Cloud Pretraining

    Cheng, Nuo / Luo, Chuanyu / Li, Xinzhe et al. | IEEE | 2024



    Undergraduate Pilot Training: Visual Discrimination Pretraining for Landing Task

    B. J. Edwards / D. C. Weyer / B. A. Smith | NTIS | 1979