Lunar exploration presents distinct challenges, particularly in the realm of human-robot collaboration. Action recognition is a key capability that enables rovers to anticipate astronaut needs, respond to unforeseen events, and collaborate more effectively to execute complex mission objectives. However, in the dynamic and unpredictable lunar environment, rovers must not only respond but adapt to their surroundings in real-time. Recent advancements in artificial intelligence (AI) and machine learning (ML) have opened new avenues for addressing these challenges using video-based action recognition models. This research aims to advance human-robot collaboration for lunar missions by leveraging Meta AI’s Video Joint Embedding Predictive Architecture (V-JEPA) for video-based action recognition. Specifically, this research evaluates V-JEPA’s performance in recognizing astronaut actions by training on simulated lunar videos and testing with actual Apollo mission footage. Five scenarios were tested: training and evaluating on simulated lunar data, training and evaluating on Apollo footage, training on simulated Earth actions and evaluating on real Earth data, and training on simulated lunar data with evaluation on Apollo footage. Training and evaluation performed on the same type of dataset, achieved high performance, while less stellar performance was observed when evaluating real lunar footage following simulated training. Additional training improved accuracy for specific actions, highlighting the need for better domain adaptation and real-time learning. Future work will focus on enhancing generalization and predictive capabilities to support effective lunar exploration.


    Access

    Check access

    Check availability in my library

    Order at Subito €


    Export, share and cite



    Title :

    Utilizing V-JEPA for action recognition on the lunar surface: advancing human-robot collaboration


    Contributors:

    Conference:

    Pattern Recognition and Prediction XXXVI ; 2025 ; Orlando, Florida, United States


    Published in:

    Proc. SPIE ; 13464


    Publication date :

    2025-05-28





    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English




    Advancing Assembly Through Human-Robot Collaboration: Framework and Implementation

    Mohammed, Abdullah / Wang, Lihui | Springer Verlag | 2020


    Legible Action Selection in Human-Robot Collaboration

    Zhu, Huaijiang / Gabler, Volker / Wollherr, Dirk | BASE | 2017

    Free access

    Intuitive Human-Robot Interface: A 3-Dimensional Action Recognition and UAV Collaboration Framework

    Chaudhary, Akash / Nascimento, Tiago / Saska, Martin | ArXiv | 2024

    Free access

    HUMAN-ROBOT COLLABORATION

    FELIP LEON JAVIER / AHUJA NILESH / CAMPOS MACIAS LEOBARDO et al. | European Patent Office | 2021

    Free access