We introduce the Prior Knowledge Robot Keypoint Detection (PK-ROKED) approach for 2D keypoint detection on a robot arm. Our proposed method comprises a Deep Learning network, which learns a representation of keypoints based on prior kinematic knowledge and monocular RGB images. This allows us to provide robust visual feedback for state estimations on a robot arm pose, as this pose can be inaccurate due to imprecise forward kinematics. We incorporate the prior kinematic knowledge about potential keypoint locations into the detection network by concatenating it to the input image. These potential keypoints are derived by forward kinematics, which can be faulty with a bounded error. Hence, this additional information can only indicate and steer the detection algorithm to assumed keypoint locations in image space. Additionally, our approach approximates the uncertainty of a keypoint detection through Monte Carlo Dropout and image moments. To this end, PK-ROKED is trained on the respective synthetic data of a robot arm, which we conduct for two different robot arm models. The resulting performance is evaluated on real-world datasets. We observe our PK-ROKED approach to outperform a baseline network, which we defined for benchmarking. Furthermore, when incorporating our prior knowledge approach into the baseline network we can observe a performance increase compared to without this additional information. To test the robustness of our algorithm, we qualitatively evaluate on challenging data from a space-analogue mission, which demonstrates our approach being potentially deployable in such an environment.


    Access

    Download


    Export, share and cite



    Title :

    Representation Learning for Robot Keypoint Detection using Prior Kinematic Knowledge


    Contributors:


    Publication date :

    2022-09-15


    Type of media :

    Theses


    Type of material :

    Electronic Resource


    Language :

    English




    Prior Knowledge Based Motion Model Representation

    Sappa, Angel D. / Aifanti, Niki / Malassiotis, Sotiris et al. | BASE | 2005

    Free access

    MONOCULAR 2D SEMANTIC KEYPOINT DETECTION AND TRACKING

    CHEN HAOFENG / ARJUN BHARGAVA / RARES ANDREI AMBRUS et al. | European Patent Office | 2023

    Free access

    Keypoint based action localization

    KADAV ASIM / LAI FARLEY / GRAF HANS PETER et al. | European Patent Office | 2025

    Free access

    A Machine-Learning Approach to Keypoint Detection and Landmarking on 3D Meshes

    Creusot, C. / Pears, N. / Austin, J. | British Library Online Contents | 2013


    KEYPOINT BASED ACTION LOCALIZATION

    KADAV ASIM / LAI FARLEY / GRAF HANS PETER et al. | European Patent Office | 2022

    Free access