Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection Article Swipe

MAKE IMPACT

Mehar Khurana , Neehar Peri , Deva Ramanan , James Hays · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2406.10115

State-of-the-art 3D object detectors are often trained on massive labeled datasets. However, annotating 3D bounding boxes remains prohibitively expensive and time-consuming, particularly for LiDAR. Instead, recent works demonstrate that self-supervised pre-training with unlabeled data can improve detection accuracy with limited labels. Contemporary methods adapt best-practices for self-supervised learning from the image domain to point clouds (such as contrastive learning). However, publicly available 3D datasets are considerably smaller and less diverse than those used for image-based self-supervised learning, limiting their effectiveness. We do note, however, that such 3D data is naturally collected in a multimodal fashion, often paired with images. Rather than pre-training with only self-supervised objectives, we argue that it is better to bootstrap point cloud representations using image-based foundation models trained on internet-scale data. Specifically, we propose a shelf-supervised approach (e.g. supervised with off-the-shelf image foundation models) for generating zero-shot 3D bounding boxes from paired RGB and LiDAR data. Pre-training 3D detectors with such pseudo-labels yields significantly better semi-supervised detection accuracy than prior self-supervised pretext tasks. Importantly, we show that image-based shelf-supervision is helpful for training LiDAR-only, RGB-only and multi-modal (RGB + LiDAR) detectors. We demonstrate the effectiveness of our approach on nuScenes and WOD, significantly improving over prior work in limited data settings. Our code is available at https://github.com/meharkhurana03/cm3d

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2406.10115

PDF

https://arxiv.org/pdf/2406.10115

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4399759380

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4399759380

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2406.10115

Digital Object Identifier

Title

Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-06-14

Full publication date if available

Authors

Mehar Khurana, Neehar Peri, Deva Ramanan, James Hays

List of authors in order

Landing page

https://arxiv.org/abs/2406.10115

Publisher landing page

PDF URL

https://arxiv.org/pdf/2406.10115

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2406.10115

Direct OA link when available

Concepts

Modal, Training (meteorology), Computer science, Artificial intelligence, Object (grammar), Off the shelf, Object detection, Computer vision, Training set, Pattern recognition (psychology), Geography, Software engineering, Meteorology, Chemistry, Polymer chemistry

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload