Long-Tailed 3D Detection via Multi-Modal Fusion Article Swipe

MAKE IMPACT

Yechi Ma , Neehar Peri , Shuoquan Wei , Wei Hua , Deva Ramanan , Yanan Li , Shu Kong · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2312.10986

Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors. While class labels naturally follow a long-tailed distribution in the real world, existing benchmarks only focus on a few common classes (e.g., pedestrian and car) and neglect many rare but crucial classes (e.g., emergency vehicle and stroller). However, AVs must reliably detect both common and rare classes for safe operation in the open world. We address this challenge by formally studying the problem of Long-Tailed 3D Detection (LT3D), which evaluates all annotated classes, including those in-the-tail. We address LT3D with hierarchical losses that promote feature sharing across classes, and introduce diagnostic metrics that award partial credit to "reasonable" mistakes with respect to the semantic hierarchy. Further, we point out that rare-class accuracy is particularly improved via multi-modal late fusion (MMLF) of independently trained uni-modal LiDAR and RGB detectors. Such an MMLF framework allows us to leverage large-scale uni-modal datasets (with more examples for rare classes) to train better uni-modal detectors. Finally, we examine three critical components of our simple MMLF approach from first principles: whether to train 2D or 3D RGB detectors for fusion, whether to match RGB and LiDAR detections in 3D or the projected 2D image plane, and how to fuse matched detections. Extensive experiments reveal that 2D RGB detectors achieve better recognition accuracy for rare classes than 3D RGB detectors, matching on the 2D image plane mitigates depth estimation errors for better matching, and score calibration and probabilistic fusion notably improves the final performance further. Our MMLF significantly outperforms prior work for LT3D, particularly improving on the six rarest classes from 12.8 to 20.0 mAP! Our code and models are available on our project page.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2312.10986

PDF

https://arxiv.org/pdf/2312.10986

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4389984048

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4389984048

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2312.10986

Digital Object Identifier

Title

Long-Tailed 3D Detection via Multi-Modal Fusion

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-12-18

Full publication date if available

Authors

Yechi Ma, Neehar Peri, Shuoquan Wei, Wei Hua, Deva Ramanan, Yanan Li, Shu Kong

List of authors in order

Landing page

https://arxiv.org/abs/2312.10986

Publisher landing page

PDF URL

https://arxiv.org/pdf/2312.10986

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2312.10986

Direct OA link when available

Concepts

Lidar, Artificial intelligence, RGB color model, Computer vision, Computer science, Leverage (statistics), Detector, Benchmark (surveying), Segmentation, Fusion, Object detection, Pattern recognition (psychology), Remote sensing, Geography, Cartography, Linguistics, Telecommunications, Philosophy

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload