A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition Article Swipe

MAKE IMPACT

Benjia Zhou , Pichao Wang , Jun Wan , Yanyan Liang , Fan Wang · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2211.09146

Motion recognition is a promising direction in computer vision, but the training of video classification models is much harder than images due to insufficient data and considerable parameters. To get around this, some works strive to explore multimodal cues from RGB-D data. Although improving motion recognition to some extent, these methods still face sub-optimal situations in the following aspects: (i) Data augmentation, i.e., the scale of the RGB-D datasets is still limited, and few efforts have been made to explore novel data augmentation strategies for videos; (ii) Optimization mechanism, i.e., the tightly space-time-entangled network structure brings more challenges to spatiotemporal information modeling; And (iii) cross-modal knowledge fusion, i.e., the high similarity between multimodal representations caused to insufficient late fusion. To alleviate these drawbacks, we propose to improve RGB-D-based motion recognition both from data and algorithm perspectives in this paper. In more detail, firstly, we introduce a novel video data augmentation method dubbed ShuffleMix, which acts as a supplement to MixUp, to provide additional temporal regularization for motion recognition. Secondly, a Unified Multimodal De-coupling and multi-stage Re-coupling framework, termed UMDR, is proposed for video representation learning. Finally, a novel cross-modal Complement Feature Catcher (CFCer) is explored to mine potential commonalities features in multimodal information as the auxiliary fusion stream, to improve the late fusion results. The seamless combination of these novel designs forms a robust spatiotemporal representation and achieves better performance than state-of-the-art methods on four public motion datasets. Specifically, UMDR achieves unprecedented improvements of +4.5% on the Chalearn IsoGD dataset. Our code is available at https://github.com/zhoubenjia/MotionRGBD-PAMI.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2211.09146

PDF

https://arxiv.org/pdf/2211.09146

OA Status

green

Cited By

3

Related Works

10

OpenAlex ID

https://openalex.org/W4309393541

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4309393541

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2211.09146

Digital Object Identifier

Title

A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-11-16

Full publication date if available

Authors

Benjia Zhou, Pichao Wang, Jun Wan, Yanyan Liang, Fan Wang

List of authors in order

Landing page

https://arxiv.org/abs/2211.09146

Publisher landing page

PDF URL

https://arxiv.org/pdf/2211.09146

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2211.09146

Direct OA link when available

Concepts

Computer science, RGB color model, Artificial intelligence, Motion (physics), Fusion mechanism, Representation (politics), Regularization (linguistics), Pattern recognition (psychology), Feature (linguistics), Machine learning, Computer vision, Fusion, Lipid bilayer fusion, Political science, Law, Politics, Linguistics, Philosophy

Top concepts (fields/topics) attached by OpenAlex

Cited by

3

Total citation count in OpenAlex

Citations by year (recent)

2025: 2, 2024: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload