Selective Structured State-Spaces for Long-Form Video Understanding Article Swipe

MAKE IMPACT

Jue Wang , Wentao Zhu , Pichao Wang , Xiang Yu , Linda Liu , Mohamed Omar , Roszilah Hamid · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2303.14526

Effective modeling of complex spatiotemporal dependencies in long-form videos remains an open problem. The recently proposed Structured State-Space Sequence (S4) model with its linear complexity offers a promising direction in this space. However, we demonstrate that treating all image-tokens equally as done by S4 model can adversely affect its efficiency and accuracy. To address this limitation, we present a novel Selective S4 (i.e., S5) model that employs a lightweight mask generator to adaptively select informative image tokens resulting in more efficient and accurate modeling of long-term spatiotemporal dependencies in videos. Unlike previous mask-based token reduction methods used in transformers, our S5 model avoids the dense self-attention calculation by making use of the guidance of the momentum-updated S4 model. This enables our model to efficiently discard less informative tokens and adapt to various long-form video understanding tasks more effectively. However, as is the case for most token reduction methods, the informative image tokens could be dropped incorrectly. To improve the robustness and the temporal horizon of our model, we propose a novel long-short masked contrastive learning (LSMCL) approach that enables our model to predict longer temporal context using shorter input videos. We present extensive comparative results using three challenging long-form video understanding datasets (LVU, COIN and Breakfast), demonstrating that our approach consistently outperforms the previous state-of-the-art S4 model by up to 9.6% accuracy while reducing its memory footprint by 23%.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2303.14526

PDF

https://arxiv.org/pdf/2303.14526

OA Status

green

Cited By

3

Related Works

10

OpenAlex ID

https://openalex.org/W4361193183

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4361193183

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2303.14526

Digital Object Identifier

Title

Selective Structured State-Spaces for Long-Form Video Understanding

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-03-25

Full publication date if available

Authors

Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, Roszilah Hamid

List of authors in order

Landing page

https://arxiv.org/abs/2303.14526

Publisher landing page

PDF URL

https://arxiv.org/pdf/2303.14526

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2303.14526

Direct OA link when available

Concepts

Computer science, Security token, Robustness (evolution), Artificial intelligence, Memory footprint, Computer vision, Machine learning, Pattern recognition (psychology), Biochemistry, Gene, Chemistry, Operating system, Computer security

Top concepts (fields/topics) attached by OpenAlex

Cited by

3

Total citation count in OpenAlex

Citations by year (recent)

2025: 2, 2024: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload