Cross-Aware Early Fusion With Stage-Divided Vision and Language Transformer Encoders for Referring Image Segmentation Article Swipe

MAKE IMPACT

Yubin Cho , Hyunwoo Yu , Suk‐Ju Kang · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.1109/tmm.2023.3340062

Referring segmentation aims to segment a target object related to a natural language expression. Key challenges of this task are understanding the meaning of complex and ambiguous language expressions and determining the relevant regions in the image with multiple objects by referring to the expression. Recent models have focused on the early fusion with the language features at the intermediate stage of the vision encoder, but these approaches have a limitation that the language features cannot refer to the visual information. To address this issue, this paper proposes a novel architecture, Cross-aware early fusion with stage-divided Vision and Language Transformer encoders (CrossVLT), which allows both language and vision encoders to perform the early fusion for improving the ability of the cross-modal context modeling. Unlike previous methods, our method enables the vision and language features to refer to each other's information at each stage to mutually enhance the robustness of both encoders. Furthermore, unlike the conventional scheme that relies solely on the high-level features for the cross-modal alignment, we introduce a feature-based alignment scheme that enables the low-level to high-level features of the vision and language encoders to engage in the cross-modal alignment. By aligning the intermediate cross-modal features in all encoder stages, this scheme leads to effective cross-modal fusion. In this way, the proposed approach is simple but effective for referring image segmentation, and it outperforms the previous state-of-the-art methods on three public benchmarks.

Type

article

Language

en

Landing Page

https://doi.org/10.1109/tmm.2023.3340062

OA Status

green

Cited By

19

References

45

Related Works

10

OpenAlex ID

https://openalex.org/W4389371459

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4389371459

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.1109/tmm.2023.3340062

Digital Object Identifier

Title

Cross-Aware Early Fusion With Stage-Divided Vision and Language Transformer Encoders for Referring Image Segmentation

Work title

Type

article

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-12-06

Full publication date if available

Authors

Yubin Cho, Hyunwoo Yu, Suk‐Ju Kang

List of authors in order

Landing page

https://doi.org/10.1109/tmm.2023.3340062

Publisher landing page

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2408.07539

Direct OA link when available

Concepts

Computer science, Encoder, Artificial intelligence, Computer vision, Segmentation, Natural language, Image segmentation, Robustness (evolution), Pattern recognition (psychology), Natural language processing, Operating system, Gene, Biochemistry, Chemistry

Top concepts (fields/topics) attached by OpenAlex

Cited by

19

Total citation count in OpenAlex

Citations by year (recent)

2025: 11, 2024: 8

Per-year citation counts (last 5 years)

References (count)

45

Number of works referenced by this work

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload