Joint Explicit and Implicit Cross-Modal Interaction Network for Anterior Chamber Inflammation Diagnosis Article Swipe

MAKE IMPACT

Qian Shao , Ye Dai , Haochao Ying , Kan Xu , Jinhong Wang , Wei Chi , Jian Wu · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2312.06171

Uveitis demands the precise diagnosis of anterior chamber inflammation (ACI) for optimal treatment. However, current diagnostic methods only rely on a limited single-modal disease perspective, which leads to poor performance. In this paper, we investigate a promising yet challenging way to fuse multimodal data for ACI diagnosis. Notably, existing fusion paradigms focus on empowering implicit modality interactions (i.e., self-attention and its variants), but neglect to inject explicit modality interactions, especially from clinical knowledge and imaging property. To this end, we propose a jointly Explicit and implicit Cross-Modal Interaction Network (EiCI-Net) for Anterior Chamber Inflammation Diagnosis that uses anterior segment optical coherence tomography (AS-OCT) images, slit-lamp images, and clinical data jointly. Specifically, we first develop CNN-Based Encoders and Tabular Processing Module (TPM) to extract efficient feature representations in different modalities. Then, we devise an Explicit Cross-Modal Interaction Module (ECIM) to generate attention maps as a kind of explicit clinical knowledge based on the tabular feature maps, then integrated them into the slit-lamp feature maps, allowing the CNN-Based Encoder to focus on more effective informativeness of the slit-lamp images. After that, the Implicit Cross-Modal Interaction Module (ICIM), a transformer-based network, further implicitly enhances modality interactions. Finally, we construct a considerable real-world dataset from our collaborative hospital and conduct sufficient experiments to demonstrate the superior performance of our proposed EiCI-Net compared with the state-of-the-art classification methods in various metrics.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2312.06171

PDF

https://arxiv.org/pdf/2312.06171

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4389708836

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4389708836

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2312.06171

Digital Object Identifier

Title

Joint Explicit and Implicit Cross-Modal Interaction Network for Anterior Chamber Inflammation Diagnosis

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-12-11

Full publication date if available

Authors

Qian Shao, Ye Dai, Haochao Ying, Kan Xu, Jinhong Wang, Wei Chi, Jian Wu

List of authors in order

Landing page

https://arxiv.org/abs/2312.06171

Publisher landing page

PDF URL

https://arxiv.org/pdf/2312.06171

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2312.06171

Direct OA link when available

Concepts

Computer science, Modal, Modality (human–computer interaction), Feature (linguistics), Focus (optics), Artificial intelligence, Encoder, Pattern recognition (psychology), Chemistry, Philosophy, Polymer chemistry, Optics, Operating system, Linguistics, Physics

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload