Shift-and-Balance Attention Article Swipe

PDF

Chunjie Luo , Jianfeng Zhan , Tianshu Hao , Lei Wang , Wanling Gao ·

YOU? · · 2021 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2103.13080

Attention is an effective mechanism to improve the deep model capability. Squeeze-and-Excite (SE) introduces a light-weight attention branch to enhance the network's representational power. The attention branch is gated using the Sigmoid function and multiplied by the feature map's trunk branch. It is too sensitive to coordinate and balance the trunk and attention branches' contributions. To control the attention branch's influence, we propose a new attention method, called Shift-and-Balance (SB). Different from Squeeze-and-Excite, the attention branch is regulated by the learned control factor to control the balance, then added into the feature map's trunk branch. Experiments show that Shift-and-Balance attention significantly improves the accuracy compared to Squeeze-and-Excite when applied in more layers, increasing more size and capacity of a network. Moreover, Shift-and-Balance attention achieves better or close accuracy compared to the state-of-art Dynamic Convolution.

Related Topics

Trunked Radio System

Computer Science

Sigmoid Function

Artificial Intelligence

Mechanical Engineering

Biology

Concepts

Balance (ability) Feature (linguistics) Power (physics) Trunk Computer science Control (management) Attention network Sigmoid function Convolution (computer science) Dynamic balance Control theory (sociology) Artificial intelligence Engineering Artificial neural network Psychology Physics Neuroscience Philosophy Quantum mechanics Ecology Mechanical engineering Biology Linguistics

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2103.13080
PDF: https://arxiv.org/pdf/2103.13080
OA Status: green
Cited By: 1
References: 39
Related Works: 10
OpenAlex ID: https://openalex.org/W3138152811

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W3138152811

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2103.13080

Digital Object Identifier
Title: Shift-and-Balance Attention

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2021

Year of publication
Publication date: 2021-03-24

Full publication date if available
Authors: Chunjie Luo, Jianfeng Zhan, Tianshu Hao, Lei Wang, Wanling Gao

List of authors in order
Landing page: https://arxiv.org/abs/2103.13080

Publisher landing page
PDF URL: https://arxiv.org/pdf/2103.13080

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2103.13080

Direct OA link when available
Concepts: Balance (ability), Feature (linguistics), Power (physics), Trunk, Computer science, Control (management), Attention network, Sigmoid function, Convolution (computer science), Dynamic balance, Control theory (sociology), Artificial intelligence, Engineering, Artificial neural network, Psychology, Physics, Neuroscience, Philosophy, Quantum mechanics, Ecology, Mechanical engineering, Biology, Linguistics

Top concepts (fields/topics) attached by OpenAlex
Cited by: 1

Total citation count in OpenAlex
Citations by year (recent): 2021: 1

Per-year citation counts (last 5 years)
References (count): 39

Number of works referenced by this work
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W3138152811
doi	https://doi.org/10.48550/arxiv.2103.13080
ids.doi	https://doi.org/10.48550/arxiv.2103.13080
ids.mag	3138152811
ids.openalex	https://openalex.org/W3138152811
fwci
type	preprint
title	Shift-and-Balance Attention
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10036
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9998999834060669
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1707
topics[0].subfield.display_name	Computer Vision and Pattern Recognition
topics[0].display_name	Advanced Neural Network Applications
topics[1].id	https://openalex.org/T11307
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9984999895095825
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Domain Adaptation and Few-Shot Learning
topics[2].id	https://openalex.org/T10812
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9961000084877014
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1707
topics[2].subfield.display_name	Computer Vision and Pattern Recognition
topics[2].display_name	Human Pose and Action Recognition
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C168031717
concepts[0].level	2
concepts[0].score	0.7262881994247437
concepts[0].wikidata	https://www.wikidata.org/wiki/Q1530280
concepts[0].display_name	Balance (ability)
concepts[1].id	https://openalex.org/C2776401178
concepts[1].level	2
concepts[1].score	0.6180199980735779
concepts[1].wikidata	https://www.wikidata.org/wiki/Q12050496
concepts[1].display_name	Feature (linguistics)
concepts[2].id	https://openalex.org/C163258240
concepts[2].level	2
concepts[2].score	0.601238489151001
concepts[2].wikidata	https://www.wikidata.org/wiki/Q25342
concepts[2].display_name	Power (physics)
concepts[3].id	https://openalex.org/C2781197403
concepts[3].level	2
concepts[3].score	0.5610832571983337
concepts[3].wikidata	https://www.wikidata.org/wiki/Q193472
concepts[3].display_name	Trunk
concepts[4].id	https://openalex.org/C41008148
concepts[4].level	0
concepts[4].score	0.5435438752174377
concepts[4].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[4].display_name	Computer science
concepts[5].id	https://openalex.org/C2775924081
concepts[5].level	2
concepts[5].score	0.5083860754966736
concepts[5].wikidata	https://www.wikidata.org/wiki/Q55608371
concepts[5].display_name	Control (management)
concepts[6].id	https://openalex.org/C2993807640
concepts[6].level	2
concepts[6].score	0.4935048818588257
concepts[6].wikidata	https://www.wikidata.org/wiki/Q103709453
concepts[6].display_name	Attention network
concepts[7].id	https://openalex.org/C81388566
concepts[7].level	3
concepts[7].score	0.4852294921875
concepts[7].wikidata	https://www.wikidata.org/wiki/Q526668
concepts[7].display_name	Sigmoid function
concepts[8].id	https://openalex.org/C45347329
concepts[8].level	3
concepts[8].score	0.4610985517501831
concepts[8].wikidata	https://www.wikidata.org/wiki/Q5166604
concepts[8].display_name	Convolution (computer science)
concepts[9].id	https://openalex.org/C166151169
concepts[9].level	2
concepts[9].score	0.41745609045028687
concepts[9].wikidata	https://www.wikidata.org/wiki/Q16946937
concepts[9].display_name	Dynamic balance
concepts[10].id	https://openalex.org/C47446073
concepts[10].level	3
concepts[10].score	0.4063819944858551
concepts[10].wikidata	https://www.wikidata.org/wiki/Q5165890
concepts[10].display_name	Control theory (sociology)
concepts[11].id	https://openalex.org/C154945302
concepts[11].level	1
concepts[11].score	0.38215699791908264
concepts[11].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[11].display_name	Artificial intelligence
concepts[12].id	https://openalex.org/C127413603
concepts[12].level	0
concepts[12].score	0.1698729693889618
concepts[12].wikidata	https://www.wikidata.org/wiki/Q11023
concepts[12].display_name	Engineering
concepts[13].id	https://openalex.org/C50644808
concepts[13].level	2
concepts[13].score	0.15301671624183655
concepts[13].wikidata	https://www.wikidata.org/wiki/Q192776
concepts[13].display_name	Artificial neural network
concepts[14].id	https://openalex.org/C15744967
concepts[14].level	0
concepts[14].score	0.15207195281982422
concepts[14].wikidata	https://www.wikidata.org/wiki/Q9418
concepts[14].display_name	Psychology
concepts[15].id	https://openalex.org/C121332964
concepts[15].level	0
concepts[15].score	0.14863234758377075
concepts[15].wikidata	https://www.wikidata.org/wiki/Q413
concepts[15].display_name	Physics
concepts[16].id	https://openalex.org/C169760540
concepts[16].level	1
concepts[16].score	0.0
concepts[16].wikidata	https://www.wikidata.org/wiki/Q207011
concepts[16].display_name	Neuroscience
concepts[17].id	https://openalex.org/C138885662
concepts[17].level	0
concepts[17].score	0.0
concepts[17].wikidata	https://www.wikidata.org/wiki/Q5891
concepts[17].display_name	Philosophy
concepts[18].id	https://openalex.org/C62520636
concepts[18].level	1
concepts[18].score	0.0
concepts[18].wikidata	https://www.wikidata.org/wiki/Q944
concepts[18].display_name	Quantum mechanics
concepts[19].id	https://openalex.org/C18903297
concepts[19].level	1
concepts[19].score	0.0
concepts[19].wikidata	https://www.wikidata.org/wiki/Q7150
concepts[19].display_name	Ecology
concepts[20].id	https://openalex.org/C78519656
concepts[20].level	1
concepts[20].score	0.0
concepts[20].wikidata	https://www.wikidata.org/wiki/Q101333
concepts[20].display_name	Mechanical engineering
concepts[21].id	https://openalex.org/C86803240
concepts[21].level	0
concepts[21].score	0.0
concepts[21].wikidata	https://www.wikidata.org/wiki/Q420
concepts[21].display_name	Biology
concepts[22].id	https://openalex.org/C41895202
concepts[22].level	1
concepts[22].score	0.0
concepts[22].wikidata	https://www.wikidata.org/wiki/Q8162
concepts[22].display_name	Linguistics
keywords[0].id	https://openalex.org/keywords/balance
keywords[0].score	0.7262881994247437
keywords[0].display_name	Balance (ability)
keywords[1].id	https://openalex.org/keywords/feature
keywords[1].score	0.6180199980735779
keywords[1].display_name	Feature (linguistics)
keywords[2].id	https://openalex.org/keywords/power
keywords[2].score	0.601238489151001
keywords[2].display_name	Power (physics)
keywords[3].id	https://openalex.org/keywords/trunk
keywords[3].score	0.5610832571983337
keywords[3].display_name	Trunk
keywords[4].id	https://openalex.org/keywords/computer-science
keywords[4].score	0.5435438752174377
keywords[4].display_name	Computer science
keywords[5].id	https://openalex.org/keywords/control
keywords[5].score	0.5083860754966736
keywords[5].display_name	Control (management)
keywords[6].id	https://openalex.org/keywords/attention-network
keywords[6].score	0.4935048818588257
keywords[6].display_name	Attention network
keywords[7].id	https://openalex.org/keywords/sigmoid-function
keywords[7].score	0.4852294921875
keywords[7].display_name	Sigmoid function
keywords[8].id	https://openalex.org/keywords/convolution
keywords[8].score	0.4610985517501831
keywords[8].display_name	Convolution (computer science)
keywords[9].id	https://openalex.org/keywords/dynamic-balance
keywords[9].score	0.41745609045028687
keywords[9].display_name	Dynamic balance
keywords[10].id	https://openalex.org/keywords/control-theory
keywords[10].score	0.4063819944858551
keywords[10].display_name	Control theory (sociology)
keywords[11].id	https://openalex.org/keywords/artificial-intelligence
keywords[11].score	0.38215699791908264
keywords[11].display_name	Artificial intelligence
keywords[12].id	https://openalex.org/keywords/engineering
keywords[12].score	0.1698729693889618
keywords[12].display_name	Engineering
keywords[13].id	https://openalex.org/keywords/artificial-neural-network
keywords[13].score	0.15301671624183655
keywords[13].display_name	Artificial neural network
keywords[14].id	https://openalex.org/keywords/psychology
keywords[14].score	0.15207195281982422
keywords[14].display_name	Psychology
keywords[15].id	https://openalex.org/keywords/physics
keywords[15].score	0.14863234758377075
keywords[15].display_name	Physics
language	en
locations[0].id	pmh:oai:arXiv.org:2103.13080
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2103.13080
locations[0].version	submittedVersion
locations[0].raw_type
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2103.13080
locations[1].id	doi:10.48550/arxiv.2103.13080
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2103.13080
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5033347503
authorships[0].author.orcid	https://orcid.org/0000-0002-6977-929X
authorships[0].author.display_name	Chunjie Luo
authorships[0].author_position	first
authorships[0].raw_author_name	Chunjie Luo
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5085963553
authorships[1].author.orcid	https://orcid.org/0000-0002-3728-6837
authorships[1].author.display_name	Jianfeng Zhan
authorships[1].author_position	middle
authorships[1].raw_author_name	Jianfeng Zhan
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5002465838
authorships[2].author.orcid	https://orcid.org/0000-0001-5540-8722
authorships[2].author.display_name	Tianshu Hao
authorships[2].author_position	middle
authorships[2].raw_author_name	Tianshu Hao
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5100435995
authorships[3].author.orcid	https://orcid.org/0000-0002-7014-2149
authorships[3].author.display_name	Lei Wang
authorships[3].author_position	middle
authorships[3].raw_author_name	Lei Wang
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5075110031
authorships[4].author.orcid	https://orcid.org/0000-0002-3911-9389
authorships[4].author.display_name	Wanling Gao
authorships[4].author_position	last
authorships[4].raw_author_name	Wanling Gao
authorships[4].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2103.13080
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Shift-and-Balance Attention
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10036
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9998999834060669
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1707
primary_topic.subfield.display_name	Computer Vision and Pattern Recognition
primary_topic.display_name	Advanced Neural Network Applications
related_works	https://openalex.org/W4385957115, https://openalex.org/W2061372042, https://openalex.org/W3047779762, https://openalex.org/W1520030019, https://openalex.org/W2071654592, https://openalex.org/W3104477175, https://openalex.org/W2088157920, https://openalex.org/W4283785902, https://openalex.org/W4392232922, https://openalex.org/W1468175696
cited_by_count	1
counts_by_year[0].year	2021
counts_by_year[0].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2103.13080
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2103.13080
best_oa_location.version	submittedVersion
best_oa_location.raw_type
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2103.13080
primary_location.id	pmh:oai:arXiv.org:2103.13080
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2103.13080
primary_location.version	submittedVersion
primary_location.raw_type
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2103.13080
publication_date	2021-03-24
publication_year	2021
referenced_works	https://openalex.org/W2905741102, https://openalex.org/W2095705004, https://openalex.org/W2963495494, https://openalex.org/W2194775991, https://openalex.org/W2895340641, https://openalex.org/W2963393494, https://openalex.org/W2963794428, https://openalex.org/W3109269658, https://openalex.org/W2962935523, https://openalex.org/W2117539524, https://openalex.org/W2963125010, https://openalex.org/W2183341477, https://openalex.org/W2962835968, https://openalex.org/W2963446712, https://openalex.org/W2597655663, https://openalex.org/W3012740297, https://openalex.org/W2963918968, https://openalex.org/W2982083293, https://openalex.org/W2612445135, https://openalex.org/W2964169985, https://openalex.org/W2970107519, https://openalex.org/W2955058313, https://openalex.org/W2890782586, https://openalex.org/W2962944050, https://openalex.org/W3118608800, https://openalex.org/W2414711238, https://openalex.org/W2922509574, https://openalex.org/W2950014519, https://openalex.org/W2752037867, https://openalex.org/W2963420686, https://openalex.org/W2963163009, https://openalex.org/W2626778328, https://openalex.org/W2778955544, https://openalex.org/W2962891704, https://openalex.org/W2963843116, https://openalex.org/W3016719260, https://openalex.org/W2279098554, https://openalex.org/W2964081807, https://openalex.org/W3034421924
referenced_works_count	39
abstract_inverted_index.a	14, 63, 118
abstract_inverted_index.It	41
abstract_inverted_index.To	55
abstract_inverted_index.an	2
abstract_inverted_index.by	35, 78
abstract_inverted_index.in	109
abstract_inverted_index.is	1, 27, 42, 76
abstract_inverted_index.of	117
abstract_inverted_index.or	125
abstract_inverted_index.to	5, 18, 45, 83, 105, 129
abstract_inverted_index.we	61
abstract_inverted_index.The	24
abstract_inverted_index.and	33, 47, 51, 115
abstract_inverted_index.new	64
abstract_inverted_index.the	7, 20, 30, 36, 49, 57, 73, 79, 85, 90, 102, 130
abstract_inverted_index.too	43
abstract_inverted_index.(SE)	12
abstract_inverted_index.deep	8
abstract_inverted_index.from	71
abstract_inverted_index.into	89
abstract_inverted_index.more	110, 113
abstract_inverted_index.show	96
abstract_inverted_index.size	114
abstract_inverted_index.that	97
abstract_inverted_index.then	87
abstract_inverted_index.when	107
abstract_inverted_index.(SB).	69
abstract_inverted_index.added	88
abstract_inverted_index.close	126
abstract_inverted_index.gated	28
abstract_inverted_index.map's	38, 92
abstract_inverted_index.model	9
abstract_inverted_index.trunk	39, 50, 93
abstract_inverted_index.using	29
abstract_inverted_index.better	124
abstract_inverted_index.branch	17, 26, 75
abstract_inverted_index.called	67
abstract_inverted_index.factor	82
abstract_inverted_index.power.	23
abstract_inverted_index.Dynamic	132
abstract_inverted_index.Sigmoid	31
abstract_inverted_index.applied	108
abstract_inverted_index.balance	48
abstract_inverted_index.branch.	40, 94
abstract_inverted_index.control	56, 81, 84
abstract_inverted_index.enhance	19
abstract_inverted_index.feature	37, 91
abstract_inverted_index.improve	6
abstract_inverted_index.layers,	111
abstract_inverted_index.learned	80
abstract_inverted_index.method,	66
abstract_inverted_index.propose	62
abstract_inverted_index.accuracy	103, 127
abstract_inverted_index.achieves	123
abstract_inverted_index.balance,	86
abstract_inverted_index.branch's	59
abstract_inverted_index.capacity	116
abstract_inverted_index.compared	104, 128
abstract_inverted_index.function	32
abstract_inverted_index.improves	101
abstract_inverted_index.network.	119
abstract_inverted_index.Attention	0
abstract_inverted_index.Different	70
abstract_inverted_index.Moreover,	120
abstract_inverted_index.attention	16, 25, 52, 58, 65, 74, 99, 122
abstract_inverted_index.branches'	53
abstract_inverted_index.effective	3
abstract_inverted_index.mechanism	4
abstract_inverted_index.network's	21
abstract_inverted_index.regulated	77
abstract_inverted_index.sensitive	44
abstract_inverted_index.coordinate	46
abstract_inverted_index.increasing	112
abstract_inverted_index.influence,	60
abstract_inverted_index.introduces	13
abstract_inverted_index.multiplied	34
abstract_inverted_index.Experiments	95
abstract_inverted_index.capability.	10
abstract_inverted_index.Convolution.	133
abstract_inverted_index.light-weight	15
abstract_inverted_index.state-of-art	131
abstract_inverted_index.significantly	100
abstract_inverted_index.contributions.	54
abstract_inverted_index.representational	22
abstract_inverted_index.Shift-and-Balance	68, 98, 121
abstract_inverted_index.Squeeze-and-Excite	11, 106
abstract_inverted_index.Squeeze-and-Excite,	72
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	5
citation_normalized_percentile