On the Performance Analysis of Momentum Method: A Frequency Domain Perspective Article Swipe
Xianliang Li
,
Jun Luo
,
Zhiwei Zheng
,
Hanxiao Wang
,
Luo Li
,
Li Wen
,
Linlong Wu
,
Sheng Xu
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2411.19671
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2411.19671
Momentum-based optimizers are widely adopted for training neural networks. However, the optimal selection of momentum coefficients remains elusive. This uncertainty impedes a clear understanding of the role of momentum in stochastic gradient methods. In this paper, we present a frequency domain analysis framework that interprets the momentum method as a time-variant filter for gradients, where adjustments to momentum coefficients modify the filter characteristics. Our experiments support this perspective and provide a deeper understanding of the mechanism involved. Moreover, our analysis reveals the following significant findings: high-frequency gradient components are undesired in the late stages of training; preserving the original gradient in the early stages, and gradually amplifying low-frequency gradient components during training both enhance performance. Based on these insights, we propose Frequency Stochastic Gradient Descent with Momentum (FSGDM), a heuristic optimizer that dynamically adjusts the momentum filtering characteristic with an empirically effective dynamic magnitude response. Experimental results demonstrate the superiority of FSGDM over conventional momentum optimizers.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2411.19671
- https://arxiv.org/pdf/2411.19671
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4405031404
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4405031404Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2411.19671Digital Object Identifier
- Title
-
On the Performance Analysis of Momentum Method: A Frequency Domain PerspectiveWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-11-29Full publication date if available
- Authors
-
Xianliang Li, Jun Luo, Zhiwei Zheng, Hanxiao Wang, Luo Li, Li Wen, Linlong Wu, Sheng XuList of authors in order
- Landing page
-
https://arxiv.org/abs/2411.19671Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2411.19671Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2411.19671Direct OA link when available
- Concepts
-
Perspective (graphical), Frequency domain, Momentum (technical analysis), Domain (mathematical analysis), Physics, Computer science, Mathematics, Economics, Mathematical analysis, Financial economics, Artificial intelligenceTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4405031404 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2411.19671 |
| ids.doi | https://doi.org/10.48550/arxiv.2411.19671 |
| ids.openalex | https://openalex.org/W4405031404 |
| fwci | |
| type | preprint |
| title | On the Performance Analysis of Momentum Method: A Frequency Domain Perspective |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11234 |
| topics[0].field.id | https://openalex.org/fields/19 |
| topics[0].field.display_name | Earth and Planetary Sciences |
| topics[0].score | 0.92330002784729 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1902 |
| topics[0].subfield.display_name | Atmospheric Science |
| topics[0].display_name | Precipitation Measurement and Analysis |
| topics[1].id | https://openalex.org/T13121 |
| topics[1].field.id | https://openalex.org/fields/22 |
| topics[1].field.display_name | Engineering |
| topics[1].score | 0.911300003528595 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2202 |
| topics[1].subfield.display_name | Aerospace Engineering |
| topics[1].display_name | Radio Wave Propagation Studies |
| topics[2].id | https://openalex.org/T11312 |
| topics[2].field.id | https://openalex.org/fields/23 |
| topics[2].field.display_name | Environmental Science |
| topics[2].score | 0.9086999893188477 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/2305 |
| topics[2].subfield.display_name | Environmental Engineering |
| topics[2].display_name | Soil Moisture and Remote Sensing |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C12713177 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7886307239532471 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q1900281 |
| concepts[0].display_name | Perspective (graphical) |
| concepts[1].id | https://openalex.org/C19118579 |
| concepts[1].level | 2 |
| concepts[1].score | 0.587531566619873 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q786423 |
| concepts[1].display_name | Frequency domain |
| concepts[2].id | https://openalex.org/C60718061 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5798393487930298 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q1414747 |
| concepts[2].display_name | Momentum (technical analysis) |
| concepts[3].id | https://openalex.org/C36503486 |
| concepts[3].level | 2 |
| concepts[3].score | 0.4391346871852875 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11235244 |
| concepts[3].display_name | Domain (mathematical analysis) |
| concepts[4].id | https://openalex.org/C121332964 |
| concepts[4].level | 0 |
| concepts[4].score | 0.37395018339157104 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[4].display_name | Physics |
| concepts[5].id | https://openalex.org/C41008148 |
| concepts[5].level | 0 |
| concepts[5].score | 0.34848207235336304 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[5].display_name | Computer science |
| concepts[6].id | https://openalex.org/C33923547 |
| concepts[6].level | 0 |
| concepts[6].score | 0.29423972964286804 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[6].display_name | Mathematics |
| concepts[7].id | https://openalex.org/C162324750 |
| concepts[7].level | 0 |
| concepts[7].score | 0.24739018082618713 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q8134 |
| concepts[7].display_name | Economics |
| concepts[8].id | https://openalex.org/C134306372 |
| concepts[8].level | 1 |
| concepts[8].score | 0.18825870752334595 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q7754 |
| concepts[8].display_name | Mathematical analysis |
| concepts[9].id | https://openalex.org/C106159729 |
| concepts[9].level | 1 |
| concepts[9].score | 0.12346208095550537 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q2294553 |
| concepts[9].display_name | Financial economics |
| concepts[10].id | https://openalex.org/C154945302 |
| concepts[10].level | 1 |
| concepts[10].score | 0.12305048108100891 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[10].display_name | Artificial intelligence |
| keywords[0].id | https://openalex.org/keywords/perspective |
| keywords[0].score | 0.7886307239532471 |
| keywords[0].display_name | Perspective (graphical) |
| keywords[1].id | https://openalex.org/keywords/frequency-domain |
| keywords[1].score | 0.587531566619873 |
| keywords[1].display_name | Frequency domain |
| keywords[2].id | https://openalex.org/keywords/momentum |
| keywords[2].score | 0.5798393487930298 |
| keywords[2].display_name | Momentum (technical analysis) |
| keywords[3].id | https://openalex.org/keywords/domain |
| keywords[3].score | 0.4391346871852875 |
| keywords[3].display_name | Domain (mathematical analysis) |
| keywords[4].id | https://openalex.org/keywords/physics |
| keywords[4].score | 0.37395018339157104 |
| keywords[4].display_name | Physics |
| keywords[5].id | https://openalex.org/keywords/computer-science |
| keywords[5].score | 0.34848207235336304 |
| keywords[5].display_name | Computer science |
| keywords[6].id | https://openalex.org/keywords/mathematics |
| keywords[6].score | 0.29423972964286804 |
| keywords[6].display_name | Mathematics |
| keywords[7].id | https://openalex.org/keywords/economics |
| keywords[7].score | 0.24739018082618713 |
| keywords[7].display_name | Economics |
| keywords[8].id | https://openalex.org/keywords/mathematical-analysis |
| keywords[8].score | 0.18825870752334595 |
| keywords[8].display_name | Mathematical analysis |
| keywords[9].id | https://openalex.org/keywords/financial-economics |
| keywords[9].score | 0.12346208095550537 |
| keywords[9].display_name | Financial economics |
| keywords[10].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[10].score | 0.12305048108100891 |
| keywords[10].display_name | Artificial intelligence |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2411.19671 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2411.19671 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2411.19671 |
| locations[1].id | doi:10.48550/arxiv.2411.19671 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2411.19671 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101803969 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-6821-146X |
| authorships[0].author.display_name | Xianliang Li |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Li, Xianliang |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5100749903 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-5122-6806 |
| authorships[1].author.display_name | Jun Luo |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Luo, Jun |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5107959210 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Zhiwei Zheng |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Zheng, Zhiwei |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5101454802 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-2439-1280 |
| authorships[3].author.display_name | Hanxiao Wang |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Wang, Hanxiao |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5044203085 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-9920-7794 |
| authorships[4].author.display_name | Luo Li |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Luo, Li |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5101413488 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-0197-3432 |
| authorships[5].author.display_name | Li Wen |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Wen, Lingkun |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5037131275 |
| authorships[6].author.orcid | https://orcid.org/0000-0003-4521-2026 |
| authorships[6].author.display_name | Linlong Wu |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Wu, Linlong |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5012441448 |
| authorships[7].author.orcid | https://orcid.org/0000-0002-0584-3096 |
| authorships[7].author.display_name | Sheng Xu |
| authorships[7].author_position | last |
| authorships[7].raw_author_name | Xu, Sheng |
| authorships[7].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2411.19671 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | On the Performance Analysis of Momentum Method: A Frequency Domain Perspective |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11234 |
| primary_topic.field.id | https://openalex.org/fields/19 |
| primary_topic.field.display_name | Earth and Planetary Sciences |
| primary_topic.score | 0.92330002784729 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1902 |
| primary_topic.subfield.display_name | Atmospheric Science |
| primary_topic.display_name | Precipitation Measurement and Analysis |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2935759653, https://openalex.org/W3105167352, https://openalex.org/W54078636, https://openalex.org/W2954470139, https://openalex.org/W1501425562, https://openalex.org/W2902782467, https://openalex.org/W3084825885, https://openalex.org/W2298861036 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2411.19671 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2411.19671 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2411.19671 |
| primary_location.id | pmh:oai:arXiv.org:2411.19671 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2411.19671 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2411.19671 |
| publication_date | 2024-11-29 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 21, 38, 49, 70, 128 |
| abstract_inverted_index.In | 33 |
| abstract_inverted_index.an | 139 |
| abstract_inverted_index.as | 48 |
| abstract_inverted_index.in | 29, 90, 100 |
| abstract_inverted_index.of | 13, 24, 27, 73, 94, 150 |
| abstract_inverted_index.on | 116 |
| abstract_inverted_index.to | 56 |
| abstract_inverted_index.we | 36, 119 |
| abstract_inverted_index.Our | 63 |
| abstract_inverted_index.and | 68, 104 |
| abstract_inverted_index.are | 2, 88 |
| abstract_inverted_index.for | 5, 52 |
| abstract_inverted_index.our | 78 |
| abstract_inverted_index.the | 10, 25, 45, 60, 74, 81, 91, 97, 101, 134, 148 |
| abstract_inverted_index.This | 18 |
| abstract_inverted_index.both | 112 |
| abstract_inverted_index.late | 92 |
| abstract_inverted_index.over | 152 |
| abstract_inverted_index.role | 26 |
| abstract_inverted_index.that | 43, 131 |
| abstract_inverted_index.this | 34, 66 |
| abstract_inverted_index.with | 125, 138 |
| abstract_inverted_index.Based | 115 |
| abstract_inverted_index.FSGDM | 151 |
| abstract_inverted_index.clear | 22 |
| abstract_inverted_index.early | 102 |
| abstract_inverted_index.these | 117 |
| abstract_inverted_index.where | 54 |
| abstract_inverted_index.deeper | 71 |
| abstract_inverted_index.domain | 40 |
| abstract_inverted_index.during | 110 |
| abstract_inverted_index.filter | 51, 61 |
| abstract_inverted_index.method | 47 |
| abstract_inverted_index.modify | 59 |
| abstract_inverted_index.neural | 7 |
| abstract_inverted_index.paper, | 35 |
| abstract_inverted_index.stages | 93 |
| abstract_inverted_index.widely | 3 |
| abstract_inverted_index.Descent | 124 |
| abstract_inverted_index.adjusts | 133 |
| abstract_inverted_index.adopted | 4 |
| abstract_inverted_index.dynamic | 142 |
| abstract_inverted_index.enhance | 113 |
| abstract_inverted_index.impedes | 20 |
| abstract_inverted_index.optimal | 11 |
| abstract_inverted_index.present | 37 |
| abstract_inverted_index.propose | 120 |
| abstract_inverted_index.provide | 69 |
| abstract_inverted_index.remains | 16 |
| abstract_inverted_index.results | 146 |
| abstract_inverted_index.reveals | 80 |
| abstract_inverted_index.stages, | 103 |
| abstract_inverted_index.support | 65 |
| abstract_inverted_index.(FSGDM), | 127 |
| abstract_inverted_index.Gradient | 123 |
| abstract_inverted_index.However, | 9 |
| abstract_inverted_index.Momentum | 126 |
| abstract_inverted_index.analysis | 41, 79 |
| abstract_inverted_index.elusive. | 17 |
| abstract_inverted_index.gradient | 31, 86, 99, 108 |
| abstract_inverted_index.methods. | 32 |
| abstract_inverted_index.momentum | 14, 28, 46, 57, 135, 154 |
| abstract_inverted_index.original | 98 |
| abstract_inverted_index.training | 6, 111 |
| abstract_inverted_index.Frequency | 121 |
| abstract_inverted_index.Moreover, | 77 |
| abstract_inverted_index.effective | 141 |
| abstract_inverted_index.filtering | 136 |
| abstract_inverted_index.findings: | 84 |
| abstract_inverted_index.following | 82 |
| abstract_inverted_index.framework | 42 |
| abstract_inverted_index.frequency | 39 |
| abstract_inverted_index.gradually | 105 |
| abstract_inverted_index.heuristic | 129 |
| abstract_inverted_index.insights, | 118 |
| abstract_inverted_index.involved. | 76 |
| abstract_inverted_index.magnitude | 143 |
| abstract_inverted_index.mechanism | 75 |
| abstract_inverted_index.networks. | 8 |
| abstract_inverted_index.optimizer | 130 |
| abstract_inverted_index.response. | 144 |
| abstract_inverted_index.selection | 12 |
| abstract_inverted_index.training; | 95 |
| abstract_inverted_index.undesired | 89 |
| abstract_inverted_index.Stochastic | 122 |
| abstract_inverted_index.amplifying | 106 |
| abstract_inverted_index.components | 87, 109 |
| abstract_inverted_index.gradients, | 53 |
| abstract_inverted_index.interprets | 44 |
| abstract_inverted_index.optimizers | 1 |
| abstract_inverted_index.preserving | 96 |
| abstract_inverted_index.stochastic | 30 |
| abstract_inverted_index.adjustments | 55 |
| abstract_inverted_index.demonstrate | 147 |
| abstract_inverted_index.dynamically | 132 |
| abstract_inverted_index.empirically | 140 |
| abstract_inverted_index.experiments | 64 |
| abstract_inverted_index.optimizers. | 155 |
| abstract_inverted_index.perspective | 67 |
| abstract_inverted_index.significant | 83 |
| abstract_inverted_index.superiority | 149 |
| abstract_inverted_index.uncertainty | 19 |
| abstract_inverted_index.Experimental | 145 |
| abstract_inverted_index.coefficients | 15, 58 |
| abstract_inverted_index.conventional | 153 |
| abstract_inverted_index.performance. | 114 |
| abstract_inverted_index.time-variant | 50 |
| abstract_inverted_index.low-frequency | 107 |
| abstract_inverted_index.understanding | 23, 72 |
| abstract_inverted_index.Momentum-based | 0 |
| abstract_inverted_index.characteristic | 137 |
| abstract_inverted_index.high-frequency | 85 |
| abstract_inverted_index.characteristics. | 62 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 8 |
| citation_normalized_percentile |