Convergence guarantees for RMSProp and ADAM in non-convex optimization and their comparison to Nesterov acceleration on autoencoders. Article Swipe
RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical foundations have remained unclear. In this work we make progress towards that by giving proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives and we give bounds on the running time. We then design experiments to compare the performances of RMSProp and ADAM against Nesterov Accelerated Gradient method on a variety of autoencoder setups. Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter $\beta_1$. We show that in terms of getting lower training and test losses, at very high values of the momentum parameter ($\beta_1 = 0.99$) (and large enough nets if using mini-batches) ADAM outperforms NAG at any momentum value tried for the latter. On the other hand, NAG can sometimes do better when ADAM's $\beta_1$ is set to the most commonly used value: $\beta_1 = 0.9$. We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms and finding weights which increase the minimum eigenvalue of the Hessian of the loss function.
Related Topics
Concepts
Acceleration
Convergence (economics)
Momentum (technical analysis)
Computer science
Mathematical optimization
Autoencoder
Mathematics
Eigenvalues and eigenvectors
Algorithm
Artificial intelligence
Applied mathematics
Artificial neural network
Physics
Economics
Finance
Economic growth
Quantum mechanics
Classical mechanics
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://arxiv.org/pdf/1807.06766.pdf
- OA Status
- green
- Cited By
- 32
- References
- 16
- Related Works
- 20
- OpenAlex ID
- https://openalex.org/W2887729414
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W2887729414Canonical identifier for this work in OpenAlex
- Title
-
Convergence guarantees for RMSProp and ADAM in non-convex optimization and their comparison to Nesterov acceleration on autoencoders.Work title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2018Year of publication
- Publication date
-
2018-07-18Full publication date if available
- Authors
-
Amitabh Basu, Soham De, Anirbit Mukherjee, Enayat UllahList of authors in order
- Landing page
-
https://arxiv.org/pdf/1807.06766.pdfPublisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/1807.06766.pdfDirect OA link when available
- Concepts
-
Acceleration, Convergence (economics), Momentum (technical analysis), Computer science, Mathematical optimization, Autoencoder, Mathematics, Eigenvalues and eigenvectors, Algorithm, Artificial intelligence, Applied mathematics, Artificial neural network, Physics, Economics, Finance, Economic growth, Quantum mechanics, Classical mechanicsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
32Total citation count in OpenAlex
- Citations by year (recent)
-
2023: 2, 2022: 2, 2021: 7, 2020: 5, 2019: 11Per-year citation counts (last 5 years)
- References (count)
-
16Number of works referenced by this work
- Related works (count)
-
20Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W2887729414 |
|---|---|
| doi | |
| ids.mag | 2887729414 |
| ids.openalex | https://openalex.org/W2887729414 |
| fwci | 5.55974885 |
| type | article |
| title | Convergence guarantees for RMSProp and ADAM in non-convex optimization and their comparison to Nesterov acceleration on autoencoders. |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11612 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 1.0 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Stochastic Gradient Optimization Techniques |
| topics[1].id | https://openalex.org/T12676 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9987999796867371 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Machine Learning and ELM |
| topics[2].id | https://openalex.org/T10500 |
| topics[2].field.id | https://openalex.org/fields/22 |
| topics[2].field.display_name | Engineering |
| topics[2].score | 0.9987000226974487 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/2206 |
| topics[2].subfield.display_name | Computational Mechanics |
| topics[2].display_name | Sparse and Compressive Sensing Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C117896860 |
| concepts[0].level | 2 |
| concepts[0].score | 0.5842130184173584 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q11376 |
| concepts[0].display_name | Acceleration |
| concepts[1].id | https://openalex.org/C2777303404 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5240521430969238 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q759757 |
| concepts[1].display_name | Convergence (economics) |
| concepts[2].id | https://openalex.org/C60718061 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5230984687805176 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q1414747 |
| concepts[2].display_name | Momentum (technical analysis) |
| concepts[3].id | https://openalex.org/C41008148 |
| concepts[3].level | 0 |
| concepts[3].score | 0.4799661636352539 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[3].display_name | Computer science |
| concepts[4].id | https://openalex.org/C126255220 |
| concepts[4].level | 1 |
| concepts[4].score | 0.45998257398605347 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q141495 |
| concepts[4].display_name | Mathematical optimization |
| concepts[5].id | https://openalex.org/C101738243 |
| concepts[5].level | 3 |
| concepts[5].score | 0.4554431736469269 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q786435 |
| concepts[5].display_name | Autoencoder |
| concepts[6].id | https://openalex.org/C33923547 |
| concepts[6].level | 0 |
| concepts[6].score | 0.45382627844810486 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[6].display_name | Mathematics |
| concepts[7].id | https://openalex.org/C158693339 |
| concepts[7].level | 2 |
| concepts[7].score | 0.4194740056991577 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q190524 |
| concepts[7].display_name | Eigenvalues and eigenvectors |
| concepts[8].id | https://openalex.org/C11413529 |
| concepts[8].level | 1 |
| concepts[8].score | 0.40997540950775146 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q8366 |
| concepts[8].display_name | Algorithm |
| concepts[9].id | https://openalex.org/C154945302 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3983847498893738 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[9].display_name | Artificial intelligence |
| concepts[10].id | https://openalex.org/C28826006 |
| concepts[10].level | 1 |
| concepts[10].score | 0.32697391510009766 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q33521 |
| concepts[10].display_name | Applied mathematics |
| concepts[11].id | https://openalex.org/C50644808 |
| concepts[11].level | 2 |
| concepts[11].score | 0.3167596459388733 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q192776 |
| concepts[11].display_name | Artificial neural network |
| concepts[12].id | https://openalex.org/C121332964 |
| concepts[12].level | 0 |
| concepts[12].score | 0.10532376170158386 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[12].display_name | Physics |
| concepts[13].id | https://openalex.org/C162324750 |
| concepts[13].level | 0 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q8134 |
| concepts[13].display_name | Economics |
| concepts[14].id | https://openalex.org/C10138342 |
| concepts[14].level | 1 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q43015 |
| concepts[14].display_name | Finance |
| concepts[15].id | https://openalex.org/C50522688 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q189833 |
| concepts[15].display_name | Economic growth |
| concepts[16].id | https://openalex.org/C62520636 |
| concepts[16].level | 1 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q944 |
| concepts[16].display_name | Quantum mechanics |
| concepts[17].id | https://openalex.org/C74650414 |
| concepts[17].level | 1 |
| concepts[17].score | 0.0 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q11397 |
| concepts[17].display_name | Classical mechanics |
| keywords[0].id | https://openalex.org/keywords/acceleration |
| keywords[0].score | 0.5842130184173584 |
| keywords[0].display_name | Acceleration |
| keywords[1].id | https://openalex.org/keywords/convergence |
| keywords[1].score | 0.5240521430969238 |
| keywords[1].display_name | Convergence (economics) |
| keywords[2].id | https://openalex.org/keywords/momentum |
| keywords[2].score | 0.5230984687805176 |
| keywords[2].display_name | Momentum (technical analysis) |
| keywords[3].id | https://openalex.org/keywords/computer-science |
| keywords[3].score | 0.4799661636352539 |
| keywords[3].display_name | Computer science |
| keywords[4].id | https://openalex.org/keywords/mathematical-optimization |
| keywords[4].score | 0.45998257398605347 |
| keywords[4].display_name | Mathematical optimization |
| keywords[5].id | https://openalex.org/keywords/autoencoder |
| keywords[5].score | 0.4554431736469269 |
| keywords[5].display_name | Autoencoder |
| keywords[6].id | https://openalex.org/keywords/mathematics |
| keywords[6].score | 0.45382627844810486 |
| keywords[6].display_name | Mathematics |
| keywords[7].id | https://openalex.org/keywords/eigenvalues-and-eigenvectors |
| keywords[7].score | 0.4194740056991577 |
| keywords[7].display_name | Eigenvalues and eigenvectors |
| keywords[8].id | https://openalex.org/keywords/algorithm |
| keywords[8].score | 0.40997540950775146 |
| keywords[8].display_name | Algorithm |
| keywords[9].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[9].score | 0.3983847498893738 |
| keywords[9].display_name | Artificial intelligence |
| keywords[10].id | https://openalex.org/keywords/applied-mathematics |
| keywords[10].score | 0.32697391510009766 |
| keywords[10].display_name | Applied mathematics |
| keywords[11].id | https://openalex.org/keywords/artificial-neural-network |
| keywords[11].score | 0.3167596459388733 |
| keywords[11].display_name | Artificial neural network |
| keywords[12].id | https://openalex.org/keywords/physics |
| keywords[12].score | 0.10532376170158386 |
| keywords[12].display_name | Physics |
| language | en |
| locations[0].id | mag:2887729414 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | arXiv (Cornell University) |
| locations[0].landing_page_url | https://arxiv.org/pdf/1807.06766.pdf |
| authorships[0].author.id | https://openalex.org/A5048617506 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-1070-2626 |
| authorships[0].author.display_name | Amitabh Basu |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Amitabh Basu |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5036322364 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-7907-1335 |
| authorships[1].author.display_name | Soham De |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Soham De |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5084835559 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-5189-8939 |
| authorships[2].author.display_name | Anirbit Mukherjee |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Anirbit Mukherjee |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5024577194 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-1395-4255 |
| authorships[3].author.display_name | Enayat Ullah |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Enayat Ullah |
| authorships[3].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/1807.06766.pdf |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Convergence guarantees for RMSProp and ADAM in non-convex optimization and their comparison to Nesterov acceleration on autoencoders. |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-10-10T17:16:08.811792 |
| primary_topic.id | https://openalex.org/T11612 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 1.0 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Stochastic Gradient Optimization Techniques |
| related_works | https://openalex.org/W2146502635, https://openalex.org/W2785523195, https://openalex.org/W2964121744, https://openalex.org/W2894692220, https://openalex.org/W2886463271, https://openalex.org/W6908809, https://openalex.org/W2903382683, https://openalex.org/W2963613486, https://openalex.org/W2963563140, https://openalex.org/W2891952073, https://openalex.org/W2805767638, https://openalex.org/W2194775991, https://openalex.org/W3118608800, https://openalex.org/W2112796928, https://openalex.org/W1994616650, https://openalex.org/W2963698657, https://openalex.org/W2808042107, https://openalex.org/W3110015897, https://openalex.org/W2893812806, https://openalex.org/W1988720110 |
| cited_by_count | 32 |
| counts_by_year[0].year | 2023 |
| counts_by_year[0].cited_by_count | 2 |
| counts_by_year[1].year | 2022 |
| counts_by_year[1].cited_by_count | 2 |
| counts_by_year[2].year | 2021 |
| counts_by_year[2].cited_by_count | 7 |
| counts_by_year[3].year | 2020 |
| counts_by_year[3].cited_by_count | 5 |
| counts_by_year[4].year | 2019 |
| counts_by_year[4].cited_by_count | 11 |
| counts_by_year[5].year | 2018 |
| counts_by_year[5].cited_by_count | 5 |
| locations_count | 1 |
| best_oa_location.id | mag:2887729414 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | arXiv (Cornell University) |
| best_oa_location.landing_page_url | https://arxiv.org/pdf/1807.06766.pdf |
| primary_location.id | mag:2887729414 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | arXiv (Cornell University) |
| primary_location.landing_page_url | https://arxiv.org/pdf/1807.06766.pdf |
| publication_date | 2018-07-18 |
| publication_year | 2018 |
| referenced_works | https://openalex.org/W152046758, https://openalex.org/W2181347294, https://openalex.org/W104184427, https://openalex.org/W2964121744, https://openalex.org/W2950726992, https://openalex.org/W2173520492, https://openalex.org/W2520472982, https://openalex.org/W2962795635, https://openalex.org/W2146502635, https://openalex.org/W2963748792, https://openalex.org/W2785523195, https://openalex.org/W2622799641, https://openalex.org/W2023695846, https://openalex.org/W2619516334, https://openalex.org/W2013918710, https://openalex.org/W1506690472 |
| referenced_works_count | 16 |
| abstract_inverted_index.= | 104, 142 |
| abstract_inverted_index.a | 66 |
| abstract_inverted_index.In | 19 |
| abstract_inverted_index.On | 123 |
| abstract_inverted_index.We | 49, 85, 144 |
| abstract_inverted_index.at | 96, 115 |
| abstract_inverted_index.be | 5 |
| abstract_inverted_index.by | 26 |
| abstract_inverted_index.do | 129 |
| abstract_inverted_index.if | 110 |
| abstract_inverted_index.in | 88, 157 |
| abstract_inverted_index.is | 134 |
| abstract_inverted_index.of | 57, 68, 100, 159, 173 |
| abstract_inverted_index.on | 45, 65, 148 |
| abstract_inverted_index.to | 4, 35, 53, 80, 136, 150 |
| abstract_inverted_index.we | 21, 42, 73 |
| abstract_inverted_index.NAG | 114, 153 |
| abstract_inverted_index.and | 1, 41, 59, 93, 163 |
| abstract_inverted_index.any | 116 |
| abstract_inverted_index.are | 33 |
| abstract_inverted_index.but | 12 |
| abstract_inverted_index.can | 127 |
| abstract_inverted_index.for | 9, 38, 120 |
| abstract_inverted_index.has | 79, 154 |
| abstract_inverted_index.its | 81 |
| abstract_inverted_index.set | 135 |
| abstract_inverted_index.the | 46, 55, 75, 101, 121, 124, 137, 168, 174 |
| abstract_inverted_index.(and | 106 |
| abstract_inverted_index.ADAM | 2, 78 |
| abstract_inverted_index.also | 145 |
| abstract_inverted_index.give | 43 |
| abstract_inverted_index.have | 16 |
| abstract_inverted_index.high | 98 |
| abstract_inverted_index.loss | 175 |
| abstract_inverted_index.make | 22 |
| abstract_inverted_index.most | 138 |
| abstract_inverted_index.nets | 11, 109 |
| abstract_inverted_index.show | 86 |
| abstract_inverted_index.test | 94 |
| abstract_inverted_index.that | 25, 29, 87, 152 |
| abstract_inverted_index.then | 50 |
| abstract_inverted_index.very | 97 |
| abstract_inverted_index.when | 131 |
| abstract_inverted_index.0.9$. | 143 |
| abstract_inverted_index.large | 107 |
| abstract_inverted_index.lower | 91 |
| abstract_inverted_index.norms | 162 |
| abstract_inverted_index.other | 125 |
| abstract_inverted_index.reach | 36 |
| abstract_inverted_index.terms | 89, 158 |
| abstract_inverted_index.their | 13 |
| abstract_inverted_index.these | 30, 71 |
| abstract_inverted_index.time. | 48 |
| abstract_inverted_index.tried | 119 |
| abstract_inverted_index.using | 111 |
| abstract_inverted_index.value | 118 |
| abstract_inverted_index.which | 166 |
| abstract_inverted_index.0.99$) | 105 |
| abstract_inverted_index.ADAM's | 132 |
| abstract_inverted_index.better | 130, 155 |
| abstract_inverted_index.bounds | 44 |
| abstract_inverted_index.design | 51 |
| abstract_inverted_index.enough | 108 |
| abstract_inverted_index.giving | 27 |
| abstract_inverted_index.method | 64 |
| abstract_inverted_index.of the | 171 |
| abstract_inverted_index.proofs | 28 |
| abstract_inverted_index.report | 146 |
| abstract_inverted_index.smooth | 39 |
| abstract_inverted_index.value: | 140 |
| abstract_inverted_index.values | 99 |
| abstract_inverted_index.Hessian | 172 |
| abstract_inverted_index.RMSProp | 0, 58 |
| abstract_inverted_index.Through | 70 |
| abstract_inverted_index.compare | 54 |
| abstract_inverted_index.finding | 164 |
| abstract_inverted_index.latter. | 122 |
| abstract_inverted_index.losses, | 95 |
| abstract_inverted_index.minimum | 169 |
| abstract_inverted_index.popular | 7 |
| abstract_inverted_index.running | 47 |
| abstract_inverted_index.towards | 24 |
| abstract_inverted_index.variety | 67 |
| abstract_inverted_index.weights | 165 |
| abstract_inverted_index.$\beta_1 | 141 |
| abstract_inverted_index.Gradient | 63 |
| abstract_inverted_index.Nesterov | 61 |
| abstract_inverted_index.continue | 3 |
| abstract_inverted_index.gradient | 161 |
| abstract_inverted_index.increase | 167 |
| abstract_inverted_index.momentum | 82, 117 |
| abstract_inverted_index.progress | 23 |
| abstract_inverted_index.remained | 17 |
| abstract_inverted_index.training | 92 |
| abstract_inverted_index.unclear. | 18 |
| abstract_inverted_index.$\beta_1$ | 133 |
| abstract_inverted_index.($\beta_1 | 103 |
| abstract_inverted_index.abilities | 156 |
| abstract_inverted_index.extremely | 6 |
| abstract_inverted_index.function. | 176 |
| abstract_inverted_index.hand, NAG | 126 |
| abstract_inverted_index.parameter | 83 |
| abstract_inverted_index.sometimes | 128 |
| abstract_inverted_index.this work | 20 |
| abstract_inverted_index.$\beta_1$. | 84 |
| abstract_inverted_index.algorithms | 8, 32 |
| abstract_inverted_index.eigenvalue | 170 |
| abstract_inverted_index.guaranteed | 34 |
| abstract_inverted_index.of getting | 90 |
| abstract_inverted_index.Accelerated | 62 |
| abstract_inverted_index.criticality | 37 |
| abstract_inverted_index.demonstrate | 74, 151 |
| abstract_inverted_index.experiments | 52, 72, 147 |
| abstract_inverted_index.foundations | 15 |
| abstract_inverted_index.interesting | 76 |
| abstract_inverted_index.theoretical | 14 |
| abstract_inverted_index.ADAM against | 60 |
| abstract_inverted_index.performances | 56 |
| abstract_inverted_index.reducing the | 160 |
| abstract_inverted_index.commonly used | 139 |
| abstract_inverted_index.mini-batches) | 112 |
| abstract_inverted_index.training neural | 10 |
| abstract_inverted_index.ADAM outperforms | 113 |
| abstract_inverted_index.sensitivity that | 77 |
| abstract_inverted_index.adaptive gradient | 31 |
| abstract_inverted_index.momentum parameter | 102 |
| abstract_inverted_index.autoencoder setups. | 69 |
| abstract_inverted_index.non-convex objectives | 40 |
| abstract_inverted_index.different autoencoders | 149 |
| cited_by_percentile_year.max | 99 |
| cited_by_percentile_year.min | 94 |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile.value | 0.95756157 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | True |