Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation Article Swipe

PDF

Felix Petersen , Christian Borgelt , Aashwin Mishra , Stefano Ermon ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2410.08125

We deal with the problem of gradient estimation for stochastic differentiable relaxations of algorithms, operators, simulators, and other non-differentiable functions. Stochastic smoothing conventionally perturbs the input of a non-differentiable function with a differentiable density distribution with full support, smoothing it and enabling gradient estimation. Our theory starts at first principles to derive stochastic smoothing with reduced assumptions, without requiring a differentiable density nor full support, and we present a general framework for relaxation and gradient estimation of non-differentiable black-box functions $f:\mathbb{R}^n\to\mathbb{R}^m$. We develop variance reduction for gradient estimation from 3 orthogonal perspectives. Empirically, we benchmark 6 distributions and up to 24 variance reduction strategies for differentiable sorting and ranking, differentiable shortest-paths on graphs, differentiable rendering for pose estimation, as well as differentiable cryo-ET simulations.

Related Topics

Concepts

Smoothing Estimation Econometrics Mathematical economics Mathematics Computer science Economics Applied mathematics Statistics Management

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2410.08125
PDF: https://arxiv.org/pdf/2410.08125
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4403365351

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4403365351

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2410.08125

Digital Object Identifier
Title: Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-10-10

Full publication date if available
Authors: Felix Petersen, Christian Borgelt, Aashwin Mishra, Stefano Ermon

List of authors in order
Landing page: https://arxiv.org/abs/2410.08125

Publisher landing page
PDF URL: https://arxiv.org/pdf/2410.08125

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2410.08125

Direct OA link when available
Concepts: Smoothing, Estimation, Econometrics, Mathematical economics, Mathematics, Computer science, Economics, Applied mathematics, Statistics, Management

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4403365351
doi	https://doi.org/10.48550/arxiv.2410.08125
ids.doi	https://doi.org/10.48550/arxiv.2410.08125
ids.openalex	https://openalex.org/W4403365351
fwci
type	preprint
title	Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10719
topics[0].field.id	https://openalex.org/fields/22
topics[0].field.display_name	Engineering
topics[0].score	0.8695999979972839
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/2206
topics[0].subfield.display_name	Computational Mechanics
topics[0].display_name	3D Shape Modeling and Analysis
topics[1].id	https://openalex.org/T10531
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.8636999726295471
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1707
topics[1].subfield.display_name	Computer Vision and Pattern Recognition
topics[1].display_name	Advanced Vision and Imaging
topics[2].id	https://openalex.org/T10638
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.8148999810218811
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1707
topics[2].subfield.display_name	Computer Vision and Pattern Recognition
topics[2].display_name	Optical measurement and interference techniques
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C3770464
concepts[0].level	2
concepts[0].score	0.750918984413147
concepts[0].wikidata	https://www.wikidata.org/wiki/Q775963
concepts[0].display_name	Smoothing
concepts[1].id	https://openalex.org/C96250715
concepts[1].level	2
concepts[1].score	0.6569744944572449
concepts[1].wikidata	https://www.wikidata.org/wiki/Q965330
concepts[1].display_name	Estimation
concepts[2].id	https://openalex.org/C149782125
concepts[2].level	1
concepts[2].score	0.4100886881351471
concepts[2].wikidata	https://www.wikidata.org/wiki/Q160039
concepts[2].display_name	Econometrics
concepts[3].id	https://openalex.org/C144237770
concepts[3].level	1
concepts[3].score	0.3944970667362213
concepts[3].wikidata	https://www.wikidata.org/wiki/Q747534
concepts[3].display_name	Mathematical economics
concepts[4].id	https://openalex.org/C33923547
concepts[4].level	0
concepts[4].score	0.3793209195137024
concepts[4].wikidata	https://www.wikidata.org/wiki/Q395
concepts[4].display_name	Mathematics
concepts[5].id	https://openalex.org/C41008148
concepts[5].level	0
concepts[5].score	0.37454313039779663
concepts[5].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[5].display_name	Computer science
concepts[6].id	https://openalex.org/C162324750
concepts[6].level	0
concepts[6].score	0.34846973419189453
concepts[6].wikidata	https://www.wikidata.org/wiki/Q8134
concepts[6].display_name	Economics
concepts[7].id	https://openalex.org/C28826006
concepts[7].level	1
concepts[7].score	0.3427053689956665
concepts[7].wikidata	https://www.wikidata.org/wiki/Q33521
concepts[7].display_name	Applied mathematics
concepts[8].id	https://openalex.org/C105795698
concepts[8].level	1
concepts[8].score	0.2699636220932007
concepts[8].wikidata	https://www.wikidata.org/wiki/Q12483
concepts[8].display_name	Statistics
concepts[9].id	https://openalex.org/C187736073
concepts[9].level	1
concepts[9].score	0.04961976408958435
concepts[9].wikidata	https://www.wikidata.org/wiki/Q2920921
concepts[9].display_name	Management
keywords[0].id	https://openalex.org/keywords/smoothing
keywords[0].score	0.750918984413147
keywords[0].display_name	Smoothing
keywords[1].id	https://openalex.org/keywords/estimation
keywords[1].score	0.6569744944572449
keywords[1].display_name	Estimation
keywords[2].id	https://openalex.org/keywords/econometrics
keywords[2].score	0.4100886881351471
keywords[2].display_name	Econometrics
keywords[3].id	https://openalex.org/keywords/mathematical-economics
keywords[3].score	0.3944970667362213
keywords[3].display_name	Mathematical economics
keywords[4].id	https://openalex.org/keywords/mathematics
keywords[4].score	0.3793209195137024
keywords[4].display_name	Mathematics
keywords[5].id	https://openalex.org/keywords/computer-science
keywords[5].score	0.37454313039779663
keywords[5].display_name	Computer science
keywords[6].id	https://openalex.org/keywords/economics
keywords[6].score	0.34846973419189453
keywords[6].display_name	Economics
keywords[7].id	https://openalex.org/keywords/applied-mathematics
keywords[7].score	0.3427053689956665
keywords[7].display_name	Applied mathematics
keywords[8].id	https://openalex.org/keywords/statistics
keywords[8].score	0.2699636220932007
keywords[8].display_name	Statistics
keywords[9].id	https://openalex.org/keywords/management
keywords[9].score	0.04961976408958435
keywords[9].display_name	Management
language	en
locations[0].id	pmh:oai:arXiv.org:2410.08125
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2410.08125
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2410.08125
locations[1].id	doi:10.48550/arxiv.2410.08125
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2410.08125
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5045582046
authorships[0].author.orcid	https://orcid.org/0000-0002-4913-8532
authorships[0].author.display_name	Felix Petersen
authorships[0].author_position	first
authorships[0].raw_author_name	Petersen, Felix
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5109526047
authorships[1].author.orcid
authorships[1].author.display_name	Christian Borgelt
authorships[1].author_position	middle
authorships[1].raw_author_name	Borgelt, Christian
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5101639379
authorships[2].author.orcid	https://orcid.org/0000-0002-7257-6784
authorships[2].author.display_name	Aashwin Mishra
authorships[2].author_position	middle
authorships[2].raw_author_name	Mishra, Aashwin
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5091179481
authorships[3].author.orcid	https://orcid.org/0000-0003-0039-2887
authorships[3].author.display_name	Stefano Ermon
authorships[3].author_position	last
authorships[3].raw_author_name	Ermon, Stefano
authorships[3].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2410.08125
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2024-10-13T00:00:00
display_name	Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10719
primary_topic.field.id	https://openalex.org/fields/22
primary_topic.field.display_name	Engineering
primary_topic.score	0.8695999979972839
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/2206
primary_topic.subfield.display_name	Computational Mechanics
primary_topic.display_name	3D Shape Modeling and Analysis
related_works	https://openalex.org/W1978572805, https://openalex.org/W2383807498, https://openalex.org/W1997992934, https://openalex.org/W1987225439, https://openalex.org/W4238188170, https://openalex.org/W2125114371, https://openalex.org/W2019977573, https://openalex.org/W2149980199, https://openalex.org/W3125766170, https://openalex.org/W2040641410
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2410.08125
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2410.08125
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2410.08125
primary_location.id	pmh:oai:arXiv.org:2410.08125
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2410.08125
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2410.08125
publication_date	2024-10-10
publication_year	2024
referenced_works_count	0
abstract_inverted_index.3	89
abstract_inverted_index.6	95
abstract_inverted_index.a	27, 31, 59, 68
abstract_inverted_index.24	100
abstract_inverted_index.We	0, 81
abstract_inverted_index.as	118, 120
abstract_inverted_index.at	47
abstract_inverted_index.it	39
abstract_inverted_index.of	5, 12, 26, 76
abstract_inverted_index.on	111
abstract_inverted_index.to	50, 99
abstract_inverted_index.up	98
abstract_inverted_index.we	66, 93
abstract_inverted_index.Our	44
abstract_inverted_index.and	16, 40, 65, 73, 97, 107
abstract_inverted_index.for	8, 71, 85, 104, 115
abstract_inverted_index.nor	62
abstract_inverted_index.the	3, 24
abstract_inverted_index.deal	1
abstract_inverted_index.from	88
abstract_inverted_index.full	36, 63
abstract_inverted_index.pose	116
abstract_inverted_index.well	119
abstract_inverted_index.with	2, 30, 35, 54
abstract_inverted_index.first	48
abstract_inverted_index.input	25
abstract_inverted_index.other	17
abstract_inverted_index.derive	51
abstract_inverted_index.starts	46
abstract_inverted_index.theory	45
abstract_inverted_index.cryo-ET	122
abstract_inverted_index.density	33, 61
abstract_inverted_index.develop	82
abstract_inverted_index.general	69
abstract_inverted_index.graphs,	112
abstract_inverted_index.present	67
abstract_inverted_index.problem	4
abstract_inverted_index.reduced	55
abstract_inverted_index.sorting	106
abstract_inverted_index.without	57
abstract_inverted_index.enabling	41
abstract_inverted_index.function	29
abstract_inverted_index.gradient	6, 42, 74, 86
abstract_inverted_index.perturbs	23
abstract_inverted_index.ranking,	108
abstract_inverted_index.support,	37, 64
abstract_inverted_index.variance	83, 101
abstract_inverted_index.benchmark	94
abstract_inverted_index.black-box	78
abstract_inverted_index.framework	70
abstract_inverted_index.functions	79
abstract_inverted_index.reduction	84, 102
abstract_inverted_index.rendering	114
abstract_inverted_index.requiring	58
abstract_inverted_index.smoothing	21, 38, 53
abstract_inverted_index.Stochastic	20
abstract_inverted_index.estimation	7, 75, 87
abstract_inverted_index.functions.	19
abstract_inverted_index.operators,	14
abstract_inverted_index.orthogonal	90
abstract_inverted_index.principles	49
abstract_inverted_index.relaxation	72
abstract_inverted_index.stochastic	9, 52
abstract_inverted_index.strategies	103
abstract_inverted_index.algorithms,	13
abstract_inverted_index.estimation,	117
abstract_inverted_index.estimation.	43
abstract_inverted_index.relaxations	11
abstract_inverted_index.simulators,	15
abstract_inverted_index.Empirically,	92
abstract_inverted_index.assumptions,	56
abstract_inverted_index.distribution	34
abstract_inverted_index.simulations.	123
abstract_inverted_index.distributions	96
abstract_inverted_index.perspectives.	91
abstract_inverted_index.conventionally	22
abstract_inverted_index.differentiable	10, 32, 60, 105, 109, 113, 121
abstract_inverted_index.shortest-paths	110
abstract_inverted_index.non-differentiable	18, 28, 77
abstract_inverted_index.$f:\mathbb{R}^n\to\mathbb{R}^m$.	80
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	4
citation_normalized_percentile