Itay Safran
YOU?
Author Swipe
View article: No Prior, No Leakage: Revisiting Reconstruction Attacks in Trained Neural Networks
No Prior, No Leakage: Revisiting Reconstruction Attacks in Trained Neural Networks Open
The memorization of training data by neural networks raises pressing concerns for privacy and security. Recent work has shown that, under certain conditions, portions of the training set can be reconstructed directly from model parameters.…
View article: Provable Privacy Attacks on Trained Shallow Neural Networks
Provable Privacy Attacks on Trained Shallow Neural Networks Open
We study what provable privacy attacks can be shown on trained, 2-layer ReLU neural networks. We explore two types of attacks; data reconstruction attacks, and membership inference attacks. We prove that theoretical results on the implicit…
View article: Depth Separations in Neural Networks: Separating the Dimension from the Accuracy
Depth Separations in Neural Networks: Separating the Dimension from the Accuracy Open
We prove an exponential size separation between depth 2 and depth 3 neural networks (with real inputs), when approximating a $\mathcal{O}(1)$-Lipschitz target function to constant accuracy, with respect to a distribution with support in th…
View article: How Many Neurons Does it Take to Approximate the Maximum?
How Many Neurons Does it Take to Approximate the Maximum? Open
We study the size of a neural network needed to approximate the maximum function over $d$ inputs, in the most basic setting of approximating with respect to the $L_2$ norm, for continuous distributions, for a network that uses ReLU activat…
View article: On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias
On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias Open
We study the dynamics and implicit bias of gradient flow (GF) on univariate ReLU neural networks with a single hidden layer in a binary classification setting. We show that when the labels are determined by the sign of a target network wit…
View article: Optimization-Based Separations for Neural Networks
Optimization-Based Separations for Neural Networks Open
Depth separation results propose a possible theoretical explanation for the benefits of deep neural networks over shallower architectures, establishing that the former possess superior approximation capabilities. However, there are no know…
View article: Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned Problems
Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned Problems Open
Recently, there has been much interest in studying the convergence rates of without-replacement SGD, and proving that it is faster than with-replacement SGD in the worst case. However, known lower bounds ignore the problem's geometry, incl…
View article: Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned\n Problems
Random Shuffling Beats SGD Only After Many Epochs on Ill-Conditioned\n Problems Open
Recently, there has been much interest in studying the convergence rates of\nwithout-replacement SGD, and proving that it is faster than with-replacement\nSGD in the worst case. However, known lower bounds ignore the problem's\ngeometry, i…
View article: The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks
The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks Open
We study the effects of mild over-parameterization on the optimization landscape of a simple ReLU neural network of the form $\mathbf{x}\mapsto\sum_{i=1}^k\max\{0,\mathbf{w}_i^{\top}\mathbf{x}\}$, in a well-studied teacher-student setting …
View article: How Good is SGD with Random Shuffling?
How Good is SGD with Random Shuffling? Open
We study the performance of stochastic gradient descent (SGD) on smooth and strongly-convex finite-sum optimization problems. In contrast to the majority of existing theoretical works, which assume that individual functions are sampled wit…
View article: A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance
A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance Open
The existence of adversarial examples in which an imperceptible change in the input can fool well trained neural networks was experimentally discovered by Szegedy et al in 2013, who called them "Intriguing properties of neural networks". S…
View article: Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
Spurious Local Minima are Common in Two-Layer ReLU Neural Networks Open
We consider the optimization problem associated with training simple ReLU neural networks of the form $\mathbf{x}\mapsto \sum_{i=1}^{k}\max\{0,\mathbf{w}_i^\top \mathbf{x}\}$ with respect to the squared loss. We provide a computer-assisted…
View article: Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks Open
We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower …
View article: Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks Open
We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower …