Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help? Article Swipe

MAKE IMPACT

Yunyang Xiong , Ronak Mehta , Vikas Singh · YOU? · · 2019 · Open Access · · DOI: https://doi.org/10.48550/arxiv.1904.03786

The design of neural network architectures is frequently either based on human expertise using trial/error and empirical feedback or tackled via large scale reinforcement learning strategies performed over distinct discrete architecture choices. In the latter case, the optimization is often non-differentiable and also not very amenable to derivative-free optimization methods. Most methods in use today require sizable computational resources. And if we want networks that additionally satisfy resource constraints, the above challenges are exacerbated because the search must now balance accuracy with certain budget constraints on resources. We formulate this problem as the optimization of a set function -- we find that the empirical behavior of this set function often (but not always) satisfies marginal gain and monotonicity principles -- properties central to the idea of submodularity. Based on this observation, we adapt algorithms within discrete optimization to obtain heuristic schemes for neural network architecture search, where we have resource constraints on the architecture. This simple scheme when applied on CIFAR-100 and ImageNet, identifies resource-constrained architectures with quantifiably better performance than current state-of-the-art models designed for mobile devices. Specifically, we find high-performing architectures with fewer parameters and computations by a search method that is much faster.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/1904.03786

PDF

https://arxiv.org/pdf/1904.03786

OA Status

green

Cited By

4

References

41

Related Works

10

OpenAlex ID

https://openalex.org/W2971491060

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W2971491060

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.1904.03786

Digital Object Identifier

Title

Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2019

Year of publication

Publication date

2019-04-08

Full publication date if available

Authors

Yunyang Xiong, Ronak Mehta, Vikas Singh

List of authors in order

Landing page

https://arxiv.org/abs/1904.03786

Publisher landing page

PDF URL

https://arxiv.org/pdf/1904.03786

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/1904.03786

Direct OA link when available

Concepts

Computer science, Mathematical optimization, Heuristic, Set (abstract data type), Differentiable function, Architecture, Resource allocation, Computation, Artificial neural network, Network architecture, Resource (disambiguation), Distributed computing, Artificial intelligence, Algorithm, Mathematics, Art, Computer network, Computer security, Visual arts, Mathematical analysis, Programming language

Top concepts (fields/topics) attached by OpenAlex

Cited by

4

Total citation count in OpenAlex

Citations by year (recent)

2021: 1, 2020: 3

Per-year citation counts (last 5 years)

References (count)

41

Number of works referenced by this work

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload