Convolutional neural networks (CNNs) are emerging as powerful tools for\nvisual recognition. Recent architecture proposals for sparse CNNs exploit zeros\nin the feature maps and filters for performance and energy without losing\naccuracy. Sparse architectures that exploit two-sided sparsity in both feature\nmaps and filters have been studied only at small scales (e.g., 1K\nmultiply-accumulate(MAC) units). However, to realize their advantages in full,\nthe sparse architectures have to be scaled up to levels of the …