Abstract Optical neural networks are emerging as a competitive alternative to their electronic counterparts, offering distinct advantages in bandwidth and energy efficiency. Despite these benefits, scaling up on-chip optical neural networks for end-to-end inference is facing significant challenges. First, network depth is constrained by the weak cascadability of optical nonlinear activation functions. Second, the input size is constrained by the scale of the optical matrix. Herein, we propose a scaling up strategy…