In the context of artificial neural networks, the rectifier or ReLU
(rectified linear unit) activation function is an activation function
defined as the positive part of its argument:
where x is the input to a neuron. This is also known as a ramp function and
is analogous to half-wave rectification in electrical engineering. This
activation function was introduced by Kunihiko Fukushima in 1969 in the
context of visual feature extraction in hierarchical neural networks. It was
later argued that it has strong biological motivations and mathematical
justifications. In 2011 it was found to enable better training of deeper
networks, compared to the widely used activation functions prior to 2011,
e.g.