I'm -1 on adding this. We don't even have dropout now, right? I feel recent advances are better suited for Keras and we don't really want or need to reimplement Keras here.

So essentially this adds an activation function and a derivative to neural_network.{ACTIVATIONS,DERIVATIVES}. Maybe we could just document that and provide a simple example? That way anyone could add the activation function they want without much maintenance effort on our side. Sometimes it's still handy to do a simple NN without installing tensorflow or pytorch.

there's no way to change the alpha, though? And the question is a bit what "simple" means. Is there evidence that this really helps in practice? In particular for dense networks?

there's no way to change the alpha, though?

Yes, that is a significant limitation of this approach.

And the question is a bit what "simple" means. Is there evidence that this really helps in practice?

Haven't searched in the literature but this is the third PR or issue about it, so there is certainly interest. Though following the dev meeting today it seems there was a consensus on not adding new DL features.

Another approach requiring a 1 line change for supporting custom activation function can be found in #14815

I think "delta[Z < 0] = alpha" should be replaced by " delta[Z < 0] *= alpha " in the computation of the derivative in order to be consistent with other derivative computations. For example, for the tanh, the function inplace derivative is computed as : delta *= (1 - Z ** 2). So, delta needs to be multiplied by the derivation function.
I used the california housing database to test this leaky_relu function. It performs well with this modification.