A minimal example would be something like,

from sklearn.neural_network.multilayer_perceptron import MLPClassifier
from sklearn.neural_network._base import ACTIVATIONS, DERIVATIVES                         

ACTIVATIONS['identity2'] = lambda x: x                                                                                                 

def inplace_identity_derivative2(Z, delta): 
     pass 
                                                                                                                                        
DERIVATIVES['identity2'] = inplace_identity_derivative2                                                                                

mlp = MLPClassifier(activation='identity2')                                                                                            

mlp.fit([[0, 1], [1, 1]], [0, 1])

changing public variables in the scikit-learn namespace is not great, but as a somewhat experimental solution it should work (also it doesn't cost much).

I'm very worried about this one. Which one of the following two are we doing?

• Allow people to use this API, which is public. If we're doing this, then we should document it in neural_networks_supervised.rst at least, and add a whats_new entry. But that means we're actually happy about this solution; are we? I rather let them pass a function instead of having to add the function to the dictionary.
• We want to have a better solution, and we'll probably do that, but later. That means we would have to deprecate this solution which is just dirty, if we already know we don't like it.

I kinda don't like not documenting it, cause we'll force ourselves to deprecate the behavior if we wanna change it anyway. Which means why not document and let people use it then?

Allow people to use this API, which is public. If we're doing this, then we should document it in neural_networks_supervised.rst at least, and add a whats_new entry

We don't have to document anything if we don't judge it necessary. The one line change to the MLP file could be done just under the justification of removal of redundant code unless you think it's controversial. It happens that you can then provide custom activation by adding to these dict variables. Is it worth explicitly disallowing that? I don't know. Technically ACTIVATIONS, DERIVATIVES are defined in sklearn.neural_network._base which is private it's just that I can't monkeypatch it there (that doesn't work).

I can locally import those inside MLPClassifier and remove top level imports in multilayer_perceptron.py (which would also allow using this mechanism by only changing private variables) if you think that would be preferrable?

I rather let them pass a function instead of having to add the function to the dictionary.

We could pass a function that returns the activation function and the derivative. The problem is that activation and derivatives are not computed at the same time (meaning it would be slower) and also add a bit more complexity to code.

I can locally import those inside MLPClassifier and remove top level imports in multilayer_perceptron.py (which would also allow using this mechanism by only changing private variables) if you think that would be preferrable?

I would prefer that.

We could pass a function that returns the activation function and the derivative. The problem is that activation and derivatives are not computed at the same time (meaning it would be slower) and also add a bit more complexity to code.

I meant passing two functions, but I'm happy with the private variable solution as well.

Are we going to start recommending users to do this in our issue board?

Are we going to start recommending users to do this in our issue board?

Allowing to do this via a private API, implying that we are not officially supporting it and it can change in the future, is not such a bad compromise. But it still leaves people the ability to experiment if they are so inclined. Documenting it would make this mechanism a bit more public, I'm not sure that is something we want. How do you think we should handle it?

Cutom

A minimal example would be something like,

from sklearn.neural_network.multilayer_perceptron import MLPClassifier
from sklearn.neural_network._base import ACTIVATIONS, DERIVATIVES                         

ACTIVATIONS['identity2'] = lambda x: x                                                                                                 

def inplace_identity_derivative2(Z, delta): 
     pass 
                                                                                                                                        
DERIVATIVES['identity2'] = inplace_identity_derivative2                                                                                

mlp = MLPClassifier(activation='identity2')                                                                                            

mlp.fit([[0, 1], [1, 1]], [0, 1])

changing public variables in the scikit-learn namespace is not great, but as a somewhat experimental solution it should work (also it doesn't cost much).

This is not helping me to work with custom activation functions in Sklearn .
Please suggest what process should I do.

@ArijitN5 The PR isn't even merged yet so it's normal that the code snippet doesn't work yet.

Also note that even though we are making this possible, we do not recommend it. You would be using a private API that is subject to change without deprecation.

Could we consider (privately) accept a tuple of callables instead of hacking the dict?

Could we consider (privately) accept a tuple of callables instead of hacking the dict?

I'd be happy to publicly accepting a tuple of callables, why not?

I'd be happy to publicly accepting a tuple of callables, why not?

The problem is that tuple of callables doesn't generalize well to the case when more than 2 callable are needed (e.g. GLM) #14720 (comment) so it's not a general solution to this problem.

hence the idea of keeping it private. it's possible, but not recommended.

OK so I reverted latest changes, and can remove the test if it bothers you. So there would be only a 2 line code simplification PR. The example in the PR still uses the private API.

To summarize, if we were to do this as part of the public API, we could

a) add a activation_derivative=None parameter, and accept a callable there and for activation.
b) have some registration mechanism for ACTIVATION (e.g. register_activation('name', activation, derivative)
c) accept a tuple of callable with (fn activation, fn derivative)
d) accept a callable returning a tuple
e) Accept an instance (or a named tuple) of the form .

 class Activation:
    def __call___(...)
    
    def derivative(...)

Can't say I'm too enthusiastic about any of them. Putting all constraints aside I would probably go with e) or c) (with a named tuple) .

I'd be fine with not having the test, and I'm also fine with C or E. After these discussions I'm also just happy with the PR as is, but ideally I'd open an issue to have one of your suggestions implemented after this one.

I'm OK with the PR as-is too.

If we ever write an example we might want to use _base.ACTIVATIONS instead of the public path.

I removed the test and changed the PR title. Should be good to go.

Will try to open an issue about possible public API implementation later today.

Merging, thanks @rth

I'm OK with a public API but I think we decided not to add more features to the NN code during the meeting? So might just not be worth it