What is preventing sklearn to achieve true model persistence? #30609
Replies: 2 comments 2 replies
-
|
Basically, it is more a maintenance burden where with the team, we estimate that we could not maintain it. However, we had recent discussion in which we think that we could have a trimmed inference estimator for each estimator, reducing the impact of potential private changes that make it to update scikit-learn versions in this setting. Basically, it would make the life easier for packages as It would be possible to working on persistence with a |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I understand the maintenance burden issue but right now a Your trimmed inference estimator idea is awesome ! |
Beta Was this translation helpful? Give feedback.
-
|
Another path would be to "just" make sure everything necessary (but nothing more) for inference is accessible as public attributes (without creating a new class for each estimator), so that tools such as |
Beta Was this translation helpful? Give feedback.
-
|
I concur with you Pierre-Bartet, it should be feasible to implement model persistence as a community effort. Issue #31143 is relevant for this discussion. There is no need for deciding on a persistence format, the only requirement is that parameters/state can be retrieved from a model, as either numpy or python native data structures. And conversely, that a model can consume the same as input for initialisation. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
What is preventing
sklearnto achieve true model persistence?For example
model.dump(..)+LogisticRegression.load(...)?All the existing solutions are brittle or force users to use exactly the same
sklearnversion for training and inference:https://scikit-learn.org/1.6/model_persistence.html
I understand that this is a deliberate choice because sklearn's team lack of resources, but offloading serialization logic to external libraries can only end up in an a much worse maintenance, communication, and interdependence nightmare.
For example
sklearn-onnxaccesses privatesklearncomponents to be able to serialize them (such asPolynomialFeatures's_min_degree, or gradient boosting's_predictors).Covering all of
sklearncomponents would be a tremendous task, but it could be done step by step, and it is also somewhat parallelizable by assigning a few models to anyone who would be happy to help.Beta Was this translation helpful? Give feedback.
All reactions