That is a known limitation. #11622 might provide a short term solution to this (the long term would be the estimator tags).

A PR to allow optionally disable certain checks of check_estimator as outlined in the above issue would be welcome.

I think it is not about disabling checks but transforming them.

For example, in the case when you want to see that an estimator pickles, the whole test depends on the input initialization:

X, y = make_blobs(n_samples=30, centers=[[0, 0, 0], [1, 1, 1]],
                  random_state=0, n_features=2, cluster_std=0.1)

By observing other tests (in order to bring them in the terms of my package) I witnessed that the most of the time the changes are all about changing the input. Considering that this input is either random/arbitrary or that it even follows a certain pattern it can be categorized in any case as the result(s) of a function or as the values of a given object parametrized for the given estimator.

Also the complexity of this checks doesn't allow someone not experienced to generalize them but maybe to make them suit their needs, which in the end may not provide a sklearn compatible solution.

That's why generalization seems a much better idea (if possible).
What are the estimator tags?

@rth how does #11622 solve this?

FYI this is the PR @rth mentioned: #8022

The current PR doesn't actually implement running the checks for texts, but it adds the mechanisms for it. There's an input type tag that will be text for a vectorizer, but the tests in the PR will just not run tests that don't want numpy array input.
A next step would be to generate the test data conditional on the model, as you suggest.

rth how does #11622 solve this?

Well, given than e.g. check_estimator(CountVectorizer) fails, if individual tests were exposed, a crude first first step could have been to at least skip those that fail for CountVectorizer and keep e.g. pickling checks etc.

I agree though estimators tags would be more general.

I think it is not about disabling checks but transforming them.
By observing other tests (in order to bring them in the terms of my package) I witnessed that the most of the time the changes are all about changing the input.

Well the problem is that this would add quite a lot of complexity. Most of checks assume that input is 2D arrays and output is 1D or 2D. Now we consider text vectorizers, in the same way one could say, well I use ML on time series or recommendation algoritm or metric learning and my input data is not a 2D matrix (n_samples, n_features) but I would still like to use check_estimator to check some consistence to the scikit-learn API to the extent it's possible (i.e. consistency between fit + predict / fit_predict, picklabilty etc.). I hope estimator tags will help, but in general it's doesn't sound that simple to generalize current estimator checks to work in all possible use case.

You could say that text vectorizers are in scope (because they are in scikit-learn) and the the rest is not, as far as estimator checks are concerned, but I don't see a fundamental difference between this and anything else that's not the usual 2D array input.