couldn't a higher n-dim with shape 1 lead to weird broadcasting?
Maybe check np.shape(y)[0] ?

With np.shape(y)[0], it will fail to recognize a row vector as a 1d vector.

With a row vector, I mean something like np.array([[1, 2, 3, 4]]).

np.array([[1, 2, 3, 4]]) == np.array([1, 2, 3, 4]) would result in broadcasting, but it doesn't.
But
np.array([[0, 1, 2]]) == np.array([[0], [1], [2]]) does, and these both go through the check.

Is this case handled correctly?

With the np.max(np.shape(x)), this is properly handle, but not with np.shape(x)[0].

In [3]: np.shape(np.array([[1, 2, 3, 4]]))[0]
Out[3]: 1

In [4]: np.max(np.shape(np.array([[1, 2, 3, 4]])))
Out[4]: 4

In [5]: np.shape(np.array([[0, 1, 2]]))[0]
Out[5]: 1

In [6]: np.max(np.shape(np.array([[0, 1, 2]])))
Out[6]: 3

Sorry, I was a bit unclear. Is this handled correctly in the accuracy?
So your check is whether there is only one dimension with shape != 1.
Checking just that might lead to surprising results:

In [4]: np.mean(np.array([[0, 1, 2]]) == np.array([[0], [1], [2]]))
Out[4]: 0.33333333333333331

I think that this case is checked in test_format_invariance_with_1d_vectors() for all metrics.

For the moment, this raised an error. I will investigate further.
But I think that it can lead to ambiguity issues.

In [4]: import numpy as np

In [5]: accuracy_score(np.array([[0, 1, 2]]) ,  np.array([[0], [1], [2]]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-c70daa59450a> in <module>()
----> 1 accuracy_score(np.array([[0, 1, 2]]) ,  np.array([[0], [1], [2]]))

/home/ajoly/git/scikit-learn/sklearn/metrics/metrics.pyc in accuracy_score(y_true, y_pred, normalize)
    932 
    933     """
--> 934     y_true, y_pred = check_arrays(y_true, y_pred, allow_lists=True)
    935 
    936     # Compute accuracy for each possible representation

/home/ajoly/git/scikit-learn/sklearn/utils/validation.pyc in check_arrays(*arrays, **options)
    192         if size != n_samples:
    193             raise ValueError("Found array with dim %d. Expected %d"
--> 194                              % (size, n_samples))
    195 
    196         if not allow_lists or hasattr(array, "shape"):

ValueError: Found array with dim 3. Expected 1

ok great :)

The reason why I don't fix those problems is that the mix of 1d row vector with other format is not allowed by check_arrays.

Furthermore, it can lead to ambiguity issue (when n_samples=1) with binary indicator format and binary classification.

# At the moment, these mix representations aren't allowed
assert_raises(ValueError, metric, y1_1d, y2_row)
assert_raises(ValueError, metric, y1_row, y2_1d)
assert_raises(ValueError, metric, y1_list, y2_row)
assert_raises(ValueError, metric, y1_row, y2_list)
assert_raises(ValueError, metric, y1_column, y2_row)
assert_raises(ValueError, metric, y1_row, y2_column)

Cosmit: no learning whitespace on the first line of the docstring.

"are vectors": don't you mean "2D arrays or more"?

What I called a vector is anything that return True for _is _1d.

Alright, I read the examples and understood afterward. The purpose of this helper is really non intuitive to me though. I need to re-read where it is actually used to understand the motivation. Maybe it could be made more explicit in the docstring.

I will add a comment about it. The main motivation was to be able to reshape, if needed, mix of vector representation. It is also used to infer correctly the number of samples with "row" vectors.

The function is indeed very specific but it is private and it is in the file it is used in, so I think that should be fine.

If you have a better way to describe what I call vector, don't hesitate!! :-)