I'd love to take this up! :)
Initial scanning leads to the following points:

• In case of imshow, _ImageBase class inherits from ScalarMappable, but overrides its set_array function as:
  

  matplotlib/lib/matplotlib/image.py

  Lines 733 to 743 in 59bfcfa

  	def set_array(self, A):
  	"""
  	Retained for backwards compatibility - use set_data instead.
  	Parameters
  	----------
  	A : array-like
  	"""
  	# This also needs to be here to override the inherited
  	# cm.ScalarMappable.set_array method so it is not invoked by mistake.
  	self.set_data(A)

• imshow doesn't raise this error as its set_data (called from set_array) copies the original array and checks for numpy casts:
  

  matplotlib/lib/matplotlib/image.py

  Lines 696 to 701 in 59bfcfa

  	self._A = cbook.safe_masked_invalid(A, copy=True)
  	if (self._A.dtype != np.uint8 and
  	not np.can_cast(self._A.dtype, float, "same_kind")):
  	raise TypeError("Image data of dtype {} cannot be converted to "
  	"float".format(self._A.dtype))

• In case of vlines, ScalarMappable is inherited again, but there's no overriding its set_array function, which evidently just accepts ndarray:
  

  matplotlib/lib/matplotlib/cm.py

  Lines 363 to 372 in 59bfcfa

  	def set_array(self, A):
  	"""
  	Set the image array from numpy array *A*.
  	Parameters
  	----------
  	A : ndarray
  	"""
  	self._A = A
  	self._update_dict['array'] = True

To solve this, I added set_array to Collection class, overriding it again. It will need some reviews, I'll issue a PR for the same.