We currently do a fair amount of ad-hoc normalization / edge case handling for when we get pandas (or xarray or TF or ...) objects that are close to numpy but not quite (for example #16347 which uses a warning context manager which is extra not-thread-safe in other places we check of obj.values).

Can we centralize / streamline this logic?