This was regressed in #2187. There was bad testing around this
before, and this should prevent a similiar situation from happening
again!