When near an item that is smaller or larger than you current size, you can right-click to swap sizes with that object :). Should work pretty good, so you are experiencing a bug I am sorry to hear that.
