The reason is the following

set_x/y/zlim() deactivate autoscaling (auto=False). set_x/y/zbound() leave the autoscaling unchanged (calling set_xlim(auto=None)).

This can be verified via

ax.set(xbound=[-1, 1], ybound=[-1, 1], zbound=[-1, 1])
ax._autoscaleXon = False
ax._autoscaleYon = False
ax._autoscaleZon = False

fixing the issue.

Questions:

• Is it reasonable that *bounds and *lim behave differently concerning auto-scaling? But even if so, it may be hard to change, because I expect that would break a lot of user code.
• Is there a reason that 3D rotation tries to auto-scale?

It's not the mouse move that triggers it, it's the draw. If you insert fig.canvas.draw() before plt.show(), the limits are [0, 1]. I don't know why this change during draw is not reflected in the plot.

I think this may be because of the new lazy autoscaling. Nothing in mplot3d sets the new ax._stale_viewlim_x, so _unstale_viewLim rescales.

This is a regression from 3.1.2; I will test some newer versions, but if it's lazy autoscaling, then it's a 3.3 3.2 regression. I think I've written most of a fix.

I don't know why this change during draw is not reflected in the plot.

And this is because Axes3D.draw first calculates the world projection, then calls super().draw(), and Axes.draw calls self._unstale_viewLim. Putting that at the beginning of Axes3D.draw makes things stabler. This is similar to #17422 for PolarAxes.