Can you give an example of where the previous code fails?

Can you give an example of where the previous code fails?

a = 3 037 000 500;
b = 2;
m = 3 037 000 501;
a * a = 3 037 000 500 × 3 037 000 500
≈ 9.223 372 037 000 025 000
which is just above LLONG_MAX = 9 223 372 036 854 775 807, so the multiplication overflows.

The same problem happens with your code: https://www.ideone.com/ZqJd1u

Typically, the assumption is M^2 fits into the data type otherwise you must expand the data type for multiplication.