PR Summary

NumPy dropped these builds first in 1.21.2 (Aug 15, 2021) for Python 3.10 only, and for all builds in 1.22 (Dec 31, 2021).
SciPy dropped these builds first in 1.8.0 (Feb 5, 2022) for Python 3.10 only, and for all builds in 1.9.2 (Oct 8, 2022).

Also, simplify workflow file to remove some duplication.

Theoretically we could join all the Build wheels for CPython 3.X steps, but I left them separate so that there wouldn't be a timeout for a single step taking too long (though I don't think that'd happen).

PR Checklist

Documentation and Tests

• [n/a] Has pytest style unit tests (and pytest passes)
• Documentation is sphinx and numpydoc compliant (the docs should build without error).
• [n/a] New plotting related features are documented with examples.

Release Notes

• [n/a] New features are marked with a .. versionadded:: directive in the docstring and documented in doc/users/next_whats_new/
• API changes are marked with a .. versionchanged:: directive in the docstring and documented in doc/api/next_api_changes/
• Release notes conform with instructions in next_whats_new/README.rst or next_api_changes/README.rst