CI: Add GHA workflow to upload nightly wheels #22733
Conversation
|
|
||
| - name: Upload x86_64.manylinux wheels to Anaconda Cloud as nightlies | ||
| run: | | ||
| anaconda --token ${{ secrets.ANACONDA_ORG_UPLOAD_TOKEN }} upload \ |
There was a problem hiding this comment.
c.f. #21637 (comment)
There was a problem hiding this comment.
ANACONDA_ORG_UPLOAD_TOKEN is still the right secret but I do not think we will know if this works until we merge it.
There was a problem hiding this comment.
Not sure if it is worth it, but you could manually download a wheel and then upload it with the token to test in advance that the ANACONDA_ORG_UPLOAD_TOKEN is valid. Though given that this PR enables workflow dispatch if there was something wrong with the token you could just change the value of the repo secret and rerun the job (which runs in under 1 minute) so probably not worth the time to test it out before.
There was a problem hiding this comment.
Thank you for opening your first PR into Matplotlib!
If you have not heard from us in a while, please feel free to ping @matplotlib/developers or anyone who has commented on the PR. Most of our reviewers are volunteers and sometimes things fall through the cracks.
You can also join us on gitter for real-time discussion.
For details on testing, writing docs, and our review process, please see the developer guide
We strive to be a welcoming and open project. Please follow our Code of Conduct.
0f78d7c to
5527d13
Compare
|
Okay, this passed CI before I rebased it to allow for uploading all the wheels (c.f. #22733 (comment)) so this should be ready for review. I'm going to tag multiple people as there as lots of discussion in Issue #9994 (apologies if this annoys anyone — not the intention). Tagging:
|
|
Oh, this is clever to re-use the existing built wheels! |
|
@QuLogic Question before I push next: Are you cool with rebasing during code review? Or would you prefer that the commit history not get revised until after final approval and then a final rebase be done to keep things more atomic and clean? The dev docs seem neutral on this point but I want to make your review as easy and not annoying as possible. 👍 |
|
We are very pro re-basing. |
d66a1b2 to
1cb0a7a
Compare
Add nightlies.yml GitHub Action workflow that uses the gh CLI API to download the 'wheels' artifact from the latest completed build of the cibuildwheel workflow from the 'main' branch. This is done by getting the fields 'event', 'status', and 'databaseId' from the output of `gh run list` as JSON and then using jq to filter for 'push' events (which correspond to merged PRs to the 'main' branch) that have completed runs. This is then sorted by the databaseIds in descending order to get the latest completed build run number. Then the 'wheels' artifact can be downloaded. The downloaded wheels are then uploaded to the scipy-wheels-nightly Anaconda Cloud organization (https://anaconda.org/scipy-wheels-nightly) using the anaconda-client CLI API.
Describe that the wheels are available on the scipy-wheels-nightly Anaconda Cloud organization (https://anaconda.org/scipy-wheels-nightly) and that a user should use --index-url to install from this package index.
1cb0a7a to
7dda3c6
Compare
|
Thanks @matthewfeickert! Congratulations on your first PR to Matplotlib 🎉 We hope to hear from you again. |
|
@tacaswell: The token is incorrect: |
Thanks for a ⚡ fast and helpful review! 😄 Truly appreciate all the help here — makes for a good first time contributor experience. 👍 I guess the "good news" RE: the token scope problem is that this shouldn't require any code changes at least. |
|
https://github.com/matplotlib/matplotlib/runs/5780119640?check_suite_focus=true <- looks like it works now. [edit to add extra word that made the comment way less obnoxious 🤦🏻 ] |
|
Looks like in CI the given command doesn't necessarily work. because it tries to pull in the matlpotlib dependencies from the same source. See https://github.com/matplotlib/ipympl/runs/5780345225?check_suite_focus=true |
|
Perhaps it should be |
|
i think |
|
follow along at matplotlib/ipympl#449 for experimentation |
YAY! :D
@ianhi @QuLogic This is what I was assuming people would be doing (given how we test $ docker run --rm -ti python:3.10 /bin/bash
root@1d34811c6d56:/# python -m venv venv && . venv/bin/activate
(venv) root@1d34811c6d56:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@1d34811c6d56:/# python -m pip --quiet install matplotlib # Get dependencies from PyPI
(venv) root@d092a0562fab:/# python -m pip install --upgrade --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple matplotlib
Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple
Requirement already satisfied: matplotlib in /venv/lib/python3.10/site-packages (3.5.1)
Collecting matplotlib
Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/matplotlib/3.6.0.dev1948%2Bgd8ede1a710/matplotlib-3.6.0.dev1948%2Bgd8ede1a710-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.9/11.9 MB 5.8 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20.0 in /venv/lib/python3.10/site-packages (from matplotlib) (21.3)
Requirement already satisfied: numpy>=1.19 in /venv/lib/python3.10/site-packages (from matplotlib) (1.22.3)
Requirement already satisfied: fonttools>=4.22.0 in /venv/lib/python3.10/site-packages (from matplotlib) (4.31.2)
Requirement already satisfied: cycler>=0.10 in /venv/lib/python3.10/site-packages (from matplotlib) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /venv/lib/python3.10/site-packages (from matplotlib) (1.4.2)
Requirement already satisfied: pillow>=6.2.0 in /venv/lib/python3.10/site-packages (from matplotlib) (9.0.1)
Requirement already satisfied: python-dateutil>=2.7 in /venv/lib/python3.10/site-packages (from matplotlib) (2.8.2)
Requirement already satisfied: pyparsing>=2.2.1 in /venv/lib/python3.10/site-packages (from matplotlib) (3.0.7)
Requirement already satisfied: six>=1.5 in /venv/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)
Though it seems that this isn't how you think people will approach it, so I guess this should get updated. edit: Oops I see that this doesn't even work properly, so yeah needs to get fixed (venv) root@d092a0562fab:/# python -m pip uninstall -y matplotlib
Found existing installation: matplotlib 3.5.1
Uninstalling matplotlib-3.5.1:
Successfully uninstalled matplotlib-3.5.1
(venv) root@d092a0562fab:/# python -m pip install --upgrade --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple matplotlib
Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple
Collecting matplotlib
Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/matplotlib/3.6.0.dev1948%2Bgd8ede1a710/matplotlib-3.6.0.dev1948%2Bgd8ede1a710-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.9/11.9 MB 18.3 MB/s eta 0:00:00
Requirement already satisfied: python-dateutil>=2.7 in /venv/lib/python3.10/site-packages (from matplotlib) (2.8.2)
Requirement already satisfied: packaging>=20.0 in /venv/lib/python3.10/site-packages (from matplotlib) (21.3)
Requirement already satisfied: fonttools>=4.22.0 in /venv/lib/python3.10/site-packages (from matplotlib) (4.31.2)
Requirement already satisfied: cycler>=0.10 in /venv/lib/python3.10/site-packages (from matplotlib) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /venv/lib/python3.10/site-packages (from matplotlib) (1.4.2)
Requirement already satisfied: numpy>=1.19 in /venv/lib/python3.10/site-packages (from matplotlib) (1.22.3)
Requirement already satisfied: pillow>=6.2.0 in /venv/lib/python3.10/site-packages (from matplotlib) (9.0.1)
ERROR: Could not find a version that satisfies the requirement pyparsing<3.0.0,>=2.2.1 (from matplotlib) (from versions: none)
ERROR: No matching distribution found for pyparsing<3.0.0,>=2.2.1 |
|
I think the pyparsing issue is one we should look into un-pinning on main. |
|
Looks like the magic working line is this one: |
|
actually no it doesn't :/ still got https://github.com/matplotlib/ipympl/runs/5780465750?check_suite_focus=true |
Yeah this is necessary. You currently have to do the following: $ docker run --rm -ti python:3.10 /bin/bash
root@ad729996566a:/# python -m venv venv && . venv/bin/activate
(venv) root@ad729996566a:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@ad729996566a:/# python -m pip --quiet install matplotlib # Get dependencies
(venv) root@ad729996566a:/# python -m pip --quiet install --upgrade 'pyparsing<3.0' # Fix pyparsing upper bound problem
(venv) root@ad729996566a:/# python -m pip install --upgrade --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre matplotlib
Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple
Collecting matplotlib
Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/matplotlib/3.6.0.dev1948%2Bgd8ede1a710/matplotlib-3.6.0.dev1948%2Bgd8ede1a710-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.9/11.9 MB 18.5 MB/s eta 0:00:00
Requirement already satisfied: pyparsing<3.0.0,>=2.2.1 in /venv/lib/python3.10/site-packages (from matplotlib) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /venv/lib/python3.10/site-packages (from matplotlib) (0.11.0)
Requirement already satisfied: packaging>=20.0 in /venv/lib/python3.10/site-packages (from matplotlib) (21.3)
Requirement already satisfied: fonttools>=4.22.0 in /venv/lib/python3.10/site-packages (from matplotlib) (4.31.2)
Requirement already satisfied: python-dateutil>=2.7 in /venv/lib/python3.10/site-packages (from matplotlib) (2.8.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /venv/lib/python3.10/site-packages (from matplotlib) (1.4.2)
Requirement already satisfied: numpy>=1.19 in /venv/lib/python3.10/site-packages (from matplotlib) (1.22.3)
Requirement already satisfied: pillow>=6.2.0 in /venv/lib/python3.10/site-packages (from matplotlib) (9.0.1)
Requirement already satisfied: six>=1.5 in /venv/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)
Installing collected packages: matplotlib
Successfully installed matplotlib-3.6.0.dev1948+gd8ede1a710
(venv) root@ad729996566a:/# python -m pip show matplotlib
Name: matplotlib
Version: 3.6.0.dev1948+gd8ede1a710
Summary: Python plotting package
Home-page: https://matplotlib.org
Author: John D. Hunter, Michael Droettboom
Author-email: matplotlib-users@python.org
License: PSF
Location: /venv/lib/python3.10/site-packages
Requires: cycler, fonttools, kiwisolver, numpy, packaging, pillow, pyparsing, python-dateutil
Required-by:
edit: I originally was uninstalling and then installing from the nightlies, but you can just use |
|
ok I think the best possible option is: this way you get the most up to date mpl dependencies (but download the deps from pypi) |
Yeah that works! $ docker run --rm -ti python:3.10 /bin/bash
root@17a63da795c7:/# python -m venv venv && . venv/bin/activate
(venv) root@17a63da795c7:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@17a63da795c7:/# python -m pip install --upgrade --pre --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple --extra-index-url https://pypi.org/simple matplotlib
Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple, https://pypi.org/simple
Collecting matplotlib
Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/matplotlib/3.6.0.dev1948%2Bgd8ede1a710/matplotlib-3.6.0.dev1948%2Bgd8ede1a710-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.9/11.9 MB 25.5 MB/s eta 0:00:00
Collecting cycler>=0.10
Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.4.2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 24.1 MB/s eta 0:00:00
Collecting pyparsing<3.0.0,>=2.2.1
Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.8/67.8 KB 5.0 MB/s eta 0:00:00
Collecting packaging>=20.0
Downloading packaging-21.3-py3-none-any.whl (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 KB 3.5 MB/s eta 0:00:00
Collecting fonttools>=4.22.0
Downloading fonttools-4.31.2-py3-none-any.whl (899 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 899.5/899.5 KB 23.8 MB/s eta 0:00:00
Collecting python-dateutil>=2.7
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 KB 23.1 MB/s eta 0:00:00
Collecting pillow>=6.2.0
Downloading Pillow-9.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 31.0 MB/s eta 0:00:00
Collecting numpy>=1.19
Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.23.0.dev0%2B967.g41cf10dcb/numpy-1.23.0.dev0%2B967.g41cf10dcb-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.0/17.0 MB 10.4 MB/s eta 0:00:00
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pyparsing, pillow, numpy, kiwisolver, fonttools, cycler, python-dateutil, packaging, matplotlib
Successfully installed cycler-0.11.0 fonttools-4.31.2 kiwisolver-1.4.2 matplotlib-3.6.0.dev1948+gd8ede1a710 numpy-1.23.0.dev0+967.g41cf10dcb packaging-21.3 pillow-9.0.1 pyparsing-2.4.7 python-dateutil-2.8.2 six-1.16.0
(venv) root@17a63da795c7:/# python -m pip show matplotlib
Name: matplotlib
Version: 3.6.0.dev1948+gd8ede1a710
Summary: Python plotting package
Home-page: https://matplotlib.org
Author: John D. Hunter, Michael Droettboom
Author-email: matplotlib-users@python.org
License: PSF
Location: /venv/lib/python3.10/site-packages
Requires: cycler, fonttools, kiwisolver, numpy, packaging, pillow, pyparsing, python-dateutil
Required-by:
I'll open a fix PR for the docs after dinner. |
|
Whoops, it looks like we fixed the compatibility in with pyparsing 3 in https://github.com/matplotlib/matplotlib/pull/21501/files and removed the pinning in the CI requirement files but forgot to remove it in setup.py 🤦🏻 |
|
#22751 to fix the pyparsing issue. |
|
PR #22752 to make the docs more robust. |
|
Duplicate uploads also fail. |
|
I think that is OK though? We have at least one build on main per day most days and I am not sure it is worth the logic to not get rejected on the days we do not (or on days when someone has pushed the button to run it early). |
If no one minds I don't mind throwing in some additional logic to check just before the upload stage. This is pretty easy with So all you'd need to do would be something like $ LAST_NIGHTLY_VERSION="$(python -m pip index \
--index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple \
--pre \
versions matplotlib | \
grep matplotlib | \
sed 's/.*(\(.*\))/\1/')"
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
$ echo "${LAST_NIGHTLY_VERSION}"
3.6.0.dev1948+gd8ede1a710and then just check if that shows up as version of the wheels that just got downloaded from the GitHub Actions workflow artifact $ [ "$(find dist -type f -iname "matplotlib-${LAST_NIGHTLY_VERSION}*.whl" | wc --lines)" -gt "0" ]
$ echo $?
0(maybe a more elegant way to do that but that's not the worst) I'll make an Issue from this and I'm happy to PR it, unless you would prefer to avoid touching it as much as possible. edit: Answer is way easier: Use the anaconda --token ${{ secrets.ANACONDA_ORG_UPLOAD_TOKEN }} upload \
--user scipy-wheels-nightly \
--skip-existing \
dist/matplotlib-*.whl |
PR Summary
This PR adds a
nightlies.ymlGitHub Action workflow that uses theghCLI API to download the 'wheels' artifact from the latest completed build of thecibuildwheelworkflow from the 'main' branch (idea taken from actions/download-artifact#3 (comment)).gh run example:
This is done by getting the fields
event,status, anddatabaseIdfrom the output ofgh run listas JSON and then usingjqto filter for 'push' events (which correspond to merged PRs to the 'main' branch) that have completed runs. This is then sorted by thedatabaseIds in descending order to get the latest completed build run number.filter and download example:
The
subset of all thedownloaded wheelsthat areare then uploaded to thex86_64.manylinux*scipy-wheels-nightlyorganization using theanaconda-clientCLI API.N.B.:
anaconda-clientmust currently be installed from GitHub as there have been no uploads to PyPI since 2016. c.f. anaconda/anaconda-client#540The workflow runs nightly as a CRON job ('schedule' in GHA parlance) at 01:23 UTC (the time has no meaning (it is just 0123) beyond making sure it misses the wall of CRON jobs that slams CI providers at 00:00 and 00:01) and on demand through workflow dispatch.
The status of the build on my fork is currently summarized in #21635 (comment).
PR Checklist
Tests and Styling
pytestpasses).flake8-docstringsand runflake8 --docstring-convention=all).(No Python code was changed, so checking these off as
mainis passing)Documentation
doc/users/next_whats_new/(follow instructions in README.rst there). (N/A as no new features)doc/api/next_api_changes/(follow instructions in README.rst there). (N/A as no API changes)