Skip to content

CI: Remove old scipy-wheels-nightly uploads to ensure space #23349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 28, 2022
Merged

CI: Remove old scipy-wheels-nightly uploads to ensure space #23349

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/nightlies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,23 @@ jobs:
--user scipy-wheels-nightly \
--skip-existing \
dist/matplotlib-*.whl

- name: Remove old uploads to save space
shell: bash
run: |
N_LATEST_UPLOADS=5
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comes from @ogrisel's suggestion #22757 (comment) to

only keep the 5 most recent dev wheels for a given project and platform spec

but the number here is arbitrary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a good number, I could see a case for going up to like 14 (2 weeks), but given that other projects are replacing their wheels nightly 5 seems pretty good!


# Remove all _but_ the last "${N_LATEST_UPLOADS}" package versions
# N.B.: `anaconda show` places the newest packages at the bottom of the output
# of the 'Versions' section and package versions are preceded with a ' + '.
anaconda show scipy-wheels-nightly/matplotlib &> >(grep '+') | \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using &> >(grep '+') here because anaconda show doesn't output to stdout.

sed 's/.* + //' | \
Comment on lines +73 to +76
Copy link
Contributor Author

@matthewfeickert matthewfeickert Jun 26, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is somewhat brittle as it relies on user facing output not changing, instead of consuming API output. Though this seems about as good/bad as trying to use

$ python -m pip index \
    --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple \
    --pre \
    versions matplotlib | \
    grep 'Available versions'
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
Available versions: 3.6.0.dev2573+g3eadeacc06, 3.6.0.dev2569+g3522217386, 3.6.0.dev2553+g3245d395d9

to get the versions available. Also the current version of anaconda-client used is frozen given anaconda/anaconda-client#540.

head --lines "-${N_LATEST_UPLOADS}" > remove-package-versions.txt

if [ -s remove-package-versions.txt ]; then
while LANG=C IFS= read -r package_version ; do
anaconda --token ${{ secrets.ANACONDA_ORG_UPLOAD_TOKEN }} remove \
--force \
"scipy-wheels-nightly/matplotlib/${package_version}"
Comment on lines +81 to +83
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While @tacaswell is (AFAIK) the only person to have removed package versions before with this API (c.f. #22757 (comment)), this is the correct syntax to remove all wheels for a given dev release on the package index given that from https://github.com/Anaconda-Platform/anaconda-client/blob/be1e14936a8e947da94d026c990715f0596d7043/binstar_client/commands/remove.py we can see that Package written as <user>[/<package>[/<version>[/<filename>]]] and for version there is the question (when run sans --force) of Are you sure you want to remove the package release %s ? (and all files under it?).

Also you can just try this and note the same clarifying prompt

$ anaconda remove scipy-wheels-nightly/matplotlib/3.6.0.dev2553+g3245d395d9
Using Anaconda API: https://api.anaconda.org
Are you sure you want to remove the package release scipy-wheels-nightly/matplotlib/3.6.0.dev2553+g3245d395d9 ? (and all files under it?) [y|N]:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been removing things via mind-less clicking on the website (which is how I took out all of the uploads at one point).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I "tested" this by running the commands locally with the token (had to trim to 3 as we only and 4 release up!) and it deleted files as expected.

done <remove-package-versions.txt
fi