Skip to content

NumPy Security roadmap proposal #29178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rgommers opened this issue Jun 11, 2025 · 8 comments
Open

NumPy Security roadmap proposal #29178

rgommers opened this issue Jun 11, 2025 · 8 comments

Comments

@rgommers
Copy link
Member

We haven't made too many major changes to how we deal with security for the NumPy project in a few years. The last significant changes were (a) requiring 2FA for everyone with repo admin privileges or PyPI access, (b) using hashes on all GitHub actions, and (c) adding OpenSSF scorecards and addressing some issues that that turned up. The security section of our roadmap is short, and currently says:

NumPy is quite secure - we get only a limited number of reports about potential vulnerabilities, and most of those are incorrect. We have made strides with a documented security policy, a private disclosure method, and maintaining an OpenSSF scorecard (with a high score). However, we have not changed much in how we approach supply chain security in quite a while. We aim to make improvements here, for example achieving fully reproducible builds for all the build artifacts we publish - and providing full provenance information for them.

The supply chain part is critical there, and the world has changed in a few significant ways:

  • The threat levels for supply chain security have gone up continuously. Examples:
    • This blog post (Jan 2024) documents an extensive supply chain attack on PyTorch.
    • This post (Mar 2025) documents how tj-actions/changed-files, used in >23,000 repositories, was compromised.
    • Tooling to automatically exploit configuration issues for GitHub Actions is commonly available, e.g. gato-x.
  • Government regulation is starting to appear. Examples:
    • A 2021 US Executive Order is mandating the use of SBOMs and documenting of processes in certain contexts.
    • The EU's Cyber Resilience Act, adopted in Oct 2024 and coming into effect in 2027, has a lot of requirements. Open source is in large part, but not fully, exempt - however numpy is distributed in lots of commercial products so a part of our user base is affected.
  • Best practices as well as Python packaging standards are evolving. Examples:

Some of the issues discussed in that blog post on PyTorch are also relevant for NumPy. For example, our GITHUB_TOKEN is still set to the default write permissions on everything, and the tokens for uploading to our nightly bucket as well as the staging bucket for releases to PyPI is accessible by anyone with write permissions on this repository. Note that it's effectively impossible to store repository secrets safely for a subset of people with commit rights, so currently our more stringent requirements for direct PyPI access are only partially effective.

Here is a useful way to think about types of security threats from Supply-chain Levels for Software Artifacts:

Image

For NumPy, we're in decent shape for "source threats" probably - commits on main and maintenance/* are quite visible so the risk of new commits being slipped in unnoticed is low, and we've never had a CVE that was actually concerning (less concerning CVEs are a pain and need mitigating, but they won't cause large-scale damage). On "build threats" I think we are not in great shape. And as one of the highest-profile Python packages and the second-most-downloaded package with compiled extensions on PyPI, we are a pretty attractive target.

The aim of this issue is to serve as a tracking issue and discuss at a high level what we want to do. Sub-issues can be created for individual actionable steps.

Proposed improvements

Related to source-level access and repository permissions:

  1. Further tighten 2FA requirements for everyone with any permissions beyond read-only on the repository.
  2. Move GITHUB_TOKEN to read-only default permissions. Any actions that need it will have to explicitly, and granularly, enable write permissions for labels, pull-requests, etc.

Related to building and distributing of release artifacts:

  1. Move building release artifacts that get uploaded to PyPI and anaconda.org to a new repository. That new repository should have a more strict approach to security:
  • Only write access for org admins and the release team (O(5) rather than O(30) people with access)
  • Enable trusted publishing to PyPI, with a staging environment with manual approval by the release manager before uploading starts.
    • So no more anaconda.org token for releases.
  • Require linear history, so commit history is easy to inspect.
  • Also branch protection etc. - apply best practices there.
  • Only wheels.yml runs, and maybe a security-related linter action
  • All release artifacts are required to be built inside this repository on GitHub Actions runners
    • No self-hosted runners allowed; cross-compiling is allowed (whether we actually do it depends on maintenance cost as always) - verification of the test suite passing after cross compilation is done either on the main repo in a self-hosted runner or under QEMU.

Related to helping downstream distributors and end users with how they approach supply chain security:

  1. Start shipping SBOMs, in the way outlined by PEP 770
  2. Start verifying that our builds are fully reproducible (this will take quite a while to implement and requires new machinery)

Envisioned benefits

The primary benefit is significantly enhanced supply chain security, which is beneficial for both end users and for NumPy as a project (an event with malicious content injected would be pretty stressful for maintainers, and bad for the project's reputation).

Also importantly, it allows the main numpy/numpy repository to continue its relatively loose approach to development, e.g.:

  • lots of people with commit rights,
  • using a host of different actions, including from sometimes unknown individual devs (this is otherwise questionable even with pinning, since no one actually rechecks the action's source code when the pin gets bumped), and from vendors with a questionable reputation like Codecov (we were lucky to not be affected by a previous Codecov breach),
  • Continue using multiple CI services (Circle CI, Cirrus CI, Azure DevOps, maybe others in the future)
  • Allow using self-hosted runners, as long as they are set up securely (i.e., ephemeral - no cross-talk between jobs or actions running on the same self-hosted runner). SciPy already uses Cirrus Runners, and on CI: Enable GitHub Actions CI for ppc64le (Power architecture) support #29125 there is a proposal for IBM-hosted runners,
  • Allowing fairly extensive usage of caching without having to worry too much about cache poisoning, because we can remove all secrets from the main repo.

The release process will also be easier to manage once trusted publishing is set up (no more anaconda-client and manually downloading/uploading wheels by the release manager).

As yet another benefit: other projects in the scientific Python ecosystem tend to follow what we do, so the effort that the proposed changes will require will be well spent - it has a multiplicative effect when maintainers of other projects can learn from what we do and copy aspects of it.


I'll also note that what this doesn't do is making use of PyPI very secure. It'll be a very significant improvement to how we safeguard our release artifacts (I'd estimate >20x fewer people having access to our release secrets, plus more ability to verify the binaries). However, while PyPI is great for development, if one really cares about security (e.g., use of Python/NumPy inside large corporations or government entities), one should have a coherent strategy like building wheels from source and hosting them on private index servers, or using a commercial vendor of Python packages (there are many options, from commercial Linux distros to Anaconda, ActiveState, Chainguard - and I know more offerings are in the making). We just aim do the best we can here with limited means.

@charris
Copy link
Member

charris commented Jun 11, 2025

LGTM. I'd be delighted to skip the download/upload to Anaconda, the only reason for it is to generate hashes for the release products, which seems something that should be automated.

@andyfaff
Copy link
Member

One thing I'm still unsure of is the ephemeral runners. Building on those depends on how the build environment is setup by the host. For example, could there be malicious tools within the environment that can inject malware into the build process? How far can we control what that environment looks like, and how far does trust extend? Of course, this argument doesn't necessarily have to be restricted to those hosts, it also applies to e.g. github.

@jorenham
Copy link
Member

I don't know much about it myself, but perhaps the documentation preview that's generated for (every?) PR is also worth looking at?

Because, roughly speaking, it can be seen as a "free" web hosting service that anyone with a gh account can use. So it might be possible to exploit that for evil things like phishing. For example, by opening a PR when the maintainers are asleep. That way, they can get a free evil website that's hosted on output.circle-artifacts.com for a couple of hours, that can't be traced back to them (if they use an anonymous gh account).

But then again, I'm not very familiar with it, so there's a good chance that I'm missing something here 🤷🏻

@rgommers
Copy link
Member Author

rgommers commented Jun 12, 2025

I'd be delighted to skip the download/upload to Anaconda, the only reason for it is to generate hashes for the release products

Indeed. For other readers: note that this is only true since very recently; our aarch64/arm64 wheels were built on Cirrus CI rather than GitHub Actions before so we also needed a staging bucket to bring all wheels together due to that (and one cannot push artifacts from Cirrus back to GHA in a reasonable way).

I don't know much about it myself, but perhaps the documentation preview that's generated for (every?) PR is also worth looking at?

Good question. I don't think the generated html pages are a worry, because (a) they can only contain malicious content if that content is present in the PR diff which is easy to spot, and (b) very few people will actually look at those links (probably zero before checking the diffs).

However, the integration of those previews (through this action) is a real security headache. For one because the action requires the most elevated permissions and because it triggers on pretty much everything. For another, because it generates pretty severe log pollution, making it really difficult to spot other irregular jobs running. A peek at https://github.com/numpy/numpy/actions will show that.

One of the benefits of moving to a separate repo is very clean logs that are easy to introspect. The CircleCI redirector action is bad enough that I'd love to be able to get rid of it completely even on the main repo - but that's a separate discussion.

One thing I'm still unsure of is the ephemeral runners. Building on those depends on how the build environment is setup by the host. For example, could there be malicious tools within the environment that can inject malware into the build process? How far can we control what that environment looks like, and how far does trust extend?

There could be of course, but that's similar to GitHub Actions with https://github.com/actions/runner-images/. Using a runner image generated by another provider should be quite safe compared to many other things we're doing. It's possible that the provider does it wrong of course, e.g. there are issues with how the container or VM is mounted so it has access to the bare metal machine it's running on. In that case there can be cross-talk between jobs, even with those from other organizations running on the same underlying infra. But again, not the biggest concern - I'd say trusting Cirrus or IBM is significantly safer than trusting the authors of a bunch of the random GitHub Actions we are using.

That said, the other issue with self-hosted runners is that configuring them and giving them permissions to write job status back to GitHub is a little fiddly. See scipy/scipy#21740 (comment) for what I wrote on doing that on the SciPy repo. If one wants to be really secure and avoid admin settings that are potentially risky if changed/misconfigured (which can go undetected for a long time), then it's best to avoid them. This is the reason why I propose we avoid them on the new repo that we do releases from, but allow them on the main repo - if there are no secrets to compromise on the main repo, the extra risk of allowing them is quite low.

Thinking about this from a "who is everybody we are trusting here", then the list for the release repo will be:

  • The NumPy devs with direct commit/admin access (I'd suggest a small group here who actually need these rights, e.g. @charris, @mattip, @andyfaff, @seberg, @rgommers)
  • GitHub Actions, its container images and the tools in them
  • The manylinux images, and cibuildwheel and trusted publishing actions
  • Other actions we use in wheels.yml (needs review, e.g. https://github.com/bus1/cabuild/)
  • YOLO installs in wheels.yml (e.g., we pull in some random Rust .exe right now)
  • Wheel build dependencies from PyPI or direct from GitHub: meson, ninja, meson-python, scipy-openblas32|64, pip, build, cython, delvewheel, auditwheel, delocate
  • Test dependencies pulled in. Currently a long list in requirements/test_requirements.txt, we should trim that down and not use optional test dependencies in wheels.yml.

And for each of those, the group of people who have access to them (EDIT: and their transitive dependencies).

For the main repo, there are many more things - I won't try to list them exhaustively here, but at a high level:

  • everyone with a commit bit
  • 4 different CI providers
  • many GHA actions, including some with elevated permissions for trivial stuff like auto-labeling
  • linting, formatting, documentation, static typing, code coverage, and benchmarking dev tools
  • many distros: Ubuntu, OpenSUSE, Fedora, Homebrew, Conda-forge, Cygwin, Pyodide
  • More YOLO installs (e.g., url -o /tmp/sde.tar.xz https://downloadmirror.intel.com/...)

@tacaswell
Copy link
Contributor

I am very supportive of this (and expect mpl will follow quickly/help).

An additional benefit is that we will be able to rebuild wheels with build numbers 1 if we had to update a vendored library, or re-build windows wheels because of c++ library things.

@andyfaff
Copy link
Member

What shall we call the repo, numpy-release? Could you create the repo Ralf?

@jorenham
Copy link
Member

What shall we call the repo, numpy-release?

numpy-build might also be worth considering

@rgommers
Copy link
Member Author

What shall we call the repo, numpy-release?

I had numpy-wheels in mind, but I like your suggestion better - numpy-release SGTM. numpy-build seems a bit too generic to me.

Could you create the repo Ralf?

Sure, but let's give it a couple more days to give people a change to comment. I'll also ping the mailing list about this proposal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants