Skip to content

Provide manylinux wheels for NumPy 2.3.0 on CPython 3.11 / AWS Lambda base image #29150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
akshit0201 opened this issue Jun 9, 2025 · 5 comments
Open

Provide manylinux wheels for NumPy 2.3.0 on CPython 3.11 / AWS Lambda base image #29150

akshit0201 opened this issue Jun 9, 2025 · 5 comments
Labels
32 - Installation Problems installing or compiling NumPy

Comments

@akshit0201
Copy link

Steps to reproduce:

  1. Use the public.ecr.aws/lambda/python:3.11 Docker base image.
  2. In a Dockerfile, copy in a requirements.txt containing numpy
  3. Run pip install -r requirements.txt.
  4. Observe that pip falls back to building NumPy 2.3.0 from source and fails due to missing C compiler.

Error message:

#7 2.842   Downloading numpy-2.3.0.tar.gz (20.4 MB)
#7 2.990      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.4/20.4 MB 132.5 MB/s eta 0:00:00
#7 5.039   Installing build dependencies: started
#7 7.540   Installing build dependencies: finished with status 'done'
#7 7.540   Getting requirements to build wheel: started
#7 7.639   Getting requirements to build wheel: finished with status 'done'
#7 7.644   Installing backend dependencies: started
#7 8.255   Installing backend dependencies: finished with status 'done'
#7 8.256   Preparing metadata (pyproject.toml): started
#7 9.146   Preparing metadata (pyproject.toml): finished with status 'error'
#7 9.151   error: subprocess-exited-with-error
#7 9.151   
#7 9.151   × Preparing metadata (pyproject.toml) did not run successfully.
#7 9.151   │ exit code: 1
#7 9.151   ╰─> [20 lines of output]
#7 9.151       + /var/lang/bin/python3.11 /tmp/pip-install-pv61bimt/numpy_a2374ad6e38d46daba33ed2d9893ca9c/vendored-meson/meson/meson.py setup /tmp/pip-install-pv61bimt/numpy_a2374ad6e38d46daba33ed2d9893ca9c /tmp/pip-install-pv61bimt/numpy_a2374ad6e38d46daba33ed2d9893ca9c/.mesonpy-rw0po2ou -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-pv61bimt/numpy_a2374ad6e38d46daba33ed2d9893ca9c/.mesonpy-rw0po2ou/meson-python-native-file.ini
#7 9.151       The Meson build system
#7 9.151       Version: 1.6.1
#7 9.151       Source dir: /tmp/pip-install-pv61bimt/numpy_a2374ad6e38d46daba33ed2d9893ca9c
#7 9.151       Build dir: /tmp/pip-install-pv61bimt/numpy_a2374ad6e38d46daba33ed2d9893ca9c/.mesonpy-rw0po2ou
#7 9.151       Build type: native build
#7 9.151       Project name: NumPy
#7 9.151       Project version: 2.3.0
#7 9.151       
#7 9.151       ../meson.build:1:0: ERROR: Unknown compiler(s): [['cc'], ['gcc'], ['clang'], ['nvc'], ['pgcc'], ['icc'], ['icx']]
#7 9.151       The following exception(s) were encountered:
#7 9.151       Running `cc --version` gave "[Errno 2] No such file or directory: 'cc'"
#7 9.151       Running `gcc --version` gave "[Errno 2] No such file or directory: 'gcc'"
#7 9.151       Running `clang --version` gave "[Errno 2] No such file or directory: 'clang'"
#7 9.151       Running `nvc --version` gave "[Errno 2] No such file or directory: 'nvc'"
#7 9.151       Running `pgcc --version` gave "[Errno 2] No such file or directory: 'pgcc'"
#7 9.151       Running `icc --version` gave "[Errno 2] No such file or directory: 'icc'"
#7 9.151       Running `icx --version` gave "[Errno 2] No such file or directory: 'icx'"
#7 9.151       
#7 9.151       A full log can be found at /tmp/pip-install-pv61bimt/numpy_a2374ad6e38d46daba33ed2d9893ca9c/.mesonpy-rw0po2ou/meson-logs/meson-log.txt
#7 9.151       [end of output]
#7 9.151   
#7 9.151   note: This error originates from a subprocess, and is likely not a problem with pip.
#7 9.153 error: metadata-generation-failed
#7 9.153 
#7 9.153 × Encountered error while generating package metadata.
#7 9.153 ╰─> See above for output.
#7 9.153 
#7 9.153 note: This is an issue with the package mentioned above, not pip.
#7 9.153 hint: See above for details.
#7 9.162 
#7 9.162 [notice] A new release of pip is available: 24.0 -> 25.1.1
#7 9.162 [notice] To update, run: pip install --upgrade pip
#7 ERROR: process "/bin/sh -c pip install -r requirements.txt" did not complete successfully: exit code: 1
------
 > [3/5] RUN pip install -r requirements.txt:
9.153 error: metadata-generation-failed
9.153 
9.153 × Encountered error while generating package metadata.
9.153 ╰─> See above for output.
9.153 
9.153 note: This is an issue with the package mentioned above, not pip.
9.153 hint: See above for details.

Additional information:

No response
@akshit0201 akshit0201 added the 32 - Installation Problems installing or compiling NumPy label Jun 9, 2025
@pangolinkeys
Copy link

pangolinkeys commented Jun 9, 2025 •
edited
Loading

We've also started experiencing this issue this morning.

      The Meson build system
      Version: 1.6.1
      Source dir: /tmp/pip-install-gf4doqkk/numpy_717adc4dac3e4268ac40ff583ce5715b
      Build dir: /tmp/pip-install-gf4doqkk/numpy_717adc4dac3e4268ac40ff583ce5715b/.mesonpy-0b0zxh5d
      Build type: native build
      Project name: NumPy
      Project version: 2.3.0
      C compiler for the host machine: cc (gcc 7.3.1 "cc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)")
      C linker for the host machine: cc ld.bfd 2.29.1-31
      C++ compiler for the host machine: c++ (gcc 7.3.1 "c++ (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17)")
      C++ linker for the host machine: c++ ld.bfd 2.29.1-31
      Cython compiler for the host machine: cython (cython 3.1.2)
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      
      ../meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 9.3

Our build step uses public.ecr.aws/sam/build-python3.11:latest-x86_64 to install dependencies so we do have a GCC compiler but the wrong version.

Edit: We are just going to pin 2.2.6 for now

@mattip
Copy link
Member

mattip commented Jun 9, 2025

We provide wheels for manylinux_2_28. Perhaps you are using an older amazon linux2 image, which does not support manylinux_2_28?

@cedricfarinazzo
Copy link

cedricfarinazzo commented Jun 9, 2025 •
edited
Loading

I had the same error with the image public.ecr.aws/lambda/python:3.11 and numpy==2.3.0.

I temporarily fixed it by installing GCC 10:

RUN yum install -y make cmake libc-dev gcc10.$(uname -m) gcc10-c++.$(uname -m)
ENV CC=/usr/bin/gcc10-gcc
ENV CXX=/usr/bin/gcc10-g++
ENV AR=/usr/bin/gcc10-ar
ENV AS=/usr/bin/gcc10-as

Now numpy==2.3.0 installs correctly, but the build takes 10 minutes because it compiles NumPy from source.

When using numpy==2.2.6, the wheel filename is numpy-2.2.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
The job building wheel for manylinux_2_17_x86_64: https://github.com/numpy/numpy/actions/runs/15086533833/job/42411405542

On numpy==2.3.0, i can only see build job for manylinux_2_28 wheels.

So I think manylinux_2_17 wheel is missing for numpy==2.3.0 and must be kept for retrocompatibility.

@andyfaff
Copy link
Member

andyfaff commented Jun 9, 2025

The 2_17 target is end of life. To use a pre built wheel you either need to use an older numpy version, or use newer Linux image to get the newest version.
Alternatively you can build from source. Since the image is EOL we won't be making a 2_17 wheel again.

A rough guide is https://github.com/mayeut/pep600_compliance/blob/master/EOL.rst, https://github.com/pypa/manylinux

@mattip
Copy link
Member

mattip commented Jun 9, 2025

The 2_17 target is end of life

Unfortunately the EOL date for amazon linux 2 was delayed to June 2026. This puts it in a difficult position since it is based on glibc 2.26, which means users of this platform will not be able to install latest versions of many scientific python packages, including NumPy and pytorch.

Do we have any knowledge of when aws will update its base images off amazon linux 2?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
32 - Installation Problems installing or compiling NumPy
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@andyfaff @mattip @pangolinkeys @cedricfarinazzo @akshit0201