Description
Describe the issue:
Hi there, we've been facing an issue wherein we have a mismatch in the function signature for the scipy.linalg.logm function (internally, this is coming from the Schur decomposition and, in turn, from the dgees LAPACK routine) for WASM/Pyodide in pyodide/pyodide#4925 when NumPy is compiled with the Emscripten toolchain.
It is to be noted that Meson sets -fvisibility=hidden by default, thus overriding what emcc sets by default (which is -fvisibility=default, confirmed using emcc --flags) – so this is the same issue as pyodide/pyodide#4310 that has been previously reported: the _npy_f2py_ARRAY_API symbol is not exposed anymore (which it shouldn't be in a normal scenario, though).
Compiling against NumPy >=1.26.4,<=2.0.2 works fine; @rgommers has pointed out that this has been coming from the changes introduced via gh-26103 or gh-26286 that first appeared in NumPy versions 2.1.0rc1 and later—which I agree are good in principle—but we have been overriding the symbol visibility anyway post the introduction of pyodide/pyodide#4313, which is something that we ought to fix, hence while this issue may inspect on a problem with numpy.f2py, it's also centred on guidance how to proceed with compiling SciPy to work correctly going forward against changes in NumPy and whether we should retain a patch that undoes such changes.
More specifically, in the WASM code, logm is defined in the relevant memory address call_indirect (param i32) (result i32) – but the value in the WASM table returns null. For versions of NumPy that work, this symbol shows up with the name ndarray_from_pyobj.
cc: @rgommers, @ryanking13, @hoodmane
For a reproducer, it is currently required to compile SciPy in-tree, since we don't build NumPy v2 and dependent packages with it yet (hence the PR to update it). Here are all the prerequisites (some system-level packages might still be required via apt/yum/dnf/Homebrew):
# Set up Emscripten
git clone https://github.com/emscripten-core/emsdk.git ~/emsdk # or any other location
pushd ~/emsdk
./emsdk install 3.1.58
./emsdk activate 3.1.58
popd
source ./emsdk_env.sh
# Set up a development environment
git clone https://github.com/agriyakhetarpal/pyodide.git
git checkout update/numpy-v2
# Set up a debug build for Pyodide, libraries, and packages
python3.12 -m virtualenv .venv
source .venv/bin/activate
python -m pip install pyodide-build
PYODIDE_DEBUG=1 PYODIDE_JOBS=$(nproc) PYODIDE_PACKAGES="scipy" makeOnce OpenBLAS and other libraries + Emscripten wheels for NumPy and SciPy get compiled, we can use them:
# Create and activate Pyodide venv
pyodide venv .venv-pyodide
source .venv-pyodide/bin/activate
python -m pip install dist/*.whl # install SciPy and all other wheelsIn this activated Python/Pyodide interpreter, please run the following code to
Reproduce the code example:
import numpy as np
from scipy.linalg import logm
np.random.seed(1234)
n = 2
scale = 1e-4
A = (np.eye(n) + np.random.rand(n, n) * scale).astype(np.float64)
logm(A)Error message:
Pyodide has suffered a fatal error. Please report this to the Pyodide maintainers.
The cause of the fatal error was:
RuntimeError: null function or function signature mismatch
at wasm://wasm/0003f1c2:wasm-function[99]:0xb4ef
at invoke_iiiiiiii (/src/dist/pyodide.asm.js:116336:40)
at wasm://wasm/0060573e:wasm-function[225]:0x3d319
at wasm://wasm/0003f1c2:wasm-function[101]:0xc3b6
at _PyEM_TrampolineCall_JS (/src/dist/pyodide.asm.js:10307:33)
at pyodide.asm.wasm._PyObject_MakeTpCall (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[1155]:0x1cc781)
at pyodide.asm.wasm.PyObject_Vectorcall (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[1164]:0x1ccf92)
at pyodide.asm.wasm._PyEval_EvalFrameDefault (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[3432]:0x2aaefa)
at pyodide.asm.wasm.PyEval_EvalCode (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[3430]:0x2a0968)
at pyodide.asm.wasm.run_mod (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[4235]:0x313fc7) {
pyodide_fatal_error: true
}
Stack (most recent call first):
File "/src/.venv-pyodide/lib/python3.12/site-packages/scipy/linalg/_decomp_schur.py", line 149 in schur
File "/src/.venv-pyodide/lib/python3.12/site-packages/scipy/linalg/_matfuncs_inv_ssq.py", line 874 in _logm
File "/src/.venv-pyodide/lib/python3.12/site-packages/scipy/linalg/_matfuncs.py", line 200 in logm
File "<stdin>", line 1 in <module>
RuntimeError: null function or function signature mismatch
at wasm://wasm/0003f1c2:wasm-function[99]:0xb4ef
at invoke_iiiiiiii (/src/dist/pyodide.asm.js:116336:40)
at wasm://wasm/0060573e:wasm-function[225]:0x3d319
at wasm://wasm/0003f1c2:wasm-function[101]:0xc3b6
at _PyEM_TrampolineCall_JS (/src/dist/pyodide.asm.js:10307:33)
at pyodide.asm.wasm._PyObject_MakeTpCall (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[1155]:0x1cc781)
at pyodide.asm.wasm.PyObject_Vectorcall (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[1164]:0x1ccf92)
at pyodide.asm.wasm._PyEval_EvalFrameDefault (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[3432]:0x2aaefa)
at pyodide.asm.wasm.PyEval_EvalCode (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[3430]:0x2a0968)
at pyodide.asm.wasm.run_mod (wasm://wasm/pyodide.asm.wasm-02aa409a:wasm-function[4235]:0x313fc7) {
pyodide_fatal_error: true
}
Python and NumPy Versions:
2.1.3
3.12.7 (main, Nov 26 2024, 19:46:55) [Clang 19.0.0git (https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b
Runtime Environment:
[{'numpy_version': '2.1.3',
'python': '3.12.7 (main, Nov 26 2024, 19:46:55) [Clang 19.0.0git '
'(https:/github.com/llvm/llvm-project 0a8cd1ed1f4f35905df318015b',
'uname': uname_result(system='Emscripten', node='emscripten', release='3.1.58', version='#1', machine='wasm32')},
{'simd_extensions': {'baseline': [], 'found': [], 'not_found': []}}]
Context for the issue:
This affects pyodide/pyodide#4925, where we are trying to upgrade NumPy to version >=2, along with all its dependents.
We have some notes on debugging a function signature mismatch in Pyodide, which is where this bug report has originated from: https://pyodide.org/en/stable/development/debugging.html#debugging-runtimeerror-function-signature-mismatch
There are probably some useful insights from here onwards, too: pyodide/pyodide#4925 (comment)