Skip to content

Cross Compile for Android and issues with PyExc_OSError #111225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
GRRedWings opened this issue Oct 23, 2023 · 15 comments · Fixed by #115780
Closed

Cross Compile for Android and issues with PyExc_OSError #111225

GRRedWings opened this issue Oct 23, 2023 · 15 comments · Fixed by #115780
Labels
build The build process and cross-build OS-android type-bug An unexpected behavior, bug, or error

Comments

@GRRedWings
Copy link

GRRedWings commented Oct 23, 2023 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

In trying to run Python 3.12 on Android. I have been cross-compiling Python for a number of years. Having built a 64bit version I am getting the following error when I try to run

I get the following error at runtime.
ImportError('dlopen failed: cannot locate symbol "PyExc_OSError" referenced by "/data/data/com.myapp/files/Python64/lib-dynload/_socket.cpython-312.so"...')

I have verified that the _socket.cpython-3.12.so exists. I'm not really sure what other info would be beneficial for people to help me understand this error so I will start with this and happily add more details if requested.

CPython versions tested on:

3.12

Operating systems tested on:

Other

Linked PRs
@GRRedWings GRRedWings added the type-bug An unexpected behavior, bug, or error label Oct 23, 2023
@AlexWaygood AlexWaygood added the build The build process and cross-build label Oct 24, 2023
@GRRedWings
Copy link
Author

I'm not sure if this is useful, but I did find this in the Android NDK 26 notes

Issue 360: thread_local variables with non-trivial destructors will cause segfaults if the containing library is dlcloseed. This was fixed in API 28, but code running on devices older than API 28 will need a workaround. The simplest fix is to stop calling dlclose. If you absolutely must continue calling dlclose, see the following table:

@mhsmith mhsmith mentioned this issue Jan 15, 2024
Closed
@mhsmith
Copy link
Member

mhsmith commented Jan 16, 2024

Issue 360: thread_local variables with non-trivial destructors will cause segfaults if the containing library is dlcloseed

That doesn't seem relevant, because you're not seeing a segfault.

More likely, _socket.cpython-3.12.so has not been linked against libpython3.12.so. You can check this using readelf -d.

@erlend-aasland
Copy link
Contributor

@mhsmith, based on your comment, can we close this issue as not-planned?

@erlend-aasland erlend-aasland added the pending The issue will be closed if no feedback is provided label Jan 16, 2024
@GRRedWings
Copy link
Author

Here is the output I got from the readelf command. I'm not exactly sure how to read it, I do not see libpython3.12.so in there though, is this something I can change to link against it?

image

@mhsmith
Copy link
Member

mhsmith commented Jan 18, 2024

I've just remembered, I also encountered this problem when building Python 3.12 for Chaquopy. I was able to fix it with this patch, which I'll turn into a pull request in the near future.

@erlend-aasland erlend-aasland removed the pending The issue will be closed if no feedback is provided label Jan 18, 2024
@GRRedWings
Copy link
Author

GRRedWings commented Jan 18, 2024 •
edited
Loading

@mhsmith - Thanks for the update. I have tried your patch as is. Something is still different because it's looking for python3.12.so instead of libpython3.12.so.

I'm playing with your patch file, I'm not an expert but looking at it and trying to figure out what my build needs for some of your changes.

/android-ndk-r26b/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android26-clang -shared -L/python3-android/src/Python-3.12.1/Android/sysroot/usr/lib -pie Modules/arraymodule.o -lpython3.12 -o Modules/array.cpython-312.so
clang-17: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument]
ld.lld: error: unable to find library -lpython3.12

@mhsmith
Copy link
Member

mhsmith commented Jan 19, 2024 •
edited
Loading

It doesn't say it's looking for python3.12.so, it says it's looking for -lpython3.12, which on Android, just like Linux, should be transformed into libpython3.12.so.

The issue is probably that this file doesn't exist in any of the -L directories on your command line. In particular, I see that the command line doesn't contain -L., even though that should have been assigned to BLDLIBRARY here when ac_sys_system=Linux-android:

cpython/configure

Lines 7500 to 7502 in 2305ca5

Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*|VxWorks*)
LDLIBRARY='libpython$(LDVERSION).so'
BLDLIBRARY='-L. -lpython$(LDVERSION)'

Are you sure you applied the whole patch?

@GRRedWings
Copy link
Author

I am pretty confident that I applied the entire patch. What I'm not sure is how I can tell if ac_sys_system is being set right. Is there a log or anything I could put into the configure? I tried using an echo and a printf, but not having any luck.

@GRRedWings
Copy link
Author

In looking at this more, and doing a before and after, I can see that it is trying to apply the patch, I don't see errors, but the configure patch is not being applied. I will look into this and make sure the patch is being applied. Sorry about that.

@mhsmith
Copy link
Member

mhsmith commented Jan 21, 2024

I haven't looked at your build process, but maybe you're regenerating the configure script from the configure.ac file after applying the patch.

@GRRedWings
Copy link
Author

You are correct. I didn't realize the script I was using was doing an auto reconfigure, and I was applying the patch before this. Once I applied the patch in the proper place, things worked as expected.

I truly appreciate your help and time looking into this with me.

@mhsmith
Copy link
Member

mhsmith commented Jan 21, 2024

That's great, but please reopen the issue, because the original bug still exists in an unpatched version of Python.

@AlexWaygood AlexWaygood reopened this Jan 21, 2024
@mhsmith mhsmith mentioned this issue Feb 16, 2024
Closed
@bedevere-app bedevere-app bot mentioned this issue Feb 21, 2024
Merged
erlend-aasland pushed a commit that referenced this issue Feb 21, 2024
@mhsmith
gh-111225: Link extension modules against libpython on Android (#115780)
7f5e3f0 
Part of the work on PEP 738: Adding Android as a supported platform.

* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately 
  reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on 
  Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a 
  note that the toolchain used it automatically, but this is no longer
  the case. Omitting it causes all linker commands to fail with an error
  like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
  symbol '_Py_FalseStruct'; recompile with -fPIC`.
woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024
@mhsmith @woodruffw
pythongh-111225: Link extension modules against libpython on Android (p…
90ac592 
…ython#115780)

Part of the work on PEP 738: Adding Android as a supported platform.

* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately 
  reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on 
  Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a 
  note that the toolchain used it automatically, but this is no longer
  the case. Omitting it causes all linker commands to fail with an error
  like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
  symbol '_Py_FalseStruct'; recompile with -fPIC`.
@mhsmith mhsmith mentioned this issue Mar 11, 2024
Closed
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@mhsmith @diegorusso
pythongh-111225: Link extension modules against libpython on Android (p…
1bd2625 
…ython#115780)

Part of the work on PEP 738: Adding Android as a supported platform.

* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately 
  reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on 
  Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a 
  note that the toolchain used it automatically, but this is no longer
  the case. Omitting it causes all linker commands to fail with an error
  like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
  symbol '_Py_FalseStruct'; recompile with -fPIC`.
@GRRedWings
Copy link
Author

It's been a while, trying to build against 3.12.4, is this patch still needed?

@mhsmith
Copy link
Member

mhsmith commented Jul 9, 2024

It's still needed on Python 3.12 and older, but it won't be needed in Python 3.13.

@GRRedWings
Copy link
Author

Perfect, I figured as much but wanted to be sure. I appreciate it.

LukasWoodtli pushed a commit to LukasWoodtli/cpython that referenced this issue Jan 22, 2025
@mhsmith @LukasWoodtli
pythongh-111225: Link extension modules against libpython on Android (p…
b86840b 
…ython#115780)

Part of the work on PEP 738: Adding Android as a supported platform.

* Rename the LIBPYTHON variable to MODULE_LDFLAGS, to more accurately 
  reflect its purpose.
* Edit makesetup to use MODULE_LDFLAGS when linking extension modules.
* Edit the Makefile so that extension modules depend on libpython on 
  Android and Cygwin.
* Restore `-fPIC` on Android. It was removed several years ago with a 
  note that the toolchain used it automatically, but this is no longer
  the case. Omitting it causes all linker commands to fail with an error
  like `relocation R_AARCH64_ADR_PREL_PG_HI21 cannot be used against
  symbol '_Py_FalseStruct'; recompile with -fPIC`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
build The build process and cross-build OS-android type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
5 participants
@mhsmith @encukou @GRRedWings @erlend-aasland @AlexWaygood