gh-137242: Add Android CI job #137186

mhsmith · 2025-07-28T19:15:14Z

See the linked issue for details:

Issue: Android CI and binary releases #137242

mhsmith · 2025-07-30T13:12:43Z

The logs are showing a number of "Scudo OOM" warnings similar to those discussed in #121595. However, none of them have been fatal yet.

In API level 35 the memory allocator no longer prints such an alarming message when this happens. Instead it will just be a one line message like this (llvm/llvm-project#68444):

I/scudo   : Can't populate more pages for size class 8720.

Luckily, in API level 35 they've also added the ability to print the allocator statistics at any time, with code like this (llvm/llvm-project@5759e3c):

import ctypes
ctypes.CDLL(None).mallopt(-205, 0)

Based on those statistics, it doesn't look like Python has a memory leak, as the "inuse" column after the test is roughly the same as it was before. So it's probably just temporary high memory use during the test, causing allocations to be directed to the next highest size class.

On the buildbot about a year ago, these warnings were often associated with crashes. As described in Android/README.md, we worked around this by running the emulator once to create it, then editing its configuration files to increase the emulator RAM from 2 GB to 4 GB. After that, the warnings continued, but the crashes stopped.

But that isn't such a good option on GitHub Actions, because the runner is not persistent, so the extra emulator restart would add about a minute to every CI run. In theory you should be able to avoid this by creating the emulator from a custom hardware profile, but in my experiments, the emulator always ended up with 2 GB of RAM regardless of what it said in the profile.

The tests that trigger this are:

test.test_ast.test_ast.AST_Tests.test_ast_recursion_limit
test.test_compile.TestSpecifics.test_compiler_recursion_limit
test.test_compile.TestSpecifics.test_stack_overflow

These tests all involve parsing an extremely long Python statement. Although they're all skipped on WASM because of limited stack space, reducing the recursion limit does not prevent the warning. What does prevent it is reducing the size of the Python statement, e.g. from 500,000 repetitions to 200,000.

Since I haven't seen these tests crash on GitHub Actions yet, I'm not going to change them just now. If they do start crashing, we can change them to reduce the statement size on Android alone, or even on all platforms if it doesn't significantly reduce the strength of the test.

freakboy3742

One fairly minor comment inline; my other comment is an extension of that one.

Although the action is broken up into steps, the context of those steps is lost when it's embedded in the workflow. Aside from the general ergonomics of being able to follow what stage the workflow is up to, there's very little visibility on the fact that some steps aren't be in invoked - unless you pay close attention, it's not clear that the arm64 run isn't running the tests. I know it's not, and why - but there's no external visibility of that fact (like a clearly skipped build step).

If the constraint against using a workflow is that we would need to reference a specific commit - isn't that commit hash one of the inputs to the release workflow? Does uses not accept variable expansions in workflow references?

.github/workflows/build.yml

freakboy3742 · 2025-07-31T02:57:06Z

.github/actions/build-android/action.yml

+# This is coded as an action rather than a workflow so the release-tools
+# repository can load it from a dynamically-chosen commit. Cross-repository
+# workflow calls must have a Git reference hard-coded in the calling workflow,
+# but actions can be run dynamically from the runner's filesystem.


This makes sense; but it also raises an eyebrow because this is something that no other platform has needed. I presume this is because no other platform that is generating binary artefacts is doing so with the tooling in release-tools?

@webknjaz, do you have any thougts here?

I'll take a look, thanks for tagging.

@mhsmith so I've tried to understand the context and this justtification didn't make sense to me.

It is perfectly possible to call reusable workflows from other repositories (in fact, this is what I'm building my reusable-tox.yml ecosystem on).

My understanding is that this is meant to be used in https://github.com/python/release-tools/blob/698deaf2ebff433a6ab9d4b5ded97a40fce109a1/.github/workflows/source-and-docs-release.yml, right?

In any case, I've been moving away from composite actions in favor of reusable workflows. This is because composite actions (or any actions for that matter) are represented as a single step in job runs. And it's rather difficult to inspect what actions are doing. So from the troubleshooting perspective, I'd strongly advise against composite actions.

It is important to make every step visible and transparent. And if you follow the practice I established with reusable workflows as modules in here, this is definitely possible.

I started with in-repo "modules" two years ago because I was solving the problem of making it possible to collect all the job statuses in a single check (for branch protection). This wasn't because it's somehow impossible to host them externally. This was just not something necessary for that purpose.

@encukou I've actually been meaning to ask if there's any workflows that are being duplicated in the python org. If yes, it'd make sense to host them in a shared repository. This could be a .github repo or even that release-tools one (although, I don't know if it makes semantic sense). This is a separate question, though.

That said, if you've faced any confusion or need help adapting this to be a reusable workflow, just let me know where you're stuck. I can help you navigate this or just rewrite it for you.

This makes sense; but it also raises an eyebrow because this is something that no other platform has needed. I presume this is because no other platform that is generating binary artefacts is doing so with the tooling in release-tools?

Right now, release-tools only creates source zips and docs artifacts. See https://github.com/python/release-tools/actions/runs/16450411678 for 3.14 RC2.

The Windows artifacts are built in Azure Pipelines, here's RC2. And Ned builds the macOS artifacts. (release-tools later takes these Windows and macOS artifacts and signs and uploads them.)

We're hoping to build the macOS artifacts using CI in the near future, so what we decide here may help inform how to do that to :)

My understanding is that this is meant to be used in python/release-tools@698deaf/.github/workflows/source-and-docs-release.yml, right?

Yes, see https://github.com/python/release-tools/pull/265/files#diff-4d14704b6b88fb06db888f96c03a8e9b3a5e07a4ee566d97d4111b2c05210e84R220.

One thing I'd like to see is CI in this repo to check Android builds okay, so we don't get caught ought on release day because we only build in release-tools.

This happened in 3.14 RC1 with the plaintext docs, which had broken back in April or so, but RC1 is the first prerelease to build docs and we hadn't caught it here.

One deciding factor for whether we have stuff over here (via composite actions or something else) might be how much difference there'll be between different versions (3.14, 3.15, etc). If a lot, we might not want it all in release-tools and would benefit from versioning things in branches over here.

On the other hand, the docs build is also versioned in branches here, release-tools CI calls a make dist command in this repo.

On the other hand, the docs build is also versioned in branches here, release-tools CI calls a make dist command in this repo.

That's more or less the way I've been trying to suggest.

bedevere-app · 2025-07-31T03:17:50Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Android/android.py

.pre-commit-config.yaml

webknjaz · 2025-07-31T15:13:52Z

Does uses not accept variable expansions in workflow references?

@freakboy3742 it does not. But I don't think that it's neccessary or that composite actions would be any different in this regard.

mhsmith · 2025-08-09T19:39:31Z

Thanks for your ideas, everyone. I've now added an android.py ci command which performs the build, package and test sequence. This has several advantages:

The command is so simple, that there's no need to wrap it in a reusable action or workflow; the cpython and release-tools repositories can invoke it directly.
It's much easier to test locally than a script in a workflow file.
It avoids the circular dependency we'd have with make, because the Makefile doesn't exist until android.py runs configure.
cibuildwheel uses the same android.py test script, so handling the GitHub Actions KVM setup at this level will save all the package maintainers from needing to do it manually.

I have made the requested changes; please review again.

I've also made the corresponding changes to python/release-tools#265.

bedevere-app · 2025-08-09T19:39:36Z

Thanks for making the requested changes!

@freakboy3742: please review the changes made to this pull request.

freakboy3742

This looks pretty good to me - the CI configuration is as about as simple as it could get; the output in CI is very navigable; and end users testing under Github will get the added benefit of not needing explicit CI configuration.

I'm happy to approve as is; holding off merging until @webknjaz has had a chance to take a look.

webknjaz

I would still like this job to be wrapped as a reusable workflow:

for in-repo use — I'd like to eventually get to a point where build.yml would only declare releationships between the jobs and matrices, keeping any steps in reusable-*.yml workflows; this is to unify the structure better + have cleaner separation of concerns/abstraction layers
this would allow having a visually collapsible group of jobs in the UI, consistent with how other groups look
with 3-4 inputs in it, it would allow relying on the same steps in python/release-tools (if an optional actions/upload-artifact is added)
the benefits of having the step wrapped in android.py would remain

However, I think it's okay to merge the PR as is and I could send a follow-up with such a change. I only ask the matrix definition to be fixed first.

.github/workflows/build.yml

webknjaz

Let's get this in and I'll look into wrapping it as a reusable workflow later.

hugovk

Thank you!

mhsmith · 2025-08-12T09:38:21Z

@hugovk: Are you able to merge this?

miss-islington-app · 2025-08-12T17:16:08Z

Thanks @mhsmith for the PR, and @hugovk for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

miss-islington-app · 2025-08-12T17:16:11Z

Sorry, @mhsmith and @hugovk, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker f660ec37531b5e368a27ba065f73d31ff6fb6680 3.14

hugovk · 2025-08-12T17:16:14Z

Yes, thanks!

miss-islington-app · 2025-08-12T17:16:17Z

Sorry, @mhsmith and @hugovk, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker f660ec37531b5e368a27ba065f73d31ff6fb6680 3.13

(cherry picked from commit f660ec3) Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Russell Keith-Magee <russell@keith-magee.com> Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

bedevere-app · 2025-08-12T17:17:59Z

GH-137683 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2025-08-12T17:19:29Z

GH-137684 is a backport of this pull request to the 3.13 branch.

(cherry picked from commit f660ec3) Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Russell Keith-Magee <russell@keith-magee.com> Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

Add Android CI job

97ffb3e

StanFromIreland added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Jul 28, 2025

mhsmith added 4 commits July 28, 2025 20:28

Add shell to each step of the composite action

bd7b642

Use single quotes

1d8dc2f

Use different artifact names in each matrix job

51ee923

Use similar arguments to --fast-ci

0d1b6d1

mhsmith added the skip news label Jul 28, 2025

mhsmith added 2 commits July 30, 2025 14:56

Suppress make install output by default

67c7af3

Strip debug information from release package by default

16f1230

mhsmith added a commit to mhsmith/release-tools that referenced this pull request Jul 30, 2025

Temporarily point inputs at python/cpython#137186

081e3be

mhsmith changed the title ~~Add Android CI job~~ gh-137242: Add Android CI job Jul 30, 2025

bedevere-app bot mentioned this pull request Jul 30, 2025

Android CI and binary releases #137242

Open

mhsmith added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jul 30, 2025

mhsmith marked this pull request as ready for review July 30, 2025 16:30

mhsmith requested review from freakboy3742, hugovk, AlexWaygood, ezio-melotti and AA-Turner as code owners July 30, 2025 16:30

bedevere-app bot added the awaiting review label Jul 30, 2025

freakboy3742 requested changes Jul 31, 2025

View reviewed changes

bedevere-app bot added awaiting changes and removed awaiting review labels Jul 31, 2025

freakboy3742 mentioned this pull request Jul 31, 2025

Add Android support python/release-tools#265

Draft

webknjaz reviewed Jul 31, 2025

View reviewed changes

Android/android.py Outdated Show resolved Hide resolved

webknjaz reviewed Jul 31, 2025

View reviewed changes

.pre-commit-config.yaml Outdated Show resolved Hide resolved

mhsmith added 2 commits August 9, 2025 19:51

Convert action into an android.py ci command

b3daf4b

Use sudo when updating udev rules

c9700e4

bedevere-app bot added awaiting change review and removed awaiting changes labels Aug 9, 2025

bedevere-app bot requested a review from freakboy3742 August 9, 2025 19:39

mhsmith added the OS-android label Aug 10, 2025

freakboy3742 approved these changes Aug 11, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting change review labels Aug 11, 2025

webknjaz reviewed Aug 11, 2025

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

mhsmith added 2 commits August 11, 2025 13:25

Adjust matrix generation

e8e2a9e

Fix whitespace

7e87375

mhsmith requested a review from webknjaz August 11, 2025 13:20

webknjaz approved these changes Aug 11, 2025

View reviewed changes

hugovk approved these changes Aug 11, 2025

View reviewed changes

hugovk merged commit f660ec3 into python:main Aug 12, 2025
44 checks passed

bedevere-app bot removed the awaiting merge label Aug 12, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Aug 12, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Aug 12, 2025

Uh oh!

gh-137242: Add Android CI job #137186

gh-137242: Add Android CI job #137186

Conversation

mhsmith commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhsmith commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

freakboy3742 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

freakboy3742 Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

encukou Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

webknjaz Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

webknjaz Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

hugovk Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

hugovk Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

webknjaz Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

bedevere-app bot commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

webknjaz commented Jul 31, 2025

Uh oh!

mhsmith commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Aug 9, 2025

Uh oh!

freakboy3742 left a comment

Choose a reason for hiding this comment

Uh oh!

webknjaz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

webknjaz left a comment

Choose a reason for hiding this comment

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

mhsmith commented Aug 12, 2025

Uh oh!

Uh oh!

miss-islington-app bot commented Aug 12, 2025

Uh oh!

miss-islington-app bot commented Aug 12, 2025

Uh oh!

hugovk commented Aug 12, 2025

Uh oh!

miss-islington-app bot commented Aug 12, 2025

Uh oh!

bedevere-app bot commented Aug 12, 2025

Uh oh!

bedevere-app bot commented Aug 12, 2025

Uh oh!

Uh oh!

mhsmith commented Jul 28, 2025 •

edited

Loading

mhsmith commented Jul 30, 2025 •

edited

Loading

mhsmith commented Aug 9, 2025 •

edited

Loading

webknjaz left a comment •

edited

Loading