Skip to content

Fix download of Windows Python toolchain on Linux #769

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 3, 2022
Merged

Fix download of Windows Python toolchain on Linux #769

merged 2 commits into from
Aug 3, 2022

Conversation

jesseschalken
Copy link
Contributor

@jesseschalken jesseschalken commented Jul 27, 2022 •
edited
Loading

PR Checklist

Please check if your PR fulfills the following requirements:

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature (please, look at the "Scope of the project" section in the README.md file)
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Documentation content changes
  • Other... Please describe:

What is the current behavior?

  1. python_repository rule crashes with: Failed to make interpreter installation read-only. 'chmod' error msg: chmod: cannot access 'lib': No such file or directory because the lib directory is actually called Lib
  2. The glob in the generated repo doesn't include the Lib directory causing no builtin libraries to be found when running the cross compiled py_binary.

Issue Number: N/A

What is the new behavior?

  1. python_repository rule implementation doesn't crash.
  2. Lib directory is included correctly.

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

@jesseschalken
Copy link
Contributor Author

I may need some help with the CI failure. The error is from build-runfiles on macOS:

mkdir 'python_x86_64-apple-darwin/lib': File exists

I don't see how this PR could have caused this, and I don't have a mac to test it on.

@UebelAndre
Copy link
Contributor

I may need some help with the CI failure. The error is from build-runfiles on macOS:

mkdir 'python_x86_64-apple-darwin/lib': File exists

I don't see how this PR could have caused this, and I don't have a mac to test it on.

Perhaps this is a flake? But MacOS is a case-insensitive filesystem by default so it can't distinguish the difference between lib and Lib. I feel like the core issue here is that the rule assumes the existence of a path based on the host OS. When instead Lib/lib should be explicitly determined by the platform attribute.

@jesseschalken
Copy link
Contributor Author

When instead Lib/lib should be explicitly determined by the platform attribute.

Isn't that what this PR does? https://github.com/bazelbuild/rules_python/pull/769/files#diff-2772f4c795e29614d2ca3c7f3302d558abe63bae2379d65285bcdf12e4ea469fR112

@UebelAndre
Copy link
Contributor

When instead Lib/lib should be explicitly determined by the platform attribute.

Isn't that what this PR does? https://github.com/bazelbuild/rules_python/pull/769/files#diff-2772f4c795e29614d2ca3c7f3302d558abe63bae2379d65285bcdf12e4ea469fR112

Ah yes, I misread the lines changed. Would you be willing to rebase and see if the failure is a flake?

@UebelAndre
Copy link
Contributor

UebelAndre commented Aug 2, 2022 •
edited
Loading

I think I found the issue and it's that if you glob for Lib and lib on a case-insensitive platform then both get included and you fail to create runfiles directories. Does something like the following patch help?

diff --git a/python/repositories.bzl b/python/repositories.bzl
index 8bb8a9c..a1fd4cd 100644
--- a/python/repositories.bzl
+++ b/python/repositories.bzl
@@ -131,6 +131,29 @@ def _python_repository_impl(rctx):
 
     python_bin = "python.exe" if ("windows" in platform) else "bin/python3"
 
+    if "windows" in platform:
+        glob_include = [
+            "*.exe",
+            "*.dll",
+            "bin/**",
+            "DLLs/**",
+            "extensions/**",
+            "include/**",
+            "Lib/**",
+            "libs/**",
+            "Scripts/**",
+            "share/**",
+        ]
+    else:
+        glob_include = [
+            "bin/**",
+            "extensions/**",
+            "include/**",
+            "lib/**",
+            "libs/**",
+            "share/**",
+        ]
+
     build_content = """\
 # Generated by python/repositories.bzl
 
@@ -141,18 +164,7 @@ package(default_visibility = ["//visibility:public"])
 filegroup(
     name = "files",
     srcs = glob(
-        include = [
-            "*.exe",
-            "*.dll",
-            "bin/**",
-            "DLLs/**",
-            "extensions/**",
-            "include/**",
-            "lib/**",
-            "libs/**",
-            "Scripts/**",
-            "share/**",
-        ],
+        include = {glob_include},
         # Platform-agnostic filegroup can't match on all patterns.
         allow_empty = True,
         exclude = [
@@ -206,6 +218,7 @@ py_runtime_pair(
     py3_runtime = ":py3_runtime",
 )
 """.format(
+        glob_include = json.encode(glob_include),
         python_path = python_bin,
         python_version = python_short_version,
     )

@jesseschalken jesseschalken force-pushed the fix-windows-toolchain-on-linux branch from 8507872 to 50ab2c6 Compare August 2, 2022 15:43
@jesseschalken
Copy link
Contributor Author

@UebelAndre Good spot. That fixed it. Thanks!

@UebelAndre
Copy link
Contributor

@groodt Do you have time to take a look at this one this week as well?

@UebelAndre UebelAndre mentioned this pull request Aug 2, 2022
Closed
groodt
groodt approved these changes Aug 3, 2022
View reviewed changes
Copy link
Collaborator

@groodt groodt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@groodt groodt merged commit 1045ca1 into bazel-contrib:main Aug 3, 2022
@jesseschalken jesseschalken deleted the fix-windows-toolchain-on-linux branch August 3, 2022 07:35
@UebelAndre UebelAndre mentioned this pull request Aug 19, 2022
Merged
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@groodt groodt groodt approved these changes

@brandjon brandjon Awaiting requested review from brandjon

@lberki lberki Awaiting requested review from lberki

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jesseschalken @UebelAndre @groodt