Skip to content

Commit e62b671

Browse files
chrislovecnmchrislovecnm
committed
feat(bzlmod): Moving register.toolchains internal
This commit moves the register.toolchains bzlmod call to inside of rules_python. Instead of a user having to call register.toolchains in their MODULE.bazel, rules_python/MODULE.bazel calls it on the internal hub. Updated various examples and removed register.toolchain calls. Include new tests in the bzlmod example in order to test for multiple toolchains.
1 parent 62e95a4 commit e62b671

File tree

20 files changed

+579
-229
lines changed

20 files changed

+579
-229
lines changed

‎.bazelrc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
# This lets us glob() up all the files inside the examples to make them inputs to tests
44
# (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
55
# To update these lines, run tools/bazel_integration_test/update_deleted_packages.sh
6-
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7-
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
6+
build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
7+
query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
88

99
test --test_output=errors
1010

‎BZLMOD_SUPPORT.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ A second example, in [examples/bzlmod_build_file_generation](examples/bzlmod_bui
3131

3232
This rule set does not have full feature partity with the older `WORKSPACE` type configuration:
3333

34-
1. Multiple python versions are not yet supported, as demonstrated in [this](examples/multi_python_versions) example.
34+
1. Multiple pip extensions are not yet supported, as demonstrated in [this](examples/multi_python_versions) example.
3535
2. Gazelle does not support finding deps in sub-modules. For instance we can have a dep like ` "@our_other_module//other_module/pkg:lib",` in a `py_test` definition.
3636

3737
Check ["issues"](/bazelbuild/rules_python/issues) for an up to date list.

‎MODULE.bazel

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,5 +47,10 @@ use_repo(
4747
"pypi__coverage_cp39_x86_64-unknown-linux-gnu",
4848
)
4949

50+
# We need to do another use_extension call to expose the "pythons_hub"
51+
# repo.
5052
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
5153
use_repo(python, "pythons_hub")
54+
55+
# This call registers the Python toolchains.
56+
register_toolchains("@pythons_hub//:all")

‎README.md

Lines changed: 22 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -53,32 +53,39 @@ To import rules_python in your project, you first need to add it to your
5353

5454
To register a hermetic Python toolchain rather than rely on a system-installed interpreter for runtime execution, you can add to the `MODULE.bazel` file:
5555

56-
```python
56+
```starlark
5757
# Find the latest version number here: https://github.com/bazelbuild/rules_python/releases
5858
# and change the version number if needed in the line below.
59-
bazel_dep(name = "rules_python", version = "0.20.0")
59+
bazel_dep(name = "rules_python", version = "0.21.0")
60+
61+
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
62+
python.toolchain(
63+
name = "python",
64+
configure_coverage_tool = True,
65+
is_default = True,
66+
python_version = "3.9",
67+
)
6068

61-
# You do not have to use pip for the toolchain, but most people
62-
# will use it for the dependency management.
63-
pip = use_extension("@rules_python//python:extensions.bzl", "pip")
69+
interpreter = use_extension("@rules_python//python/extensions:interpreter.bzl", "interpreter")
70+
interpreter.install(
71+
name = "interpreter",
72+
python_name = "python",
73+
)
74+
use_repo(interpreter, "interpreter")
6475

76+
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
6577
pip.parse(
6678
name = "pip",
79+
incompatible_generate_aliases = True,
80+
python_interpreter_target = "@interpreter//:python",
6781
requirements_lock = "//:requirements_lock.txt",
82+
requirements_windows = "//:requirements_windows.txt",
6883
)
69-
7084
use_repo(pip, "pip")
71-
72-
# Register a specific python toolchain instead of using the host version
73-
python = use_extension("@rules_python//python:extensions.bzl", "python")
74-
75-
use_repo(python, "python3_10_toolchains")
76-
77-
register_toolchains(
78-
"@python3_10_toolchains//:all",
79-
)
8085
```
8186

87+
For more documentation see the bzlmod examples under the [examples](examples) folder.
88+
8289
### Using a WORKSPACE file
8390

8491
To import rules_python in your project, you first need to add it to your

‎examples/bzlmod/BUILD.bazel

Lines changed: 0 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,6 @@
88
load("@bazel_skylib//rules:build_test.bzl", "build_test")
99
load("@pip//:requirements.bzl", "all_requirements", "all_whl_requirements", "requirement")
1010
load("@python_39//:defs.bzl", py_test_with_transition = "py_test")
11-
12-
# This is not working yet till the toolchain hub registration is working
13-
# load("@python_310//:defs.bzl", py_binary_310 = "py_binary")
1411
load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
1512
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
1613

@@ -50,22 +47,6 @@ py_binary(
5047
],
5148
)
5249

53-
# This is still WIP. Not working till we have the toolchain
54-
# registration functioning.
55-
56-
# This is used for testing mulitple versions of Python. This is
57-
# used only when you need to support multiple versions of Python
58-
# in the same project.
59-
# py_binary_310(
60-
# name = "main_310",
61-
# srcs = ["__main__.py"],
62-
# main = "__main__.py",
63-
# visibility = ["//:__subpackages__"],
64-
# deps = [
65-
# ":lib",
66-
# ],
67-
# )
68-
6950
# see https://bazel.build/reference/be/python#py_test
7051
py_test(
7152
name = "test",

‎examples/bzlmod/MODULE.bazel

Lines changed: 3 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,10 @@ local_path_override(
1515
# We also use the same value in the python.host_python_interpreter call.
1616
PYTHON_NAME_39 = "python_39"
1717

18-
PYTHON_39_TOOLCHAINS = PYTHON_NAME_39 + "_toolchains"
19-
2018
INTERPRETER_NAME_39 = "interpreter_39"
2119

2220
PYTHON_NAME_310 = "python_310"
2321

24-
PYTHON_310_TOOLCHAINS = PYTHON_NAME_310 + "_toolchains"
25-
2622
INTERPRETER_NAME_310 = "interpreter_310"
2723

2824
# We next initialize the python toolchain using the extension.
@@ -50,25 +46,9 @@ python.toolchain(
5046
python_version = "3.10",
5147
)
5248

53-
# use_repo imports one or more repos generated by the given module extension
54-
# into the scope of the current module. We are importing the various repos
55-
# created by the above python.toolchain calls.
56-
use_repo(
57-
python,
58-
PYTHON_NAME_39,
59-
PYTHON_39_TOOLCHAINS,
60-
PYTHON_NAME_310,
61-
PYTHON_310_TOOLCHAINS,
62-
)
63-
64-
# This call registers the Python toolchains.
65-
# Note: there is work under way to move this code to within
66-
# rules_python, and the user won't have to make this call,
67-
# unless they are registering custom toolchains.
68-
register_toolchains(
69-
"@{}//:all".format(PYTHON_39_TOOLCHAINS),
70-
"@{}//:all".format(PYTHON_310_TOOLCHAINS),
71-
)
49+
# You only need to load this repositories if you are using muiltple Python versions.
50+
# See the tests folder for various examples.
51+
use_repo(python, PYTHON_NAME_39, "python_aliases")
7252

7353
# The interpreter extension discovers the platform specific Python binary.
7454
# It creates a symlink to the binary, and we pass the label to the following

‎examples/bzlmod/other_module/MODULE.bazel

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,8 @@ bazel_dep(name = "rules_python", version = "")
1212
# testing purposes.
1313
PYTHON_NAME_39 = "python_39"
1414

15-
PYTHON_39_TOOLCHAINS = PYTHON_NAME_39 + "_toolchains"
16-
1715
PYTHON_NAME_311 = "python_311"
1816

19-
PYTHON_311_TOOLCHAINS = PYTHON_NAME_311 + "_toolchains"
20-
2117
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
2218
python.toolchain(
2319
# This name is used in the various use_repo statements
@@ -42,16 +38,5 @@ python.toolchain(
4238
use_repo(
4339
python,
4440
PYTHON_NAME_39,
45-
PYTHON_39_TOOLCHAINS,
4641
PYTHON_NAME_311,
47-
PYTHON_311_TOOLCHAINS,
48-
)
49-
50-
# This call registers the Python toolchains.
51-
# Note: there is work under way to move this code to within
52-
# rules_python, and the user won't have to make this call,
53-
# unless they are registering custom toolchains.
54-
register_toolchains(
55-
"@{}//:all".format(PYTHON_39_TOOLCHAINS),
56-
"@{}//:all".format(PYTHON_311_TOOLCHAINS),
5742
)

‎examples/bzlmod/tests/BUILD.bazel

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
load("@python_aliases//3.10:defs.bzl", py_binary_3_10 = "py_binary", py_test_3_10 = "py_test")
2+
load("@python_aliases//3.11:defs.bzl", py_binary_3_11 = "py_binary", py_test_3_11 = "py_test")
3+
load("@python_aliases//3.9:defs.bzl", py_binary_3_9 = "py_binary", py_test_3_9 = "py_test")
4+
load("@rules_python//python:defs.bzl", "py_binary", "py_test")
5+
6+
py_binary(
7+
name = "version_default",
8+
srcs = ["version.py"],
9+
main = "version.py",
10+
)
11+
12+
py_binary_3_9(
13+
name = "version_3_9",
14+
srcs = ["version.py"],
15+
main = "version.py",
16+
)
17+
18+
py_binary_3_10(
19+
name = "version_3_10",
20+
srcs = ["version.py"],
21+
main = "version.py",
22+
)
23+
24+
py_binary_3_11(
25+
name = "version_3_11",
26+
srcs = ["version.py"],
27+
main = "version.py",
28+
)
29+
30+
# This is a work in progress and the commented
31+
# tests will not work until we can support
32+
# multiple pips with bzlmod.
33+
34+
#py_test(
35+
# name = "my_lib_default_test",
36+
# srcs = ["my_lib_test.py"],
37+
# main = "my_lib_test.py",
38+
# deps = ["//libs/my_lib"],
39+
#)
40+
41+
#py_test_3_9(
42+
# name = "my_lib_3_9_test",
43+
# srcs = ["my_lib_test.py"],
44+
# main = "my_lib_test.py",
45+
# deps = ["//libs/my_lib"],
46+
#)
47+
48+
#py_test_3_10(
49+
# name = "my_lib_3_10_test",
50+
# srcs = ["my_lib_test.py"],
51+
# main = "my_lib_test.py",
52+
# deps = ["//libs/my_lib"],
53+
#)
54+
55+
#py_test_3_11(
56+
# name = "my_lib_3_11_test",
57+
# srcs = ["my_lib_test.py"],
58+
# main = "my_lib_test.py",
59+
# deps = ["//libs/my_lib"],
60+
#)
61+
62+
py_test(
63+
name = "version_default_test",
64+
srcs = ["version_test.py"],
65+
env = {"VERSION_CHECK": "3.9"}, # The default defined in the WORKSPACE.
66+
main = "version_test.py",
67+
)
68+
69+
py_test_3_9(
70+
name = "version_3_9_test",
71+
srcs = ["version_test.py"],
72+
env = {"VERSION_CHECK": "3.9"},
73+
main = "version_test.py",
74+
)
75+
76+
py_test_3_10(
77+
name = "version_3_10_test",
78+
srcs = ["version_test.py"],
79+
env = {"VERSION_CHECK": "3.10"},
80+
main = "version_test.py",
81+
)
82+
83+
py_test_3_11(
84+
name = "version_3_11_test",
85+
srcs = ["version_test.py"],
86+
env = {"VERSION_CHECK": "3.11"},
87+
main = "version_test.py",
88+
)
89+
90+
py_test(
91+
name = "version_default_takes_3_10_subprocess_test",
92+
srcs = ["cross_version_test.py"],
93+
data = [":version_3_10"],
94+
env = {
95+
"SUBPROCESS_VERSION_CHECK": "3.10",
96+
"SUBPROCESS_VERSION_PY_BINARY": "$(rootpath :version_3_10)",
97+
"VERSION_CHECK": "3.9",
98+
},
99+
main = "cross_version_test.py",
100+
)
101+
102+
py_test_3_10(
103+
name = "version_3_10_takes_3_9_subprocess_test",
104+
srcs = ["cross_version_test.py"],
105+
data = [":version_3_9"],
106+
env = {
107+
"SUBPROCESS_VERSION_CHECK": "3.9",
108+
"SUBPROCESS_VERSION_PY_BINARY": "$(rootpath :version_3_9)",
109+
"VERSION_CHECK": "3.10",
110+
},
111+
main = "cross_version_test.py",
112+
)
113+
114+
sh_test(
115+
name = "version_test_binary_default",
116+
srcs = ["version_test.sh"],
117+
data = [":version_default"],
118+
env = {
119+
"VERSION_CHECK": "3.9", # The default defined in the WORKSPACE.
120+
"VERSION_PY_BINARY": "$(rootpath :version_default)",
121+
},
122+
)
123+
124+
sh_test(
125+
name = "version_test_binary_3_9",
126+
srcs = ["version_test.sh"],
127+
data = [":version_3_9"],
128+
env = {
129+
"VERSION_CHECK": "3.9",
130+
"VERSION_PY_BINARY": "$(rootpath :version_3_9)",
131+
},
132+
)
133+
134+
sh_test(
135+
name = "version_test_binary_3_10",
136+
srcs = ["version_test.sh"],
137+
data = [":version_3_10"],
138+
env = {
139+
"VERSION_CHECK": "3.10",
140+
"VERSION_PY_BINARY": "$(rootpath :version_3_10)",
141+
},
142+
)

‎examples/bzlmod/tests/cross_version_test.py

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import os
16+
import subprocess
17+
import sys
18+
19+
process = subprocess.run(
20+
[os.getenv("SUBPROCESS_VERSION_PY_BINARY")],
21+
stdout=subprocess.PIPE,
22+
universal_newlines=True,
23+
)
24+
25+
subprocess_current = process.stdout.strip()
26+
subprocess_expected = os.getenv("SUBPROCESS_VERSION_CHECK")
27+
28+
if subprocess_current != subprocess_expected:
29+
print(
30+
f"expected subprocess version '{subprocess_expected}' is different than returned '{subprocess_current}'"
31+
)
32+
sys.exit(1)
33+
34+
expected = os.getenv("VERSION_CHECK")
35+
current = f"{sys.version_info.major}.{sys.version_info.minor}"
36+
37+
if current != expected:
38+
print(f"expected version '{expected}' is different than returned '{current}'")
39+
sys.exit(1)

‎examples/bzlmod/tests/version.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Copyright 2023 The Bazel Authors. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import sys
16+
17+
print(f"{sys.version_info.major}.{sys.version_info.minor}")

0 commit comments

Comments
 (0)