Skip to content

RECORD file is broken if strip_path_prefixes used #745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ppodolsky opened this issue Jul 5, 2022 · 0 comments · Fixed by #789
Closed

RECORD file is broken if strip_path_prefixes used #745

ppodolsky opened this issue Jul 5, 2022 · 0 comments · Fixed by #789

Comments

@ppodolsky
Copy link

ppodolsky commented Jul 5, 2022 •
edited
Loading

Hi! Following rule

py_wheel(
    ...
    strip_path_prefixes = ["prefix"],
)

creates wheel with RECORD file containing following records:

/blabla/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
/blabla/language_detect.py,sha256=RTufRn4WInDHn_5W4CThkFp6RV0ZYaserNa1rBLJJ-s,445
/blabla/morph.py,sha256=54gzoFL3oJQWq-cKuhJfXojEOU6Knw0ZR4peX8u7zaY,845
/blabla/regex.py,sha256=LqhF6JmKWihvi02XIFMFZOvmzqMxyixIUl1aaYn0BkQ,1488
/blabla/utils.py,sha256=kYW4zrRvksvXzardHjlZgwOLCPJrfVjG72k-gmIqzf4,2726
package-1.0.4.dist-info/METADATA,sha256=ogEnrEFMwzwSuA9V_uCzMViGbgnxb-UQDHLatLVBUgM,462
package-1.0.4.dist-info/RECORD,,
package-1.0.4.dist-info/WHEEL,sha256=sobxWSyDDkdg_rinUth-jxhXHqoNqlmNMJY3aTZn2Us,91

The issue is that remaining leading slash in first files break the package. If you will try to install such wheel with requirement it failes at
https://github.com/pypa/installer/blob/main/src/installer/sources.py#L158
The solution is to merely lstrip this slash for every line in RECORD
@UebelAndre UebelAndre mentioned this issue Aug 13, 2022
Merged
12 tasks
rdesgroppes added a commit to rdesgroppes/bookingcom-buildtools that referenced this issue Nov 24, 2022
@rdesgroppes
Bump rules_python to fix RECORD in .whl file
6b08f83 
Prior to `rules_python` version 0.11.0, `py_wheel` used to generate
invalid RECORD entries for file paths outside of the current package:
```
Traceback (most recent call last):
  File "[...]/pip_install/extract_wheels/extract_single_wheel.py", line 105, in <module>
    main()
  File "[...]/pip_install/extract_wheels/extract_single_wheel.py", line 93, in main
    bazel.extract_wheel(
  File "[...]/pip_install/extract_wheels/bazel.py", line 364, in extract_wheel
    whl.unzip(directory)
  File "[...]/pip_install/extract_wheels/wheel.py", line 90, in unzip
    installer.install(
  File "[...]/pypi__installer/installer/_core.py", line 96, in install
    for record_elements, stream, is_executable in source.get_contents():
  File "[...]/pypi__installer/installer/sources.py", line 158, in get_contents
    assert record is not None, "In {}, {} is not mentioned in RECORD".format(
AssertionError: In unused_deps_py-0.0.5-py3-none-any.whl, compatibility_tests/v2.5.0/tests/__init__.py is not mentioned in RECORD
```

This was reported through
bazel-contrib/rules_python#745, then fixed by
bazel-contrib/rules_python#789.

`rules_python`'s version gets here conservatively bumped to the
corresponding bugfix version:
[0.11.0](https://github.com/bazelbuild/rules_python/releases/tag/0.11.0)

Practically, the main difference between
unused_deps_py-0.0.5.dist-info/RECORD and
unused_deps_py-0.0.6.dist-info/RECORD consists in leading `/` being
stripped:
```diff
-/compatibility_tests/v2.5.0/tests/__init__.py,sha256=8SXBh8YGxPXGDmFtR_n90wt5xWvc5CEp_L5Pn687VaU,150
[...]
-/compatibility_tests/v2.5.0/tests/google/protobuf/internal/__init__.py,sha256=H-9Le6XWW7Ikwaq9qPneZts4ZitfM2XgeK4x-B5joTA,1867
+compatibility_tests/v2.5.0/tests/__init__.py,sha256=8SXBh8YGxPXGDmFtR_n90wt5xWvc5CEp_L5Pn687VaU,150
[...]
+compatibility_tests/v2.5.0/tests/google/protobuf/internal/__init__.py,sha256=H-9Le6XWW7Ikwaq9qPneZts4ZitfM2XgeK4x-B5joTA,1867
```
and:
```diff
-/google/__init__.py,sha256=8SXBh8YGxPXGDmFtR_n90wt5xWvc5CEp_L5Pn687VaU,150
[...]
-/google/protobuf/wrappers_pb2.py,sha256=xybMLGFD_xtm-GZ0owcWN-44dDkI5oR2U8Tl_ZngwpQ,12473
+google/__init__.py,sha256=8SXBh8YGxPXGDmFtR_n90wt5xWvc5CEp_L5Pn687VaU,150
[...]
+google/protobuf/wrappers_pb2.py,sha256=xybMLGFD_xtm-GZ0owcWN-44dDkI5oR2U8Tl_ZngwpQ,12473
```

Note: with latest version of `rules_python` (0.14.0 as of Nov 24, 2022),
the build would fail on:
> ERROR: $PWD/unused_deps_py/unused_deps_py_lib/BUILD.bazel:5:11: no
> such package '@unused_deps_py_pip_java_manifest//': The repository
> '@unused_deps_py_pip_java_manifest' could not be resolved: Repository
> '@unused_deps_py_pip_java_manifest' is not defined and referenced by
> '//unused_deps_py/unused_deps_py_lib:unused_deps_py_lib'
@rdesgroppes rdesgroppes mentioned this issue Nov 24, 2022
Merged
rdesgroppes added a commit to rdesgroppes/bookingcom-buildtools that referenced this issue Nov 24, 2022
@rdesgroppes
fix(unused_deps_py): Bump rules_python to fix whl RECORD
f025d78 
Prior to `rules_python` version 0.11.0, `py_wheel` used to generate
invalid RECORD entries for file paths outside of the current package:
```
Traceback (most recent call last):
  File "[...]/pip_install/extract_wheels/extract_single_wheel.py", line 105, in <module>
    main()
  File "[...]/pip_install/extract_wheels/extract_single_wheel.py", line 93, in main
    bazel.extract_wheel(
  File "[...]/pip_install/extract_wheels/bazel.py", line 364, in extract_wheel
    whl.unzip(directory)
  File "[...]/pip_install/extract_wheels/wheel.py", line 90, in unzip
    installer.install(
  File "[...]/pypi__installer/installer/_core.py", line 96, in install
    for record_elements, stream, is_executable in source.get_contents():
  File "[...]/pypi__installer/installer/sources.py", line 158, in get_contents
    assert record is not None, "In {}, {} is not mentioned in RECORD".format(
AssertionError: In unused_deps_py-0.0.5-py3-none-any.whl, compatibility_tests/v2.5.0/tests/__init__.py is not mentioned in RECORD
```

This was reported through
bazel-contrib/rules_python#745, then fixed by
bazel-contrib/rules_python#789.

`rules_python`'s version gets here conservatively bumped to the
corresponding bugfix version:
[0.11.0](https://github.com/bazelbuild/rules_python/releases/tag/0.11.0)

Practically, the main difference between
unused_deps_py-0.0.5.dist-info/RECORD and
unused_deps_py-0.0.6.dist-info/RECORD consists in leading `/` being
stripped:
```diff
-/compatibility_tests/v2.5.0/tests/__init__.py,sha256=8SXBh8YGxPXGDmFtR_n90wt5xWvc5CEp_L5Pn687VaU,150
[...]
-/compatibility_tests/v2.5.0/tests/google/protobuf/internal/__init__.py,sha256=H-9Le6XWW7Ikwaq9qPneZts4ZitfM2XgeK4x-B5joTA,1867
+compatibility_tests/v2.5.0/tests/__init__.py,sha256=8SXBh8YGxPXGDmFtR_n90wt5xWvc5CEp_L5Pn687VaU,150
[...]
+compatibility_tests/v2.5.0/tests/google/protobuf/internal/__init__.py,sha256=H-9Le6XWW7Ikwaq9qPneZts4ZitfM2XgeK4x-B5joTA,1867
```
and:
```diff
-/google/__init__.py,sha256=8SXBh8YGxPXGDmFtR_n90wt5xWvc5CEp_L5Pn687VaU,150
[...]
-/google/protobuf/wrappers_pb2.py,sha256=xybMLGFD_xtm-GZ0owcWN-44dDkI5oR2U8Tl_ZngwpQ,12473
+google/__init__.py,sha256=8SXBh8YGxPXGDmFtR_n90wt5xWvc5CEp_L5Pn687VaU,150
[...]
+google/protobuf/wrappers_pb2.py,sha256=xybMLGFD_xtm-GZ0owcWN-44dDkI5oR2U8Tl_ZngwpQ,12473
```

Note: with latest version of `rules_python` (0.14.0 as of Nov 24, 2022),
the build would fail on:
> ERROR: $PWD/unused_deps_py/unused_deps_py_lib/BUILD.bazel:5:11: no
> such package '@unused_deps_py_pip_java_manifest//': The repository
> '@unused_deps_py_pip_java_manifest' could not be resolved: Repository
> '@unused_deps_py_pip_java_manifest' is not defined and referenced by
> '//unused_deps_py/unused_deps_py_lib:unused_deps_py_lib'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix leading forward slashes in RECORD files produced by py_wheel.
1 participant
@ppodolsky