Skip to content

Refactor and separate concerns of external python package handling code #953

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 10, 2023

Conversation

groodt
Copy link
Collaborator

@groodt groodt commented Dec 29, 2022

Refactors the code that deals with external python packages retrieved from repositories (i.e. PyPI, artifactory etc) into 3 main concerns:

  1. wheel_installer: The intention is for this to be only Wheel unpacking, which would be a hermetic operation and could run as a build action. However, at the moment, there is also a pip interaction that happens here to build wheels. This can be refactored out into a wheel_builder in a future PR
  2. lock_file_generator: This generates a requirements.bzl lock file for the external python packages from a fully resolved requirements.txt file (optionally generated from dependency_resolver. See 3)
  3. dependency_resolver: Fully resolves a requirements.in file of direct dependencies into a requirements.txt file.

I think the refactor makes the code easier to work with and understand. This should make further improvements easier over time.

@groodt groodt marked this pull request as ready for review December 29, 2022 10:16
@groodt groodt requested a review from f0rmiga December 29, 2022 10:17
Copy link
Collaborator

@rickeylev rickeylev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just nits, really; otherwise LGTM

@@ -0,0 +1,51 @@
load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be "@rules_python//" or just "//"?

srcs = [
"namespace_pkgs_test.py",
],
tags = ["unit"],
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this tags=unit used by something elsewhere (didn't see it)? I'd think that size=small was sufficient info to filter (small usually means unit test)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's probably unused. I saw a few test_tag_filters to exclude integration test tags. I'll try remove it and see.

@groodt groodt merged commit 70cce26 into main Jan 10, 2023
@alexeagle alexeagle deleted the groodt-refactor-tools branch May 23, 2023 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants