Skip to content

Make python interpreter configurable to support python3 #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 5, 2018
Merged

Conversation

scele
Copy link
Owner

@scele scele commented Aug 30, 2018

  • Add pip_version_proxy to simplify picking correct pip deps

    The new pip_version_proxy rule generates a requirements.bzl file that provides
    a requirements() function that resolves to the correct version
    of python dependencies depending on the current python runtime.
    This is used via the WORKSPACE pattern:

      pip_import(
          name = "pip_deps2",
          requirements = ["//:requirements-pip.txt"],
          python = "@python2//:bin/python",
      )
      pip_import(
          name = "pip_deps3",
          requirements = ["//:requirements-pip.txt"],
          python = "@python3//:bin/python",
      )
      load("@pip_deps2//:requirements.bzl", pip_install2 = "pip_install")
      pip_install2()
      load("@pip_deps3//:requirements.bzl", pip_install3 = "pip_install")
      pip_install3()
    
      pip_version_proxy(
          name = "pip_deps",
          define = "python",
          values = {
              "py2": "@pip_deps2",
              "py3": "@pip_deps3",
          },
      )
    

    With this code, requirement("foo") would map to @pip_deps2 when running
    "bazel build --define python=py2", and to @pip_deps3 when running with
    "bazel build --define python=py3".

  • Make python interpreter configurable

    We need to propagate the python interpreter through all pip_import
    and wheel rules, to support using two different interpreters
    (read: python2 and python3) in the same workspace.

  • Check that python headers are installed

    Lack of python headers (/usr/include/pythonX/...) on the host machine
    may affect how wheels are built, which in turn can change the filenames
    of the wheels. For example, "pip wheel wrapt==1.10.11" produces
    "wrapt-1.10.11-py3-none-any.whl" if headers are not available, and
    "wrapt-1.10.11-cp35-cp35m-linux_x86_64.whl" if they are. Since we
    bake the wheel name into the locked requirements.bzl file, the build
    would break depending on header availability.

  • Fix BUILD file generation for dirty wheels

    When extracting foo_dirty wheel repository, we pass all wheels as an
    argument to piptool that transitively depend on foo, so that piptool
    will extract all of them to the foo_dirty repository. After that,
    piptool generates a BUILD file from the first wheel, expecting
    that to be foo.whl. This was not happening, since we were passing
    wheels in random order to piptool extract.

  • Generate missing init.py files when extracting wheels

    This allows us to stop relying on legacy_create_init=True attribute.

This allows us to stop relying on legacy_create_init=True attribute.
@scele scele force-pushed the python3 branch 2 times, most recently from 179eb95 to 892aa0d Compare August 30, 2018 08:39
scele added 2 commits August 30, 2018 12:16
We need to propagate the python interpreter through all pip_import
and wheel rules, to support using two different interpreters
(read: python2 and python3) in the same workspace.

Also add some sorting to maintain deterministic order when running
in python3.
When extracting foo_dirty wheel repository, we pass all wheels as an
argument to piptool that transitively depend on foo, so that piptool
will extract all of them to the foo_dirty repository.  After that,
piptool generates a BUILD file from the *first* wheel, expecting
that to be foo.whl.  This was not happening, since we were passing
wheels in random order to piptool extract.
scele added 2 commits August 31, 2018 12:20
Lack of python headers (/usr/include/pythonX/...) on the host machine
may affect how wheels are built, which in turn can change the filenames
of the wheels.  For example, "pip wheel wrapt==1.10.11" produces
"wrapt-1.10.11-py3-none-any.whl" if headers are not available, and
"wrapt-1.10.11-cp35-cp35m-linux_x86_64.whl" if they are.  Since we
bake the wheel name into the locked requirements.bzl file, the build
would break depending on header availability.
The new pip_version_proxy rule generates a requirements.bzl file that provides
a requirements() function that resolves to the correct version
of python dependencies depending on the current python runtime.
This is used via the WORKSPACE pattern:

    pip_import(
        name = "pip_deps2",
        requirements = ["//:requirements-pip.txt"],
        python = "@python2//:bin/python",
    )
    pip_import(
        name = "pip_deps3",
        requirements = ["//:requirements-pip.txt"],
        python = "@python3//:bin/python",
    )
    load("@pip_deps2//:requirements.bzl", pip_install2 = "pip_install")
    pip_install2()
    load("@pip_deps3//:requirements.bzl", pip_install3 = "pip_install")
    pip_install3()

    pip_version_proxy(
        name = "pip_deps",
        define = "python",
        values = {
            "py2": "@pip_deps2",
            "py3": "@pip_deps3",
        },
    )

With this code, requirement("foo") would map to @pip_deps2 when running
"bazel build --define python=py2", and to @pip_deps3 when running with
"bazel build --define python=py3".
@scele scele merged commit 3883a60 into master Sep 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant