@dslomov @meteorcloudy @laszlocsomor This doesn't look like my bug.

Here's what I'm seeing in the Bazel CI logs once I fix the path delimiters in zipfile:

...
INFO: Building...
[0 / 18] Writing file rules_python/piptool.zip-2.params
[12 / 126] Compiling external/protobuf/src/google/protobuf/compiler/js/embed.cc [for host]
[12 / 140] Executing genrule @protobuf//:protos_python_genrule [for host]
[13 / 179] Compiling external/protobuf/src/google/protobuf/io/gzip_stream.cc [for host]
ERROR: C:/temp/_bazel_system/_wrknaru/external/protobuf/BUILD:113:1: C++ compilation of rule '@protobuf//:protobuf' failed (Exit 2): cl.exe failed: error executing command 
  cd C:/temp/_bazel_system/_wrknaru/execroot/io_bazel_rules_python
  SET INCLUDE=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE;C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um;C:\Program Files (x86)\Windows Kits\8.1\include\\shared;C:\Program Files (x86)\Windows Kits\8.1\include\\um;C:\Program Files (x86)\Windows Kits\8.1\include\\winrt;
    SET LIB=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\LIB\amd64;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.10240.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64;C:\Program Files (x86)\Windows Kits\8.1\lib\winv6.3\um\x64;
    SET PATH=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\amd64;C:\windows\Microsoft.NET\Framework64\v4.0.30319;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\VCPackages;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Performance Tools\x64;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Performance Tools;C:\Program Files (x86)\Windows Kits\8.1\bin\x64;C:\Program Files (x86)\Windows Kits\8.1\bin\x86;C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\;c:\tools\msys64\usr\bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\Google\Compute Engine\sysprep\;C:\Program Files\Google\Compute Engine\metadata_scripts\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\ProgramData\chocolatey\bin;;;C:\ProgramData\chocolatey\bin;;c:\python_27_amd64\files;C:\Program Files\Java\jdk1.8.0_144\bin;;C:\windows\system32
    SET PWD=/proc/self/cwd
    SET TEMP=c:\TEMP
    SET TMP=c:\TEMP
  C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/amd64/cl.exe /DCOMPILER_MSVC /DNOMINMAX /D_WIN32_WINNT=0x0600 /D_CRT_SECURE_NO_DEPRECATE /D_CRT_SECURE_NO_WARNINGS /D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS /bigobj /Zm500 /J /Gy /GF /EHsc /wd4351 /wd4291 /wd4250 /wd4996 /nologo /Iexternal/protobuf /Ibazel-out/host/genfiles/external/protobuf /Iexternal/bazel_tools /Ibazel-out/host/genfiles/external/bazel_tools /Iexternal/protobuf/src /Ibazel-out/host/genfiles/external/protobuf/src /Iexternal/bazel_tools/tools/cpp/gcc3 /showIncludes /MT /O2 /c external/protobuf/src/google/protobuf/io/gzip_stream.cc /Fobazel-out/host/bin/external/protobuf/_objs/protobuf/external/protobuf/src/google/protobuf/io/gzip_stream.o -DHAVE_PTHREAD -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function.
cl : Command line error D8021 : INFO: Building complete.

I think the protobuf dependency is coming from skydoc. /cc @davidzchen @damienmg Does that work on Windows?

@duggelz FYI I wasn't gonna merge it before it works, I wanted to use this as a place to triage and discuss issues in the related tooling.

@mattmoor The error is because you have invalid warning arguments in your command line, such as
-Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function.
They are all gcc options, you should exclude them from BUILD file.
An example change:
bazelbuild/bazel@86a9a50#diff-eb50ae634b51ccba3d129d82412f25e6R92

@meteorcloudy This repo has zero C++ code, this is coming through a dependency (@protobuf), which is likely coming through skydoc deps).

Have you guys done any outreach to the owners of common dependencies yet?

We do compile protobuf as part of building Bazel, so there is at some version that should build on Windows.

Disabling skydoc stuff I am now getting:

...
ERROR: C:/temp/_bazel_system/_wrknaru/external/subpar/compiler/BUILD:40:1: Building par file @subpar//compiler:compiler.par failed (Exit 1): compiler.cmd failed: error executing command 
  cd C:/temp/_bazel_system/_wrknaru/execroot/io_bazel_rules_python
bazel-out/host/bin/external/subpar/compiler/compiler.cmd --manifest_file bazel-out/host/bin/external/subpar/compiler/compiler.par_SOURCES --outputpar bazel-out/host/bin/external/subpar/compiler/compiler.par --stub_file bazel-out/host/bin/external/subpar/compiler/compiler.cmd external/subpar/compiler/compiler.py.
'"python"' is not recognized as an internal or external command,
operable program or batch file.

It looks like the py_binary script that is generated isn't properly resolving the Python interpreter?

On a local box, here's what that contains:

@rem Copyright 2017 The Bazel Authors. All rights reserved.
@rem
@rem Licensed under the Apache License, Version 2.0 (the "License");
@rem you may not use this file except in compliance with the License.
@rem You may obtain a copy of the License at
@rem
@rem    http://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@rem This script was generated from python_stub_template_windows.txt.  Please
@rem don't edit it directly.

@SETLOCAL ENABLEEXTENSIONS

@rem launcher=${$0%.cmd}
@set launcher=%~dp0%~n0

@call "python" "%launcher%" %*

I can work around this locally by supplying: --python_path=c:\Python27\python.exe

However, the next thing I run into is (very likely) a PAR issue, where the PAR compiler seems to parse the .cmd stub file.

I opened this to track subpar support for Windows.

If I comment out the build of piptool.par, the bazel build ... passes (small victories, it is Python).

However, all of the tests fail with (variations of):

WindowsError: [Error 206] The filename or extension is too long: 'c:\\users\\mattmoor\\appdata\\local\\temp\\2\\_bazel_mattmoor\\czlhytfa\\execroot\\io_bazel_rules_python\\bazel-out\\msvc_x64-fastbuild\\bin\\examples\\helloworld\\helloworld_test.runfiles\\io_bazel_rules_python\\Bazel.runfiles_0dwpqt\\runfiles\\io_bazel_rules_python\\examples'�

@dslomov Do you guys have anything planned to address this?

@duggelz pointed me to this black magic.

@mattmoor :

@dslomov Do you guys have anything planned to address this?

subpar wasn't on my radar until now, thanks for filing the bug!
We learned a lot about Windows path handling as we ported Bazel to Windows. I'd be happy to share my knowledge regarding \\?\ and the caveats around it.

@laszlocsomor I think the other major one you guys should make sure is on your radar is pubref/rules_*, which @pcj kindly curates.

@laszlocsomor To be clear, I don't think the long path has anything to do with subpar, this is happening in a py_test trying to open test data.

@mattmoor Can you give me some instructions on how to reproduce your failure?
And which version of Python are you using? IIRC, only Python 3 supports long path.

@meteorcloudy IIRC on top of my windows branch I had to comment out the par_binary rule in rules_python/BUILD and invoke Bazel with:

bazel test --python_path=c:\Python27\python.exe ...

I believe the paths that ultimately fail are all coming through here. Would it make sense to use the weird trick in TEST_SRCDIR?

@mattmoor :

Would it make sense to use the weird trick in TEST_SRCDIR?

It would only work with Python 3. To be clear, the \\?\ thing is not a trick, it is the proper way to tell the Windows API that you intend to use a long path.

@mattmoor
With Python3, I can at least run the test, but got error message:

pcloudy@pcloudy0-w MSYS ~/workspace/rules_python
$ bazel test rules_python:whl_test
INFO: Analysed target //rules_python:whl_test.
INFO: Found 1 test target...
FAIL: //rules_python:whl_test (see C:/tmp/_bazel_pcloudy/oey3yzqk/execroot/io_bazel_rules_python/bazel-out/msvc_x64-fastbuild/testlogs/rules_python/whl_test/test.log)
INFO: From Testing //rules_python:whl_test:
==================== Test output for //rules_python:whl_test:
EEE
======================================================================
ERROR: test_futures_whl (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "\\?\C:\tmp\Bazel.runfiles_g0ifw578\runfiles\io_bazel_rules_python\rules_python\whl_test.py", line 40, in test_futures_whl
    self.assertEqual(wheel.name(), 'futures')
  File "\\?\C:\tmp\Bazel.runfiles_g0ifw578\runfiles\io_bazel_rules_python\rules_python\whl.py", line 65, in name
    return self.metadata().get('name')
  File "\\?\C:\tmp\Bazel.runfiles_g0ifw578\runfiles\io_bazel_rules_python\rules_python\whl.py", line 59, in metadata
    with zipfile.ZipFile(self.path(), 'r') as whl:
  File "C:\Program Files\Anaconda3\lib\zipfile.py", line 1009, in __init__
    self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:/tmp/_bazel_pcloudy/oey3yzqk/execroot/io_bazel_rules_python/bazel-out/msvc_x64-fastbuild/bin/rules_python/whl_test.runfiles\\futures_whl/file/futures-3.1.1-py2-none-any.whl'

This is because we don't have runfiles on Windows, but I believe you can still access futures_whl/file/futures-3.1.1-py2-none-any.whl by this relative path. Because we compress everything under runfiles into a zip file and unzip it during runtime.

This is because we don't have runfiles on Windows

ugh, I feel like I just figured out runfiles, and now you are changing it on me! ;-)

Are you suggesting that this relative path is the portable way to access files from tests?

On Linux/macOS, we can create symlinks, so Bazel creates the runfiles directory tree with symlinks in the directories, and the symlinks point to the actual files. Binaries and tests can access the runfiles via <runfiles_root>/path/to/runfile.

On Windows, we can't create symlinks, so Bazel just writes a manifest file, which describes where the symlinks would be and where they'd point to. Binaries and tests can access the runfiles by grep path/to/runfile <runfiles_root>/MANIFEST | awk '{print $2}'.

Example: https://github.com/bazelbuild/bazel/blob/e6b6d7c2312a6ccca365db1832b12a0b6704cf5b/tools/test/test-setup.sh#L100-L118

Because we compress everything under runfiles into a zip file and unzip it during runtime.

@meteorcloudy , this is new to me -- is this logic implemented in the rule, or in Bazel?

@mattmoor @laszlocsomor Basically, there are two ways of accessing runfiles in Bazel's native python rule.

1. As @laszlocsomor pointed out, using the manifest file to map runfiles to its real absolute path.

2. According to this design doc, py_binary is a zip file on Windows. It unzips itself to a temp directory during execution, all the runfiles are under <temp>\\Bazel.runfiles_XXXX\\runfiles

I prefer users to go with the first way, because it's the general way to access runfiles on Windows, which also works for other rules(cc_binary, java_binary). Look how we access runfiles in our python tests, _LoadRunfiles and Rlocation

@dslomov @meteorcloudy @laszlocsomor So after the recent skydoc update I rebased this change and I'm now hitting the issue I worked around locally by passing --python_path to Bazel.

Is there a reason the Bazel CI on Windows isn't passing this? Is 0.6 expected to fix this? Does google/subpar need some sort of fix?

@mattmoor Unfortunately, bazelbuild/bazel@213a52a didn't make it into 0.6.

@meteorcloudy Should Bazel CI be passing --python_path to Bazel invocations? Perhaps this can be done via .bazelrc?

@mattmoor You can also add --python_path in .ci/rules_python.json, like https://github.com/bazelbuild/bazel/blob/master/scripts/ci/bazel-tests.json#L88

Cool, do you know the path to Python on the CI setup?

Should be --python_path="C:/Program Files/Anaconda3/python.exe"

Finally got a chance to update the CI config, and it looks like the next issue is coming from PAR (I don't recall hitting this locally):

ERROR: C:/windows/temp/_bazel_system/fkfyb_ar/external/subpar/compiler/BUILD:40:1: Building par file @subpar//compiler:compiler.par failed (Exit 1): compiler.cmd failed: error executing command 
  cd C:/windows/temp/_bazel_system/fkfyb_ar/execroot/io_bazel_rules_python
bazel-out/host/bin/external/subpar/compiler/compiler.cmd --manifest_file bazel-out/host/bin/external/subpar/compiler/compiler.par_SOURCES --outputpar bazel-out/host/bin/external/subpar/compiler/compiler.par --stub_file bazel-out/host/bin/external/subpar/compiler/compiler.cmd external/subpar/compiler/compiler.py.
Traceback (most recent call last):
  File "C:\Program Files\Anaconda3\lib\site.py", line 570, in <module>
    main()
  File "C:\Program Files\Anaconda3\lib\site.py", line 561, in main
    os.environ["PATH"])
  File "C:\Program Files\Anaconda3\lib\os.py", line 725, in __getitem__
    raise KeyError(key) from None
KeyError: 'PATH'
Failed to import the site module

The PATH assumption clearly doesn't work on Windows. I haven't dug into this at all yet, but I can try to reproduce it on my Windows box when I'm in.

retest this please.