Description

• The https://github.com/pytorch/benchmark seems to be the official PyTorch performance benchmark running on the CI.
• This is a subtask of https://github.com/microsoft/ARM-on-Windows-Ecosystem/issues/187

Acceptance Criteria

• We can compare results of https://github.com/pytorch/benchmark on Windows and WSL.

Documentation

Steps to make this benchmark working:

• Enable long file names policy Make Windows 11 Accept File Paths over 260 Characters (thegeekpage.com)
  • Still an issue Add long path support for Windows [Approach #2] by majaeger · Pull Request #2056 · ninja-build/ninja (github.com)
• conda create -n pytorch-benchmark-py3.8 python=3.8
• conda activate pytorch-benchmark-py3.8
• conda install -y -c pytorch pytorch torchvision torchtext
• conda install git-lfs pyyaml
• git clone https://github.com/pytorch/benchmark
• Apply the following patch:

  diff --git a/torchbenchmark/__init__.py b/torchbenchmark/__init__.py
  index 897dbe23..92c1aca9 100644
  --- a/torchbenchmark/__init__.py
  +++ b/torchbenchmark/__init__.py
  @@ -43,10 +43,11 @@ internal_model_dir = "fb"
   install_file = 'install.py'
  
  
  -def _test_https(test_url: str = 'https://github.com', timeout: float = 0.5) -> bool:
  +def _test_https(test_url: str = 'https://github.com', timeout: float = 5.0) -> bool:
       try:
           request.urlopen(test_url, timeout=timeout)
  -    except OSError:
  +    except OSError as e:
  +        print(e)
           return False
       return True
  
  @@ -57,7 +58,7 @@ def _install_deps(model_path: str, verbose: bool = True) -> Tuple[bool, Any]:
           [sys.executable, install_file],
       ]
       run_env = os.environ.copy()
  -    run_env["PYTHONPATH"] = this_dir.parent
  +    run_env["PYTHONPATH"] = str(this_dir.parent)
       run_kwargs = {
           'cwd': model_path,
           'check': True,
  @@ -88,7 +89,10 @@ def _install_deps(model_path: str, verbose: bool = True) -> Tuple[bool, Any]:
           return (False, e, io.FileIO(stdout_fpath, mode="r").read().decode())
       finally:
           del output_buffer
  -        os.remove(stdout_fpath)
  +        try:
  +            os.remove(stdout_fpath)
  +        except:
  +            pass
  
       return (True, None, None)
  
  diff --git a/torchbenchmark/models/fambench_xlmr/requirements.txt 
  b/torchbenchmark/models/fambench_xlmr/requirements.txt
  index da75cfcd..06cb9fef 100644
  --- a/torchbenchmark/models/fambench_xlmr/requirements.txt
  +++ b/torchbenchmark/models/fambench_xlmr/requirements.txt
  @@ -1,8 +1,7 @@
   sacrebleu
   bitarray
   # pin fairseq version
  -fairseq==0.10.2
  +fairseq==0.10.0
   omegaconf==2.1.1
   hydra-core==1.1.2
   sentencepiece
  -xformers
  diff --git a/torchbenchmark/models/opacus_cifar10/requirements.txt 
  b/torchbenchmark/models/opacus_cifar10/requirements.txt
  index 106f0456..6a68fd02 100644
  --- a/torchbenchmark/models/opacus_cifar10/requirements.txt
  +++ b/torchbenchmark/models/opacus_cifar10/requirements.txt
  @@ -1,3 +1,2 @@
  -git+https://github.com/pytorch/functorch.git
  -# must include the fix https://github.com/pytorch/opacus/pull/426
  +functorch
   opacus>=1.1.2
  diff --git a/torchbenchmark/util/framework/detectron2/requirements.txt 
  b/torchbenchmark/util/framework/detectron2/requirements.txt
  index a7f6a571..3c17e0b6 100644
  --- a/torchbenchmark/util/framework/detectron2/requirements.txt
  +++ b/torchbenchmark/util/framework/detectron2/requirements.txt
  @@ -1,3 +1,2 @@
  -git+https://github.com/facebookresearch/detectron2.git@c470ca3
   omegaconf==2.1.1
   numpy
  

• python .\install.py (Must be Run as Administrator...)
• TODO: This is just WIP.