Description

Add vllm build into pytorch ci pipeline.

Details: how we do this

This pr setup vllm build logics that pass around wheels generated by the docker run and docker build. We set up a cli tool with customized logics for external build such as vllm, the vllm build process in brief:

• the torch build generates a sm80/sm90 torch ci whl based on pr
• the vllm-build job uses the torch dockerImage as base image and install torch whls from step 1.
• the vllm-build job run docker build and generate xformer whl, vllm whl and flashinfer whl
• the artifacts stored in s3 waiting for usage in test [next step]

Code implementation

This code covers 3 sections

• dockerfile.base from vllm
• torch_cli cli tool for build & configuration yml file
• vllm yaml file to run vllm x torch ci tests

Dockerfile.torch_nightly in .github/docker/external/vllm/

• Modified the vllm docker_file.torch_nightly to work both for default image and torch ci one
• Add extra stage to output the developed whls from the stages
• Upadted the flashinfer version aligned with vllm stable

Originally Made a copy of the Dockerfile.torch_nightly from vllm to accelerate the code development, then a RCE happend last week regarding the dependency fro external repo such as vlllm.
So leave it in torch now to decide the further discussion regarding install xfn repo's dependencies in torch

Setup cli tool scripts/torch_cli

set up a cli tool for torch ci build, this also set up a way to do our build and tests for new features.
it normally comes with a config yml files to store specifc parameters which affect the build/test results

pip install -e . scripts/torch_cli
python3 -m cli.build --config ".github/configs/vllm.yml" external vllm 

the cli tool must be run at root of pytorch repo.

vllm yaml file

vllm.yaml: main yaml file to set up vllm test workflow
_linux-external-build.yml: build yml file to handle external lib build such as vllm, which depends on torch ci from the pr.

next step

add vllm test step.
see experimental pr [not final, just a workable one]: https://github.com/pytorch/pytorch/actions/runs/16759050517/job/47453916017