✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	a19e7a3
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67d81c8882d6c00008f73e31
😎 Deploy Preview	https://deploy-preview-511--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

cc @kfswain

/lgtm
/hold

2 questions that I think should be nonblocking:

1. Is the hash directory q9t5s3a7 going to be persistent? Seems like it could be temporary
2. I assume we have tested the LoRA adapters that are included?

Holding for @ahg-g to finish review and for questions, thanks Nir!!

/lgtm /hold

2 questions that I think should be nonblocking:

1. Is the hash directory q9t5s3a7 going to be persistent? Seems like it could be temporary
2. I assume we have tested the LoRA adapters that are included?

Holding for @ahg-g to finish review and for questions, thanks Nir!!

1. from my understanding and what they wrote in the issue I’ve opened, I think in the longer term vllm team will maintain the cpu images in docker hub. once that happens we will need to update the image once again, but until then I believe this hash is going to be stable, otherwise it will be hard for them to see adoption of their image.
2. yes, definitely. I’ve tested the LoRA adapters to verify it works

This is still not working for me, and I get the same error (see below). Is this expected to work on specific CPU architectures?

Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 911, in
uvloop.run(run_server(args))
File "/usr/local/lib/python3.10/dist-packages/uvloop/init.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.10/dist-packages/uvloop/init.py", line 61, in wrapper
return await main
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 875, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 230, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.

/approve
/hold cancel

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, nirrozenbaum

The full list of commands accepted by this bot can be found here.

The pull request process is described here

ok, deployed on an intel machine and it worked, previously I was testing on N2D (an AMD architecture machine). We need to clarify that in the guide if indeed the image is built to work with specific architecture.

ok, deployed on an intel machine and it worked, previously I was testing on N2D (an AMD architecture machine). We need to clarify that in the guide if indeed the image is built to work with specific architecture.

Is there an image which supports AMD architecture?