Skip to content

Pd filter #87

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 29, 2025
Merged

Pd filter #87

merged 4 commits into from
Apr 29, 2025

Conversation

mayabar
Copy link
Collaborator

@mayabar mayabar commented Apr 29, 2025

Ref #83

@mayabar mayabar requested review from elevran and shmuelk April 29, 2025 14:39
mayabar added 2 commits April 29, 2025 17:58
* dev:
  Read pod role from kubernetes pod info in reconciler (kubernetes-sigs#82)
  feat: Add scripts for kubernetes dev env using vLLM and vLLM-p2p  (kubernetes-sigs#60)
  Add local_config.go which adds custom plugins (kubernetes-sigs#86)
  Fixed merge error
  Removed local version of max score scorer
  Fixed file formatting
  Deleted local version of max scorer
  Add max score picker
  Bump the kubernetes group with 6 updates (kubernetes-sigs#754)
  Add GetEnvString helper function (kubernetes-sigs#758)
  add max score picker (kubernetes-sigs#752)
  Updates from merge
  fixed broken link to implemenations (kubernetes-sigs#750)
  Removed unneeded import
  Added simple tests
  Update expected Results in tests to reflect the updated object
  Add support for Filters and Scorers to have access to all of the request headers and to be able to add request headers
  Weighted scorers (kubernetes-sigs#737)
  fixed broken link to implemenations (kubernetes-sigs#750)

# Conflicts:
#pkg/epp/backend/metrics/pod_metrics.go
#pkg/epp/backend/metrics/types.go
Copy link
Collaborator

@shmuelk shmuelk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mayabar mayabar merged commit 183c047 into neuralmagic:dev Apr 29, 2025
1 check passed
@mayabar
Copy link
Collaborator Author

mayabar commented Apr 29, 2025

To use this:
1 - add label 'llmd.org/role' with values 'prefill' or 'decode' for the prefill and decode vllm pods respectively
2 - uncomment pkg/epp/scheduling/local_config.go line 27, save the file, ensure that imports are added
3 - rebuild image

  • Header "X-Prefill-Dns-Name" will contain the prefill pod name

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants