24 Pull requests merged by 9 people

• Implemented KVCacheAwareScorer

  #34 merged Apr 30, 2025

• [fix]: Add oc check to kubernetes-dev-env script

  #90 merged Apr 30, 2025

• Minor fixes in instructions for deploying dev k8s env

  #58 merged Apr 30, 2025

• Sessions support is removed from datastore, server and scheduler

  #88 merged Apr 30, 2025

• pd filter returns full url of the port

  #94 merged Apr 30, 2025

• test: Test load based scorer

  #81 merged Apr 30, 2025

• update prefill header name

  #89 merged Apr 29, 2025

• Pd filter

  #87 merged Apr 29, 2025

• Read pod role from kubernetes pod info in reconciler

  #82 merged Apr 29, 2025

• feat: Add scripts for kubernetes dev env using vLLM and vLLM-p2p

  #60 merged Apr 29, 2025

• Add local_config.go which adds custom plugins

  #86 merged Apr 29, 2025

• Dev rebase 2025 04 29

  #84 merged Apr 29, 2025

• Added support for plugins to receive request headers and to modify the headers that are finally forwarded

  #79 merged Apr 29, 2025

• Add load based scorer

  #77 merged Apr 29, 2025

• Rebase main into dev - 2025-04-28

  #74 merged Apr 28, 2025

• Rebased latest K8S Main into dev

  #61 merged Apr 27, 2025

• Report lint errors on new code only

  #54 merged Apr 27, 2025

• feat: add cleanup for kubernetes dev env

  #59 merged Apr 24, 2025

• Add max score picker

  #51 merged Apr 24, 2025

• Makefile target setting prod and dev versions to the same value

  #55 merged Apr 24, 2025

• fix: Correct the setup of the KGateway Envoy service to be a proper NodePort service

  #57 merged Apr 24, 2025

• tweak instructions for accessing inference service on kind cluster

  #56 merged Apr 24, 2025

• test: Scorers test with dummy scorer and picker

  #53 merged Apr 24, 2025

• Add Kubernetes dev environment deployments with KGateway

  #50 merged Apr 24, 2025

4 Pull requests opened by 3 people

• Session affinity scorer

  #52 opened Apr 24, 2025

• Partial implementation of Session Affinity (without headers in response)

  #97 opened Apr 30, 2025

• Added unit test for pod role loading

  #99 opened Apr 30, 2025

• Change quay.io path

  #102 opened Apr 30, 2025

11 Issues closed by 6 people

• Change prefill header value to URL

  #92 closed Apr 30, 2025

• Remove sessions support from datastore, server and scheduler

  #95 closed Apr 30, 2025

• Test load based scorer

  #80 closed Apr 30, 2025

• Read pod's role (p/d/both) from kubernetes label

  #78 closed Apr 30, 2025

• Add initial version of p/d filter

  #83 closed Apr 30, 2025

• Create local_config.go which will contain all specific plugins

  #85 closed Apr 30, 2025

• Development Environments

  #3 closed Apr 29, 2025

• Deployments - Cleanup & Documentation

  #28 closed Apr 29, 2025

• OpenShift Dev Environment - Full Gateway+GIE Stack Deployment with VLLM and VLLM-P2P mode

  #72 closed Apr 29, 2025

• Endpoint Picker logger not showing traces

  #45 closed Apr 29, 2025

• Add load based routing support

  #66 closed Apr 29, 2025

17 Issues opened by 8 people

• Migrate to llm-d image repo

  #101 opened Apr 30, 2025

• Required env vars not blocked in kubernetes dev env

  #100 opened Apr 30, 2025

• Add unit test for pod role loading

  #98 opened Apr 30, 2025

• Create sesssion affinity scorer

  #96 opened Apr 30, 2025

• EPP should be able to set full pod url in header for P/D purposes

  #93 opened Apr 30, 2025

• Add unit tests for p-d filter

  #91 opened Apr 30, 2025

• Adding Prometheus and Grafana for vLLM Monitoring in the Deployment Setup

  #75 opened Apr 28, 2025

• Scorers/Filters needs access (read/update) to Request/Response Headers/Body

  #73 opened Apr 28, 2025

• Support for KV cache aware routing

  #71 opened Apr 27, 2025

• Incorporate post-build tests in CICD pipeline

  #70 opened Apr 27, 2025

• Add support for disaggregated Prefill/Decode (PD)

  #69 opened Apr 27, 2025

• Support multiple instances of the EPP for HA/load

  #68 opened Apr 27, 2025

• Add internal queueing of requests

  #67 opened Apr 27, 2025

• Explore reuse of GIE reference implementation to support proposed filter/scorer paradigm

  #65 opened Apr 27, 2025

• Document process for deploying router EPP

  #64 opened Apr 27, 2025

• Use Kubernetes controller runtime

  #63 opened Apr 27, 2025

• Run K8s GIE benchmark on a regular basis

  #62 opened Apr 27, 2025

5 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Prefix Aware Scorer

  #48 commented on Apr 30, 2025 • 16 new comments

• Add scorer based on active loras

  #43 commented on Apr 27, 2025 • 4 new comments

• Prefix Aware Routing

  #42 commented on Apr 27, 2025 • 0 new comments

• Fix in scorer manager in picking the best target

  #35 commented on Apr 27, 2025 • 0 new comments

• Add second vLLM-sim instance to allow for routing

  #41 commented on Apr 27, 2025 • 0 new comments