Skip to content

Add filtering to vector index #21916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 32 commits into
base: devel
Choose a base branch
from
Draft

Add filtering to vector index #21916

wants to merge 32 commits into from
+1,437 −157

Conversation

jbajic
Copy link
Contributor

@jbajic jbajic commented Aug 19, 2025 •
edited
Loading

Scope & Purpose

Introduces filtering to the vector index. This way, we will push the filtering expression to the vector index iterator and evaluate it there. A continuation of this would be an investigation into whether the batching of documents would be beneficial, and if so, implementing it.
Now queries like this should work, and the FilterNode will be merged with the EnumerateNearVectorIndexNode:

FOR d IN col
FILTER d.val > 3
LET dist = APPROX_NEAR_L2(d.vector, @q)
SORT dist LIMIT 3 RETURN d

This implementation still does not support storedValues usage and compound indexes.

  • 💩 Bugfix
  • 🍕 New feature
  • 🔥 Performance improvement
  • 🔨 Refactoring/simplification

Checklist

  • Tests
    • Regression tests
    • C++ Unit tests
    • integration tests
    • resilience tests
  • 📖 CHANGELOG entry made
  • 📚 documentation written (release notes, API changes, ...)
  • Backports
    • Backport for 3.12.0: (Please link PR)
    • Backport for 3.11: (Please link PR)
    • Backport for 3.10: (Please link PR)

Related Information

(Please reference tickets / specification / other PRs etc)

  • Docs PR:
  • Enterprise PR:
  • GitHub issue / Jira ticket:
  • Design document:

@jbajic jbajic self-assigned this Aug 19, 2025
@cla-bot cla-bot bot added the cla-signed label Aug 19, 2025
jbajic added 21 commits August 19, 2025 15:02
@jbajic
Merge branch 'devel' into feature/vector-index-filtering
f3cc8f2
@jbajic
Remove logs
3c53778
@jbajic
Add L2 tests
71e3ba2
@jbajic
Remove Advanced and Mixed tests
8813fe7
@jbajic
Fix cosine tests
851df4b
@jbajic
Fix inner product tests
d17f516
@jbajic
Use same parameter for nProbe and nLists
89f39f1
@jbajic
Clone filter expression in optimizer
c8cd3bb
@jbajic
Make filter expression a unique_ptr
090bcba
@jbajic
Add filterExpression to ser/deser
3cd28ac
@jbajic
Fix cloning filterExpression
d35fcc2
@jbajic
Remove redundant part
9b02ff3
@jbajic
Move expression evaluation in callback
0dc7e82
@jbajic
Add changelog entry
2d71bc3
@jbajic
Merge branch 'devel' into feature/vector-index-filtering
c837ac3
@jbajic
Fix iteration condition
be790ad
@jbajic
Add loop filter test
50f4712
@jbajic
Merge branch 'feature/vector-index-filtering' of github.com:arangodb/…
408693b 
…arangodb into feature/vector-index-filtering
@jbajic
Cleanup
db81118
@jbajic
Small refactoring
893d09b
@jbajic
Merge branch 'devel' into feature/vector-index-filtering
68774d3
@maierlars
Copy link
Contributor

Interesting, so we are materializing the document twice? It appears to me that filtering based on stored values would be even easier and more efficient. Just my five cents 😁 but I like the progress 👏🏽

@jbajic
Copy link
Contributor Author

jbajic commented Sep 3, 2025

Interesting, so we are materializing the document twice? It appears to me that filtering based on stored values would be even easier and more efficient. Just my five cents 😁 but I like the progress 👏🏽

Yes, we are materializing twice. This would be the non-optimized case that would support all FILTER clauses next to the vector index. We are keeping in mind to work on supporting storedValues in vector index 😃 to avoid double materialization. Thanks for the comment and for keeping track of the vector index work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jbajic jbajic

Labels
cla-signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jbajic @maierlars