Skip to content

PillarCache: reimplement using salt.cache #68030

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

mattp-
Copy link
Contributor

@mattp- mattp- commented May 19, 2025

commit 8fd858d (HEAD -> cachepillar, origin/cachepillar)
Author: Matt Phillips mphillips81@bloomberg.net
Date: Thu Apr 4 17:51:13 2024 -0400

PillarCache: reimplement using salt.cache

took the liberty of making it a proper subclass in the process. this now
uses the salt.cache infrastructure such that it can be driven by the
base cache driver or a different one if so desired. functionality should
be equivalent, including using the base bank=pillar key=minion_id for
merged pillar, such that minion_data_cache can take advantage of the
same cache. because we are updating the cache at the source, we no
longer need to update the cache in master/masterapi.

commit 4b20c8d
Author: Matt Phillips mphillips81@bloomberg.net
Date: Thu Aug 1 19:36:36 2019 -0400

fix minion data cache organization

previously 'bank' was minions/$minion, and key was 'data', which was a
merge of both pillar and grains. rather than do that, we organize pillar
and grains into separate banks for better index optimizations for
backends that can take advantage of it. it also just makes more sense
this way

commit fac04fc
Author: Matt Phillips mphillips81@bloomberg.net
Date: Tue Apr 15 11:55:10 2025 -0400

salt.cache: allow cache.store() to set expires per key

rather than force a cache.cache() interface and a uniform expiry across
the cache, we add a more common per store() expiry that can also be used
instead or in addition to the original cache-wide expiry.

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

Commits signed with GPG?

Yes

twangboy
twangboy previously approved these changes May 19, 2025
@twangboy twangboy added this to the Argon v3008.0 milestone May 19, 2025
@twangboy twangboy added the test:full Run the full test suite label May 19, 2025
@mattp- mattp- changed the title PillarCache: reimplement using salt.cache WIP: PillarCache: reimplement using salt.cache May 20, 2025
@mattp- mattp- changed the title WIP: PillarCache: reimplement using salt.cache PillarCache: reimplement using salt.cache May 24, 2025
mattp- added 4 commits May 27, 2025 11:13
rather than force a cache.cache() interface and a uniform expiry across
the cache, we add a more common per store() expiry that can also be used
instead or in addition to the original cache-wide expiry.
previously 'bank' was minions/$minion, and key was 'data', which was a
merge of both pillar and grains. rather than do that, we organize pillar
and grains into separate banks for better index optimizations for
backends that can take advantage of it. it also just makes more sense
this way
took the liberty of making it a proper subclass in the process. this now
uses the salt.cache infrastructure such that it can be driven by the
base cache driver or a different one if so desired. functionality should
be equivalent, including using the base bank=pillar key=minion_id for
merged pillar, such that minion_data_cache can take advantage of the
same cache. because we are updating the cache at the source, we no
longer need to update the cache in master/masterapi.
the loops involved dont use sets which seems to have an oversized impact
on lookup performance. using sets for containment checks makes this
function go 15x faster on a 50k dataset (from 30+ sec to 2~ sec).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test:full Run the full test suite
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants