-
Notifications
You must be signed in to change notification settings - Fork 875
feat: implement reconciliation loop #17261
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
evgeniy-scherbina
merged 158 commits into
main
from
yevhenii/510-reconciliation-loop-v2
Apr 17, 2025
Merged
Changes from 1 commit
Commits
Show all changes
158 commits
Select commit
Hold shift + click to select a range
300e80f
add prebuilds system user database changes and associated changes
SasSwart b788237
optionally prevent system users from counting to user count
dannykopping 8122595
appease the linter
dannykopping bfb7c28
add unit test for system user behaviour
dannykopping 6639167
reverting RBAC changes; not relevant here
dannykopping 769ae1d
removing unnecessary changes
dannykopping e7e9c27
exclude system user db tests from non-linux OSs
dannykopping 3936047
Rename prebuild system user reference
SasSwart 8bdcafb
ensure that users.IsSystem is not nullable
SasSwart 412d198
feat: add migrations and queries to support prebuilds
SasSwart 51773ec
Simplify workspace_latest_build view
dannykopping 23773c2
Revert test change
dannykopping bc3ff44
make gen
dannykopping 7ff747e
mark prebuilds as such and set their preset ids
SasSwart baa3076
refactor: add comments to SQL queries
evgeniy-scherbina ed14fb3
test: added get-presets-backoff test
evgeniy-scherbina 3cc74fb
refactor: add comment to SQL query
evgeniy-scherbina fc32154
refactor: add comments + improve tests
evgeniy-scherbina d7b4ec4
fix: bug in SQL
evgeniy-scherbina e8b53f7
test: minor changes to the test
evgeniy-scherbina 9df6554
refactor: remove job_status from SQL query
evgeniy-scherbina ccc309e
refactor: embed preset_prebuilds table into presets table
evgeniy-scherbina ee1f16a
refactor: rename sql table
evgeniy-scherbina d040ddd
refactor: remove unnecessary JOIN
evgeniy-scherbina 83a6722
refactor: remove unnecessary JOIN
evgeniy-scherbina cd70710
refactor: use INNER JOIN for consistency
evgeniy-scherbina 97cc4ff
refactor: simplify GetPresetsBackoff SQL Query
evgeniy-scherbina 4d59039
Revert "refactor: simplify GetPresetsBackoff SQL Query"
evgeniy-scherbina 205d6af
refactor: improve GetPresetsBackoff query
evgeniy-scherbina e489e1b
Merge remote-tracking branch 'origin/main' into prebuilds-db
evgeniy-scherbina 1b29686
Merge remote-tracking branch 'origin/main' into prebuilds-db
evgeniy-scherbina 20470e4
fix: bump migration numbers
evgeniy-scherbina 7b9c8ce
test: remove deprecated test
evgeniy-scherbina e189a0b
fix: fix linter
evgeniy-scherbina 692c0e5
fix: fix 000310_prebuilds.down migration
evgeniy-scherbina f747db0
fix: fix fixture migration
evgeniy-scherbina 3166a42
fix: fix get-presets-backoff test
evgeniy-scherbina aa6b490
fix: fix linter
evgeniy-scherbina bc4e7d2
fix: fix linter
evgeniy-scherbina f167b92
correctly select for the latest built with a preset in latest_prebuil…
SasSwart 8fd34ab
Merge remote-tracking branch 'origin/main' into prebuilds-db
SasSwart 7a8ec49
Properly label and filter metrics for prebuilds
SasSwart a64d661
test: fix db tests
evgeniy-scherbina c787cd2
test: added tests for workspaces with multiple agents
evgeniy-scherbina bd38603
refactor: avoid code duplication
evgeniy-scherbina 097f9c3
clarify query clause
SasSwart 4cfdd6f
tidy up dbauthz_test.go
SasSwart 4a34d52
refactor: remove * usage from prebuilds.sql queries
evgeniy-scherbina 8d9cd45
refactor: remove * usage from prebuilds views
evgeniy-scherbina f870d7e
refactor: join wlb with pj
evgeniy-scherbina 18ad931
refactor: Rename SQL query
evgeniy-scherbina 4667171
Added comments for SQL query
evgeniy-scherbina a26c094
refactor: fix down migration
evgeniy-scherbina 6ed4121
Merge remote-tracking branch 'origin/main' into prebuilds-db
SasSwart 2312f41
renumber migrations
SasSwart 8da7f47
Merge remote-tracking branch 'origin/prebuilds-db' into 16930
SasSwart 5150a5c
refactor: clarify comment for SQL query
evgeniy-scherbina bff34ea
refactor: fix indentations
evgeniy-scherbina ef462b6
refactor: rename helper func in test package
evgeniy-scherbina dc45165
refactor: database level tests
evgeniy-scherbina 9c8a352
refactor: database level tests
evgeniy-scherbina eb80919
refactor: helper funcs in db-level tests
evgeniy-scherbina 0b2bbee
refactor: minor improvement in SQL query
evgeniy-scherbina 3a97bf6
refactor: rename SQL queries
evgeniy-scherbina 2eeb884
refactor: rename SQL queries
evgeniy-scherbina 73f99e8
refactor: rename fields in SQL query
evgeniy-scherbina c942753
add tests to ensure workspace builds that include a preset have it se…
SasSwart 9badf7c
test to ensure we mark prebuilds as such
SasSwart 6763ba2
make -B gen fmt
SasSwart e5117d7
add template_version_preset_id to mock types
SasSwart dfec884
Merge branch 'prebuilds-db' into 16930
SasSwart 5065ad6
fix dbmem tests
SasSwart e354956
review notes. mostly rename isPrebuild to prebuild
SasSwart 4d3aab6
Merge remote-tracking branch 'origin/main' into yevhenii/510-reconcil…
evgeniy-scherbina 97b3886
fix dbmem tests
SasSwart fe60b56
feat: implement reconciliation loop
evgeniy-scherbina eeb0407
refactor: temporary commit - tests are passing
evgeniy-scherbina e807d02
refactor: remove DeterminePrebuildsState lock
evgeniy-scherbina d78675e
refactor: use TryAcquireLock
evgeniy-scherbina 7f60a5d
refactor: Verify ActionType in state_test.go
evgeniy-scherbina 8f5c9f9
refactor: minor refactor
evgeniy-scherbina 42582e1
refactor: minor refactoring
evgeniy-scherbina 14d924c
added comments
evgeniy-scherbina 82e016c
mark prebuilds as such and set their preset ids
SasSwart c03ea52
add tests to ensure workspace builds that include a preset have it se…
SasSwart 3693d45
test to ensure we mark prebuilds as such
SasSwart beb814f
make -B gen fmt
SasSwart a5418ac
add template_version_preset_id to mock types
SasSwart f4f9b17
fix dbmem tests
SasSwart d11fd58
go mod tidy && make -B gen
SasSwart 31d3bf6
test: added few more tests
evgeniy-scherbina 5007a83
refactor: remove redundant check for consistency
evgeniy-scherbina a79fe4c
refactor: use slice.Find instead of slice.Filter for backoff
evgeniy-scherbina c0246f4
refactor: make sure InProgress works on preset level as well
evgeniy-scherbina ed608cb
refactor: remove irrelevant comment
evgeniy-scherbina 70223e4
refactor: add BackoffUntil to validateActions func
evgeniy-scherbina 07808c2
refactor: CR's fixes
evgeniy-scherbina a2ceeb6
refactor: CR's fixes
evgeniy-scherbina 73fb414
Remove todo
evgeniy-scherbina 7e9c65f
minor refactoring
evgeniy-scherbina 2ca8030
refactor: remove deprecated TODOs
evgeniy-scherbina ce83f92
refactor: slighly adjust comment
evgeniy-scherbina 3bc4d8a
refactor: CR fixes
evgeniy-scherbina 2986574
Remove deprecated TODO
evgeniy-scherbina aa22a8a
refactor: update comments
evgeniy-scherbina f41b19e
refactor: remove deprecated TODO
evgeniy-scherbina 61a88e4
deduplicate test and add back proto field
SasSwart 9ac7a2c
refactor: add RunLoop test
evgeniy-scherbina d0c2094
Merge remote-tracking branch 'origin/16930' into yevhenii/510-reconci…
evgeniy-scherbina 474fc06
Fixes after merge
evgeniy-scherbina 9fac5d7
refactor: simplify SQL query
evgeniy-scherbina 868d0b6
refactor
evgeniy-scherbina 5f204f2
minor fix
evgeniy-scherbina 40b3e5f
minor fix
evgeniy-scherbina 5e3adbc
make fmt
evgeniy-scherbina 108720f
make lint
evgeniy-scherbina 9d8f6b1
chore: fix gpg forwarding test (#17355)
deansheather b994eec
refactor: fix linter
evgeniy-scherbina eff754e
refactor: fix linter
evgeniy-scherbina 4b052be
refactor: fix linter
evgeniy-scherbina bc5297c
refactor: fix linter
evgeniy-scherbina 0989636
refactor: fix linter
evgeniy-scherbina 2b57ac4
refactor: fix linter
evgeniy-scherbina 41d7e07
fix: linter
evgeniy-scherbina ab5fa8f
fix: linter
evgeniy-scherbina f485bc0
refactor: fix linter
evgeniy-scherbina 074f768
refactor: fix linter
evgeniy-scherbina a47627a
refactor: fix imports
evgeniy-scherbina eebb298
refactor: fix SQL comment
evgeniy-scherbina 9a672dd
refactor: rename dblock
evgeniy-scherbina c2f4561
Merge remote-tracking branch 'origin/main' into yevhenii/510-reconcil…
evgeniy-scherbina 62fb3f4
refactor: add test when create-prebuild helper fails
evgeniy-scherbina 742d0d3
refactor: add additional check for create-prebuilds flow
evgeniy-scherbina f3e24b1
refactor: minor fixes in util.go
evgeniy-scherbina 08aed24
refactor: minor fix in noop.go
evgeniy-scherbina 951c8b5
refactor: add TestFilter in util/slice package
evgeniy-scherbina 9c1e82f
refactor: add doc comments
evgeniy-scherbina 46e240c
refactor: update comment
evgeniy-scherbina 6dc1f68
fix: minor bug and add correspoding test
evgeniy-scherbina 23964aa
fix: minor fix for logging
evgeniy-scherbina 0a4d053
fix: CR's fixes
evgeniy-scherbina 98d203d
fix: CR's fixes
evgeniy-scherbina 145b9ff
fix: CR's fixes
evgeniy-scherbina 8b91668
CR's fixes
evgeniy-scherbina cccdab2
Update coderd/prebuilds/preset_snapshot_test.go
evgeniy-scherbina a2e5643
Update coderd/prebuilds/preset_snapshot_test.go
evgeniy-scherbina 1771c84
refactor: CR's fixes
evgeniy-scherbina 1fc551d
refactor: CR's fixes
evgeniy-scherbina d99c5cb
refactor: CR's fixes
evgeniy-scherbina b98ccd4
fix: make sure prebuild is owned by prebuild user before deleting
evgeniy-scherbina 936cc38
refactor: CR's fixes
evgeniy-scherbina 32da1ab
refactor: CR's fixes
evgeniy-scherbina 5a403c0
Update enterprise/coderd/prebuilds/reconcile.go
evgeniy-scherbina 908e6eb
refactor: CR's fixes
evgeniy-scherbina 77e4472
refactor: CR's fixes
evgeniy-scherbina 853af80
refactor: CR's fixes
evgeniy-scherbina 172177f
Merge remote-tracking branch 'origin/main' into yevhenii/510-reconcil…
evgeniy-scherbina a825bf0
refactor: reduce number of methods in Reconciler interface
evgeniy-scherbina File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
refactor: temporary commit - tests are passing
- Loading branch information
commit eeb0407d783cdda71ec2418c113f325542c47b1c
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current API feels too low-level. Ideally, we should expose higher-level methods such as:
ReconcileAll()
ReconcilePreset(preset)
Note:
SnapshotState
is currently used only byMetricsCollector
.CalculateActions
is currently unused.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's been a couple weeks but I believe we only expose these for tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe make it an internal interface and internal test?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a detailed comments, explaining how this API can be used, and why it may sense to expose all these API.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@johnstcn I also wanted to do it, but:
MetricsCollector
: DO NOT REVIEW/MERGE #16523So removing them means:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no need for this interface to be exported for the metrics collector to work correctly, since it lives in the same package as the
StoreReconciler
.Tests can be inside the same package if they need to access unexported functions. Also, you don't need to export an interface to call exported functions on, say, the
StoreReconciler
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we do this refactor in a follow-up? Let's try keep the momentum on this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't agree that "momentum" should be a reason to check in code that doesn't meet our standards. What is to be gained by fixing it later rather than now? IMO it just makes it more likely that we don't fix the issue at all.
Also, this interface isn't actually used anywhere within this PR, so fixing it within the scope of this PR should be trivial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@spikecurtis I reduce number of methods in
Reconciler
interface.Now it looks like this:
In future PRs maybe we'll need additional methods, but we can deal with it when we'll be working on those PRs.
Also we'll have more information at that point.
cc @dannykopping