Skip to content

Implement Quotas v3 #5012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 33 commits into from
Nov 14, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
917129d
Implement Quotas v3
ammario Nov 10, 2022
e317d53
provisioner/terraform: add cost to resource_metadata
ammario Nov 10, 2022
208b9eb
provisionerd/runner: use Options struct
ammario Nov 10, 2022
fdac30b
Complete provisionerd implementation
ammario Nov 11, 2022
c0cb103
Begin working on inserting plan state
ammario Nov 11, 2022
d1a69aa
provisioner: begin splitting out Apply and Plan
ammario Nov 11, 2022
e200ffd
Merge remote-tracking branch 'origin/main' into quotas-v3
ammario Nov 11, 2022
7a988ea
coderd/provisionerdserver: support quotas
ammario Nov 11, 2022
7a97198
Get failure case working
ammario Nov 12, 2022
3b5b422
Begin database work
ammario Nov 12, 2022
940f905
Add quota_allowance to groups
ammario Nov 12, 2022
dbc2f48
Get Quota test passing
ammario Nov 12, 2022
77c86f6
Permit builds if they are not a net quota increase
ammario Nov 12, 2022
6afe3b2
Add group editing to frontend
ammario Nov 12, 2022
9670f05
WIP frontend
ammario Nov 12, 2022
3b0a6ea
Wire budget into the frontend
ammario Nov 12, 2022
3b02ca0
Combine Quota and RBAC licenses
ammario Nov 12, 2022
fa1dc9a
fixup! Combine Quota and RBAC licenses
ammario Nov 12, 2022
4998a8f
Write docs
ammario Nov 12, 2022
4dc4eb6
Rename to Daily cost
ammario Nov 13, 2022
8c9fe6d
Daily cost
ammario Nov 13, 2022
2dc2dac
Condense inTx interfaces
ammario Nov 13, 2022
2c61e81
Cost -> DailyCost
ammario Nov 13, 2022
b030d47
Fix tests
ammario Nov 14, 2022
4c2338c
Merge remote-tracking branch 'origin/main' into quotas-v3
ammario Nov 14, 2022
cee73f7
Fix typo
ammario Nov 14, 2022
de64d7b
Fix FE tests
ammario Nov 14, 2022
31e629f
make gen
ammario Nov 14, 2022
c1613c1
Fix null sum bug
ammario Nov 14, 2022
4582e44
Fix null column bug again
ammario Nov 14, 2022
75bd7b0
Clean up docs
ammario Nov 14, 2022
c6180e9
Merge remote-tracking branch 'origin/main' into quotas-v3
ammario Nov 14, 2022
d75c469
Fix flakey test
ammario Nov 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions cli/deployment/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -362,12 +362,6 @@ func newConfig() *codersdk.DeploymentConfig {
Enterprise: true,
Secret: true,
},
UserWorkspaceQuota: &codersdk.DeploymentConfigField[int]{
Name: "User Workspace Quota",
Usage: "Enables and sets a limit on how many workspaces each user can create.",
Flag: "user-workspace-quota",
Enterprise: true,
},
Provisioner: &codersdk.ProvisionerConfig{
Daemons: &codersdk.DeploymentConfigField[int]{
Name: "Provisioner Daemons",
Expand Down
8 changes: 3 additions & 5 deletions cli/deployment/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,16 +79,14 @@ func TestConfig(t *testing.T) {
}, {
Name: "Enterprise",
Env: map[string]string{
"CODER_AUDIT_LOGGING": "false",
"CODER_BROWSER_ONLY": "true",
"CODER_SCIM_API_KEY": "some-key",
"CODER_USER_WORKSPACE_QUOTA": "10",
"CODER_AUDIT_LOGGING": "false",
"CODER_BROWSER_ONLY": "true",
"CODER_SCIM_API_KEY": "some-key",
},
Valid: func(config *codersdk.DeploymentConfig) {
require.Equal(t, config.AuditLogging.Value, false)
require.Equal(t, config.BrowserOnly.Value, true)
require.Equal(t, config.SCIMAPIKey.Value, "some-key")
require.Equal(t, config.UserWorkspaceQuota.Value, 10)
},
}, {
Name: "TLS",
Expand Down
2 changes: 1 addition & 1 deletion coderd/activitybump.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ func activityBumpWorkspace(log slog.Logger, db database.Store, workspace databas
return xerrors.Errorf("update workspace build: %w", err)
}
return nil
})
}, nil)
if err != nil {
log.Error(
ctx, "bump failed",
Expand Down
2 changes: 1 addition & 1 deletion coderd/autobuild/executor/lifecycle_executor.go
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ func (e *Executor) runOnce(t time.Time) Stats {
}

return nil
})
}, nil)
if err != nil {
log.Error(e.ctx, "workspace scheduling failed", slog.Error(err))
}
Expand Down
14 changes: 4 additions & 10 deletions coderd/coderd.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@ import (
"github.com/coder/coder/coderd/rbac"
"github.com/coder/coder/coderd/telemetry"
"github.com/coder/coder/coderd/tracing"
"github.com/coder/coder/coderd/workspacequota"
"github.com/coder/coder/coderd/wsconncache"
"github.com/coder/coder/codersdk"
"github.com/coder/coder/provisionerd/proto"
"github.com/coder/coder/site"
"github.com/coder/coder/tailnet"
)
Expand All @@ -66,7 +66,6 @@ type Options struct {
CacheDir string

Auditor audit.Auditor
WorkspaceQuotaEnforcer workspacequota.Enforcer
AgentConnectionUpdateFrequency time.Duration
AgentInactiveDisconnectTimeout time.Duration
// APIRateLimit is the minutely throughput rate limit per user or ip.
Expand Down Expand Up @@ -145,9 +144,6 @@ func New(options *Options) *API {
if options.Auditor == nil {
options.Auditor = audit.NewNop()
}
if options.WorkspaceQuotaEnforcer == nil {
options.WorkspaceQuotaEnforcer = workspacequota.NewNop()
}

siteCacheDir := options.CacheDir
if siteCacheDir != "" {
Expand All @@ -174,12 +170,10 @@ func New(options *Options) *API {
Authorizer: options.Authorizer,
Logger: options.Logger,
},
metricsCache: metricsCache,
Auditor: atomic.Pointer[audit.Auditor]{},
WorkspaceQuotaEnforcer: atomic.Pointer[workspacequota.Enforcer]{},
metricsCache: metricsCache,
Auditor: atomic.Pointer[audit.Auditor]{},
}
api.Auditor.Store(&options.Auditor)
api.WorkspaceQuotaEnforcer.Store(&options.WorkspaceQuotaEnforcer)
api.workspaceAgentCache = wsconncache.New(api.dialWorkspaceAgentTailnet, 0)
api.TailnetCoordinator.Store(&options.TailnetCoordinator)
oauthConfigs := &httpmw.OAuth2Configs{
Expand Down Expand Up @@ -590,8 +584,8 @@ type API struct {
ID uuid.UUID
Auditor atomic.Pointer[audit.Auditor]
WorkspaceClientCoordinateOverride atomic.Pointer[func(rw http.ResponseWriter) bool]
WorkspaceQuotaEnforcer atomic.Pointer[workspacequota.Enforcer]
TailnetCoordinator atomic.Pointer[tailnet.Coordinator]
QuotaCommitter atomic.Pointer[proto.QuotaCommitter]
HTTPAuth *HTTPAuthorizer

// APIHandler serves "/api/v2"
Expand Down
3 changes: 2 additions & 1 deletion coderd/coderdtest/coderdtest.go
Original file line number Diff line number Diff line change
Expand Up @@ -528,7 +528,8 @@ func AwaitWorkspaceBuildJob(t *testing.T, client *codersdk.Client, build uuid.UU
t.Logf("waiting for workspace build job %s", build)
var workspaceBuild codersdk.WorkspaceBuild
require.Eventually(t, func() bool {
workspaceBuild, err := client.WorkspaceBuild(context.Background(), build)
var err error
workspaceBuild, err = client.WorkspaceBuild(context.Background(), build)
return assert.NoError(t, err) && workspaceBuild.Job.CompletedAt != nil
}, testutil.WaitShort, testutil.IntervalFast)
return workspaceBuild
Expand Down
62 changes: 61 additions & 1 deletion coderd/database/databasefake/databasefake.go
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ func (*fakeQuerier) Ping(_ context.Context) (time.Duration, error) {
}

// InTx doesn't rollback data properly for in-memory yet.
func (q *fakeQuerier) InTx(fn func(database.Store) error) error {
func (q *fakeQuerier) InTx(fn func(database.Store) error, _ *sql.TxOptions) error {
q.mutex.Lock()
defer q.mutex.Unlock()
return fn(&fakeQuerier{mutex: inTxMutex{}, data: q.data})
Expand Down Expand Up @@ -2246,6 +2246,7 @@ func (q *fakeQuerier) InsertWorkspaceResource(_ context.Context, arg database.In
Name: arg.Name,
Hide: arg.Hide,
Icon: arg.Icon,
DailyCost: arg.DailyCost,
}
q.provisionerJobResources = append(q.provisionerJobResources, resource)
return resource, nil
Expand Down Expand Up @@ -2757,6 +2758,20 @@ func (q *fakeQuerier) UpdateWorkspaceBuildByID(_ context.Context, arg database.U
}
return database.WorkspaceBuild{}, sql.ErrNoRows
}
func (q *fakeQuerier) UpdateWorkspaceBuildCostByID(_ context.Context, arg database.UpdateWorkspaceBuildCostByIDParams) (database.WorkspaceBuild, error) {
q.mutex.Lock()
defer q.mutex.Unlock()

for index, workspaceBuild := range q.workspaceBuilds {
if workspaceBuild.ID != arg.ID {
continue
}
workspaceBuild.DailyCost = arg.DailyCost
q.workspaceBuilds[index] = workspaceBuild
return workspaceBuild, nil
}
return database.WorkspaceBuild{}, sql.ErrNoRows
}

func (q *fakeQuerier) UpdateWorkspaceDeletedByID(_ context.Context, arg database.UpdateWorkspaceDeletedByIDParams) error {
q.mutex.Lock()
Expand Down Expand Up @@ -2858,6 +2873,7 @@ func (q *fakeQuerier) UpdateGroupByID(_ context.Context, arg database.UpdateGrou
if group.ID == arg.ID {
group.Name = arg.Name
group.AvatarURL = arg.AvatarURL
group.QuotaAllowance = arg.QuotaAllowance
q.groups[i] = group
return group, nil
}
Expand Down Expand Up @@ -3230,6 +3246,7 @@ func (q *fakeQuerier) InsertGroup(_ context.Context, arg database.InsertGroupPar
Name: arg.Name,
OrganizationID: arg.OrganizationID,
AvatarURL: arg.AvatarURL,
QuotaAllowance: arg.QuotaAllowance,
}

q.groups = append(q.groups, group)
Expand Down Expand Up @@ -3430,3 +3447,46 @@ func (q *fakeQuerier) UpdateGitAuthLink(_ context.Context, arg database.UpdateGi
}
return nil
}

func (q *fakeQuerier) GetQuotaAllowanceForUser(_ context.Context, userID uuid.UUID) (int64, error) {
q.mutex.Lock()
defer q.mutex.Unlock()
var sum int64
for _, member := range q.groupMembers {
if member.UserID != userID {
continue
}
for _, group := range q.groups {
if group.ID == member.GroupID {
sum += int64(group.QuotaAllowance)
}
}
}
return sum, nil
}

func (q *fakeQuerier) GetQuotaConsumedForUser(_ context.Context, userID uuid.UUID) (int64, error) {
q.mutex.Lock()
defer q.mutex.Unlock()
var sum int64
for _, workspace := range q.workspaces {
if workspace.OwnerID != userID {
continue
}
if workspace.Deleted {
continue
}

var lastBuild database.WorkspaceBuild
for _, build := range q.workspaceBuilds {
if build.WorkspaceID != workspace.ID {
continue
}
if build.CreatedAt.After(lastBuild.CreatedAt) {
lastBuild = build
}
}
sum += int64(lastBuild.DailyCost)
}
return sum, nil
}
2 changes: 1 addition & 1 deletion coderd/database/databasefake/databasefake_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func TestInTx(t *testing.T) {
})
assert.NoError(t, err)
return nil
})
}, nil)
assert.NoError(t, err)
}()
var nums []int
Expand Down
6 changes: 3 additions & 3 deletions coderd/database/db.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ type Store interface {
customQuerier

Ping(ctx context.Context) (time.Duration, error)
InTx(func(Store) error) error
InTx(func(Store) error, *sql.TxOptions) error
}

// DBTX represents a database connection or transaction.
Expand Down Expand Up @@ -68,7 +68,7 @@ func (q *sqlQuerier) Ping(ctx context.Context) (time.Duration, error) {
}

// InTx performs database operations inside a transaction.
func (q *sqlQuerier) InTx(function func(Store) error) error {
func (q *sqlQuerier) InTx(function func(Store) error, txOpts *sql.TxOptions) error {
if _, ok := q.db.(*sqlx.Tx); ok {
// If the current inner "db" is already a transaction, we just reuse it.
// We do not need to handle commit/rollback as the outer tx will handle
Expand All @@ -80,7 +80,7 @@ func (q *sqlQuerier) InTx(function func(Store) error) error {
return nil
}

transaction, err := q.sdb.BeginTxx(context.Background(), nil)
transaction, err := q.sdb.BeginTxx(context.Background(), txOpts)
if err != nil {
return xerrors.Errorf("begin transaction: %w", err)
}
Expand Down
4 changes: 2 additions & 2 deletions coderd/database/db_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,8 @@ func TestNestedInTx(t *testing.T) {
LoginType: database.LoginTypeGithub,
})
return err
})
})
}, nil)
}, nil)
require.NoError(t, err, "outer tx: %w", err)

user, err := db.GetUserByID(context.Background(), uid)
Expand Down
13 changes: 10 additions & 3 deletions coderd/database/dbtestutil/db.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,16 @@ func NewDB(t *testing.T) (database.Store, database.Pubsub) {
db := databasefake.New()
pubsub := database.NewPubsubInMemory()
if os.Getenv("DB") != "" {
connectionURL, closePg, err := postgres.Open()
require.NoError(t, err)
t.Cleanup(closePg)
connectionURL := os.Getenv("CODER_PG_CONNECTION_URL")
if connectionURL == "" {
var (
err error
closePg func()
)
connectionURL, closePg, err = postgres.Open()
require.NoError(t, err)
t.Cleanup(closePg)
}
sqlDB, err := sql.Open("postgres", connectionURL)
require.NoError(t, err)
t.Cleanup(func() {
Expand Down
9 changes: 6 additions & 3 deletions coderd/database/dump.sql

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions coderd/database/migrations/000076_cost.down.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
ALTER TABLE workspace_builds DROP COLUMN daily_cost;
ALTER TABLE workspace_resources DROP COLUMN daily_cost;
ALTER TABLE groups DROP COLUMN quota_allowance;
3 changes: 3 additions & 0 deletions coderd/database/migrations/000076_cost.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
ALTER TABLE workspace_builds ADD COLUMN daily_cost int NOT NULL DEFAULT 0;
ALTER TABLE workspace_resources ADD COLUMN daily_cost int NOT NULL DEFAULT 0;
ALTER TABLE groups ADD COLUMN quota_allowance int NOT NULL DEFAULT 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might make this daily_quota_allowance as well... although since it's not time based, that gets a bit odd... maybe maximum_concurrent_quota?

3 changes: 3 additions & 0 deletions coderd/database/models.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions coderd/database/querier.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading