Skip to content

Issue Deleting Abandoned Workspaces #7872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
baxters-ctc opened this issue Jun 6, 2023 · 13 comments
Closed

Issue Deleting Abandoned Workspaces #7872

baxters-ctc opened this issue Jun 6, 2023 · 13 comments
Assignees
Labels
s2 Broken use cases or features (with a workaround). Only humans may set this.
Milestone

Comments

@baxters-ctc
Copy link

In the process of moving from local users to OIDC, I ended up deleting and recreating a user. There were still workspaces attached to the user and I can't find a way to delete them.

When I access the workspace from the main Workspaces section in the UI I get this message -
"user" must be an existing uuid or username.

None of the typical UI items are visible including the delete button.

I tried recreating the username as a local user and now when I try to load the workspace it just redirects me back to the main Workspaces tab.

Is there any way to force delete these workspaces without going through the UI?

@MrPeacockNLB
Copy link
Contributor

Did you try coder delete?

coder delete --orphan true -y WORKSPACE

@baxters-ctc
Copy link
Author

@MrPeacockNLB Sorry I didn't catch that in the docs earlier. I tried running this and got an unexpected output -

coder delete --orphan true -y 2nd-test
wanted 1 args but got 2 [true 2nd-test]
coder delete --orphan true -y test-afs
wanted 1 args but got 2 [true test-afs]

If I run without the workspace arg it prompts but doesn't tell me what it's going to delete -

coder delete --orphan true 
> Confirm delete workspace? (yes/no) no

Any suggestions on how to proceed?

@ammario
Copy link
Member

ammario commented Jun 6, 2023

@baxters-ctc

--orphan by itself is all you need, so coder delete --orphan test-afs. The CLI doesn't know whether true is for the boolean flag or the name of the workspace.

@matifali matifali added the waiting-for-info The issue creator is asked to provide more information. label Jun 6, 2023
@baxters-ctc
Copy link
Author

baxters-ctc commented Jun 6, 2023

@ammario Thanks for following up and clarifying! I'm now running into a slightly different error . If I use just the workspace name, I get a "Resource not found or you do not have access to this resource -

coder delete --orphan -y 2nd-test
Resource not found or you do not have access to this resource

I'm a full admin on the coder instance so I expect I would have access. When I try to give the fully qualified username/workspace I get a different error that seems similar to what's shown the UI -

coder delete --orphan -y admin-somayira/2nd-test
"user" must be an existing uuid or username.

I was able to delete another workspace from that user's new OIDC account using the same fully qualified username/password format.

coder delete --orphan aldensomayire/terraform-upload

Confirm delete workspace? (yes/no) yes
✔ Queued [564ms]
✔ Setting up [28ms]
✔ Planning infrastructure [21ms]
⧗ Destroying workspace
The terraform state does not exist, there is nothing to do
✔ Destroying workspace [48ms]
✔ Cleaning Up [38ms]

The terraform-upload workspace has been deleted at Jun 6 23:43:06!

Any ideas why it doesn't work for the orphan case?

@matifali matifali added s2 Broken use cases or features (with a workaround). Only humans may set this. and removed waiting-for-info The issue creator is asked to provide more information. labels Jun 7, 2023
@ammario
Copy link
Member

ammario commented Jun 9, 2023

Since this issues appears related to auth perhaps @Emyrk can assist.

@sreya sreya added this to the ❓Sprint 1 milestone Jun 12, 2023
@Emyrk
Copy link
Member

Emyrk commented Jun 12, 2023

Looks like it 🤔

Looking into it

@Emyrk
Copy link
Member

Emyrk commented Jun 13, 2023

Solution

@atnomoverflow I think this solution will work for you.

coder delete --orphan -y <deleted_user_id>/2nd-test

Explanation

When you run coder delete --orphan -y 2nd-test, the cli assumes you own the workspace and tries <your_username>/2nd-test.

To delete the correct workspace, you need to specify it with:

  • <deleted_user_id>/2nd-test

You must use the deleted user's ID and not their username. This is because when a user is deleted, we no longer guarantee their username to be unique. So it can conflict with an active or other deleted users.


Below this line is just an explanation for any future readers or those who are curious.

The abandoned workspace state

Being able to delete a user who has workspaces should not be allowed. We have a check to prevent deleting a user if the user has workspaces:

coder/coderd/users.go

Lines 433 to 438 in f13632c

if len(workspaces) > 0 {
httpapi.Write(ctx, rw, http.StatusExpectationFailed, codersdk.Response{
Message: "You cannot delete a user that has workspaces. Delete their workspaces and try again!",
})
return
}

So there might be a bug or avenue that allowed this to happen.

How to reproduce

I can force this state in a unit test by bypassing the check. I get the same error when trying to delete the deleted user's workspace:

unexpected status code 400: \"user\" must be an existing uuid or username."

I am running from an admin authenticated cli

coder delete <deleted_username>/<workspace_name> -y --orphan

To make the test pass, you must do:

coder delete <deleted_user_id>/<workspace_name> -y --orphan

What is happening

What is happening is that this line in the cli that fetches the workspace:

workspace, err := namedWorkspace(inv.Context(), client, inv.Args[0])

Hits this route:

func (api *API) workspaceByOwnerAndName(rw http.ResponseWriter, r *http.Request) {

Which uses this middleware:

func ExtractUserParam(db database.Store, redirectToLoginOnMe bool) func(http.Handler) http.Handler {

We currently do not allow this middleware to return deleted users by username. This is because a deleted user does not guarantee a unique username. So the username could refer to any number of deleted users, or conflict with an active user.

@baxters-ctc
Copy link
Author

@Emyrk Is there a way for me to find out what that value of "deleted_user_id" is?

@Emyrk
Copy link
Member

Emyrk commented Jun 14, 2023

@Emyrk Is there a way for me to find out what that value of "deleted_user_id" is?

Great question. Right now the /users endpoint excludes deleted users:

users.deleted = false

We might want to make that an option 🤔. I don't think there is a way to get the user_id except from maybe audit logs.

@bpmct Is there any reason to prevent being able to query for deleted users?

@matifali
Copy link
Member

Should we add a new category to the user state?
We currently have active and suspended; we can add deleted and show their user-id instead of username.
This should be listed then in the "Users" tab when filtered with the deleted filter

@Emyrk
Copy link
Member

Emyrk commented Jun 15, 2023

@matifali that does avoid adding a new knob

@Emyrk
Copy link
Member

Emyrk commented Jun 21, 2023

@baxters-ctc I was working on implementing this, and might have found a solution for you to get the deleted user id.

If this is still an issue, can you try this?

coder ls --search 2nd-test -o json

You can grep for the owner_id, then run the delete

coder ls --search 2nd-test -o json | grep \"owner_id\"
# Use the output from above for the deleted user.
coder delete <deleted_user_id>/2nd-test -y --orphan

@Emyrk
Copy link
Member

Emyrk commented Jun 21, 2023

After doing some work on this, I think we should leave the code how it is and not add any functionality regarding this issue.

First: This edge case should not happen, so it should be rare.

Second: It can be resolved, as mentioned above (#7872 (comment))

Third: Deleted user IDs can be found via database directly, audit logs, or through the resource itself (#7872 (comment))


The code solution to this would be to add the ability to list deleted users. However I think listing deleted users gets into privacy/security questions around who should be able to see all deleted users.

Since there exists work arounds for this edge case, I am going to close this issue.

If the problem still exists, or the mentioned workarounds do not work, anyone can feel free to reopen this PR with additional information.

@Emyrk Emyrk closed this as completed Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
s2 Broken use cases or features (with a workaround). Only humans may set this.
Projects
None yet
Development

No branches or pull requests

6 participants