Comprehensive side effect audit logging #5419

Emyrk · 2022-12-14T16:59:43Z

Emyrk
Dec 14, 2022
Collaborator

Problem

Currently audit logging is initiated per route. This means that each route has at most 1 audit event created. However, single http request can cause multiple changes to multiple entities. This is currently not being tracked.

Examples

Create workspace

Create workspace has multiple db changes; workspace, parameters, provisioner_job, and workspace_build. Only workspace is logged even though 4 different db changes are made.

These tables can be updated in other routes and generate different audit log events (eg workspace_build). But because it is a side effect in the create workspace route, it is not logged the same as it is in update workspace.

Question

Should we design/implement a way to audit log side effects?

Example

Create workspace

An audit log for the create workspace should be made, as that is the user action. Side effects of the provisioner job and workspace build should also be logged so that they can be tracked in the context of the original user action.

Now a provisioner can be searched to see what jobs it ran and see the original context for each job. Without the linking and side effect audit logs, these jobs are impossible to track the origin because only the workspace diff and id are logged.

Proposal

Linked audit logs (parent -> child or primary -> secondary)

Allow creating linked audit logs with an optional parent_id field. Setting a parent_id of an audit log allows relating to a primary action. In the UI only parent logs should be shown. Children logs should be hidden under some sort of "toggle" to see all side effects of a primary action.

We should not allow infinite nesting, and only allow 1 level of "children" logs. We do not want to build full tracing here.

Linked audit log table

A new audit log table for these secondary audit logs would be created.

audit_diffs
- primary_log (id)
- resource_id
- resource_type (string)
- diff
- succeeded true/false

Other ideas?

You got any?

Kira-Pilot · 2022-12-14T17:16:57Z

Kira-Pilot
Dec 14, 2022
Collaborator

So glad you brought this up! It's definitely a pain point. Some color:

As you may be aware, we do actually log workspace build jobs, but you're absolutely right: they're not really associated with the workspace resource in any way. As a result, we are missing some information that would be nice to have, e.g. the IP address associated with the request.
I'm running into this issue right now as I try to audit group_members, which exist in a table separate from group. You can see some discussion in the open PR where we consider the pros and cons of adding another auditable resource.

So pain points from my end are scaling (how many auditable resources do we really want) and UX (how many audit log lines do we really want; how can we make associations between log items clear; how do we handle filtering).

2 replies

Emyrk Dec 14, 2022
Collaborator Author

Thank you for the extra context links.

A thought on secondary audit events is that many can be implemented on the persistence layer itself. So every change to the db will automatically drop these logs.

You bring up even more points of "secondary events" such as failed jobs. This would give the provisioner a way to drop an audit logs as a failed event relating to the primary user action.

This does bring up a question though. Should the provisioner log show in the UI as it's own row? Or underneath the Create Workspac audit log row in some toggle?

Kira-Pilot Dec 14, 2022
Collaborator

I like the idea of toggling because I actually think it might keep the UI simpler. What I would like to avoid is having to put in a bunch of filtering so users can make sense of the view - as @spikecurtis mentioned below, most filtering is probably going to be done in Splunk. IDK though - warrants more thought.

spikecurtis · 2022-12-14T18:20:38Z

spikecurtis
Dec 14, 2022
Collaborator

In the RFC, we set this as a requirement, and the proposed solution was to generate a RequestID that would tie together multiple logs if the request resulted in multiple actions.

8 replies

Emyrk Dec 14, 2022
Collaborator Author

It does seem like we might have to store the request_id or whatever ID is linking the audit logs in the db somewhere.

Originally I was thinking it can all go on the ctx, but @Kira-Pilot brings up examples where that is not the case.

spikecurtis Dec 14, 2022
Collaborator

I mean, we should already be storing it on the Audit Logs, but in the case that a request writes something to the database which then gets picked up by some background process (e.g. builds) then yes, the job that gets picked up would need the request ID.

Emyrk Dec 16, 2022
Collaborator Author

I do like the idea of request_id since it allows linking audit logs to other things too.

I still think there is value in primary and secondary, not just for a UI purpose. The primary action should be something a user can do in the product and makes intuitive sense. secondary audit logs should be granular and might only make sense to a more technical user.

bpmct Dec 19, 2022
Maintainer

Have we done any investigation into how other products that often involve linked, async requests handle their audit logs (e.g. CI vendors (e.g. circleci), git providers (e.g. gitlab), or identify-based access tools (e.g. teleport)?

I'm personally curious if there are any patterns in how they link/relate logs, mostly to ensure that we don't do something bespoke that may be confusing for customers who are used to consuming logs from these other platforms (via the UI, consuming via the API, or ingesting into Splunk/StackDriver)

Emyrk Dec 19, 2022
Collaborator Author

@bpmct I have not, but this is essentially tracing which is rather normal. The only flair is how we present it on our UI

Emyrk · 2022-12-21T16:00:38Z

Emyrk
Dec 21, 2022
Collaborator Author

Another reason for this is we log Login events for v1, but signing-out is logged as an api-key delete action. It would be better to be consistent and have a logout audit log, but the actual action deletes 1 or more api keys.

If we can link audit log actions, then we could have a login and logout event with the api key creation/deletes logged separately.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comprehensive side effect audit logging #5419

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Comprehensive side effect audit logging #5419

Emyrk Dec 14, 2022 Collaborator

Problem

Examples

Create workspace

Question

Example

Create workspace

Proposal

Linked audit logs (parent -> child or primary -> secondary)

Linked audit log table

Other ideas?

Replies: 3 comments · 10 replies

Kira-Pilot Dec 14, 2022 Collaborator

Emyrk Dec 14, 2022 Collaborator Author

Kira-Pilot Dec 14, 2022 Collaborator

spikecurtis Dec 14, 2022 Collaborator

Emyrk Dec 14, 2022 Collaborator Author

spikecurtis Dec 14, 2022 Collaborator

Emyrk Dec 16, 2022 Collaborator Author

bpmct Dec 19, 2022 Maintainer

Emyrk Dec 19, 2022 Collaborator Author

Emyrk Dec 21, 2022 Collaborator Author

Emyrk
Dec 14, 2022
Collaborator

Replies: 3 comments 10 replies

Kira-Pilot
Dec 14, 2022
Collaborator

Emyrk Dec 14, 2022
Collaborator Author

Kira-Pilot Dec 14, 2022
Collaborator

spikecurtis
Dec 14, 2022
Collaborator

Emyrk Dec 14, 2022
Collaborator Author

spikecurtis Dec 14, 2022
Collaborator

Emyrk Dec 16, 2022
Collaborator Author

bpmct Dec 19, 2022
Maintainer

Emyrk Dec 19, 2022
Collaborator Author

Emyrk
Dec 21, 2022
Collaborator Author