Well, the logs aren't in the database...

That's peculiar. AFAIK we only delete logs if the agent hasn't connected for >7 days. 🤔

Yeah, my current suspicion is some agent problem where it doesn't send any...

If this happens again, can you verify if there are script logs in /tmp and if the agent logs contain anything interesting? It feels unlikely that all startup scripts would fail to send logs (each module usually adds its own script with log output, and each script output is handled separately).

Oh snap!, we're deleting them. My logs where there yesterday and I refreshed the page this morning and they're gone.

We should at least add a message that logs are only being persisted for n days instead of it being blank. +1 for this being a (hidden) server flag

We could also just retain latest build logs indefinitely, should fully solve the issue and is a simple query change.

Unsure what the overhead is for this exactly and what our recommendation would be for large deployments with large scripts, but generally open to having these persisted.

@bpmct, if we always keep the log from latest build, and retain the behavior that we otherwise keep logs for up to 7 days. I don't think this changes much in terms of recommendations for customers.

Why? I believe it's fairly uncommon that workspaces are kept alive indefinitely. So once a workspace is stopped, the logs are eligible for deletion (after 7 days from start).

There's another protection in place, too, currently we cut off the logs at 1 MB written to DB, so the logs won't grow indefinitely either.

I think we should just notify the users that logs are gone, that seems fine to me.

We could also always persist logs for the latest build, that doesn't seem too bad.

We could also always persist logs for the latest build, that doesn't seem too bad.

This seems like the simplest option to me (easy win and doesn't require notifying the user).

#14460 should hopefully address this.