Another scenario idea from @bpmct:

1. Upload a template.
2. Start a workspace.
3. Push the new template version.
4. Update the workspace.

Ben posted a few harmful issues from the past. It would be great to cover them tests:

#6799 - root cause: new CLI introduced a regression. Reproduce: create a template, create a workspace, update coder instance, and push a new template.

#6807 - RBAC misconfiguration. Reproduce: create a workspace, using a regular user account try to start the workspace

#7034 - git askpass stopped working for multiple weeks, TBD

Re: Askpass, I'm not familiar with that flow. I might need to understand the context first. @bpmct Did it occur only in Windows?

#6746 - blank page after the update, depending on the environment and DERP connectivity

Summing up:

1. We may want to extend scenarios with a mid-scenario Coder update.
2. It would be beneficial to perform some actions as a regular user.

Re: Askpass, I'm not familiar with that flow. I might need to understand the context first. @bpmct Did it occur only in Windows?

Nope, new git clone using Git Authentication would hang inside workspaces

Status update -

I didn't have much time to work on e2e this week, and I see that the reviving PR has already diverged (modified site header, I guess?). Anyway, let me share my thoughts:

1. Running E2E test as part of the GitHub pipeline doesn't make debugging easier. Flaky tests can be retried, but developer won't be able to hop on the host the monitor the execution (for instance dump db state).
2. I'm having problems with setting up Docker host, which means that I have to adjust/hack standard host settings.
3. We don't control GitHub infra, so e2e tests will be vulnerable to transient network issues.

The new approach is as follows:

1. Create the E2E Coder template that can pull e2e tests from the repository, and binaries from releases.
2. The template configures a cloud machine (OSX, Linux, Windows), and installs Coder.
3. E2E tests can be started automatically after startup or by the operator.

EDIT:

We found one more issue - currently e2e tests don't use release binary, but go run the source code, which means that they don't test if the release is correct, whether it contains slim binaries, and if it's possible to download agent from the coderd.

> GET /bin/coder-linux-amd64 HTTP/2
> Host: p5fhcggssugps.pit-1.try.coder.app
> user-agent: curl/7.81.0
> accept: */*
> accept-encoding: deflate, gzip, br, zstd
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [130 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
< HTTP/2 404 
< alt-svc: h3=":443"; ma=2592000
< content-encoding: zstd
< content-security-policy: script-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; default-src 'self'; font-src 'self' data:; worker-src 'self' blob:; manifest-src 'self' blob:; connect-src 'self' wss://p5fhcggssugps.pit-1.try.coder.app ws://p5fhcggssugps.pit-1.try.coder.app; frame-src 'self'; frame-ancestors 'none'; child-src 'self'; img-src 'self' https: data:; form-action 'self'; media-src 'self'; report-uri /api/v2/csp/reports;
< content-type: text/plain; charset=utf-8
< date: Fri, 28 Apr 2023 12:53:53 GMT
< permissions-policy: accelerometer=(), autoplay=(), battery=(), camera=(), document-domain=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), payment=(), usb=(), vr=(), screen-wake-lock=(), xr-spatial-tracking=()
< referrer-policy: no-referrer
< server: Caddy
< set-cookie: csrf_token=ftJYcMILpy3FTWy/x/pb50s+lmSew4zimft7/mEEby8=; Path=/; HttpOnly; SameSite=Lax
< vary: Accept-Encoding
< x-coder-build-version: v0.0.0-devel
< x-coder-request-id: 73760b8b-bd94-49a1-972f-5077ee944a8b
< x-content-type-options: nosniff
< content-length: 32
* The requested URL returned error: 404
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* stopped the pause stream!
*

I believe now we are at a point when we can test an end to end a new release. We can use the same approach we use for deploying PR environments.
This currently includes,

1. A real database
2. Creating a template
3. Creating a workspace
4. Stopping the workspace
5. Upgrading the deployment with a new build.

Alright, looks like we tweaked E2E tests, so this can be closed.