Status update: Base logic is mostly done, however, some timeframe edge cases are not yet handled and there are a few open questions. Tests also need to be added.

• We currently accept input times with date+clock, we should perhaps either remove clock or enforce that it's 00:00:00.
• Allowing clock skews the interval report since we're not locked into whole days

Questions:

• What timezone should we use**?**, we could use deployment, user, or viewer
  • For simplicity/flexibility we can allow the dashboard to request whatever timezone it feels is best (as it is now)
  • For the most accurate (IMO), we'd use the user timezone so that when user A's Monday and users B's Monday appear on Monday in the data, even though they're on opposite sides of the globe
• App usage, should this be cumulative or an average of all users? (Current implementation is cumulative/per user)

Issues:

• We're relying heavily on workspace_agents_stats, however, user latency based on this table is skewed since it includes agent<->coderd latency
  • Also, if an admin e.g. does ssh otheruser/workspace, the admins session will be counted towards user stats as well (both latency and apps usage), that said, fixing this is not a high priority

Current response samples:

GET http://localhost:3000/api/v2/insights/user-latency?start_time=2023-07-18T00:00:00Z&end_time=2023-07-20T00:00:00Z

{
  "report": {
    "start_time": "2023-07-18T00:00:00Z",
    "end_time": "2023-07-20T00:00:00Z",
    "template_ids": [
      "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
    ],
    "users": [
      {
        "template_ids": [
          "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
        ],
        "user_id": "a6d24688-2c59-423a-a8b0-41d56331d4a1",
        "username": "admin",
        "latency_ms": {
          "p50": 0.327,
          "p95": 0.5352
        }
      },
      {
        "template_ids": [],
        "user_id": "e7217d35-9ce1-4c98-ae09-c6c9ad6f5686",
        "username": "member",
        "latency_ms": null
      },
      {
        "template_ids": [
          "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
        ],
        "user_id": "d9fb2aa8-fedb-4c76-a759-27e4e4c5d550",
        "username": "woot",
        "latency_ms": {
          "p50": 0.59,
          "p95": 0.59
        }
      }
    ]
  }
}

GET http://localhost:3000/api/v2/insights/templates?start_time=2023-07-18T00:00:00Z&end_time=2023-07-20T00:00:00Z&interval=day

{
  "report": {
    "start_time": "2023-07-18T00:00:00Z",
    "end_time": "2023-07-20T00:00:00Z",
    "template_ids": [
      "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
    ],
    "active_users": 2,
    "apps_usage": [
      {
        "template_ids": [
          "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
        ],
        "type": "builtin",
        "display_name": "Visual Studio Code",
        "slug": "vscode",
        "icon": "/icons/code.svg",
        "seconds": 0
      },
      {
        "template_ids": [
          "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
        ],
        "type": "builtin",
        "display_name": "JetBrains",
        "slug": "jetbrains",
        "icon": "/icons/intellij.svg",
        "seconds": 0
      },
      {
        "template_ids": [
          "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
        ],
        "type": "builtin",
        "display_name": "Web Terminal",
        "slug": "reconnecting-pty",
        "icon": "/icons/terminal.svg",
        "seconds": 4800
      },
      {
        "template_ids": [
          "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
        ],
        "type": "builtin",
        "display_name": "SSH",
        "slug": "ssh",
        "icon": "/icons/terminal.svg",
        "seconds": 1800
      }
    ]
  },
  "interval_reports": [
    {
      "start_time": "2023-07-18T00:00:00Z",
      "end_time": "2023-07-19T00:00:00Z",
      "template_ids": [
        "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
      ],
      "interval": "day",
      "active_users": 2
    },
    {
      "start_time": "2023-07-19T00:00:00Z",
      "end_time": "2023-07-20T00:00:00Z",
      "template_ids": [
        "bb20c5b7-3b3c-4031-868b-3aee8b80e79b"
      ],
      "interval": "day",
      "active_users": 1
    }
  ]
}

We currently accept input times with date+clock, we should perhaps either remove clock or enforce that it's 00:00:00.
Allowing clock skews the interval report since we're not locked into whole days

Thinking in terms of MVP, it is something we can iterate later. IMHO we should start with whole days, and exceptionally allow for the current day to be counted in hours (not 24h yet).

For simplicity/flexibility we can allow the dashboard to request whatever timezone it feels is best (as it is now)

Alternatively, you can depend on the deployment time+TZ, and just share it over Insights API, so that UI can consider it.

For the most accurate (IMO), we'd use the user timezone so that when user A's Monday and users B's Monday appear on
Monday in the data, even though they're on opposite sides of the globe

DevOps people are probably aware of this issue, so they will be fine with any option :) Do you find any of them easier to implement?

App usage, should this be cumulative or an average of all users? (Current implementation is cumulative/per user)
Issues:

cumulative/per user 👍 BUT... how will the API response look like if we are doing scale tests for 1000 users? AFAIR I asked about this before.

Also, if an admin e.g. does ssh otheruser/workspace, the admins session will be counted towards user stats as well (both latency and apps usage), that said, fixing this is not a high priority

I would park it for later 👍

Thinking in terms of MVP, it is something we can iterate later. IMHO we should start with whole days, and exceptionally allow for the current day to be counted in hours (not 24h yet).

Agreed, I'll add the "today" exception so that it's possible for the site to show stable numbers for today.

Alternatively, you can depend on the deployment time+TZ, and just share it over Insights API, so that UI can consider it.

Sure, we can do that. We'll have to remove TZ from the start_time/end_time request format as well.

DevOps people are probably aware of this issue, so they will be fine with any option :) Do you find any of them easier to implement?

Since we don't have knowledge of user TZ, let's go with the simple approach for now. 😄

[...] how will the API response look like if we are doing scale tests for 1000 users? AFAIR I asked about this before.

I guess the scaletest numbers would be included, unless there's some special handling to not gather stats in this case? It's unclear to me whether or not we'd want to filter them out, will scale testing be a common/recurring practice on live deployments?

I guess the scaletest numbers would be included, unless there's some special handling to not gather stats in this case? It's unclear to me whether or not we'd want to filter them out, will scale testing be a common/recurring practice on live deployments?

IMHO we have to look at scaletesting from a different angle. Running load tests with traffic ~1000 users means that any customer can potentially create such a large deployment. Coder API must sustain this load too, and in this case, I'm mostly concerned about the response size for the Insights endpoint.

What timezone should we use**?**, we could use deployment, user, or viewer

The principle of least surprise would be that the FE specifies the timezone to be used, auto-detecting from the browser.

App usage, should this be cumulative or an average of all users? (Current implementation is cumulative/per user)

Cumulative across all users makes the most sense here, I think. Per-user averages are too widely skewed.

I guess the scaletest numbers would be included, unless there's some special handling to not gather stats in this case? It's unclear to me whether or not we'd want to filter them out, will scale testing be a common/recurring practice on live deployments?

The scaletest command is hidden, so we should not expect it to be used regularly in the field.Th at having been said, all scaletest users and workspaces are prefixed with the string scaletest-, so excluding them from the /insights endpoint should not be too difficult. But maybe we do want to leverage scaletests to generate data for the insights endpoint? It should be performant even with a large amount of data.

@mtojek

IMHO we have to look at scaletesting from a different angle. Running load tests with traffic ~1000 users means that any customer can potentially create such a large deployment. Coder API must sustain this load too, and in this case, I'm mostly concerned about the response size for the Insights endpoint.

Ah right, this mainly affects the user-latency endpoint which we can add paginations to when/if needed. I think we can punt pagination for later, though. All stats in the templates endpoint are aggregated so we don't have the same scale issue there.

@johnstcn

The principle of least surprise would be that the FE specifies the timezone to be used, auto-detecting from the browser.

I'd be fine with this, it's easier to handle on the backend too (as-is currently) 😄, thoughts @mtojek?

Cumulative across all users makes the most sense here, I think. Per-user averages are too widely skewed.

Ah, I believe I've worded that too open ended. It's currently cumulative across all users, however, with per-user I meant group by, this e.g. comes into play in the following scenario:

User A has two workspaces and a VSCode sessions open to both workspaces from 9am to 10am. User A contributes to cumulative stats with 1h of VSCode usage (vs 2h). Same if it's two VSCode sessions open in one workspace, etc.

It's perhaps a weird line to draw, though, since the user can contribute with 1h vscode + 1h jetbrains + 1h terminal usage. The idea is anyway that a user can only focus on one app at a time.

But maybe we do want to leverage scaletests to generate data for the insights endpoint? It should be performant even with a large amount of data.

I agree. 👍🏻

The idea is anyway that a user can only focus on one app at a time.

I wouldn't necessarily make that assumption ;-)

I wouldn't necessarily make that assumption ;-)

I think it's a matter of making the data useful. For instance, take a deployment of one user with 8 workspaces based on one template. At the start of the day they launch a VSCode for each workspace and during the day keeps hopping between editor windows. Given the current implementation, we will see 8h of VSCode usage (logical, one employee working a full day), whereas without this assumption, we see 64h of VSCode usage for one day (confusing).

I think this is a question for @BrunoQuaresma and @bpmct, though, which matches their vision better?

Decisions made:

• User latency pagination will be left for later (/ as needed)
• Client provides timezone as part of start_time/end_time, can use deployment tz if that's preferred
• Time ranges can't be in the future, except "Today" which has a special case and if the clock is 15:33:00, an end time of 16:00:00 is OK (this includes all the latest data), although 15:00:00 may be preferable. Using anything other than 00:00:00 outside of "Today" is not allowed
• Assuming admin permissions, so all data is "fair game"

Things we may want to improve in the future:

• User latency, the data stored in workspace_agent_stats is suboptimal and includes agent<->coderd latency as well, not only user
• App usage calculations, right now we only know that a connection from e.g. VSCode is open, but not whether or not the user is actively using VSCode