Skip to content

DAU graph misleads admins on license utilization #15297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 of 6 tasks
stirby opened this issue Oct 30, 2024 · 29 comments · Fixed by #16134
Closed
3 of 6 tasks

DAU graph misleads admins on license utilization #15297

stirby opened this issue Oct 30, 2024 · 29 comments · Fixed by #16134
Assignees
Labels
customer-requested Features requested by enterprise customers. Only humans may set this. observability Issues related to observability (metrics, dashboards, alerts, opentelemetry) site Area: frontend dashboard

Comments

@stirby
Copy link
Collaborator

stirby commented Oct 30, 2024

Credit to @coadler for distilling this issue from customer feedback.

Problem statement

The General deployment settings page (dev link) contains a graph of daily active users across the deployment. This graph is purely intended to measure the adoption of the product within an organization and summarize user activity. Here's what this graph looks like in our dogfood deployment:

Image

However, for enterprise deployments with a license cap, a "user limit" bar appears at the top. This misleads admins into thinking this is a way to gauge license usage over time and judge whether expansion/contraction is necessary:

Image

The (?) tooltip doesn't clarify the purpose of this graph to admins either, simply explaining how activity is calculated:

Image

This ambiguity is a problem for administrators. To summarize

  1. The graph is only titled Active Users , which is also the terminology we use for license seats
  2. Nothing mentions this being DAU, except for hovering over a data point. The tool tip only explains what a user needs to do to be considered active
  3. The user limit bar on the top to me implies that this is connected to license usage

Proposal

We should clarify that this graph is purely to measure concurrent user activity.

Requirements:

Must have:

  • Clearly display this as Daily Active Users
  • Remove the user limit bar at the top for licensed deployments
  • Explain in the tooltip that this is for measuring user activity and has no connection to license consumption

Should have:

  • Add a new "Registered Users" chart as the first graph on the top with the upper limit bar, with the existing active users graph below it with no upper limit bar

Nice to have:

  • On the registered users graph, overlay "suspended" and "dormant" users. We may want to be careful here since SCIM creates a ton of initial dormant users who never log in so we would want to filter those out and only have dormant users who logged in.
  • Change Daily Active Users to configurable by day, week, and month just like template insights. Weekly is much more useful IMO
@coder-labeler coder-labeler bot added customer-requested Features requested by enterprise customers. Only humans may set this. docs Area: coder.com/docs observability Issues related to observability (metrics, dashboards, alerts, opentelemetry) labels Oct 30, 2024
@coadler coadler added site Area: frontend dashboard and removed docs Area: coder.com/docs observability Issues related to observability (metrics, dashboards, alerts, opentelemetry) labels Oct 30, 2024
@matifali
Copy link
Member

Instead of calling it licensed utilization chart, I suggest we call registered users count graph. It's helpful to know how the total registered users go over time on both OSS and licensed deployments.

We can put a horizontal line to indicate the limit for licensed deployments.

@bpmct
Copy link
Member

bpmct commented Oct 31, 2024

Additionally, we should consider creating a license utilization chart and adding it directly below this chart. That work will be tracked in another issue.

Instead of calling it licensed utilization chart, I suggest we call registered users count graph. It's helpful to know how the total registered users go over time on both OSS and licensed deployments.

We can put a horizontal line to indicate the limit for licensed deployments.

I just added these to this issue as should have and nice to have. We want to remove the confusion quickly, but if the should haves are relatively easy we get to do that now too. If it is more difficult, we can extract to a separate issue.

I totally agree that with just the should have, many users may still confuse that chart as utilization and registered users would be a very useful chart.

@bpmct bpmct added the must-do Issues that must be completed by the end of the Sprint. Or else. Only humans may set this. label Oct 31, 2024
@bpmct
Copy link
Member

bpmct commented Oct 31, 2024

@BrunoQuaresma do you think it is possible to do the "must have" part at least before the release? The "should have" can follow after the release as subsequent PRs?

@BrunoQuaresma
Copy link
Collaborator

@bpmct sure thing. I just would need the copy for the "Explain in the tooltip that this is for measuring user activity and has no connection to license consumption".

@BrunoQuaresma
Copy link
Collaborator

Since we are touching on this part of the UI, I think would be nice to have some thoughts from @chrifro on how to better display this data to the user.

@matifali
Copy link
Member

Add a new "Registered Users" chart as the first graph on the top with the upper limit bar, with the existing active users graph below it with no upper limit bar

This should be on /deployment/general page right?

@stirby
Copy link
Collaborator Author

stirby commented Oct 31, 2024

I don't think we need much redesign, we can use the existing graph component for "registered users" IMO.

@BrunoQuaresma
Copy link
Collaborator

We said the same about the forgot password 😆 I would like to have @chrifro decide.

@chrifro
Copy link

chrifro commented Nov 1, 2024

👋 I'd suggest that we add a design polish as a "nice to have". It shouldn't block the content changes that are requested by the customer and are more time-sensitive. At the same time, we can definitely make the graph visually more appealing and help to grasp the information quickly.

Some ideas:

  • use a chart design similar to shadcn
  • we could use the design with two lines (one for active user/per day and one for total registered users). The registered user graph could include a split for "suspended" and "dormant" users in the tooltip
  • extend the timeframe from the last two months to all-time data
  • allow filtering for days, weeks, months, and years. Maybe show monthly by default

I could create the design early next week. Let me know if I should focus on that.

Btw, thanks for the the great issue description @stirby, makes it easy to dive into this 🙏

@mtojek mtojek removed the must-do Issues that must be completed by the end of the Sprint. Or else. Only humans may set this. label Nov 4, 2024
@mtojek
Copy link
Member

mtojek commented Nov 4, 2024

FYI not a "must-do" anymore

@BrunoQuaresma
Copy link
Collaborator

@chrifro from the shadcn example charts, do you have a type preference? I think for this case, a stacked area chart could work very well. What do you think?

@chrifro
Copy link

chrifro commented Nov 8, 2024

@chrifro from the shadcn example charts, do you have a type preference? I think for this case, a stacked area chart could work very well. What do you think?

That's exactly the one I had in mind as well 👌
Note: I didn't look into a design for this as I didn't hear back on the priority. Let me know if you need/prefer a mockup

@matifali
Copy link
Member

matifali commented Nov 8, 2024

Maybe something like

This is AI generated at https://v0.dev

Image

@BrunoQuaresma
Copy link
Collaborator

BrunoQuaresma commented Nov 8, 2024

Here’s the current situation: we have not implemented a redesign for the charts or this page, so I'm focusing on making only the necessary changes. @chrifro, please let me know if you would like to redesign the component now or if you prefer to do it later.

Image

EDITED: I just realized that we need the total number of registered users, not just the new users. Making the correction now...

@chrifro
Copy link

chrifro commented Nov 11, 2024

We can stick to your version for now. It's a great improvement ✨

For the colors, are you using existing colors? If not, could you use the blue and green that are defined in the new design system?
For dark theme (see Figma for light theme)
blue-bg: ##082f49
blue-highlight: #38bdf8
green-bg: #052e16
green-highlight: #BBF7D0

Related issue #14780

@bpmct
Copy link
Member

bpmct commented Nov 11, 2024

Yep, we need a chart that shows the total users against the seat count. We do not have to put everything in the same chart, however.

@SasSwart
Copy link
Contributor

Potential partial blocker that I need input on. In short, I need to propose that we opt for a point in time gauge instead of a graph over time for total users against seat count. At least for this release.

This is because:

We don't track history for when users change status. We can't accurately display historical license count, because we only know whether a user is dormant or active today. We don't know when that changed. The same goes for when users where deleted. We know whether or not they were deleted. We don't know when they were deleted.

I can get a gauge done by release. The changes that need to be made to the db schema to support historical stats for dormancy and deletion require more consideration.

cc @matifali @chrifro @mafredri

@matifali
Copy link
Member

matifali commented Nov 26, 2024

Potential partial blocker that I need input on. In short, I need to propose that we opt for a point in time gauge instead of a graph over time for total users against seat count. At least for this release.

We already have a point-in-time value for registered users on the /deployment/licenses page. See the screenshot or are you talking about something else?
Image

Yep, we need a chart that shows the total users against the seat count. We do not have to put everything in the same chart, however.

What we need new is as @bpmct said a new chart to show the growth of registered users.

Can you paste a mockup of what is your proposal for what to put in this release?

@SasSwart
Copy link
Contributor

SasSwart commented Nov 26, 2024

What you've pasted there is what I had in mind. I'm glad we already have it.
As for the requirement of graphing over time:

Do we care about registered users or about license utilization? The original problem statement above is this:

This misleads admins into thinking this is a way to gauge license usage over time

If we just want registered users over time regardless of whether or not they are consuming a license, then I can go ahead and fulfil the requirement. If we care about license utilisation, then I need to make larger changes to the DB schema that will take time.

https://coder.com/docs/admin/users#:~:text=Dormant%20accounts%20do%20not%20count%20towards%20the%20total%20number%20of%20licensed%20seats%20in%20a%20Coder%20subscription%2C%20allowing%20organizations%20to%20optimize%20their%20license%20usage

@BrunoQuaresma BrunoQuaresma removed their assignment Dec 2, 2024
SasSwart added a commit that referenced this issue Dec 2, 2024
This PR is the first iteration towards #15297

We cannot yet show license utilization over time, so we show current
license utilization.
This is because we don't track user states over time. We only track the
current user state. A graph over time filtering by active users would
therefore not account for day to day changes in user state and be
inaccurate.
DB schema migrations and related updates will follow that allow us to
show license utilization over time.


![image](https://github.com/user-attachments/assets/91bd6e8c-e74c-4ef5-aa6b-271fd245da37)

---------

Co-authored-by: ケイラ <mckayla@hey.com>
stirby pushed a commit that referenced this issue Dec 2, 2024
This PR is the first iteration towards #15297

We cannot yet show license utilization over time, so we show current
license utilization.
This is because we don't track user states over time. We only track the
current user state. A graph over time filtering by active users would
therefore not account for day to day changes in user state and be
inaccurate.
DB schema migrations and related updates will follow that allow us to
show license utilization over time.

![image](https://github.com/user-attachments/assets/91bd6e8c-e74c-4ef5-aa6b-271fd245da37)

---------

Co-authored-by: ケイラ <mckayla@hey.com>
(cherry picked from commit 7e1ac2e)
@SasSwart
Copy link
Contributor

SasSwart commented Dec 4, 2024

Based on the conversations above, I've created this issue:
#15740 (comment)

I created a new one because I would like to clarify priority for the remaining graph. It would be easier to track in isolation.

It specifically captures the outstanding feature and documents how to achieve it. Everything else in this current issue has been accomplished.

I propose we close this issue and shift it's focus to the new one. Objections?

@matifali
Copy link
Member

matifali commented Dec 5, 2024

I am ok with closing this. But we can keep it open to track the nice to haves.
cc: @stirby

@mtojek mtojek closed this as completed Dec 11, 2024
@stirby stirby reopened this Dec 18, 2024
@coder-labeler coder-labeler bot added needs-triage Issue that require triage observability Issues related to observability (metrics, dashboards, alerts, opentelemetry) labels Dec 18, 2024
@stirby
Copy link
Collaborator Author

stirby commented Dec 18, 2024

Keeping open until we have a new time series graph for license utilization.

@matifali matifali removed the needs-triage Issue that require triage label Dec 19, 2024
@SasSwart
Copy link
Contributor

Latest mockup for the next iteration:
Image

Notes:

  • A tooltip would explain the meaning of active, dormant and suspended. It would mention that only active users consume a license seat.
  • I've tweaked the wording on the bottom graph to mention activity, not active users, because active users has a specific meaning, with which activity cannot be confused
  • The issue mentioned a specific charting library. I stuck with the one we're currently using for visual consistency.
  • As long as the green line is below the white line, the user is in a valid state. Red and gray are allowed to exceed the white line because they do not consume license seats.
  • If any of the three exceeds the user limit annotation line (white), then we can switch iconography and border colors to warning orange. if active exceeds the limit, we can switch to error red
  • TODO: I'd like to give the lines a background to more explicitly show that the lines are stacked.

@SasSwart
Copy link
Contributor

SasSwart commented Jan 3, 2025

@SasSwart SasSwart marked this as a duplicate of #15740 Jan 3, 2025
SasSwart added a commit that referenced this issue Jan 13, 2025
RE: #15740,
#15297

In order to add a graph to the coder frontend to show user status over
time as an indicator of license usage, this PR adds the following:

* a new `api.insightsUserStatusCountsOverTime` endpoint to the API
* which calls a new `GetUserStatusCountsOverTime` query from postgres
* which relies on two new tables `user_status_changes` and
`user_deleted`
* which are populated by a new trigger and function that tracks updates
to the users table

The chart itself will be added in a subsequent PR

---------

Co-authored-by: Mathias Fredriksson <mafredri@gmail.com>
@SasSwart
Copy link
Contributor

Status update. Changes have been merged that allow the database and backend to support this feature. A frontend PR is ready, but designs have since shifted and some more work on the frontend is required. To my knowledge, @BrunoQuaresma has agreed to support here. I've co-assigned him to the issue.

@mtojek
Copy link
Member

mtojek commented Jan 13, 2025

@SasSwart @BrunoQuaresma Can you describe the scope of remaining changes for transparency purposes? Unless they are documented somewhere else.

@BrunoQuaresma
Copy link
Collaborator

BrunoQuaresma commented Jan 13, 2025

The design has been updated and should now follow the design outlined here.

Image

@bartekgatzcoder
Copy link
Contributor

The General page is going to use the old API, but we rename the series on the UI to unconfuse the users. We also add additional explanation box.

The new API that Sas created is for the License page. In scope of the discussion we decided to use only the "Active Users" series from the API, so the "dormant" and "suspended" users will not be displayed. But the active users there are in fact the "users consuming the license seats" which is different from the "General page", where we show the daily user engagement.

To sum up:

  • general page: use old API that was there before
  • license page: use new API that Sas created, but only for the "active" series.
  • we need to touch both of the screens with the changes, consequently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer-requested Features requested by enterprise customers. Only humans may set this. observability Issues related to observability (metrics, dashboards, alerts, opentelemetry) site Area: frontend dashboard
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants