Skip to content

Add per db metrics #4183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

benjaminjb
Copy link
Contributor

Checklist:

  • Have you added an explanation of what your changes do and why you'd like them to be included?
  • Have you updated or added documentation for the change, as applicable?
  • Have you tested your changes on all related environments with successful results, as applicable?
    • Have you added automated tests?

Type of Changes:

  • New feature
  • Bug fix
  • Documentation
  • Testing enhancement
  • Other

What is the current behavior (link to any open issues here)?

No per-db metrics

What is the new behavior (if this is a feature change)?

  • Breaking change (fix or feature that would cause existing functionality to change)

  • Add per-db metrics (based on pgMonitor v5.2.1)

  • Allow per-db custom metrics

Other Information:
Issues: [PGO-21]

@benjaminjb benjaminjb changed the title Benjb/per db metrics Add per db metrics May 19, 2025
@benjaminjb benjaminjb marked this pull request as ready for review May 20, 2025 15:21
@benjaminjb
Copy link
Contributor Author

Some notes on this PR:

  • I based the per db metrics on pgmonitor 5.2.1, which Drew is bumping to elsewhere
  • some per db metrics were already included in internal/collector/gte_pg16_slow_metrics.yaml/internal/collector/lt_pg16_slow_metrics.yaml, so I moved them over to the per db metric list; this let me delete those files and update some of the logic for adding metrics
  • I added steps to our OTel KUTTL test (beginning to strain at the bounds of a reasonable test, might break up in the future) to test the different functionality added: targeting certain dbs with the defaults; removing metrics from the defaults; adding custom per db metrics.

@@ -2165,6 +2170,12 @@ spec:
type: string
type: array
type: object
perDBMetricTargets:
Copy link
Member

@cbandy cbandy May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 IIUC, this and customQueries only make sense at/on Postgres. Is there a way to arrange structs so the PGAdmin API doesn't have these fields? Not a blocker; v1beta1.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want to make any too big changes with the spec, but I wonder if the pgadmin and postgrescluster versions will diverge even more (or if we'll eventually want to slice up instrumentation by target-type, eg., postgres-instrumentation, pgbouncer-instrumentation, pgadmin-instrumentation).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm yeah good point... Probably makes sense to at least have a pgadmin-instrumentation and a postgrescluster-instrumentation and then pick and choose which structs make sense in each

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I add "taking apart the instrumentation struct" to this PR?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might make sense if you have the bandwidth...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the other work due this sprint, I'm going to make a ticket for this topic and not include it in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made that other ticket.

@benjaminjb
Copy link
Contributor Author

@dsessler7 This failed at finding patroni logs -- didn't we run into that before?

@dsessler7
Copy link
Contributor

dsessler7 commented May 27, 2025

@dsessler7 This failed at finding patroni logs -- didn't we run into that before?

It looks like it is still using the 5.8.1 postgres-operator image even though the 5.8.2 image is specified for RELATED_IMAGE_COLLECTOR in the kuttl-k3d gh action job... 🤔 We aren't prefetching the postgres-operator image (which we should), but it should still pull the image specified by the related image...

UPDATE: Okay, it looks like your branch still has the 5.8.1 image set in the GH action, so I think you just need to rebase on main.

"datasource": fmt.Sprintf(
`host=localhost dbname=%s port=5432 user=%s password=${env:PGPASSWORD}`,
db,
MonitoringUser),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Will the monitoring user always have access to whatever databases the user specifies?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking into this -- I got the metrics I expected, but why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be because we GRANT pg_monitor to ccp_monitoring; in the setup.

// Default behavior is to target `postgres`.
// ---
// +optional
Database string `json:"database,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want the user to be able to specify multiple databases for a given batch of custom queries? 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question, I went back and forth a bit, as a user could just add another customQueries section with a ref to the same configMap/queries and a new db -- but I'm not sold on that interface as more correct than just adding an array of DBs to this field.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Want to table this discussion and get feedback from the demo tomorrow?

This PR adds changes to allow per-db metrics in OTel:

- change API for per-db metrics
- add default metrics for per-db metrics based on pgmonitor 5.2.1
- remove unusused metrics
- add kuttl test
@benjaminjb benjaminjb force-pushed the benjb/per-db-metrics branch from bdbb541 to 954784a Compare May 28, 2025 16:35
Co-authored-by: Drew Sessler <36803518+dsessler7@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants