Skip to content

Cardinality limit on metrics (otel_metrics_overflow) #2997

@lalitb

Description

@lalitb

Discussed in #2993

Originally posted by xjanin July 8, 2024
Hi,
I've instrumented my application with opentelemetry-cpp (metrics only), and I use tags in my metrics. When testing with a low rate of metrics recording, I see my metrics with the expected tags in the prometheus export on the opentelemetry collector. However, when testing the application with a high rate of requests, and so a high rate of metrics recording, I get metrics aggregated in time series with no tags except "otel_metrics_overflow".

My understanding is that this should happens if the sdk collector has to collect metrics with a tag cardinality superior to 2000 in a collection cycle (by default).
Edit: https://opentelemetry.io/docs/specs/otel/metrics/sdk/#cardinality-limits

However, my tag cardinality doesn't increase with the number of request processed, and in the opentelemetry collector that my application uses, I don't see 2000 time series in the prometheus export.

So my questions are :

  • Do I have a wrong understanding of what "otel_metrics_overflow" means ?
  • Is there a way of monitoring the cardinality of the metrics produced by my application ?

Thank you and best regards,
Xavier

Metadata

Metadata

Assignees

Labels

triage/acceptedIndicates an issue or PR is ready to be actively worked on.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions