Hello Jon,
In the techdocs for the Telemetry Metric assertion, we do warn about cardinality. In particular, for backends and based on how they store metric data, but there can clearly be an impact on the gateway too. I worry that by setting this experimental property higher, you're just delaying the problem while using more gateway resources.
"It's important to be mindful of high cardinality attributes (e.g., timestamp, tags, etc.) and their potential impact on the backend storage's performance and scalability. In choosing what attributes to add, consider the trade-offs between granularity and scalability."
------------------------------
Ben Urbanski
Product Manager, API Gateway
Layer7 API Management
------------------------------
Original Message:
Sent: May 01, 2025 07:48 PM
From: Jon Schendt
Subject: Modification on OpenTelemetry Default Cardinality
Joe, that was my idea as well :) Glad that we're on the same wavelength!
otel.experimental.metrics.cardinality.limit does change the cardinality limit after a Gateway restart. I do wonder if those system.properties arguments get passed directly through to the various services, so the Gateway doesn't have to explicitly express them in configuration. If that's the case, there would be potentially many more options that I haven't looked at yet.
Thanks!
Original Message:
Sent: May 01, 2025 03:46 PM
From: Joseph Fry
Subject: Modification on OpenTelemetry Default Cardinality
Hey Jon,
I will take an inital stab at this. I'm sure others will provide the right answers later.
First of all, 2000 is the default in the SDK per https://opentelemetry.io/docs/languages/java/configuration/:
Properties for cardinality limits:
| System property | Description | Default |
|---|
otel.experimental.metrics.cardinality.limit | If set, configure cardinality limit. The value dictates the maximum number of distinct points per metric. This option is experimental and subject to change or removal. | 2000 |
And as you can see, the property to change it is experimental.
That said, I would would have to wait for someone with lower level knowledge to let us know if this property is exposed.
I have had some luck guessing with some of the other libraries used in the product. A CWP like this might work?:
io.opentelemetry.sdk.metrics.cardinalty.limit = 4000
Original Message:
Sent: May 01, 2025 12:38 PM
From: Jon Schendt
Subject: Modification on OpenTelemetry Default Cardinality
I'm running APIGW 11.1.1, and when turning on some metrics via telemetry metric assertion and eventually (sometimes minutes, sometimes hours) get a warning:
io.opentelemetry.sdk.metrics.internal.state.AsynchronousMetricStroage: Instrument has exceeded the maximum allowed cardinality (1999).
Is there a way through configuration to increase the maximum allowed cardinality? 2000 seems small and I can't reduce the uniqueness of the metrics. I know there's SDK configs, but I'm not sure if Layer7 supports that modification through system.properties.
Please let me know! I've got some other ideas but want to see what the community has to say about it!