You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consumer group name cannot be overridden in block-builder and live-store #7897
Describe the bug
In Tempo 3.0 microservices mode (Kafka ingest path), the consumer group names used by the block-builder, live-store, and metrics-generator components cannot be prefixed or customised. This makes Tempo unusable on shared/multi-tenant Kafka clusters
Observed consumer group names (Tempo v3.0.3):
Component │ Consumer group (on the wire)
block-builder │ block-builder
live-store │ tempo-distributed-live-store
metrics-generator │ metrics-generator
None of these can be given a tenant prefix (e.g., mytenant.block-builder).
The CLI flag -ingest.consumer-group exists (-help-all) and can be set, but the block-builder ignores it. Verified on the wire: with -ingest.consumer-group=mytenant.block-builder set on the container, the FindCoordinator request still uses group=block-builder.
Same outcome for live-store (-ingest.consumer-group ignored; group stays tempo-distributed-live-store) and metrics-generator (-generator.consumer-group ignored; group stays metrics-generator).
There is an open PR which fixes it for metrics-generator. #7613
To Reproduce
Steps to reproduce the behavior:
Deploy tempo-distributed (chart 3.5.1, appVersion Tempo 3.0.3) in microservices mode.
Set -ingest.consumer-group=mytenant.block-builder on the block-builder and live-store (and -generator.consumer-group=mytenant.metrics-generator on metrics-generator).
Observe via Kafka broker logs / a protocol-aware proxy that the FindCoordinator/OffsetFetch requests still use groups block-builder, tempo-distributed-live-store, metrics-generator — the override has no effect.
On a Kafka cluster with a PREFIXED/tenant-prefix consumer-group ACL, consumption fails with GROUP_AUTHORIZATION_FAILED error.
Expected behavior
Allow the consumer group name for each Kafka-consuming component to be configurable, e.g.:
Honor -ingest.consumer-group / ingest.kafka.consumer_group for block-builder and live-store, and -generator.consumer-group for metrics-generator; or
Add a global ingest.kafka.consumer_group_prefix that is prepended to all internally-derived group names.
Ideally the derived name would still keep per-component distinctness (e.g., .block-builder, .live-store, .metrics-generator) so the components remain separate consumer groups.
Environment:
Infrastructure: EKS 1.36
Deployment tool: Helm
Tempo version: v3.0.3 (chart grafana-community/tempo-distributed 3.5.1).
Activity
tomaszpapirzwierz commentedon Sep 28, 2026
Huge surprise to me as well to find out the inconsistency in consumer group naming across Tempo components. What I also found out is:
metrics-generator is not universally hardcoded:
• hardcoded in partition ring mode
• custom group survives in generator ring mode
live-store is not a pure fixed literal:
• derived from instance ID prefix-stripping
• still effectively not directly configurable