Skip to content

Consumer group name cannot be overridden in block-builder and live-store #7897

Open

Description

@asif-mahmood-nw

Describe the bug
In Tempo 3.0 microservices mode (Kafka ingest path), the consumer group names used by the block-builder, live-store, and metrics-generator components cannot be prefixed or customised. This makes Tempo unusable on shared/multi-tenant Kafka clusters

Observed consumer group names (Tempo v3.0.3):
Component │ Consumer group (on the wire)
block-builder │ block-builder
live-store │ tempo-distributed-live-store
metrics-generator │ metrics-generator

None of these can be given a tenant prefix (e.g., mytenant.block-builder).

The CLI flag -ingest.consumer-group exists (-help-all) and can be set, but the block-builder ignores it. Verified on the wire: with -ingest.consumer-group=mytenant.block-builder set on the container, the FindCoordinator request still uses group=block-builder.
Same outcome for live-store (-ingest.consumer-group ignored; group stays tempo-distributed-live-store) and metrics-generator (-generator.consumer-group ignored; group stays metrics-generator).

There is an open PR which fixes it for metrics-generator. #7613

To Reproduce
Steps to reproduce the behavior:

  1. Deploy tempo-distributed (chart 3.5.1, appVersion Tempo 3.0.3) in microservices mode.
  2. Set -ingest.consumer-group=mytenant.block-builder on the block-builder and live-store (and -generator.consumer-group=mytenant.metrics-generator on metrics-generator).
  3. Observe via Kafka broker logs / a protocol-aware proxy that the FindCoordinator/OffsetFetch requests still use groups block-builder, tempo-distributed-live-store, metrics-generator — the override has no effect.
  4. On a Kafka cluster with a PREFIXED/tenant-prefix consumer-group ACL, consumption fails with GROUP_AUTHORIZATION_FAILED error.

Expected behavior
Allow the consumer group name for each Kafka-consuming component to be configurable, e.g.:

  • Honor -ingest.consumer-group / ingest.kafka.consumer_group for block-builder and live-store, and -generator.consumer-group for metrics-generator; or
  • Add a global ingest.kafka.consumer_group_prefix that is prepended to all internally-derived group names.
    Ideally the derived name would still keep per-component distinctness (e.g., .block-builder, .live-store, .metrics-generator) so the components remain separate consumer groups.

Environment:

  • Infrastructure: EKS 1.36
  • Deployment tool: Helm
  • Tempo version: v3.0.3 (chart grafana-community/tempo-distributed 3.5.1).

Activity

  1. tomaszpapirzwierz commented on Sep 28, 2026

    @tomaszpapirzwierz

    Huge surprise to me as well to find out the inconsistency in consumer group naming across Tempo components. What I also found out is:

    metrics-generator is not universally hardcoded:
    • hardcoded in partition ring mode
    • custom group survives in generator ring mode

    live-store is not a pure fixed literal:
    • derived from instance ID prefix-stripping
    • still effectively not directly configurable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions