Skip to content

S3 backend fails to start on S3 directory buckets (Local Zones, S3 Express One Zone) #8035

Open

Description

@akin-ozer

Describe the bug

Tempo won't start when the S3 backend points at an S3 directory bucket. Ours is in the Istanbul Local Zone (<name>--euc1-ist1-az1--x-s3). The usage-report module fails during startup:

msg="error running Tempo" err="failed to init module services: error initialising module: usage-report: failed to initialize usage report: unexpected error from ListObjects on <name>--euc1-ist1-az1--x-s3: This bucket does not support ListObjects API. Consider using ListObjectsV2 API."

Loki runs fine against the same bucket.

Directory buckets only support ListObjectsV2. The S3 backend still calls the V1 ListObjects API in three places: the bucket check in New(), List() and ClearBlock(). Switching those over isn't enough on its own. ListBlocks() already uses V2, but it passes start-after and relies on keys coming back sorted. Directory buckets support neither.

Authentication is the other gap. Directory buckets expect S3 Express session auth: you call CreateSession, then sign requests for the s3express service with the session token. minio-go never does this for Local Zone buckets, because its bucket-name regex only matches Availability Zone IDs like use1-az4. Loki uses AWS SDK v2, which handles all of this.

To Reproduce

  1. Create an S3 directory bucket, in a Local Zone or with S3 Express One Zone.
  2. Start Tempo <version> with storage.trace.backend: s3 pointing at that bucket.
  3. Tempo exits during module init with the error above.

Expected behavior

Tempo starts and works against directory buckets, the same way Loki does.

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: Helm

Additional Context

I've opened a fix: #8034. It switches all listing to ListObjectsV2 and lists directory buckets without start-after or relying on sort order. It also adds S3 Express session signing on top of minio-go.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions