You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
S3 backend fails to start on S3 directory buckets (Local Zones, S3 Express One Zone) #8035
Tempo won't start when the S3 backend points at an S3 directory bucket. Ours is in the Istanbul Local Zone (<name>--euc1-ist1-az1--x-s3). The usage-report module fails during startup:
msg="error running Tempo" err="failed to init module services: error initialising module: usage-report: failed to initialize usage report: unexpected error from ListObjects on <name>--euc1-ist1-az1--x-s3: This bucket does not support ListObjects API. Consider using ListObjectsV2 API."
Loki runs fine against the same bucket.
Directory buckets only support ListObjectsV2. The S3 backend still calls the V1 ListObjects API in three places: the bucket check in New(), List() and ClearBlock(). Switching those over isn't enough on its own. ListBlocks() already uses V2, but it passes start-after and relies on keys coming back sorted. Directory buckets support neither.
Authentication is the other gap. Directory buckets expect S3 Express session auth: you call CreateSession, then sign requests for the s3express service with the session token. minio-go never does this for Local Zone buckets, because its bucket-name regex only matches Availability Zone IDs like use1-az4. Loki uses AWS SDK v2, which handles all of this.
To Reproduce
Create an S3 directory bucket, in a Local Zone or with S3 Express One Zone.
Start Tempo <version> with storage.trace.backend: s3 pointing at that bucket.
Tempo exits during module init with the error above.
Expected behavior
Tempo starts and works against directory buckets, the same way Loki does.
Environment:
Infrastructure: Kubernetes
Deployment tool: Helm
Additional Context
I've opened a fix: #8034. It switches all listing to ListObjectsV2 and lists directory buckets without start-after or relying on sort order. It also adds S3 Express session signing on top of minio-go.
Activity