Distributor merges instrumentation scopes with different attributes #8110

Describe the bug

When two ScopeSpans groups in the same ResourceSpans entry contain spans from the same trace and have identical scope names/versions but different scope attributes, Tempo returns both spans under the first scope's attributes. The second scope's attributes are lost.

To Reproduce

1. Start Tempo v3.0.3 (revision 1900ed7bb) with its OTLP/HTTP receiver enabled.
2. Send the request below directly to that receiver. It contains one resource entry, one trace, and two spans in different scopes with the same scope name/version but different scope attributes.
3. Retrieve the trace by ID using the final command. Both spans appear under a single scope.

Run in Bash with curl, openssl, and GNU date. Replace the addresses below with your Tempo endpoints. This test uses port 4418 for OTLP/HTTP and 3200 for queries.

trace_id=$(openssl rand -hex 16)
start_ns=$(date +%s%N)
end_ns=$((start_ns + 1000000))

curl --noproxy '*' --fail-with-body -sS \
  http://tempo:4418/v1/traces \
  -H 'Content-Type: application/json' \
  --data-binary @- <<JSON
{
  "resourceSpans": [{
    "resource": {
      "attributes": [{"key": "service.name", "value": {"stringValue": "scope-repro"}}]
    },
    "scopeSpans": [
      {
        "scope": {
          "name": "repro", "version": "1.0",
          "attributes": [{"key": "scope.label", "value": {"stringValue": "alpha"}}]
        },
        "spans": [{
          "traceId": "$trace_id", "spanId": "1111111111111111",
          "name": "expected-alpha", "kind": 1,
          "startTimeUnixNano": "$start_ns", "endTimeUnixNano": "$end_ns"
        }]
      },
      {
        "scope": {
          "name": "repro", "version": "1.0",
          "attributes": [{"key": "scope.label", "value": {"stringValue": "beta"}}]
        },
        "spans": [{
          "traceId": "$trace_id", "spanId": "2222222222222222",
          "name": "expected-beta", "kind": 1,
          "startTimeUnixNano": "$start_ns", "endTimeUnixNano": "$end_ns"
        }]
      }
    ]
  }]
}
JSON

printf '\nTrace ID: %s\n' "$trace_id"
curl --noproxy '*' --fail-with-body -sS \
  -H 'Accept: application/json' \
  "http://tempo:3200/api/traces/$trace_id"

Expected behavior

Span expected-alpha retains scope.label alpha and Span expected-beta retains the scope.label beta.

Observed behavior

The trace response contains one scope group holding both spans:

{
  "scope": {
    "name": "repro",
    "version": "1.0",
    "attributes": [{"key": "scope.label", "value": {"stringValue": "alpha"}}]
  },
  "spans": [
    {"name": "expected-alpha"},
    {"name": "expected-beta"}
  ]
}

Environment:

• Ubuntu 24.04
• Docker Compose, monolithic Tempo (-target=all).
• Tempo: v3.0.3, revision 1900ed7bb

Additional Context

In requestsByTraceID(), the scope-group map is shared across incoming scopes within each resource entry. Its key includes the trace ID, scope name, and version, but not scope attributes.

The first matching group stores Scope: ils.Scope and subsequent spans are appended without comparing attributes.