Slack groupingKey thread_ts not persisted - threading breaks after controller restart or leader change #4809

Describe the bug

When using groupingKey in a Slack notification template, subsequent notifications for the same rollout are expected to be sent as replies in the same Slack thread. This works correctly within a single controller process lifetime, but threading breaks whenever the controller restarts or leadership changes in a multi-replica setup.

The root cause is that when a Slack message is sent with a groupingKey, the Slack API returns a thread_ts - the unique timestamp ID of the sent message. Currently, thread_ts is stored only in the memory of the controller pod that sent the original message. It is never persisted to Kubernetes (not in annotations, Secrets, or ConfigMaps).

As a result, when the controller pod restarts or leadership changes in a multi-replica deployment, the thread_ts is lost and the next notification creates a new top-level Slack message instead of replying in the existing thread.

To Reproduce

1. Deploy argo-rollouts with replicas: 2
2. Configure a Rollout with a Slack notification template using groupingKey
3. Trigger a rollout - observe the "Start Running" Slack message and thread
4. Let the rollout reach a Pause step - observe the Pause notification sent as a reply in the thread ✓
5. Restart the argo-rollouts controller pods (or wait for a natural leader change)
6. Restart the Rollout(e.g. via kubectl argo rollouts restart )
7. Observe the new Pause notification - it creates a new top-level Slack message instead of replying in the existing thread ✗

Expected behavior

All notifications for the same groupingKey are sent as replies in the same Slack thread, regardless of which controller pod processes them or how many times the controller restarts.

The groupingKey is configured as:

groupingKey: "...{{ .rollout.status.currentPodHash }}"
The intent is that all notifications for the same rollout revision (same pod hash) are grouped into a single Slack thread.
This works correctly within a single controller process lifetime but breaks as described.

Failure conditions

The threading breaks in any of the following cases:

1. Leader pod restarts - slackState is reset on startup regardless of whether the same pod wins the leader election again or not. The thread_ts is gone either way.
2. Leader changes without a restart - In a multi-replica setup, leader election can switch between pods due to a lease timeout or network partition. The new leader pod has its own separate slackState with an empty ThreadTSs map, with no knowledge of thread_ts values stored by the previous leader.

In all cases, the next notification creates a new top-level Slack message instead of a reply in the existing thread.

Version

v1.9.0