[Bug]: LogService bootstrap briefly resurrects a removed initial HAKeeper replica #32135

Description

After an initial HAKeeper replica has been removed from the current Raft membership and from the store's local mo-logservice.metadata, restarting that store with its original bootstrap-cluster=true configuration can briefly start the removed initial replica from old Dragonboat files. HAKeeper later sends L/Kill, but the removed replica has already run locally and attempted PreVote/ReadIndex.

Environment

• MatrixOne upstream/main at d61a7206c16b30baa4efc525eb6067c4364035aa.
• Isolated three-voter LogService and HAKeeper cluster on 10.222.1.51, with a fourth fresh-directory spare. The old store retains its original data directory and UUID. Its config has bootstrap-cluster=true, init-hakeeper-members including initial HAKeeper ID 131072, and concrete service-addresses for the three initial nodes. This is not a discovery-only configuration.
• No TN/CN was started. The tested current HAKeeper shard membership before restart was node1 385028, node2 395265, node3 477186; it did not contain node0 or 131072.

Reproduction

1. Start the three initial LogService stores. Keep node0's old Dragonboat Raft files and original bootstrap config.
2. Keep node0 down long enough for HAKeeper to remove its initial shard-0 replica 131072 from current membership. Restart node0 and wait for L/Kill ...:0:131072:0 to finish removing that ID from mo-logservice.metadata.
3. Stop node0 again. Start a spare store with a new, empty directory and wait for HAKeeper shard 0 to move to it, so the current three voters exclude node0.
4. Before each retry, verify all of these conditions: 131072 is absent from node0's local metadata and the authoritative three-member view; the old Dragonboat files for 131072 still exist; node0's original bootstrap-cluster=true config still lists 131072.
5. Restart node0 with its old directory and configuration. Inspect its startup log and NodeHost activity until the later L/Kill completes.

For the already prepared isolated directory, the test runner is:

root=/home/mo/jxm/logservice-chaos-20261103-d61a7206
python3 "$root/reproduce_bootstrap_zombie.py" "$root" \
  --rounds 3 --old-id 131072 \
  --peer-addresses 127.0.0.1:56411,127.0.0.1:56421,127.0.0.1:56431

The script is also in test/logservice-chaos/reproduce_bootstrap_zombie.py. It waits for metadata cleanup after each L/Kill; stopping at the first applying cmd: L/Kill log is too early because that log precedes stopReplica/removeMetadata.

Actual Result and Repetition

With the complete preconditions above, 3/3 consecutive restarts replayed the removed 131072 Raft instance, attempted PreVote, and later received L/Kill. The replay-to-kill windows were approximately 5.9, 3.0 and 3.2 seconds. An earlier trial that killed node0 before metadata cleanup observed only 2/3; that trial lacked the required precondition and is not counted as a stable reproduction.

The original incident showed repeated PreVote/ReadIndex without a leader, plus local errors such as failed to get HAKeeper Leader ID: shard not found and failed to get cluster details: cannot locate ha keeper. In these tests, the old ID did not enter the current three-member table or become leader, and there was no confirmed data loss or final client write/read failure. The current voters remained node1/node2/node3 after the trials.

Expected Result

When authoritative current membership excludes an initial HAKeeper ID, a warm restart must skip that removed ID before starting any local Raft instance or advertising the store as its HAKeeper host. If the store should host HAKeeper again, HAKeeper must assign a current replica ID. A genuine first-cluster cold bootstrap, with no authoritative membership yet, must still start the configured initial members.

Code Evidence and Impact

pkg/logservice/store.go checks zombie replicas only among IDs loaded from local mo-logservice.metadata. Once 131072 has been removed from metadata, it cannot enter skippedZombies. pkg/logservice/service_bootstrap.go later reads 131072 from the unchanged initial config, sees no skip marker and calls startHAKeeperReplica, writing the removed ID back into local metadata. HAKeeper eventually detects and kills it. The existing TestServiceBootstrap/skips zombie HAKeeper replica pre-populates the skip set and does not cover the metadata-absent/config-present path.

The confirmed impact is a transient stale Raft instance and local HAKeeper errors during recovery. Before cleanup, the service's nonzero haKeeperReplicaID can also make the stale address discoverable, plausibly causing request errors or long tails; that downstream effect was not confirmed here. A simultaneous return of enough old initial stores to form an obsolete majority was not tested and must not be presented as observed.

Full preconditions, timeline and retained evidence: LogService three-node failure report. Raw evidence on the test host is in zombie-repeat-verified-20261104.log and node0's stdout.log under /home/mo/jxm/logservice-chaos-20261103-d61a7206/.

Related Issue and Regression Boundary

Related closed #27803 addressed a different path: stale replica IDs were still present in local metadata, but discovery-only configuration left the startup zombie check without concrete peers. Here metadata is already missing the old ID while the original bootstrap config still lists it, and concrete peer addresses are available. The prior fix and regression test do not exercise this bootstrap-only gap.

A regression should model metadata-absent/config-present/old-Raft-files-present with authoritative membership excluding the old ID, assert the old NodeHost replica never starts, then also verify valid cold bootstrap and assignment of a new current ID.