Use Redis Hash Field Expiration to keep cache tag indexes always consistent and eliminate stale-tag maintenance (Redis >= 7.4)

[vladvildanov]

Problem

RedisTagSet tracks which cache keys belong to a tag using a sorted set per tag (tag:{name}:entries), where the member is the cache key and the score is its absolute expiration timestamp (RedisTagSet::addEntry(), src/Illuminate/Cache/RedisTagSet.php:19-29). Redis sorted sets have no concept of per-member expiration, so this score is purely bookkeeping data that Laravel itself must act on.

When a tagged cache entry's TTL elapses, the entry's own key disappears — but its reference inside the tag's sorted set does not. Nothing removes it automatically. The only cleanup mechanism is:

• RedisTagSet::flushStaleEntries() (RedisTagSet.php:86-101), which issues ZREMRANGEBYSCORE to prune members whose score is in the past
• the cache:prune-stale-tags artisan command (src/Illuminate/Cache/Console/PruneStaleTagsCommand.php), which application owners must remember to add to their own scheduler.

If that command is never scheduled — which is easy to forget, since nothing fails or warns when it isn't — tag sorted sets grow unbounded with references to cache entries that no longer exist. This has two visible costs as it grows:

1. Memory: the sorted sets accumulate dead references indefinitely.
2. Work: RedisTaggedCache::flush() and RedisTagSet::entries() (RedisTagSet.php:37-79) iterate every member via ZSCAN, including all the dead ones, issuing wasted DEL calls against keys that vanished long ago.

This is a workaround for something Redis didn't support when this code was written: per-member expiration on a collection type.

Proposal

Redis 7.4 introduced Hash Field Expiration (HEXPIRE, HPEXPIRE, HPERSIST, HTTL, plus HSCAN over fields) — TTLs scoped to individual hash fields rather than the whole key. This is the primitive that was missing.

For servers that support it, back each tag with a hash instead of a sorted set: field = cache key, and the field's own TTL (set via HEXPIRE, using the same relative $ttl already passed into addEntry()) mirrors the underlying cache entry's expiration. Redis then expires the field itself — no score bookkeeping, no scheduled prune job, no possibility of unbounded ghost accumulation. flushStaleEntries() and cache:prune-stale-tags become unnecessary for correctness on these servers.

Benefits

• Always-consistent tag index, not just less cleanup. Today the tag's sorted set is only as accurate as the last time flushStaleEntries() ran — between prunes, it's a mix of live and dead references, and on a long-running or unscheduled app it trends toward mostly dead. With HFE, a field is gone the instant it expires: entries() and flush() are always operating on the true live set, not a snapshot that's stale by however long it's been since the last cron run.
• Faster flush()/entries() as a direct consequence, not a side effect. Because the index never accumulates dead references, ZSCAN-then-DEL-against-already-gone-keys work disappears entirely — this scales with how long a tag has been alive and how much traffic has churned through it, so the win compounds over the life of a long-running app rather than being a one-time fixed saving.
• Removes the maintenance burden of cache:prune-stale-tags — a command that's easy to forget to schedule, and whose omission silently degrades the two points above over time rather than failing loudly.
• Simpler addEntry() — no need to compute an absolute expiry timestamp as a sorted-set score; pass the relative TTL straight to HEXPIRE.

Would be happy to contribute a PR for this!

[mostafanajee-coder]

I think HFE is a very good fit for this, but there are a few implementation details worth addressing before replacing the current sorted-set approach.

The biggest one is that this would eliminate TTL-induced stale references, but not necessarily make the tag index universally "always consistent".

For example, a cache entry may disappear before its TTL because of an explicit forget(), eviction, or a failed/partial write. In those cases the hash field could still exist until its own TTL expires. So I would probably describe the benefit as eliminating stale references caused by normal TTL expiration, rather than eliminating every possible stale reference.

There are also a few important compatibility/implementation details:

1. Existing tag keys cannot simply change from ZSET to HASH

Today Laravel uses keys such as:

tag:{name}:entries

as sorted sets.

If an upgraded application starts issuing HSET against an existing tag key that is still a ZSET, Redis will return WRONGTYPE.

So the HFE implementation would need either:

• a versioned/new tag-index key,
• a migration strategy for existing ZSET indexes,
• or an explicit cache/tag reset during the transition.

This is probably the most important upgrade-path issue.

2. HSET clears an existing field expiration

Redis documents that overwriting a hash field with HSET clears that field's TTL.

That matters because Laravel currently has this behavior:

$this->tags->addEntry($this->itemKey($key), updateWhen: 'NX');

for increment() / decrement().

The current ZADD NX behavior intentionally avoids replacing an existing index entry. A hash implementation should preserve that semantic.

For example, HSETNX would be more appropriate there. Otherwise doing an ordinary HSET could accidentally turn an expiring tag reference into a persistent one before a new HEXPIRE is applied.

3. HSET + HEXPIRE should be atomic

For a normal finite-TTL entry, the implementation would roughly need:

HSET tag-key cache-key ""
HEXPIRE tag-key ttl FIELDS 1 cache-key

But if the process dies between those two commands, the field becomes persistent and we have recreated the stale-entry problem in a different form.

For Redis 7.4, I would therefore execute the write + expiration atomically, probably using Lua or a transaction.

Redis 8 has HSETEX, which makes this cleaner, but if Redis 7.4 is the target then Laravel cannot depend on it.

4. forever() and TTL transitions need explicit tests

The implementation should cover at least these transitions:

finite TTL -> new finite TTL
finite TTL -> forever
forever    -> finite TTL
forever    -> forever

In particular, changing an existing finite entry to forever() must ensure the hash-field expiration is removed.

5. Server support is not the only compatibility concern

Laravel supports both PhpRedis and Predis.

So capability detection should account for both:

• Redis server >= 7.4
• client support / invocation path for HEXPIRE, HPEXPIRE, HPERSIST, etc.

The existing sorted-set implementation should remain the fallback for older Redis installations.

That also means cache:prune-stale-tags would still need to exist for the legacy path, even though it could become unnecessary for the HFE-backed path.

6. entries() / flush() map nicely to hashes

Once the compatibility details are solved, the read/flush side becomes attractive:

ZSCAN -> HSCAN
ZREM  -> HDEL

and expired fields naturally disappear from the index, so normal TTL churn no longer causes the index to grow indefinitely.

I would definitely consider this worth pursuing.

My preferred design would probably be:

1. Keep the existing ZSET implementation as the legacy backend.
2. Add an HFE-backed implementation for supported Redis/client combinations.
3. Use a versioned tag-index key or a safe migration path instead of changing the Redis type of an existing key in place.
4. Make field creation + expiration atomic.
5. Preserve the current NX semantics used by increment/decrement.
6. Add tests specifically covering TTL transitions, forever(), increment/decrement, migration from legacy tag indexes, PhpRedis, Predis, and clustered connections.

With those pieces in place, HFE should remove the main reason Laravel currently needs stale-tag pruning for expiring entries, while keeping backward compatibility with older Redis installations.

[vladvildanov]

@mostafanajee-coder Thanks for your feedback! All points are pretty much valid and I agree we need: backward compatibility, clear migration path, consistency between data and metadata

[vladvildanov]

Related: #61328

Recently fixed the write/flush race in the same Redis tagged-cache path. This proposal addresses the separate accumulation of expired finite-TTL references.