YAML config apply resolves change_retentions against a stale table cache, crash-looping self-hosted deployments when tables are new #2150

Summary

When applying declarative YAML config (Sequin.YamlLoader.apply_from_stdin! via start_commands.sh), change_retentions source/destination tables are resolved against the cached table list stored on the postgres_databases record — not against the live database. The cache is only refreshed on create_db and by a 6-hourly cron (EnqueueDatabaseUpdateWorker). If a table was created after the last refresh, config apply fails even though the table exists and the connection role can see it:

** (RuntimeError) Failed to apply config: %Sequin.Error.BadRequestError{message: "Error setting up change retention '...': Not found: No `table` found matching params: `name=..., schema=public`", code: nil}

Because config apply runs at container start (before start_application), the failure crash-loops the container — which in turn largely prevents the 6-hourly refresh from ever running, so the deployment cannot self-heal.

Environment

• Sequin v0.14.1, self-hosted, headless (SERVER_CHECK disabled — no web console available)
• Config delivered via CONFIG_FILE_PATH, applied on every container start
• Source database on Cloud SQL for PostgreSQL 16

Steps to reproduce

1. Run self-hosted Sequin with a YAML config including a database and change_retentions.
2. After Sequin has been running for a while (so tables_refreshed_at is in the past), create a new table in the source database.
3. Add a change_retentions entry for the new table and restart the container so the config re-applies.
4. Config apply fails with No table found matching params, the container exits, and it keeps crash-looping until the cache happens to be refreshed.

Where this happens in the code

• lib/sequin/yaml_loader.ex — parse_wal_pipeline_attrs/2 calls fetch_table(source_database.tables, ...), which searches the cached list only.
• lib/sequin/databases/databases.ex — update_db/2 (the path taken for an existing database during config apply) never calls update_tables/1.
• config/config.exs — the only periodic refresh is {"0 */6 * * *", Sequin.Databases.EnqueueDatabaseUpdateWorker}.

Workarounds we tried

• Granting privileges / verifying the table is visible to the replication role: no effect (the lookup never touches the live DB).
• Hand-inserting an Oban job for Sequin.Databases.DatabaseUpdateWorker into oban_jobs: the job was never picked up inside the short crash-loop windows (manual inserts don't fire the insert notification, and flipping the row to scheduled for the stager didn't get it dispatched in time either).
• What finally works is either manually writing the table entry into the postgres_databases.tables jsonb, or waiting for a boot to happen to straddle the 6-hour cron tick — neither is great operationally.

Request

Any of the following would solve this class of problem:

1. Refresh on miss: when fetch_table misses during YAML config apply, call Databases.update_tables/1 once and retry the lookup before failing. This keeps the fast path cached but makes declarative config deterministic.
2. Refresh on apply: unconditionally refresh tables for each upserted database during apply_from_yml (config apply is rare and already talks to the database).
3. An operator escape hatch: a documented release task, e.g. bin/sequin eval "Sequin.Release.refresh_tables", and/or a CLI command (sequin databases refresh-tables) so headless deployments can force a refresh without touching internal tables.

Option 1 seems the most aligned with the existing design. Happy to contribute a PR if you'd accept one — thanks for the great tool!