Skip to content

Add run health watcher Helm deployment - #97

Open
yanhemage wants to merge 4 commits into
masterfrom
yan/stuck-run-health-diagnosis-recovery-alerts
Open

yanhemage wants to merge 4 commits into
masterfrom
yan/stuck-run-health-diagnosis-recovery-alerts

Conversation

@yanhemage

@yanhemage yanhemage commented Aug 7, 2026 •

Copy link
Copy Markdown
Contributor

This PR is based on stories Alert on stuck runs affecting subsequent scheduled triggers and Detect, diagnose, and terminate pipeline runs stuck in Running.

The associated feature PR : https://github.com/mage-ai/mage-pro/pull/3803

Summary

Adds optional Helm support for deploying the run health watcher as a dedicated Kubernetes workload.

This includes:

  • new runHealthWatcher values, disabled by default;
  • a dedicated run-health-watcher Deployment that runs python -m mage_ai.orchestration.run_health.watcher;
  • namespace-scoped ServiceAccount, Role, and RoleBinding for pod/job/event diagnosis and safe workload termination;
  • shared env/config/secret wiring so the watcher can access the same Mage, DB, Redis, workspace, and executor configuration as the main deployment;
  • Helm unit tests covering disabled rendering, Deployment rendering, mounts/env, and RBAC rendering.

Tests

Ran Helm unit tests:

helm unittest charts/mageai

Results:
Charts:      1 passed, 1 total
Test Suites: 3 passed, 3 total
Tests:       15 passed, 15 total

@yanhemage yanhemage changed the title [yh] Add run health watcher Helm deployment Add run health watcher Helm deployment Aug 7, 2026
@yanhemage

Copy link
Copy Markdown
Contributor Author

Run Health Watcher Staging Deployment Guide on AWS EKS V2

This guide documents the staging deployment path used for the run-health watcher
Helm chart validation on the existing workspace-mode Mage Pro staging cluster.

Use this v2 flow for validating the Helm chart PR against:

Staging URL: https://cluster-staging.mage.ai/manage
AWS account: 679849156117
AWS region: us-west-2
EKS cluster: mage-pro-staging
Namespace: default
Helm release: my-mageai
Mage image repository: 679849156117.dkr.ecr.us-west-2.amazonaws.com/mage-pro
Mage image tag: 20280806-stuck-run-health-diagnosis-recovery-alerts-py3.12

The run-health watcher is a dedicated Kubernetes Deployment, not a sidecar and
not an in-process background thread inside the Mage web/scheduler pod.

Why V2 Uses default/my-mageai

The earlier staging attempt targeted a separate release in the mage namespace.
That path tried to provision new workspace EFS PVCs and hit staging storage
provisioning issues. The cluster owner recommended using the existing workspace
mode release instead:

Namespace: default
Release: my-mageai
Manage URL: https://cluster-staging.mage.ai/manage

This existing release already has the intended workspace storage model:

WORKSPACE_PVC_NAME=pvc-mageai-staging
WORKSPACE_PVC_SUBPATH_PREFIX=mage_pro/workspaces/mageai-staging

That means child workspaces reuse the shared EFS PVC and a workspace-specific
subpath instead of provisioning a new PVC per workspace.

1. Refresh AWS And Kubernetes Access

The yan-he IAM user is protected by MFA. Use an MFA-backed session before
running Helm or kubectl against mage-pro-staging.

Create a helper script once:

cat > /tmp/refresh-mage-staging-kube.sh <<'SH'
#!/usr/bin/env zsh
set -euo pipefail

MFA_CODE="${1:-}"

if [[ -z "$MFA_CODE" ]]; then
  echo "Usage: source /tmp/refresh-mage-staging-kube.sh <YOUR_6_DIGIT_MFA_CODE>"
  return 1 2>/dev/null || exit 1
fi

unset AWS_PROFILE
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN

read AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_SESSION_EXPIRATION <<EOF
$(aws sts get-session-token \
  --serial-number arn:aws:iam::679849156117:mfa/yan-he \
  --token-code "$MFA_CODE" \
  --duration-seconds 43200 \
  --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken,Expiration]' \
  --output text)
EOF

export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN

echo "AWS session expires at: $AWS_SESSION_EXPIRATION"

aws sts get-caller-identity
aws eks update-kubeconfig --region us-west-2 --name mage-pro-staging

echo "Current kube context:"
kubectl config current-context

echo "Kubernetes access check:"
kubectl get ns
SH

chmod +x /tmp/refresh-mage-staging-kube.sh

Refresh the current terminal session:

source /tmp/refresh-mage-staging-kube.sh <YOUR_6_DIGIT_MFA_CODE>

Do not paste MFA codes into shared logs, PR comments, or chat.

2. Set Deployment Variables

Run from the Helm chart feature worktree:

cd /Users/yanhe/mage/helm-charts-stuck-run-health-diagnosis-recovery-alerts

export MAGE_NAMESPACE=default
export MAGE_RELEASE=my-mageai
export HELM_CHART=/Users/yanhe/mage/helm-charts-stuck-run-health-diagnosis-recovery-alerts
export MAGE_IMAGE_REPOSITORY=679849156117.dkr.ecr.us-west-2.amazonaws.com/mage-pro
export MAGE_IMAGE_TAG=20280806-stuck-run-health-diagnosis-recovery-alerts-py3.12
export RUN_HEALTH_VALUES=/tmp/run-health-watcher-default-values.yaml

Confirm the existing release:

helm list -n default

Expected release:

my-mageai  default  deployed  mageai-0.2.15

3. Confirm The Mage Pro Image Exists

aws ecr describe-images \
  --region us-west-2 \
  --repository-name mage-pro \
  --image-ids imageTag="$MAGE_IMAGE_TAG"

Expected image from this validation:

imageDigest: sha256:f382916e7399c18e195ef209bf23a4cb83a6e3333b59cd84fe1d5f5f03cda37d
imageStatus: ACTIVE

4. Back Up Existing Helm Values

The cluster owner specifically recommended starting from the live my-mageai
values. This preserves the staging workspace-mode configuration.

helm get values "$MAGE_RELEASE" -n "$MAGE_NAMESPACE" -o yaml \
  > /tmp/my-mageai-values-backup.yaml

Inspect workspace storage settings:

rg -n "PROJECT_TYPE|WORKSPACE|workspace|storage|persistence|extraEnvs|image:" \
  /tmp/my-mageai-values-backup.yaml

Important expected values:

PROJECT_TYPE=main
WORKSPACE_PVC_NAME=pvc-mageai-staging
WORKSPACE_PVC_SUBPATH_PREFIX=mage_pro/workspaces/mageai-staging

Do not replace this values file with a minimal values file. Helm arrays replace
rather than merge, so a partial extraEnvs override can accidentally remove
existing staging configuration.

5. Create The Run-Health Watcher Overlay

Create a run-health-only overlay. Do not override workspace storage settings in
this file.

cat > "$RUN_HEALTH_VALUES" <<'YAML'
runHealthWatcher:
  enabled: true
  replicaCount: 1
  env:
  - name: MAGE_RUN_HEALTH_WATCHER_ENABLED
    value: "true"
  - name: MAGE_RUN_HEALTH_WATCHER_AUTO_TERMINATE_ENABLED
    value: "false"
  - name: MAGE_RUN_HEALTH_WATCHER_AUTO_TERMINATE_MAX_ACTIONS_PER_SCAN
    value: "0"
  - name: MAGE_RUN_HEALTH_WATCHER_CADENCE_SECONDS
    value: "60"
  - name: MAGE_RUN_HEALTH_WATCHER_LEASE_TTL_SECONDS
    value: "120"
  - name: MAGE_RUN_HEALTH_WATCHER_LONG_RUNNING_MINUTES
    value: "120"
  - name: MAGE_RUN_HEALTH_WATCHER_MINIMUM_IMPACT_COUNT
    value: "1"
  - name: MAGE_RUN_HEALTH_WATCHER_NO_PROGRESS_MINUTES
    value: "30"
  - name: MAGE_RUN_HEALTH_WATCHER_NOTIFICATION_COOLDOWN_MINUTES
    value: "60"
  - name: MAGE_RUN_HEALTH_WATCHER_OLD_ACTIVE_BACKFILL_LIMIT
    value: "25"
  - name: MAGE_RUN_HEALTH_WATCHER_SCAN_WINDOW_MINUTES
    value: "1440"
  - name: MAGE_RUN_HEALTH_WATCHER_SCHEDULER_HEARTBEAT_STALE_MINUTES
    value: "10"
YAML

6. Render And Dry-Run

Render/dry-run using the live staging values plus the run-health overlay:

helm upgrade --install "$MAGE_RELEASE" "$HELM_CHART/charts/mageai" \
  --namespace "$MAGE_NAMESPACE" \
  --values /tmp/my-mageai-values-backup.yaml \
  --values "$RUN_HEALTH_VALUES" \
  --set image.repository="$MAGE_IMAGE_REPOSITORY" \
  --set image.tag="$MAGE_IMAGE_TAG" \
  --dry-run --debug \
  > /tmp/run-health-default-dry-run.yaml

Inspect the rendered output:

rg -n "kind: Deployment|name: run-health-watcher|MAGE_RUN_HEALTH_WATCHER|image:|PROJECT_TYPE|WORKSPACE_|pvc|claimName" \
  /tmp/run-health-default-dry-run.yaml

Expected signals:

Deployment/run-health-watcher is rendered
run-health-watcher uses the PR image tag
MAGE_RUN_HEALTH_WATCHER_* env vars are rendered
WORKSPACE_PVC_NAME=pvc-mageai-staging is preserved
claimName: pvc-mageai-staging is preserved

Treat /tmp/run-health-default-dry-run.yaml as sensitive because it can include
user-supplied values and inherited secrets/tokens from the existing Helm release.

7. Apply To default/my-mageai

helm upgrade --install "$MAGE_RELEASE" "$HELM_CHART/charts/mageai" \
  --namespace "$MAGE_NAMESPACE" \
  --values /tmp/my-mageai-values-backup.yaml \
  --values "$RUN_HEALTH_VALUES" \
  --set image.repository="$MAGE_IMAGE_REPOSITORY" \
  --set image.tag="$MAGE_IMAGE_TAG" \
  --wait --timeout 10m

Successful output:

Release "my-mageai" has been upgraded. Happy Helming!
STATUS: deployed
DESCRIPTION: Upgrade complete

If Helm or kubectl reports that the cluster is unreachable or credentials are
required, refresh the current terminal again:

source /tmp/refresh-mage-staging-kube.sh <YOUR_6_DIGIT_MFA_CODE>

8. Verify Deployment And Watcher Health

kubectl -n default get deploy mageai run-health-watcher

Expected:

mageai               1/1
run-health-watcher   1/1

Check watcher logs:

kubectl -n default logs deploy/run-health-watcher --tail=100

Startup logs may be quiet after Alembic/PostgreSQL initialization:

INFO:alembic.runtime.migration:Context impl PostgresqlImpl.
INFO:alembic.runtime.migration:Will assume transactional DDL.

Verify the authoritative watcher heartbeat:

kubectl -n default exec deploy/run-health-watcher -- python -c "from mage_ai.orchestration.db import db_connection; db_connection.start_session(); from mage_ai.orchestration.db.models.schedules import RunHealthWatcherHeartbeat; print([r.to_dict() for r in RunHealthWatcherHeartbeat.query.all()])"

Expected:

lease_owner starts with run-health-watcher-
last_scan_status: success
last_heartbeat_at is recent

Verify shared workspace PVC settings were preserved:

kubectl -n default get deploy mageai \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="mageai")].env[?(@.name=="WORKSPACE_PVC_NAME")].value}{"\n"}'

kubectl -n default get deploy mageai \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="mageai")].env[?(@.name=="WORKSPACE_PVC_SUBPATH_PREFIX")].value}{"\n"}'

Expected:

pvc-mageai-staging
mage_pro/workspaces/mageai-staging

9. Create A Workspace For QA

Open:

https://cluster-staging.mage.ai/manage

Create a workspace such as:

run-health-ws-2

Leave Configure container off unless the cluster owner gives a specific
container config.

Verify:

kubectl -n default get pods -o wide | grep run-health-ws-2
kubectl -n default get sts,svc,ingress,pvc | grep run-health-ws-2

For the default/my-mageai shared-PVC path, the workspace should not create a
new workspace PVC. It should use pvc-mageai-staging.

10. Workspace Creation Troubleshooting

Pending From Node Selector

If the workspace pod is Pending with:

0/3 nodes are available: node(s) didn't match Pod's node affinity/selector

Check node instance types:

kubectl get nodes -L node.kubernetes.io/instance-type

In this validation, all available nodes were:

m5.xlarge

but the new workspace initially requested:

node.kubernetes.io/instance-type=m5.2xlarge

Patch only the QA workspace if needed:

kubectl -n default patch sts mageai-staging-workspace-run-health-ws-2 \
  --type='json' \
  -p='[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'

kubectl -n default delete pod mageai-staging-workspace-run-health-ws-2-0

ContainerCreating From Extra /app PVC Mount

If the pod schedules but stays ContainerCreating, compare the QA workspace
against an existing Running workspace without modifying the existing workspace:

kubectl -n default get sts mageai-staging-workspace-test2 \
  -o jsonpath='test2 annotations={.spec.template.metadata.annotations}{"\n"}volumes={.spec.template.spec.volumes}{"\n"}mounts={.spec.template.spec.containers[0].volumeMounts}{"\n"}'

kubectl -n default get sts mageai-staging-workspace-run-health-ws-2 \
  -o jsonpath='run-health-ws-2 annotations={.spec.template.metadata.annotations}{"\n"}volumes={.spec.template.spec.volumes}{"\n"}mounts={.spec.template.spec.containers[0].volumeMounts}{"\n"}'

In this validation, the stuck workspace had an extra PVC mount:

volume: workspace-source-code -> pvc-mageai-staging
mount: /app from workspace-source-code readOnly

The known-good test2 workspace did not have that mount. Removing it allowed
the container to start:

kubectl -n default patch sts mageai-staging-workspace-run-health-ws-2 \
  --type='json' \
  -p='[
    {"op":"remove","path":"/spec/template/spec/containers/0/volumeMounts/3"},
    {"op":"remove","path":"/spec/template/spec/volumes/4"}
  ]'

kubectl -n default delete pod mageai-staging-workspace-run-health-ws-2-0

CrashLoop From Custom /app Command

After removing the extra /app mount, the workspace may crash with:

python: can't open file '/app/mage_ai/cli/main.py': [Errno 2] No such file or directory

Compare commands:

kubectl -n default get sts mageai-staging-workspace-test2 \
  -o jsonpath='test2 command={.spec.template.spec.containers[0].command}{"\n"}args={.spec.template.spec.containers[0].args}{"\n"}'

The known-good test2 workspace used the image default command. Remove the QA
workspace custom command:

kubectl -n default patch sts mageai-staging-workspace-run-health-ws-2 \
  --type='json' \
  -p='[
    {"op":"remove","path":"/spec/template/spec/containers/0/command"}
  ]'

kubectl -n default delete pod mageai-staging-workspace-run-health-ws-2-0

Expected final state:

mageai-staging-workspace-run-health-ws-2-0   1/1   Running

Open:

https://cluster-staging.mage.ai/run-health-ws-2/pipelines

11. Workspace-Mode Run-Health QA

Once the workspace is Running, use the workspace URL:

https://cluster-staging.mage.ai/run-health-ws-2/...

Suggested high-level checks:

  1. Create a manual stuck sleep pipeline inside the workspace.
  2. Trigger it manually and wait for the watcher threshold.
  3. Confirm the run detail page shows a non-healthy run-health panel.
  4. Confirm Terminate run appears for an authorized user while the run is active.
  5. Click Terminate run and confirm the run becomes cancelled.
  6. Create a healthy periodic-progress pipeline and confirm it does not show a
    false stuck/runaway diagnosis.
  7. Create a scheduled-trigger blocked pipeline with per-trigger concurrency 1
    and skip-on-limit behavior.
  8. Confirm the trigger detail page shows Stuck run blocking scheduled triggers
    with a link to the blocking run.

Use the Mage Pro feature QA guide for exact pipeline code snippets and expected
UI states.

12. Rollback

Disable only the watcher:

helm upgrade "$MAGE_RELEASE" "$HELM_CHART/charts/mageai" \
  --namespace "$MAGE_NAMESPACE" \
  --reuse-values \
  --set runHealthWatcher.enabled=false \
  --wait --timeout 10m

Rollback the full Helm release:

helm -n "$MAGE_NAMESPACE" history "$MAGE_RELEASE"
helm -n "$MAGE_NAMESPACE" rollback "$MAGE_RELEASE" <previous-revision> --wait --timeout 10m

Clean up temporary debug/test pods:

kubectl -n default delete pod pvc-debug --ignore-not-found=true
kubectl -n default delete pod run-health-image-pull-test --ignore-not-found=true

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant