Skip to content

Add dependency validation init container - #99

Merged
wangxiaoyou1993 merged 8 commits into
masterfrom
codex/dependency-validation-init-container
Oct 24, 2026
Merged

wangxiaoyou1993 merged 8 commits into
masterfrom
codex/dependency-validation-init-container

Conversation

@wangxiaoyou1993

@wangxiaoyou1993 wangxiaoyou1993 commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Summary

Add an opt-in dependency-validation init container to the Mage chart for combined and standalone web-server/scheduler deployments. The validator uses the Mage application image and startup script, installs the project requirements with fail-fast behavior, and imports the server and scheduler startup paths without launching either process.

The chart version is bumped from 0.2.16 to 0.2.17. The feature is disabled by default and supports both Mage OSS and Mage Pro images.

Why

Mage currently installs project dependencies during application startup. An incompatible or unavailable dependency can therefore be discovered only after Kubernetes has begun replacing application pods. This change adds a preflight gate: Kubernetes starts the main Mage container only after dependency installation and the critical server/scheduler imports complete successfully.

The validator is a readiness gate, not a replacement for normal startup setup. Mage still performs its usual dependency setup when the main container starts. The validator's managed dependency environment is isolated, while project volumes remain shared and custom validation commands must avoid mutating project state.

Behavior

When dependencyValidation.enabled=true, the chart adds validate-dependencies after the existing Redis/PostgreSQL readiness init containers. It:

  1. Runs /app/run_app.sh so dependency-manager setup follows the normal Mage startup path.
  2. Re-runs the project requirements.txt installation under /bin/bash -ec, ensuring installation failures propagate as a nonzero exit.
  3. Imports mage_ai.server.server.
  4. Imports the scheduler startup path: scheduler_manager, job_manager, and loop_time_trigger.

The chart now applies the existing digest-first image selection consistently across combined and standalone web-server/scheduler workloads: when image.digest is non-empty, it takes precedence over image.tag. The validator uses the same selected Mage image reference and pull policy as its workload. It forces MAGE_DATA_DIR to a validator-only emptyDir, preventing dependency setup from reading or mutating the cached virtual environment used by existing replicas. Split deployments also receive their matching INSTANCE_TYPE. A custom command and validator resource requests/limits can be supplied through dependencyValidation.command and dependencyValidation.resources. Existing releases that enable the feature with --reuse-values fall back to the safe default command when the new command value is absent.

This deliberately does not run environment-wide pip check: live testing showed that it reports unrelated, pre-existing metadata conflicts among packages bundled in an otherwise working Mage image. The acceptance condition is instead that project requirements install and both startup paths import successfully.

Configuration examples and impact

Enable the default preflight

dependencyValidation:
  enabled: true

Impact:

  • Adds one validate-dependencies init container to the combined Mage Deployment.
  • With standaloneScheduler=true, adds the validator to both the web-server and scheduler Deployments.
  • Delays main-container startup until requirements installation and server/scheduler imports pass.
  • Prevents the new Mage container from starting when validation exits nonzero; during a rolling Deployment, the previous ready pod remains available.
  • Performs dependency setup in both the init container and normal Mage startup, so startup can take longer and may perform duplicate installation work.

For an existing release using reused values:

helm upgrade RELEASE ./charts/mageai \
  --namespace NAMESPACE \
  --reuse-values \
  --set dependencyValidation.enabled=true

The template supplies the safe default command when an older release has no dependencyValidation.command value.

Set validator resources

dependencyValidation:
  enabled: true
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: "1"
      memory: 1Gi

Impact:

  • Applies requests and limits only to the validation init container.
  • Higher CPU can shorten dependency resolution/import time; insufficient memory or CPU can cause validation failure or a slower rollout.
  • Does not change the main Mage container resources.

Supply a custom validation command

dependencyValidation:
  enabled: true
  command:
    - /app/run_app.sh
    - /bin/bash
    - -ec
    - |
      PROJECT_PATH="${USER_CODE_PATH:-default_repo}"
      REQUIREMENTS_FILE="${PROJECT_PATH}/requirements.txt"
      if [ -f "${REQUIREMENTS_FILE}" ]; then
        python -m pip install -r "${REQUIREMENTS_FILE}"
      fi
      python -c "import mage_ai.server.server"
      python -c "import mage_ai.server.scheduler_manager"
      python -c "import my_project.validation"

Impact:

  • command replaces the entire default command; it does not append additional checks.
  • Keeping /app/run_app.sh preserves Mage dependency-environment setup.
  • Keeping /bin/bash -ec ensures the first failed install or import exits nonzero and blocks startup.
  • Project-specific imports can verify optional plugins or integrations, but any side effects in those imports run before the main container starts.

Leave validation disabled

dependencyValidation:
  enabled: false

Impact:

  • No validation init container is added.
  • Mage retains its existing startup behavior.
  • Redis and PostgreSQL readiness init containers, when enabled, are unchanged.

Live EKS validation

The current PR head (fb4b2a8) was tested against a shared EKS test release using the target Mage Pro image.

Executable failure-path result (revision 215):

  • Applied a validator command that intentionally exited 42.
  • wait-for-postgres completed with exit code 0.
  • validate-dependencies logged the expected failure and entered Init:CrashLoopBackOff with exit code 42.
  • The candidate Mage container remained PodInitializing with started=false.
  • The rollout timed out as expected while the previous application pod remained Ready 1/1 and available.

Current-head recovery and success result (revision 217):

  • Restored the chart's default dependency-validation command.
  • The validator created its managed environment under /tmp/mage-validation-data/venvs/..., proving that it did not reuse the running replica's /home/src/mage_data/venvs cache.
  • Project requirements installation passed.
  • Server and scheduler startup imports passed.
  • validate-dependencies completed with exit code 0.
  • The main Mage container started and became Ready.
  • /api/status returned HTTP 200.
  • The Deployment completed its rolling update, retaining the previous Ready pod until the new pod passed validation and readiness.
  • Final candidate pod: Ready 1/1.
  • Final release status: deployed; dependencyValidation.enabled=true.

Automated testing

  • helm unittest charts/mageai — 24/24 tests passed
  • helm lint charts/mageai — passed
  • GitHub chart-testing — passed
  • GitHub kubeconform for charts/mageai — passed
  • git diff --check — passed
  • Verified feature-disabled rendering when dependencyValidation is absent or null, including combined and split workloads
  • Rendered and inspected combined, standalone web-server, and standalone scheduler configurations
  • Covered default-disabled behavior, existing readiness-container ordering, env/config/secret and project-mount propagation, custom commands, missing/empty command fallback, and digest precedence for both standalone workloads
  • Executed a failing validator on EKS and verified nonzero exit, blocked main-container startup, stalled rollout, and continued availability of the previous replica
  • Restored the current-head default validator and verified isolated environment creation, successful imports, rollout completion, and HTTP readiness

Residual compatibility note

The optional securityContext.readOnlyRootFilesystem=true configuration remains incompatible with Mage startup temporary-file creation unless writable temporary storage is also configured. This limitation already applies to the main Mage container on the base chart because /app/run_app.sh uses /tmp; the validation feature does not introduce it. Setting TMPDIR to writable storage would be useful chart hardening but would need to cover the main container as well as the validator.

Human QA

  1. Enable dependencyValidation.enabled=true on a test release.
  2. Confirm readiness init containers complete before validate-dependencies.
  3. Confirm validation exits 0 and the main container starts.
  4. Confirm /api/status becomes ready and the old pod remains available until then.
  5. Introduce a deliberately invalid requirement or import in an isolated test release and confirm the main container never starts and the existing pod remains available.

No screenshot or video is applicable: this change affects Helm rendering and Kubernetes init-container/startup behavior. Runtime evidence is provided through Helm history, init-container logs and exit statuses, and Deployment rollout status.

@wangxiaoyou1993

Copy link
Copy Markdown
Member Author

EKS runtime validation completed on mage-data-prep-production-eks, namespace mageai-1, release mageai-1-workspace-test2.

  • Initial live upgrade using --reuse-values exposed that existing releases do not have dependencyValidation.command. The template now falls back to the safe default command when that value is missing or empty; commit d9f638f adds regression coverage.
  • Retried with --atomic --wait; wait-for-postgres completed successfully.
  • validate-dependencies reused the startup dependency environment, performed the strict requirements check, and correctly blocked the new Mage container after pip check detected installed-package conflicts involving dbt, PyIceberg, and Singer dependencies.
  • The previous Mage pod remained healthy throughout.
  • The test release was rolled back immediately. Final state: Helm revision 213, chart mageai-0.2.16, status deployed, Mage deployment 1/1 available, PostgreSQL 1/1 ready, and dependencyValidation absent from restored release values.

No screenshot or video applies: this change and validation are limited to Helm rendering and Kubernetes init-container/runtime behavior, with evidence captured through Helm history, pod state, and init-container logs.

@wangxiaoyou1993

Copy link
Copy Markdown
Member Author

Follow-up EKS validation after narrowing the acceptance gate to the requested behavior:

  • Removed the environment-wide pip check, which was reporting unrelated pre-existing image package metadata conflicts.
  • The validator now requires the project requirements install to succeed, imports mage_ai.server.server, and imports the scheduler startup path (scheduler_manager, job_manager, and loop_time_trigger).
  • Runtime result on mage-data-prep-production-eks / mageai-1 / mageai-1-workspace-test2: wait-for-postgres=Completed/0, validate-dependencies=Completed/0.
  • The main Mage container started and passed /api/status; Deployment rollout completed successfully.
  • Final release: Helm revision 214, chart mageai-0.2.17, status deployed, new pod mageai-1-workspace-test2-5687c886f5-vgvvt Ready 1/1, and dependencyValidation.enabled=true.

No screenshot or video applies because this is Helm/Kubernetes init-container and startup behavior; evidence is from Helm history, init-container exit statuses/logs, and Deployment rollout status.

@wangxiaoyou1993

Copy link
Copy Markdown
Member Author

README documentation expanded in 27a9494 from a feature-only note into a repository guide. It now covers chart inventory, prerequisites, published repository setup, a rendered quick start, image/storage/Secret/split-workload/Redis/service configuration, dependency-validation behavior and caveats, upgrades, rollback, uninstall retention, examples, contribution checks, and licensing.

Validation: the minimal quick-start values rendered successfully; helm lint charts/mageai passed; helm unittest charts/mageai passed 20/20; git diff --check passed. Independent documentation review verified repository/chart names, commands, links, storage guidance, OSS/Pro wording, and exact validator configuration inheritance.

No screenshot or video applies: this is repository Markdown documentation and does not change a rendered Mage product interface.

@wangxiaoyou1993

Copy link
Copy Markdown
Member Author

Resolved [P1][SR-02/SR-03/SR-09] Existing releases can fail upgrades when validation remains disabled in commit 15b4086.

The templates now use a nil-safe mageai.dependencyValidationEnabled helper, so an older release whose reused values omit the entire dependencyValidation map—or provide it as null—renders without dereferencing .enabled. The same guard is used by the combined Deployment and both standalone web-server/scheduler Deployments. Enabled behavior is unchanged.

Validation:

  • helm unittest charts/mageai — 21/21 passed, including a regression case with dependencyValidation: null and validation disabled
  • helm lint charts/mageai — passed
  • git diff --check — passed
  • Manual combined render with a null map — passed; no validator rendered
  • Manual split-workload render with a null map — passed; neither workload rendered the validator
  • Manual enabled split render — passed; both workloads rendered the validator
  • Independent focused review — accepted with no actionable findings

No screenshot or video applies because this fix affects Helm template evaluation and rendered Kubernetes manifests only.

@wangxiaoyou1993

Copy link
Copy Markdown
Member Author

Resolved [P1][SR-02/SR-08/SR-09] Pre-validation can mutate the dependency environment of the still-running deployment in commit 91e6afa.

The validator now forces MAGE_DATA_DIR=/tmp/mage-validation-data, filters conflicting direct env/extraEnvs entries, and mounts a validator-only emptyDir at that path. Explicit container environment values override matching ConfigMap/Secret envFrom values. The existing replicas therefore retain their cached virtual environments untouched, while failed or successful preflight environments are discarded with the candidate pod. The explicit volume also keeps validation working with readOnlyRootFilesystem: true.

Validation:

  • helm unittest charts/mageai — 22/22 passed
  • helm lint charts/mageai — passed
  • git diff --check — passed
  • Combined render with a conflicting user MAGE_DATA_DIR — only the isolated path remained
  • Combined render with readOnlyRootFilesystem: true — writable validator-only mount rendered correctly
  • Split web-server/scheduler render — both validators received the isolated path and volume
  • Independent correctness re-review — accepted with no actionable findings
  • Independent documentation review — README updated and accepted

This uses the shared OSS/Pro MAGE_DATA_DIR startup contract and requires no Mage Pro runtime change. Custom imports can still mutate shared project files, so the README retains the read-only custom-command guidance.

No screenshot or video applies because this change affects Helm manifests and init-container filesystem isolation only.

@wangxiaoyou1993

Copy link
Copy Markdown
Member Author

Addressed [P2][SR-01/SR-03/SR-10] Validation-disabled standalone deployments can unexpectedly change images in commit b0e22ef by making the behavior explicit and covered rather than reverting it.

The consistency change is intentional: a non-empty image.digest takes precedence over image.tag for the combined Deployment and both standalone web-server/scheduler Deployments. Dependency validation uses that same selected image.

Reviewer-facing updates:

  • Added focused Helm tests for standalone web-server digest precedence
  • Added focused Helm tests for standalone scheduler digest precedence
  • Declared and documented image.digest in values.yaml
  • Documented digest precedence in the README
  • Added an upgrade warning that standalone releases carrying both fields may roll to the digest-selected image even when validation is disabled
  • Updated the PR description and removed the item from current findings

Validation:

  • helm unittest charts/mageai — 24/24 passed
  • helm lint charts/mageai — passed
  • git diff --check — passed
  • Independent documentation/correctness review — accepted with no actionable findings

No screenshot or video applies because this is Helm image-reference rendering behavior.

@wangxiaoyou1993

Copy link
Copy Markdown
Member Author

Resolved the documentation-contract and executable failure-coverage findings at current PR head fb4b2a8.

Documentation now enumerates the validator inputs it actually inherits and states that unrelated feature-specific environment variables and mounts are not copied.

Executable EKS evidence on the shared test release:

  • deliberate validator failure exited 42
  • candidate remained in init and the Mage container never started
  • rollout stalled while the previous replica stayed Ready
  • restoring the default validator created the isolated environment under the validator-only temporary mount
  • requirements and server/scheduler startup imports passed
  • validator exited 0, the replacement pod became Ready, rollout completed, and /api/status returned HTTP 200

Local validation: helm unittest charts/mageai 24/24, helm lint, and git diff --check. The PR description contains the updated redacted runtime evidence. No screenshot or video applies; validation is evidenced by init exit states, logs, rollout status, and API readiness.

@wangxiaoyou1993
wangxiaoyou1993 merged commit 4bde8a4 into master Oct 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants