Repository navigation
Add dependency validation init container - #99
Conversation
|
EKS runtime validation completed on
No screenshot or video applies: this change and validation are limited to Helm rendering and Kubernetes init-container/runtime behavior, with evidence captured through Helm history, pod state, and init-container logs. |
|
Follow-up EKS validation after narrowing the acceptance gate to the requested behavior:
No screenshot or video applies because this is Helm/Kubernetes init-container and startup behavior; evidence is from Helm history, init-container exit statuses/logs, and Deployment rollout status. |
|
README documentation expanded in Validation: the minimal quick-start values rendered successfully; No screenshot or video applies: this is repository Markdown documentation and does not change a rendered Mage product interface. |
|
Resolved [P1][SR-02/SR-03/SR-09] Existing releases can fail upgrades when validation remains disabled in commit The templates now use a nil-safe Validation:
No screenshot or video applies because this fix affects Helm template evaluation and rendered Kubernetes manifests only. |
|
Resolved [P1][SR-02/SR-08/SR-09] Pre-validation can mutate the dependency environment of the still-running deployment in commit The validator now forces Validation:
This uses the shared OSS/Pro No screenshot or video applies because this change affects Helm manifests and init-container filesystem isolation only. |
|
Addressed [P2][SR-01/SR-03/SR-10] Validation-disabled standalone deployments can unexpectedly change images in commit The consistency change is intentional: a non-empty Reviewer-facing updates:
Validation:
No screenshot or video applies because this is Helm image-reference rendering behavior. |
|
Resolved the documentation-contract and executable failure-coverage findings at current PR head Documentation now enumerates the validator inputs it actually inherits and states that unrelated feature-specific environment variables and mounts are not copied. Executable EKS evidence on the shared test release:
Local validation: |
Summary
Add an opt-in dependency-validation init container to the Mage chart for combined and standalone web-server/scheduler deployments. The validator uses the Mage application image and startup script, installs the project requirements with fail-fast behavior, and imports the server and scheduler startup paths without launching either process.
The chart version is bumped from
0.2.16to0.2.17. The feature is disabled by default and supports both Mage OSS and Mage Pro images.Why
Mage currently installs project dependencies during application startup. An incompatible or unavailable dependency can therefore be discovered only after Kubernetes has begun replacing application pods. This change adds a preflight gate: Kubernetes starts the main Mage container only after dependency installation and the critical server/scheduler imports complete successfully.
The validator is a readiness gate, not a replacement for normal startup setup. Mage still performs its usual dependency setup when the main container starts. The validator's managed dependency environment is isolated, while project volumes remain shared and custom validation commands must avoid mutating project state.
Behavior
When
dependencyValidation.enabled=true, the chart addsvalidate-dependenciesafter the existing Redis/PostgreSQL readiness init containers. It:/app/run_app.shso dependency-manager setup follows the normal Mage startup path.requirements.txtinstallation under/bin/bash -ec, ensuring installation failures propagate as a nonzero exit.mage_ai.server.server.scheduler_manager,job_manager, andloop_time_trigger.The chart now applies the existing digest-first image selection consistently across combined and standalone web-server/scheduler workloads: when
image.digestis non-empty, it takes precedence overimage.tag. The validator uses the same selected Mage image reference and pull policy as its workload. It forcesMAGE_DATA_DIRto a validator-onlyemptyDir, preventing dependency setup from reading or mutating the cached virtual environment used by existing replicas. Split deployments also receive their matchingINSTANCE_TYPE. A custom command and validator resource requests/limits can be supplied throughdependencyValidation.commandanddependencyValidation.resources. Existing releases that enable the feature with--reuse-valuesfall back to the safe default command when the new command value is absent.This deliberately does not run environment-wide
pip check: live testing showed that it reports unrelated, pre-existing metadata conflicts among packages bundled in an otherwise working Mage image. The acceptance condition is instead that project requirements install and both startup paths import successfully.Configuration examples and impact
Enable the default preflight
Impact:
validate-dependenciesinit container to the combined Mage Deployment.standaloneScheduler=true, adds the validator to both the web-server and scheduler Deployments.For an existing release using reused values:
The template supplies the safe default command when an older release has no
dependencyValidation.commandvalue.Set validator resources
Impact:
Supply a custom validation command
Impact:
commandreplaces the entire default command; it does not append additional checks./app/run_app.shpreserves Mage dependency-environment setup./bin/bash -ecensures the first failed install or import exits nonzero and blocks startup.Leave validation disabled
Impact:
Live EKS validation
The current PR head (
fb4b2a8) was tested against a shared EKS test release using the target Mage Pro image.Executable failure-path result (revision
215):42.wait-for-postgrescompleted with exit code0.validate-dependencieslogged the expected failure and enteredInit:CrashLoopBackOffwith exit code42.PodInitializingwithstarted=false.1/1and available.Current-head recovery and success result (revision
217):/tmp/mage-validation-data/venvs/..., proving that it did not reuse the running replica's/home/src/mage_data/venvscache.validate-dependenciescompleted with exit code0./api/statusreturned HTTP200.1/1.deployed;dependencyValidation.enabled=true.Automated testing
helm unittest charts/mageai— 24/24 tests passedhelm lint charts/mageai— passedcharts/mageai— passedgit diff --check— passeddependencyValidationis absent ornull, including combined and split workloadsResidual compatibility note
The optional
securityContext.readOnlyRootFilesystem=trueconfiguration remains incompatible with Mage startup temporary-file creation unless writable temporary storage is also configured. This limitation already applies to the main Mage container on the base chart because/app/run_app.shuses/tmp; the validation feature does not introduce it. SettingTMPDIRto writable storage would be useful chart hardening but would need to cover the main container as well as the validator.Human QA
dependencyValidation.enabled=trueon a test release.validate-dependencies.0and the main container starts./api/statusbecomes ready and the old pod remains available until then.No screenshot or video is applicable: this change affects Helm rendering and Kubernetes init-container/startup behavior. Runtime evidence is provided through Helm history, init-container logs and exit statuses, and Deployment rollout status.