feat: Fleet Intelligence & Model Lifecycle Management (F19/F20)

[leocamello]

Summary

Ship Model Lifecycle Management (F20) and Pre-warming & Fleet Intelligence (F19) — RFC-001 Phase 3.

Enables Nexus to control model loading/unloading across the fleet and predict demand for proactive model placement.

What Changed

New API Endpoints

• POST /v1/models/load — Load model on specific backend
• DELETE /v1/models/{id} — Unload model from specific backend
• POST /v1/models/migrate — Coordinated migration (load on target, operator unloads source)
• GET /v1/fleet/recommendations — Pre-warming recommendations based on request patterns

New Modules

• src/api/lifecycle.rs — HTTP handlers with X-Nexus-Lifecycle-Status headers
• src/routing/reconciler/fleet.rs — Background pattern analysis, 15 unit tests
• src/routing/reconciler/lifecycle.rs — Routing filter for loading backends, 11 unit tests
• src/config/lifecycle.rs / src/config/fleet.rs — Configuration types

Ollama NII Implementation

• load_model() → POST /api/pull
• unload_model() → POST /api/generate (keep_alive=0)
• resource_usage() → GET /api/ps (VRAM tracking)

Key Design Decisions

• FleetReconciler: Background analysis task (not per-request pipeline) — avoids latency impact
• LifecycleReconciler: Early pipeline filter before PrivacyReconciler
• Migration: Source preserved in routing until operator manually unloads
• Fleet intelligence: Suggestion-first — recommends, never auto-loads

Verification

• Tests: 1287 passed, 0 failed (1272 existing + 15 new)
• Clippy: 0 warnings
• Fmt: Clean
• Benchmarks: Routing latency 6-160µs (well under 1ms constitution target)
• Verification checklist: 143 pass, 67 N/A, 0 unchecked

Red Team Review

Verdict: [PASS] (after fixes)

Initial review found 4 issues, all fixed in commit 6cda049:

#	Severity	Issue	Fix
1	Critical	Memory leak — record_request() grew unbounded when fleet disabled	Added if !self.config.enabled { return; } guard
2	High	TOCTOU race — update_operation() set after load_model()	Moved operation tracking before agent call; clear on failure
3	Medium	VRAM heuristic dead code — Ollama returns None for total VRAM	Made threshold configurable via vram_heuristic_max_gb
4	Medium	Migration rollback left orphaned load on target	Set both operations before load_model(); rollback both on failure

Re-review confirmed all fixes correct, no new issues introduced.

Configuration

[lifecycle]
timeout_ms = 300000
vram_headroom_percent = 20
vram_heuristic_max_gb = 16

[fleet]
enabled = false
min_sample_days = 3
min_request_count = 10
analysis_interval_seconds = 3600
max_recommendations = 10

Spec & Docs

• Spec: specs/022-fleet-intelligence-model-lifecycle/
• API docs: docs/api/lifecycle.md
• Walkthrough: specs/022-fleet-intelligence-model-lifecycle/walkthrough.md

Closes #194
Closes #195
Closes #196
Closes #197
Closes #198
Closes #199
Closes #200