Home Heroes Models ⚙️ MLOps Quantum ⚙️ MLOps 🧠 NPEAD

⚙️ MLOps — Model Lifecycle & Operations

How E-DICE models are trained, validated, deployed, and monitored in production.

📊 Current State — What We Have Today

Capability	Status	Implementation	Gap
Model Registry	✅ Live	`model_registry.py` — versioned NPEAD states in DynamoDB + S3	No UI, no approval gate
Automated Training	✅ Live	Nightly: GCN (800 epochs), FusionWeightLearner (500 epochs), Meta-Learning Policy	No experiment comparison
Immediate Learning	✅ Live	`immediate_learn.py` — policy updates after every GPU batch job	No validation before deploy
Outcome Tracking	✅ Live	Outcome Memory (Phase 4) — every prediction vs actual result logged	No drift detection
Audit Trail	✅ Live	`audit_logger.py` — all autonomous actions traced with decision reason	Not queryable (raw DDB)
CI/CD Containers	✅ Live	GitHub Actions → ECR → Lambda/Batch auto-deploy on push	No staging environment
Rollback	⚠️ Partial	Model registry supports version rollback; `locked=true` prevents regression	Manual trigger only
Performance Monitoring	⚠️ Partial	NPEAD accuracy tracked (89.7%), FusionWeightLearner (86.8%)	No alerts, no dashboard
Data Validation	⚠️ Partial	CortexQ validation (6 rules), FRSI rank guards, chemistry BLOCKED_SMARTS	No schema enforcement
Feature Store	❌ Missing	Features computed inline at training time	No reuse, no versioning
Experiment Tracking	❌ Missing	Stdout logs only	No MLflow/W&B/comparison
A/B Testing	❌ Missing	Single model path, no shadow deployment	Can't compare model versions
Model Approval Gate	❌ Missing	Nightly auto-deploys if accuracy improves	No human sign-off
Data Drift Detection	❌ Missing	—	Training distribution not monitored
Reproducibility	❌ Missing	No seed pinning, no DVC, no pipeline DAGs	Can't reproduce past training

🏗️ Architecture — Current Model Pipeline

┌─────────────────────────────────────────────────────────────────────┐
│                        NIGHTLY ORCHESTRATOR                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────┐   ┌──────────────┐   ┌───────────────┐              │
│  │ Outcome  │──→│ Meta-Learning│──→│ NPEAD-DLE v6.1│              │
│  │ Memory   │   │ Policy       │   │ (action select)│              │
│  └──────────┘   └──────────────┘   └───────────────┘              │
│       ↑                                     │                       │
│       │         ┌──────────────┐            ↓                       │
│       │         │ FusionWeight │      ┌───────────┐                │
│       └─────────│ Learner      │      │ CortexQ   │                │
│                 └──────────────┘      │ Validation│                │
│                                       └───────────┘                │
│  ┌──────────┐   ┌──────────────┐                                   │
│  │ CIFOG GCN│   │ QGCN         │   ← Quantum escalation           │
│  │ (276node)│──→│ (SV1 refine) │                                   │
│  └──────────┘   └──────────────┘                                   │
│                                                                     │
│  ┌──────────────────────────────┐                                   │
│  │ Immediate Learn (per-batch)  │   ← Real-time weight updates     │
│  └──────────────────────────────┘                                   │
└─────────────────────────────────────────────────────────────────────┘
         │                    │                      │
         ↓                    ↓                      ↓
   cortex-heroes        edice-platform-state    cortex-brain-bus
   (193 targets)        (model weights)         (learning signals)

🎯 Models in Production

Model	Type	Accuracy	Training	Samples	Last Updated
NPEAD-DLE v6.1	Evidence-directed action selector	89.7%	Continuous (immediate learn)	837 outcomes	Nightly
CIFOG GCN	Graph Convolutional Network	—	800 epochs nightly	276 nodes, 3517 edges	Nightly
FusionWeightLearner	Logistic regression (L2)	86.8%	500 epochs nightly	106 heroes	Nightly
Meta-Learning Policy	Per-class weight optimizer	—	Nightly (outcome aggregation)	51 outcomes	Nightly
CortexQ Gen v4	6-layer Transformer (GPT)	100% validity	Offline (40 epochs, 10K mols)	3.2M params	Static
Cortex Master v7	3-layer FFN (53→128→64→32)	—	Offline	53-dim input	Static
QGCN	GCN + QAOA quantum layer	—	Nightly (post-GCN)	272 low-confidence nodes	Nightly

🗺️ Roadmap to Formal MLOps

Phase 1 — Observability (Week 1-2)

Can't improve what you can't measure.

1.1 Model Performance Dashboard
NPEAD accuracy over time FusionWeight accuracy trend GCN loss curve Prediction confidence distribution
Implementation: CloudWatch custom metrics + simple HTML dashboard reading from DDB

1.2 Automated Alerts
Alert if accuracy drops >2% Alert if nightly fails Alert if FRSI regresses Alert if GPU job fails silently
Implementation: SNS topic + CloudWatch alarms on platform-state metrics

1.3 Training Lineage
Log training data hash per run Log hyperparameters Log before/after weights S3 artifact per training run
Implementation: Extend nightly to write training manifest to S3 per run

Phase 2 — Governance (Week 3-4)

Human oversight before models go live.

2.1 Model Approval Gate
Nightly computes new weights → writes to model:candidate:{name} → waits for approval → only then promotes to model:active:{name}.
Auto-approve if: accuracy improves AND no regression on holdout set AND within 5% of previous weights.

2.2 Shadow Deployment
New model version runs in parallel (shadow mode) for 24h. Compare predictions against production model. Promote only if shadow ≥ production on key metrics.

2.3 Automated Rollback
If accuracy drops >3% within 24h of promotion → auto-rollback to previous version. Already have model_registry.py rollback mechanism — just needs trigger.

Phase 3 — Reproducibility (Week 5-6)

Any past training run can be exactly reproduced.

3.1 Feature Store
Versioned feature sets stored in S3. Training reads from feature store, not raw DDB. Features: molecular (7-dim), NPEAD (12-dim), enrichment (30-dim), brain signals (20-dim).

3.2 Training Pipeline DAG
Step Functions workflow: extract features → validate data → train → evaluate → compare → gate → deploy. Each step idempotent, retryable, logged.

3.3 Data Drift Detection
Compare feature distributions between training windows. Alert if KL-divergence > threshold on any dimension. Triggers retraining with expanded data.

📐 Metrics We Track Today

NPEAD accuracy: 89.7% FusionWeight accuracy: 86.8% Total outcomes: 837 Heroes: 193 GCN nodes: 276 GCN edges: 3517 Molecules: 2355 Capsules: 41 Repurposing: 221 QGCN escalated: 272

🔧 What Changes Today to Get There

Immediate (already in place):

Model registry with version + rollback
Locked records prevent nightly regression
Outcome memory tracks every decision
Audit logger traces autonomous actions
FRSI rank guards (can never downgrade)
CortexQ validation (6 override rules)

Next sprint (Phase 1 — this week):

Write accuracy + loss to CloudWatch custom metrics after each nightly
SNS alert on nightly failure or accuracy drop
Training manifest (JSON) to S3 per nightly run
This page auto-populates metrics from API

Critical gap to close:

Dev exists but not synced: 16 edicer-dev-* Lambdas + 7 tables (CDK stack) — cortex pipeline deploys direct to prod, bypassing dev entirely
No holdout validation set — accuracy measured on training data
Nightly auto-deploys without human review

🔄 Dev/Prod Environment

Same account (865950114883), isolated by prefix. CI auto-deploys to dev. Prod only on manual trigger.

Component	Dev	Prod
DynamoDB (8 tables)	`cortex-dev-*`	`cortex-*`
API Lambda	`cortex-dev-api`	`cortex-api`
Nightly Lambda	`cortex-dev-nightly`	`cortex-intelligence-nightly`
Schedule	4am UTC	3am UTC
Batch Job Def	`cortex-dev-docking-gpu`	`cortex-docking-gpu`
Web (S3)	`cortex-dev-web-865950114883`	`edicer-api-docs-865950114883`
API Gateway	`/dev/` stage	`/prod/` stage
ECR Tags	`:dev`	`:latest`

# Setup (one-time):
AWS_PROFILE=edice-r python3 scripts/create_dev_env.py

# CI flow:
push → build → auto-deploy to dev → you validate → you say "deploy" → promote to prod

# Code reads:
STAGE=dev  → cortex-dev-heroes, cortex-dev-molecules, cortex-dev-repurposing
STAGE=prod → cortex-heroes, cortex-molecules, cortex-repurposing