Live cost tracking. 4-6 week plan starts June 13, 2026. Scale-down: July 25. Updates dynamically from GPU job data.
Loading...
Based on actual EventBridge schedules + A-only grade gate
| Day | RL (4am) | Auto-dock (11pm) | Physician | Research | Total Submitted | Pass Gate (~30%) |
|---|---|---|---|---|---|---|
| Mon | 1 | 2 | 8 (prostate) | 3 | 14 | 6 |
| Tue | 1 | 2 | — | — | 3 | 3 |
| Wed | 1 | 2 | — | — | 3 | 3 |
| Thu | 1 | 2 | 8 (pancreatic) | 3 | 14 | 6 |
| Fri | 1 | 2 | — | — | 3 | 3 |
| Sat | 1 | 2 | — | 3 | 6 | 4 |
| Sun | 1 | 2 | — | — | 3 | 3 |
| WEEK | 7 | 14 | 16 | 9 | 46 | 28 |
| Metric | Per Week | 4 Weeks | 6 Weeks |
|---|---|---|---|
| Jobs that actually dock | 28 | 112 | 168 |
| GPU hours (28 × 0.5h) | 14h | 56h | 84h |
| GPU cost (On-Demand $0.53/h) | $7.42 | $29.68 | $44.52 |
| GPU cost (Spot $0.16/h) | $2.24 | $8.96 | $13.44 |
| Lambda + DDB + S3 + CF | $4.50 | $18.00 | $27.00 |
| TOTAL (On-Demand) | $11.92 | $47.68 | $71.52 |
| TOTAL (Spot) | $6.74 | $26.96 | $40.44 |
⚠️ Budget assumes: grade gate = A only, watchdog kills hung jobs at 90min, spot quota 64 vCPUs pending approval.
Loading from Batch history...
Loading...
| Phase | What | Runtime | Cost/Night |
|---|---|---|---|
| 1-12 | Brain updates, gene discovery, NPEAD learning, teacher eval | ~45s Lambda | ~$0.001 |
| 12a8 | NPEAD-MAML training (ECS cortex-rs) | ~90s ECS | ~$0.01 |
| 13-20 | Omics enrichment, BioBrain retrain, gates, scoring | ~30s Lambda | ~$0.001 |
| 21-31 | Intelligence briefings, backups, evidence integrity | ~20s Lambda | ~$0.001 |
| 32 | NPEAD distillation (MAML → BWM) | ~5s Lambda | ~$0.0001 |
| 33 | Auto-repurpose discovery | ~10s Lambda | ~$0.0001 |
| 34 | State reconciliation | ~5s Lambda | ~$0.0001 |
| 35 | Self-governance proposals | ~3s Lambda | ~$0.0001 |
| 36 | Public ChEMBL IC50 ingestion | ~15s Lambda | ~$0.001 |
| 37 | Regulatory layer (TF activity) | ~5s Lambda | ~$0.0001 |
| 38 | TCGA survival enrichment | ~5s Lambda | ~$0.0001 |
| — | Qdrant keepalive ping | ~1s Lambda | $0 |
| — | Cortex-RS nightly (ECS) | ~90s Fargate | ~$0.01 |
| — | Intelligence nightly (Lambda) | ~30s | ~$0.001 |
| NIGHTLY TOTAL | ~3 min | ~$0.03/night | |
| MONTHLY NIGHTLY COST | ~90 min | $0.90/month | |
Nightly pipeline is essentially free (~$1/month). GPU docking is 97% of total cost.
| Schedule | Frequency | Service | Cost/Month |
|---|---|---|---|
| Watchdog | Every 5 min (8640/mo) | Lambda | $0.05 |
| Model backup | 2x/day | Lambda + S3 | $0.10 |
| Omics cache refresh | Daily | Lambda | $0.30 |
| Paper reader | 3x/week | Lambda | $0.50 |
| Evidence integrity | Weekly | Lambda | $0.20 |
| Archive (2x/day) | Daily | Lambda + S3 | $0.10 |
| RL trigger | Daily | Lambda | $0.01 |
| TOTAL OTHER SCHEDULED | $1.26/month | ||
| Control | Setting | Savings |
|---|---|---|
| Grade gate | — | ~60% fewer jobs |
| Watchdog (hung jobs) | Kill at 90 min | Prevents $100+ overruns |
| Watchdog (stuck queue) | Cancel at 10 min | No idle billing |
| Spot recovery | Auto-resubmit | No lost work |
| Job timeout (docking) | 1 hour max | Hard cap per job |
| Job timeout (MD) | 2 hours max | Hard cap per job |
| Monthly budget cap | $30 (spot) — hard cutoff | All jobs paused at cap |
| Spot only after | June 20 (no on-demand) | 3x cost reduction |
| Retry limit (per job) | Max 3 retries | No infinite retry loops |
| Retry limit (per day) | Max 10 total | Caps mass retry cost at $0.79/day |
| Budget alert | Email at 80% ($24) | Early warning |
| Budget cutoff alert | Email at 100% ($30) | Jobs paused + notified |
| Retry limit alert | Email when 10/day hit | Spot issues flagged |
| Watchdog dead alert | Email if no run in 15min | Safety net monitored |
Loading...
June 8-12: $190 anomaly — 4 IDH1 MD jobs hung 47h + 1000 failed jobs (wrong Docker image). ✅ FIXED: watchdog + timeouts + image fix.