HomeHeroes🎛️ Control💰 Costs🖥️ GPU Fleet🧪 Tests

🖥️ GPU Fleet Status

5-region spot fleet with multi-region watchdog. Auto-reroutes on capacity failure. Parallel spread across idle regions.

5
Regions
512
Max vCPUs
SPOT
Strategy
—
Running
—
Queued
—
Best Region

Compute Environments

RegionNameTypeMax vCPUsInstance TypesAMIStatus
us-east-1cortex-gpu-spotSPOT (Capacity Optimized)256g4dn.xl, g4dn.2xl, g5.xl, g5.2xl, g6.xlECS_AL2023_NVIDIA✅ Healthy
us-east-1cortex-gpuOn-Demand256g4dn.xlargeECS_AL2023_NVIDIA✅ Healthy
us-east-1cortex-gpu-flexOn-Demand (Best Fit)64g4dn.xl, g5.xl, g4dn.2xl, g5.2xlECS_AL2_NVIDIA✅ Healthy
us-east-2cortex-gpu-spot-east2SPOT (Capacity Optimized)64g4dn.xl, g4dn.2xl, g5.xl, g5.2xlECS_AL2023_NVIDIA✅ Healthy
us-west-2cortex-gpu-spotSPOT (Capacity Optimized)64g4dn.xl, g4dn.2xl, g5.xl, g5.2xlECS_AL2023_NVIDIA✅ Healthy
ca-central-1cortex-gpu-spotSPOT (Capacity Optimized)64g4dn.xl, g4dn.2xl, g5.xl, g5.2xlECS_AL2023_NVIDIA✅ Healthy
eu-west-1cortex-gpu-spotSPOT (Capacity Optimized)64g4dn.xl, g4dn.2xl, g5.xl, g5.2xlECS_AL2023_NVIDIA✅ Healthy
us-east-1cortex-fargateFargate16——✅ Healthy

Spot Pricing (Live Estimates)

Regiong4dn.xlargeg4dn.2xlargeg5.xlargeg5.2xlarge
us-east-2 🏆$0.14$0.19$0.48$0.36
us-west-2$0.20$0.27$0.63$0.59
ca-central-1$0.21$0.29$0.36$0.44
us-east-1$0.25$0.26$0.52$0.54
eu-west-1$0.29$0.35$0.73$1.14

Watchdog Architecture

ComponentRegionScheduleRole
Master edice-job-watchdogus-east-1Every 5 minScan all regions, reroute, dedup, auto-submit, budget enforce
Failover edice-watchdog-failoverus-west-2Every 10 minActivates if master heartbeat stale >30 min

Watchdog Capabilities

  1. Multi-region scan (5 regions, single pass)
  2. Cancel stuck RUNNABLE >15 min + reroute with full env vars
  3. Parallel spread — distribute queued jobs across idle regions
  4. Terminate hung RUNNING >3 hrs
  5. Budget enforcement (region-aware pricing, $30/mo cap)
  6. Auto-reject failed heroes after 72 hrs (also in nightly pipeline)
  7. Reset stuck docking_in_progress heroes
  8. Auto-submit top A+/A/B+ heroes to best region
  9. Cross-region dedup detection
  10. Daily stat decay (spot success/fail counters)

Job Queue: cortex-gpu-queue

Job Definition: cortex-docking-gpu (v17)

Container: 865950114883.dkr.ecr.us-east-1.amazonaws.com/edice-agents:batch-nosns

vCPUs: 4 per job | GPU: 1 per job | Timeout: 3 hrs

Budget Cap: $30/month spot | Alert at: 80% ($24)

Spot Quota

RegionQuotaType
us-east-140 vCPUsAll G and VT Spot
us-east-264 vCPUsAll G and VT Spot
us-west-264 vCPUsAll G and VT Spot
ca-central-164 vCPUsAll G and VT Spot
eu-west-164 vCPUsAll G and VT Spot