Pipeline optimization: caching, parallelism, Rust migration, multi-region GPU distribution.
| Step | Shard Strategy | Workers | Before | After |
|---|---|---|---|---|
| Gene Enrichment | Chunks of 10 genes | 5 parallel shards × 10 threads each | ~45s sequential | ~5s |
| FRSI Scoring | By cancer type | 4 parallel (1 per cancer) | ~30s sequential | ~8s |
| FRSI DDB Writes | Per cancer shard | 10 threaded writers per shard | 89 sequential | ~2s |
| GPU Docking | By region | 5 regions parallel (watchdog spread) | 1 region, 129 queued | 5 regions, distributed |
| Rust Enrichment (Fargate) | 20 async tasks | tokio JoinSet | n/a | ~5s cold, <1s cached |
| Optimization | Before | After | Technique |
|---|---|---|---|
| DDB scans | 5× full table scan | 1× single scan | Pre-load all_heroes, pass to steps |
| Gene enrichment HTTP | 534 new TCP connections | Connection pooling (keep-alive) | requests.Session() |
| Enrichment workers | 4 threads | 10 threads | ThreadPoolExecutor |
| Enrichment batch | 20 genes/night | 50 genes/night | Raised BATCH_SIZE |
| HTTP timeout | 12s (blocks thread) | 8s (fail fast) | Lower timeout + retry |
| DDB writes per gene | 3 sequential | 3 parallel | ThreadPoolExecutor per gene |
| FRSI DDB writes | 89 sequential update_item | 10 threaded | ThreadPoolExecutor |
| Lambda timeout | 120s (crashing) | 900s | Configuration fix |
| Total nightly | >3 min (timeout) | ~45s | All combined |
Table: cortex-enrichment-cache
Key: {api}:{gene} (e.g. chembl:BRAF)
TTL: 7 days (DynamoDB auto-expiry enabled)
APIs cached: UniProt, ChEMBL, OpenTargets
Impact: First run ~5s (cold), subsequent runs <1s (all cache hits)
Cost: ~$0.01/month (89 genes × 3 APIs = 267 items, each <1KB)
| Module | Status | Concurrency | Speed vs Python |
|---|---|---|---|
| Gene Enrichment | ✅ Scaffolded | 20 async tasks (tokio) | ~10× faster |
| Watchdog | Python (deployed) | Sequential per region | Fine at 5 regions |
| FRSI Scoring | Migratable (just math) | — | ~5× faster |
| FGAO / RDKit | Keep Python | — | Needs RDKit bindings |
| GCN Training | Keep Python | — | numpy is fast enough |
┌─────────────────────────────────────────────────┐ │ cortex-rs enrichment module │ ├─────────────────────────────────────────────────┤ │ │ │ scan_unenriched_heroes() ──→ Vec<String> │ │ │ │ │ ▼ (20 concurrent tasks) │ │ ┌──────────────────────────────────────┐ │ │ │ fetch_cached(ddb, http, api, gene) │ │ │ │ ├── cache_get() → HIT → return │ │ │ │ └── MISS → fetch API → cache_put │ │ │ └──────────────────────────────────────┘ │ │ │ │ │ ▼ (parallel DDB writes) │ │ write_enrichment() ──→ cortex-heroes table │ │ │ └─────────────────────────────────────────────────┘
| Feature | Logic | Impact |
|---|---|---|
| Parallel spread | If 1 region has >3 queued + others idle → distribute | 129 jobs → 5 regions in 1 cycle |
| Stuck reroute | RUNNABLE >15min → cancel + reroute to best region | No more 69-min waits |
| Region scoring | Track success/fail rate per region, pick lowest failure | Auto-avoids bad capacity zones |
| Spread cap | Keep 4 in source, spread up to 60 per cycle | Fleet fully utilized in 1 cycle |
| Dedup | Cross-region duplicate detection | No wasted GPU time |
| Metric | Value |
|---|---|
| Execution time | ~3s per cycle |
| API calls per cycle | 15 (3 per region × 5 regions) |
| Schedule | Every 5 min (master), every 10 min (failover) |
| Monthly cost | ~$0.50 (288 invocations/day × 3s × 256MB) |
| ECS Fargate (current) | K8s (EKS) | |
|---|---|---|
| Ops overhead | Zero — no cluster | Cluster, nodes, upgrades |
| Cost at our scale | ~$0.002/run | $75/mo minimum (control plane) |
| Spin-up | ~10s | ~10s (warm nodes) |
| GPU support | EC2 mode (Batch) | Full pod scheduling |
| Already deployed | ✅ cortex-fargate exists | Would need setup |
| Multi-region | Separate task per region | Federation complexity |
Decision: Stay on ECS. Migrate to EKS only when:
| Optimization | Impact | When |
|---|---|---|
| Rust nightly binary | Full pipeline in ~10s | Burst mode session |
| NVIDIA DGX Cloud burst | 100+ concurrent docking jobs | Tomorrow |
| Google Cloud TPU | GCN training acceleration | Tomorrow |
| Sharded Lambda fan-out | 5 parallel workers × 18 genes each | At 200+ heroes |
| S3 Select for large datasets | Skip full TCGA downloads | At FRSI v0.3 |