Gen 5 — Reinforcement Learning Architecture # Gen 5 — Reinforcement Learning Architecture > **Updated 2026-06-30** — Status sections revised. Design decisions retained as reference. ## Current Status ### Implemented - `gen5/features.rs` — 280-dim feature vector (equity, board, betting, opponent, action history) - `gen5/actions.rs` — 11-discrete-action space with legality masking - `gen5/recorder.rs` — JSONL transition recorder (thread-safe, bounded) - `gen5/network.rs` — Candle MLP policy (280→512→LN→512→LN→256→11+1), training + inference (behind `gen5_nn` feature) - `gen5/rl_strategy.rs` — DAgger strategy (teacher fallback + model inference + recording) - `bin/train_gen5.rs` — CLI trainer with GPU/CUDA, class-weighted loss, gradient clipping, cosine LR - Registered as `gen5_collect` and `gen5_rl` bot types ### NOT Implemented - Reward assignment (end-of-hand outcome → per-transition reward) — needed for PPO - Showdown hand recording (revealed cards for range training) - Model hot-reload for live inference - PPO training loop (self-play RL) --- ## Design Decisions ### Q1: NN to Predict Ranges vs Current Design (NN takes range as input)? **Decision: Start with Approach B** (current design — NN takes equity + opponent stats as input, outputs action). Pragmatic, builds on Gen 4. If range prediction quality proves to be the bottleneck (Gen 4 ranges miss >30% of showdown hands), add a separate range-prediction NN in Phase 2. A hybrid (range-prediction NN as feature extractor feeding policy NN) is also possible. ### Q2: Train Preflop AND Postflop? **Decision: Unified network** for both. Feature vector already covers both — preflop zeroes board features, postflop populates them. Split only if convergence is slow. The NN's value-add over NG preflop tables: opponent-specific opening ranges, 3-bet/4-bet play, stack-depth-dependent adjustments. ### Q3: NN with Player Model as Input? **Decision: Yes — current design.** `OpponentFeatures` (20 fields) mirrors Gen 4 `PlayerStats`. Cold-start (unknown opponent) = all-zeros → network defaults to baseline. As stats accumulate, network adapts automatically. ### Q4: RL in Live Play Alone? **Decision: No.** Hybrid approach: 1. Phase 1 (offline): DAgger imitation learning from Gen 4 teacher 2. Phase 2 (offline): Self-play RL via PPO 3. Phase 3 (live): Fine-tune on Torn data Live play is too slow (~100 hands/hour vs ~50k in simulation) for meaningful weight updates. ### Q5: Training Curriculum (Gen 1 → 2 → 3 → 4)? **Plan:** ``` Stage 1: Beat Gen 1 (chump_bot + g1 rules) → +30 BB/100 over 50k hands Stage 2: Beat Gen 2 (g2_postflop + flock_bot) → +10 BB/100 over 50k hands Stage 3: Beat Gen 3 (g3_postflop + mixed pool) → +5 BB/100 over 100k hands Stage 4: Beat Gen 4 (g4_adaptive + all opponents)→ +2 BB/100 over 200k hands Stage 5: Self-play convergence → Nash equilibrium approximation ``` Advancement criterion: statistically significant win rate (p < 0.05) over 50k+ hands. ### Q6: Table Hopper Based on Performance? **Phase 3 feature.** Requires performance tracking in live server + table lobby scraping. Defer until Gen 5 is stable and profitable. ### Q7: Capture Complete DOM? **Planned for debugging + training data.** Userscript "snapshot mode" saves full table DOM HTML at each decision. Useful for scraper debugging and future DOM→features auto-extraction. --- ## Architecture Diagram ``` ┌─────────────────┐ │ Live Server │ │ (Axum :8088) │ └────────┬────────┘ │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ ┌──────────────┐ ┌───────────┐ ┌──────────────┐ │ Gen4 Strategy│ │ Gen5 │ │ Showdown │ │ (decisions) │ │ Recorder │ │ Recorder │ │ │ │ (JSONL) │ │ (TODO) │ └──────────────┘ └─────┬─────┘ └──────────────┘ │ ▼ ┌─────────────────────────┐ │ Training Pipeline │ │ │ │ 1. DAgger Imitation │ │ (Gen4 teacher) │ │ │ │ 2. Self-Play PPO │ │ (Gen1→2→3→4) │ │ │ │ 3. Live Fine-Tuning │ │ (Torn data) │ └───────────┬─────────────┘ │ ┌──────────▼──────────┐ │ Trained Model │ │ (safetensors) │ │ 280→512→LN→512→LN │ │ →256→11 actions+1V │ └──────────┬──────────┘ │ ┌──────────▼──────────┐ │ Inference in │ │ Live Server │ │ (<1ms per decision)│ └─────────────────────┘ ``` --- ## Implementation Roadmap ### Phase 1: DAgger Imitation Learning (in progress) 1. ✅ Feature extraction, action space, recorder, network, strategy 2. ✅ Opponent features bridge 3. ✅ Training CLI with class-weighted loss + grad clipping + cosine LR 4. ✅ GPU/CUDA support 5. 🔄 Collect ~8M transitions from Gen 4 teacher (running) 6. ⬜ Retrain v5 with all fixes, verify non-degeneracy 7. ⬜ Evaluate v5 vs Gen 4 in simulation ### Phase 2: Self-Play RL (future) 8. ⬜ Implement reward assignment 9. ⬜ PPO training loop with curriculum 10. ⬜ Showdown hand recording (for range NN) ### Phase 3: Live Deployment (future) 11. ⬜ Model hot-reload in live server 12. ⬜ Live fine-tuning on Torn data 13. ⬜ Table hopper prototype id: bac569872f9149acbfa4832eb553312c parent_id: 5a06903f7db44bfcb4c8c8a9cf0d2326 created_time: 2026-06-25T18:51:41.725Z updated_time: 2026-07-17T08:37:46.707Z is_conflict: 0 latitude: 0.00000000 longitude: 0.00000000 altitude: 0.0000 author: source_url: is_todo: 0 todo_due: 0 todo_completed: 0 source: joplin-desktop source_application: net.cozic.joplin-desktop application_data: order: 1782413501725 user_created_time: 2026-06-25T18:51:41.725Z user_updated_time: 2026-06-30T05:33:53.789Z encryption_cipher_text: encryption_applied: 0 markup_language: 1 is_shared: 0 share_id: conflict_original_id: master_key_id: user_data: deleted_time: 1784277466707 type_: 1