id: 093b781a8bf24fe5b3edf2fb7e409f16
parent_id: 
item_type: 1
item_id: bac569872f9149acbfa4832eb553312c
item_updated_time: 1782586529311
title_diff: "[{\"diffs\":[[1,\"Gen 5 — Reinforcement Learning Architecture\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":43}]"
body_diff: "[{\"diffs\":[[1,\"# Gen 5 — Reinforcement Learning Architecture\\\n\\\n> **Updated 2026-06-27** — Added answers to design questions, training curriculum, and implementation status.\\\n\\\n## Current Status\\\n\\\n### Code Implemented (compiles, tests pass)\\\n- `gen5/features.rs` — 280-dim feature vector extraction (equity, board, betting, opponent, action history)\\\n- `gen5/actions.rs` — 11-discrete-action space with legality masking\\\n- `gen5/recorder.rs` — JSONL transition recorder (thread-safe, bounded)\\\n- `gen5/network.rs` — Candle MLP policy (280→512→512→256→11+1), training + inference (behind `gen5_nn` feature)\\\n- `gen5/rl_strategy.rs` — `Strategy` trait impl (teacher fallback + optional model inference + transition recording)\\\n- Registered as `gen5_collect` bot type (collects transitions via Gen3 teacher)\\\n\\\n### What's NOT Done Yet\\\n- Training loop (generate → train → evaluate cycle)\\\n- Opponent features bridge (from Gen4 PlayerModel → OpponentFeatures struct)\\\n- Reward assignment (end-of-hand outcome → per-transition reward)\\\n- Model hot-reload for live inference\\\n- Hand database for live fine-tuning\\\n\\\n---\\\n\\\n## Design Decisions (Answers to Key Questions)\\\n\\\n### Q1: NN to Predict Ranges (Weight Tables) vs Current Heuristic?\\\n\\\n**Two viable approaches:**\\\n\\\n| Approach | A: NN predicts full range | B: NN takes range as input (current design) |\\\n|---|---|---|\\\n| **What it learns** | Maps (opponent stats, action sequence, board) → weighted 1326-hand range | Maps (equity vs predicted range, opponent stats) → best action |\\\n| **Training data** | Needs **showdown hands** (where opponent cards are revealed) — rare, ~1 per 5 hands | Needs **decision + outcome** pairs — every hand generates data |\\\n| **Complexity** | High — output space is 1326-dimensional | Low — output space is 11-dimensional |\\\n| **When it helps** | If range prediction is the bottleneck (Gen 4's heuristic ranges are wrong) | If strategy/sizing is the bottleneck (we don't exploit ranges well enough) |\\\n| **Verdict** | **Phase 2** — ambitious, powerful, but needs lots of showdown data | **Phase 1** — pragmatic, fast to implement, builds on Gen 4 |\\\n\\\n**Recommendation**: Start with Approach B (current design). If analysis shows range prediction quality is the bottleneck (Gen 4 ranges miss >30% of showdown hands), add a separate range-prediction NN in Phase 2.\\\n\\\nA **hybrid** is also possible: train a range-prediction NN as a *feature extractor* whose output feeds into the policy NN. This is essentially multi-task learning.\\\n\\\n### Q2: Should We Start Recording All Hands?\\\n\\\n**YES — immediately.** Even without Gen 5, recorded hands are valuable for:\\\n\\\n1. **Imitation learning data** — every (state, action, outcome) tuple is a training sample\\\n2. **Opponent profiling** — track real opponent tendencies across sessions\\\n3. **Strategy validation** — replay hands with new configs to measure improvement\\\n4. **Range accuracy analysis** — compare predicted ranges to showdown cards\\\n5. **Bug detection** — replay problematic hands (hand_replay crate already does this)\\\n\\\n**Implementation plan:**\\\n- The `gen5_collect` bot type already records transitions to JSONL\\\n- Need to wire it into the **live server** as a secondary recorder (Gen4 strategy makes decisions, Gen5 recorder captures features alongside)\\\n- Also record **showdown hands** separately (hand_id, all hole cards, board, winner) for range training\\\n- Store in `data/hands/YYYY-MM-DD/` with daily rotation\\\n\\\n### Q3: Train Preflop AND Postflop?\\\n\\\n**Yes — both, with the same network.** The feature vector already covers both:\\\n\\\n- **Preflop**: board features are zeroed, equity computed vs random/range, action history is just the preflop betting round\\\n- **Postflop**: board features populated, equity computed vs board+ranges, action history includes all streets\\\n\\\n**Alternative**: Separate preflop/postflop networks\\\n- Pro: each converges faster (simpler input space)\\\n- Con: loses transfer learning (postflop play depends on preflop tendencies), more infrastructure\\\n- Verdict: **Unified network first**. Split only if convergence is slow.\\\n\\\n**For preflop specifically**: the existing NG preflop (169-hand tables) is already quite good. The NN's value-add is in:\\\n- Adapting to opponent-specific opening ranges (tight/loose players)\\\n- 3-bet/4-bet pot play (where static tables are weakest)\\\n- Stack-depth-dependent adjustments (current tables don't handle 40-60 BB well)\\\n\\\n### Q4: NN with Current Player Model as Input?\\\n\\\n**Yes — this is exactly the current design.** The `OpponentFeatures` struct (20 fields) mirrors the Gen 4 `PlayerModel`:\\\n\\\n```\\\nvpip, pfr, af, fold_to_bet, bet_freq, raise_freq, call_freq,\\\ncbet_freq, threebet, wtsd, hands_observed, stack_bb, wagered_bb,\\\nis_active, distance_from_hero, range_width, range_strength,\\\nis_aggressor, has_acted, confidence\\\n```\\\n\\\n**This is much easier than learning ranges from scratch** because:\\\n- We don't need showdown labels (unsupervised features work)\\\n- The model already condenses the action history into meaningful stats\\\n- Cold-start (unknown opponent) = all-zeros features → network defaults to baseline play\\\n- As stats accumulate, the network automatically adapts\\\n\\\n### Q5: RL in Live Play — Enough? How Fast?\\\n\\\n**RL in live play alone is NOT enough.** Here's why:\\\n\\\n| Factor | Live Play | Simulation |\\\n|---|---|---|\\\n| Hands/hour | ~100 | ~50,000 (32 cores) |\\\n| To reach 10M hands | ~3 years | ~8 days |\\\n| Opponent variety | 5-9 regulars | Unlimited bot pool |\\\n| Variance | High (real money at stake) | Controlled (equity-adjusted) |\\\n| Non-stationarity | Opponents adapt to us | We control the pool |\\\n\\\n**Recommended hybrid approach:**\\\n1. **Phase 1 (offline)**: Imitation learning from Gen 3/4 teacher (~2M hands, ~8 hours)\\\n2. **Phase 2 (offline)**: Self-play RL via PPO (~50M hands, ~3 days)\\\n3. **Phase 3 (live)**: Fine-tune on real Torn data (~10k hands/week, lightweight updates)\\\n\\\n**Adaptation speed:**\\\n- **Per-opponent**: Instant via features (network sees updated VPIP/PFR/AF → adjusts)\\\n- **Weight updates**: Slow (~1k hands for meaningful gradient signal, impractical live)\\\n- **Cold start**: Network uses default features → plays baseline Gen 3/4 style until stats accumulate (~20 hands)\\\n\\\n### Q6: Training Curriculum (Gen 1 → 2 → 3 → 4)\\\n\\\n**Excellent idea — standard curriculum learning.** Revised plan:\\\n\\\n```\\\nStage 1: Beat Gen 1 (chump_bot + g1 rules)\\\n  → Network learns basic ABC poker\\\n  → Target: +30 BB/100 over 50k hands\\\n  \\\nStage 2: Beat Gen 2 (g2_postflop_rollout + flock_bot)  \\\n  → Network learns equity-based decisions\\\n  → Target: +10 BB/100 over 50k hands\\\n  \\\nStage 3: Beat Gen 3 (g3_postflop_rollout + mixed pool)\\\n  → Network learns range-aware play\\\n  → Target: +5 BB/100 over 100k hands (harder)\\\n  \\\nStage 4: Beat Gen 4 (g4_adaptive + all opponents)\\\n  → Network learns opponent exploitation\\\n  → Target: +2 BB/100 over 200k hands (much harder)\\\n  \\\nStage 5: Self-play convergence\\\n  → Network plays itself + all previous gens\\\n  → Target: Nash equilibrium approximation\\\n```\\\n\\\n**Advancement criterion**: Move to next stage when win rate is statistically significant (p < 0.05) over 50k+ hands. If losing, drop back a stage.\\\n\\\n### Q7: Table Hopper Based on Performance\\\n\\\n**Phase 2 feature.** Implementation plan:\\\n\\\n```python\\\n# Meta-controller logic:\\\nif win_rate_last_500_hands > 10 BB/100:\\\n    move up in stakes (if table available)\\\nelif win_rate_last_500_hands < -15 BB/100:\\\n    move down in stakes\\\nif current_table_vpip < 0.20 for 100+ hands:\\\n    find looser table (table selection)\\\n```\\\n\\\nThis requires:\\\n- Performance tracking in the live server (rolling BB/100 window)\\\n- Table lobby scraping (available tables, their stats)\\\n- Blind level/stake detection\\\n- Userscript table-switching capability\\\n\\\nDefer until Gen 5 is stable and profitable.\\\n\\\n### Q8: Capture Complete DOM to Files\\\n\\\n**Yes — for debugging AND training data.** Plan:\\\n\\\n1. **Userscript \\\"snapshot mode\\\"**: At each `waiting_for_action`, save the full table DOM HTML to a file\\\n2. **Storage**: `data/dom_snapshots/YYYY-MM-DD/hand_XXX.html`\\\n3. **Uses**:\\\n   - Debug parsing edge cases (new Torn UI changes)\\\n   - Extract features the scraper missed\\\n   - Build labeled dataset for scraper improvement\\\n   - Training data for DOM→features auto-extraction (future)\\\n\\\nAlso useful for the **standalone poker screen** — capture and replay full game states.\\\n\\\n---\\\n\\\n## Implementation Roadmap (Priority Order)\\\n\\\n### Immediate (next session)\\\n1. **Wire Gen5 recorder into live server** — record transitions alongside Gen4 decisions\\\n2. **Record showdown hands** — separate file with revealed cards for range training\\\n3. **Bridge PlayerModel → OpponentFeatures** — fill the 20-field structs from Gen4 observer\\\n4. **Add reward assignment** — end-of-hand outcome propagates to all transitions in that hand\\\n\\\n### Short-term (1-2 weeks)\\\n5. **Imitation learning Phase 1** — collect ~500k hands from Gen3/4 teacher, train MLP\\\n6. **Evaluate imitation model** — compare to Gen3/4 in simulation\\\n7. **DOM snapshot mode** — add to userscript for debugging\\\n\\\n### Medium-term (2-4 weeks)\\\n8. **Self-play RL (PPO)** — curriculum learning Gen 1→2→3→4\\\n9. **Model serving in live server** — load trained weights, forward pass for decisions\\\n10. **Table hopper prototype** — basic win-rate-based stake changes\\\n\\\n### Long-term (1-3 months)\\\n11. **Live fine-tuning** — adapt to Torn player pool\\\n12. **Range prediction NN** (if needed) — separate model for opponent ranges\\\n13. **Multi-site deployment** — generalize beyond Torn\\\n\\\n---\\\n\\\n## Architecture Diagram\\\n\\\n```\\\n                    ┌─────────────────┐\\\n                    │  Live Server    │\\\n                    │  (Axum :8088)   │\\\n                    └────────┬────────┘\\\n                             │\\\n              ┌──────────────┼──────────────┐\\\n              ▼              ▼              ▼\\\n     ┌──────────────┐ ┌───────────┐ ┌──────────────┐\\\n     │ Gen4 Strategy│ │ Gen5      │ │ Showdown     │\\\n     │ (decisions)  │ │ Recorder  │ │ Recorder     │\\\n     │              │ │ (JSONL)   │ │ (hands)      │\\\n     └──────────────┘ └─────┬─────┘ └──────┬───────┘\\\n                            │              │\\\n                            ▼              ▼\\\n                   ┌─────────────────────────┐\\\n                   │  Training Pipeline      │\\\n                   │                         │\\\n                   │  1. Imitation Learning  │\\\n                   │     (Gen3/4 teacher)    │\\\n                   │                         │\\\n                   │  2. Self-Play PPO       │\\\n                   │     (Gen1→2→3→4)        │\\\n                   │                         │\\\n                   │  3. Live Fine-Tuning    │\\\n                   │     (Torn data)         │\\\n                   └───────────┬─────────────┘\\\n                               │\\\n                    ┌──────────▼──────────┐\\\n                    │  Trained Model      │\\\n                    │  (safetensors)      │\\\n                    │  280→512→512→256    │\\\n                    │  →11 actions + 1 V  │\\\n                    └──────────┬──────────┘\\\n                               │\\\n                    ┌──────────▼──────────┐\\\n                    │  Inference in       │\\\n                    │  Live Server        │\\\n                    │  (<1ms per decision)│\\\n                    └─────────────────────┘\\\n```\\\n\\\n---\\\n\\\n## Training Data Requirements\\\n\\\n| Data Type | Source | Volume Needed | Status |\\\n|---|---|---|---|\\\n| Imitation (state, action) | Gen3/4 teacher in sim | ~2M transitions | Recorder ready, needs wiring |\\\n| Self-play (state, action, reward) | NN vs bot pool | ~50M transitions | Needs training loop |\\\n| Showdown hands (cards revealed) | Torn live | ~10k hands | Needs showdown recorder |\\\n| Real opponent stats | Torn live (Gen4 observer) | Ongoing | Already saved to profiles/ |\\\n\\\n---\\\n\\\n## Feature Vector Layout (280 dims, implemented)\\\n\\\n| Section | Dims | Description |\\\n|---|---|---|\\\n| Card/Equity | 20 | hs, ppot, npot, nutpot, rpot + hand category (9 one-hot) + draws (6) |\\\n| Board texture | 10 | board_count, wetness, suit_count, connectivity, high_card + padding |\\\n| Betting/Situation | 25 | pot_bb, to_call_bb, spr, stack_bb + position (9) + street (4) + counts (3) + padding |\\\n| Hero state | 5 | wagered_bb, is_aggressor, is_pfr_aggressor, aggressor_present, facing_bet |\\\n| Opponent models (×3) | 60 | 20 fields each from Gen4 PlayerModel |\\\n| Action history (×8) | 160 | 20 fields each (player, street, action_type, amounts, padding) |\\\n| **Total** | **280** | |\\\n\\\n---\\\n\\\n## Network Architecture (implemented, Candle)\\\n\\\n```\\\nInput(280) → Linear(512) → ReLU\\\n           → Linear(512) → ReLU  \\\n           → Linear(256) → ReLU\\\n           → Actor head: Linear(11) → masked softmax\\\n           → Value head: Linear(1)\\\n```\\\n\\\n- **Actor-critic** (A2C/PPO compatible)\\\n- **Action masking**: illegal actions get -1e9 logits before softmax\\\n- **Inference**: <1ms on CPU (280→512→512→256→11 = ~1M params)\\\n- **Training**: AdamW optimizer, cross-entropy loss for imitation phase\\\n\\\n---\\\n\\\n## Open Questions (Updated)\\\n\\\n1. **Rust-native (candle) vs Python (PyTorch) for training?**\\\n   - Candle works for inference (already implemented)\\\n   - For PPO training loop, Python/PyTorch is more mature (stable-baselines3, etc.)\\\n   - **Recommendation**: Generate transitions in Rust (fast), train in Python (mature), deploy in Rust (candle for inference)\\\n\\\n2. **Equity MC in training loop?**\\\n   - Pre-compute and cache per decision point (current approach works)\\\n   - For self-play, each table computes MC independently (no sharing needed)\\\n\\\n3. **Reward design?**\\\n   - Phase 1 (imitation): no reward needed (supervised)\\\n   - Phase 2 (PPO): BB/hand with all-in equity adjustment (credit expected value, not runout luck)\\\n   - Consider per-street reward shaping: small positive for good folds, small negative for bad calls\\\n\\\n4. **Multi-table training?**\\\n   - Yes — run 100+ tables in parallel via rayon, collect transitions centrally\\\n   - This is how we reach 50k hands/hour on 32 cores\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":13801}]"
metadata_diff: {"new":{"id":"bac569872f9149acbfa4832eb553312c","parent_id":"e13f1845de9b4b6392ad866354fbd562","latitude":"0.00000000","longitude":"0.00000000","altitude":"0.0000","author":"","source_url":"","is_todo":0,"todo_due":0,"todo_completed":0,"source":"joplin-desktop","source_application":"net.cozic.joplin-desktop","application_data":"","order":1782413501725,"markup_language":1,"is_shared":0,"share_id":"","conflict_original_id":"","master_key_id":"","user_data":"","deleted_time":0},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-06-27T18:57:38.054Z
created_time: 2026-06-27T18:57:38.054Z
type_: 13