id: 7575eb0ce9e74d4784488b89332fb312
parent_id: 
item_type: 1
item_id: c7a39e16770542a08004fd73e971ca1f
item_updated_time: 1782797557406
title_diff: "[{\"diffs\":[[1,\"Gen 5 — Reinforcement Learning (State of the Art)\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":49}]"
body_diff: "[{\"diffs\":[[1,\"# Gen 5 — Reinforcement Learning\\\n\\\n> **Current state, 2026-06-30. Training pipeline implemented. Four model versions trained (v1–v4), all degenerate. Root causes identified and fixed in code. Fresh data collection running; v5 retraining pending.**\\\n\\\n## Role in the Framework\\\n\\\nGen 5 replaces the heuristic decision logic (Gen 3/4 thresholds) with a **neural network** trained via imitation learning (DAgger). It does not replace the Gen 4 observer — it uses the observer's stats as input features and learns when to trust equity vs exploit opponent tendencies, optimal bet sizing, and bluffing frequencies.\\\n\\\n## Architecture\\\n\\\n### Network\\\n\\\n```\\\nInput(280) → Linear(512) → ReLU → LayerNorm(512)\\\n           → Linear(512) → ReLU → LayerNorm(512)\\\n           → Linear(256) → ReLU\\\n           → Actor head: Linear(11) → masked softmax\\\n           → Value head: Linear(1)\\\n```\\\n\\\n~1M parameters. Inference <1ms on CPU. LayerNorm added (was documented but missing — fixed 2026-06-29).\\\n\\\n### Feature Vector (280 dims)\\\n\\\n| Section | Dims | Content |\\\n|---|---|---|\\\n| Card/Equity | 20 | hs, ppot, npot, nutpot, rpot + hand category (9 one-hot) + draws (6) |\\\n| Board texture | 10 | board_count, wetness, suit_count, connectivity, high_card |\\\n| Betting/Situation | 25 | pot_bb, to_call_bb, spr, stack_bb + position (9) + street (4) + counts |\\\n| Hero state | 5 | wagered_bb, is_aggressor, is_pfr_aggressor, facing_bet |\\\n| Opponent models (×3) | 60 | 20 fields each from Gen 4 PlayerStats (bridged via `from_player_stats()`) |\\\n| Action history (×8) | 160 | Per-action: player_offset, street, action_type, amount_bb, pot_after_bb |\\\n| **Total** | **280** | |\\\n\\\n### Action Space (11 discrete)\\\n\\\n`Fold, Check, Call, BetThird(0.33), BetHalf(0.50), BetTwoThird(0.67), BetPot(1.0), Bet2x(2.0), Raise25x(2.5×), RaisePot(pot), AllIn`\\\n\\\nIllegal actions masked with -∞ logits before softmax. DAgger mode: teacher always computed first; model plays if `prob ≥ confidence_threshold` (0.3 default).\\\n\\\n## What's Implemented\\\n\\\n| Component | File | Status |\\\n|---|---|---|\\\n| Feature extraction (280 dims) | `gen5/features.rs` | ✅ Complete + opponent bridge |\\\n| Action space (11 discrete + masking) | `gen5/actions.rs` | ✅ Complete |\\\n| Transition recorder (JSONL) | `gen5/recorder.rs` | ✅ Complete |\\\n| NN policy network (Candle MLP + LayerNorm) | `gen5/network.rs` | ✅ Complete |\\\n| RL strategy (DAgger + teacher fallback) | `gen5/rl_strategy.rs` | ✅ Complete |\\\n| Training CLI (`train_gen5`) | `bin/train_gen5.rs` | ✅ Complete (GPU/CUDA) |\\\n| Data collection bot (`gen5_collect`) | registered | ✅ Complete |\\\n| Self-play loop script | `scripts/gen5_selfplay_loop.sh` | ✅ Complete |\\\n| Opponent features bridge | `gen5/features.rs` | ✅ Complete (`extract_opponent_features()`) |\\\n\\\n### Training Features\\\n\\\n- **GPU support**: CUDA on RTX 5060 Ti (build with `CUDA_COMPUTE_CAP=89`, JIT-compiled at runtime)\\\n- **Class-weighted loss**: inverse-frequency weighting (capped at 10×) to prevent majority-class collapse\\\n- **Gradient clipping**: global norm clipping at 1.0\\\n- **Cosine LR schedule**: warmup (10% of epochs) + decay to 10% base LR, applied at epoch boundaries only\\\n- **Per-class accuracy analysis**: action distribution + weight reporting\\\n\\\n## What's NOT Implemented\\\n\\\n| Component | Status | Priority |\\\n|---|---|---|\\\n| Reward assignment (outcome → per-transition) | ❌ Not started | High — needed for PPO |\\\n| Showdown hand recording (revealed cards) | ❌ Not started | Medium — needed for range-prediction NN |\\\n| Model hot-reload for live inference | ❌ Not started | Medium |\\\n| PPO self-play training loop | ❌ Not started | Phase 2 |\\\n\\\n## Model History\\\n\\\n| Version | Training Data | Accuracy | Status |\\\n|---|---|---|---|\\\n| v1 | 818K transitions (buggy opponent features) | 50.2% | Degenerate (AllIn bias) |\\\n| v2 | 1.2M transitions (opponent features bridged) | 45.2% | Degenerate (majority-class collapse) |\\\n| v3 | — | 51.8% | Degenerate (AllIn policy) |\\\n| v4 | 1.08M transitions (LayerNorm added) | 66.3% | **Degenerate** — collapsed to Check/Fold. Loss 0.996 ≈ always-Check baseline 0.955 |\\\n\\\n### v4 Degeneracy — Root Causes (all fixed)\\\n\\\n1. **LayerNorm missing**: Network architecture documented LayerNorm but it wasn't implemented in code. Model had no normalization → unstable training.\\\n2. **`engine_action_to_discrete` broken**: All bets classified as BetHalf, all raises as Raise25x. Training labels were wrong → model learned degenerate policy.\\\n3. **Opponent features not bridged**: `compute_features()` passed `&[]` for opponent stats → 60/280 dims always zero.\\\n4. **Class imbalance**: Check/Fold dominate (60%+ of transitions). Plain cross-entropy loss → model collapses to majority class.\\\n5. **Per-batch optimizer recreation**: `set_lr()` called every batch (~47K times/epoch), destroying AdamW momentum state each time.\\\n\\\n**Fixes applied**: LayerNorm implemented, discretization fixed (classifies by pot fraction / bet multiple), opponent bridge added, class-weighted loss implemented, LR schedule moved to epoch boundaries.\\\n\\\n## Current Data Collection\\\n\\\nFresh self-play collection running (8 tables × 1M hands) with fixed discretization. New data shows **10/11 action classes active** (was 6/11 with buggy discretization). Action distribution:\\\n\\\n| Action | Share |\\\n|---|---|\\\n| Fold | 27.5% |\\\n| Check | 35.6% |\\\n| Call | 9.9% |\\\n| BetThird | 4.0% |\\\n| BetHalf | 4.5% |\\\n| BetTwoThird | 14.3% |\\\n| BetPot | 3.0% |\\\n| Raise25x | 0.3% |\\\n| RaisePot | 0.5% |\\\n| AllIn | 0.6% |\\\n\\\nRate: ~330 transitions/sec. Target: ~8M transitions.\\\n\\\n## Training Pipeline (planned)\\\n\\\n### Phase 1: DAgger Imitation Learning (current)\\\n1. Collect transitions from Gen 4 teacher in simulation\\\n2. Train MLP with class-weighted cross-entropy loss\\\n3. Deploy model as new teacher, collect again (DAgger iteration)\\\n4. Repeat until model matches or exceeds Gen 4\\\n\\\n### Phase 2: Self-Play RL via PPO (future)\\\n- Curriculum: Gen 1 → 2 → 3 → 4 (progressively harder opponents)\\\n- Pool: mix of all generations + earlier checkpoints\\\n- Reward: BB/hand with all-in equity adjustment\\\n- Requires reward assignment implementation\\\n\\\n### Phase 3: Live Fine-Tuning\\\n- Collect Torn session data\\\n- Fine-tune on real opponent distributions\\\n\\\n## Open Issues\\\n\\\n1. **v5 training pending**: All fixes are in code but need rebuild with CUDA + retrain on fresh data. Must verify per-class accuracy improves for minority classes (not just overall accuracy).\\\n\\\n2. **`gen5_pipeline.sh` bash arithmetic bug**: Line 164 has `+56.30: syntax error`. DAgger iterations never start. Needs fix for automation.\\\n\\\n3. **Small MLP capacity**: 280→512→512→256→11 may be too small for opponent-aware features. If v5 still underperforms, consider larger hidden layers or attention mechanism.\\\n\\\n4. **No validation gate**: Model deploys regardless of quality. Need automated check: if model loses to Gen 4 in simulation, reject and keep previous.\\\n\\\n5. **Action discretization is coarse** (11 actions): Sizing buckets may miss optimal bet sizes. Phase 2 consideration.\\\n\\\n## Key Files\\\n\\\n| File | Content |\\\n|---|---|\\\n| `holdem_bots/src/gen5/features.rs` | 280-dim feature extraction + OpponentFeatures bridge |\\\n| `holdem_bots/src/gen5/actions.rs` | 11-action space + legality masking |\\\n| `holdem_bots/src/gen5/recorder.rs` | JSONL transition recorder |\\\n| `holdem_bots/src/gen5/network.rs` | Candle MLP policy (LayerNorm + Trainer with class weights, grad clipping, cosine LR) |\\\n| `holdem_bots/src/gen5/rl_strategy.rs` | DAgger strategy (teacher fallback + model inference + recording) |\\\n| `holdem_bots/src/bin/train_gen5.rs` | CLI trainer (GPU/CUDA, class-weighted loss, per-class analysis) |\\\n| `configs/bots/gen5_collect.toml` | Data collection config (Gen 4 teacher + recording) |\\\n| `configs/bots/gen5_rl.toml` | Inference config (model-driven + teacher fallback) |\\\n| `scripts/gen5_selfplay_loop.sh` | DAgger self-play automation |\\\n| `scripts/gen5_train_collect.sh` | Parallel data collection |\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":7842}]"
metadata_diff: {"new":{"id":"c7a39e16770542a08004fd73e971ca1f","parent_id":"5a06903f7db44bfcb4c8c8a9cf0d2326","latitude":"0.00000000","longitude":"0.00000000","altitude":"0.0000","author":"","source_url":"","is_todo":0,"todo_due":0,"todo_completed":0,"source":"joplin-desktop","source_application":"net.cozic.joplin-desktop","application_data":"","order":1782623846889,"markup_language":1,"is_shared":0,"share_id":"","conflict_original_id":"","master_key_id":"","user_data":"","deleted_time":0},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-06-30T05:37:45.265Z
created_time: 2026-06-30T05:37:45.265Z
type_: 13