id: e38808ddbef44ac1bc7c2c9d611b9706
parent_id: 
item_type: 1
item_id: 486b52d6fe994be1ab6d74735bfe322f
item_updated_time: 1782506037457
title_diff: "[{\"diffs\":[[1,\"Gen 5 Implementation Progress — 2026-06-26\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":42}]"
body_diff: "[{\"diffs\":[[1,\"# Gen 5 Implementation Progress — 2026-06-26\\\n\\\n## Status: Framework skeleton complete + feature extraction working\\\n\\\n## What's Implemented\\\n\\\n### Module Structure (`holdem_bots/src/gen5/`)\\\n- `mod.rs` — module declaration + re-exports\\\n- `actions.rs` — 11-action discrete space with legality masking\\\n- `features.rs` — 280-dim feature vector extraction (card/equity + hand category + draws + board texture + betting/situation + hero state)\\\n- `recorder.rs` — JSONL transition recorder for offline training\\\n- `rl_strategy.rs` — `Strategy` trait impl (teacher delegation + recording)\\\n\\\n### Registered as `gen5_rl` (strategy #20)\\\n\\\n### Action Space (`actions.rs`)\\\n11 discrete actions:\\\n```\\\n0: Fold       3: Bet 1/3    6: Bet pot    8: Raise 2.5x\\\n1: Check      4: Bet 1/2    7: Bet 2x     9: Raise pot\\\n2: Call       5: Bet 2/3                  10: All-in\\\n```\\\n- `legal_action_mask()` — returns `[bool; 11]` based on to_call, pot, stack\\\n- `action_to_engine()` — maps DiscreteAction → engine `Action`, clamps to stack\\\n- 7 unit tests\\\n\\\n### Feature Vector (`features.rs`, 280 dims)\\\n| Section | Dims | Status | Source |\\\n|---------|------|--------|--------|\\\n| Card/Equity | 5 | ✅ DONE | PotentialResult (hs, ppot, npot, nutpot, rpot) |\\\n| Hand category | 9 | ✅ DONE | HandHelper::evaluate_hand → one-hot (0=HighCard…8=StraightFlush/RoyalFlush) |\\\n| Draw flags | 6 | ✅ DONE | flush_draw, OESD, gutshot, paired_board, monotone_board, connected_board |\\\n| Board texture | 10 | ✅ DONE | board_count, wetness, max_suit, straight_conn, high_card + 5 padding |\\\n| Betting/Situation | 25 | ✅ DONE | pot_bb, to_call_bb, SPR, stack_bb, position(9), street(4), counts(3) |\\\n| Hero state | 5 | ✅ DONE | wagered_bb, aggressor flags, facing bet |\\\n| Opponent models | 60 | ⬜ PENDING | 3 × 20 (vpip, pfr, af, frequencies, range, stack_bb) |\\\n| Action history | 160 | ⬜ PENDING | 8 × 20 (player, street, action type one-hot, amounts) |\\\n\\\n**Active features: 55/280** (card/equity + hand category + draws + board texture + betting + hero)\\\n\\\n### Transition Recorder (`recorder.rs`)\\\n- JSONL format: one `Transition` per line\\\n- Fields: hand_id, decision_idx, features[], legal_mask[], action, is_teacher, reward, hand_reward, is_terminal, street, position\\\n- Thread-safe via `Mutex<BufWriter>`\\\n- Flushes every 1000 transitions + on Drop\\\n- 3 unit tests\\\n\\\n### RL Strategy (`rl_strategy.rs`)\\\n- Implements `Strategy` trait (`&self` only — uses `Mutex<RlState>`)\\\n- **Mode 1 (current):** Records transitions while delegating to Gen 3 fallback (teacher)\\\n- **Mode 2 (TODO):** Loads model weights, runs forward pass for inference\\\n- Config from TOML: `model_path`, `transitions_path`, `record_transitions`, `use_teacher`, `epsilon`\\\n- 3 unit tests\\\n\\\n### Python Training Script (`scripts/train_gen5.py`)\\\n- **PokerPolicy**: MLP (280→512→512→256→Actor(11)+Value(1)) with legal action masking\\\n- **TransitionDataset**: loads JSONL → tensors\\\n- **train()**: cross-entropy loss, Adam optimizer, cosine LR, train/val split\\\n- **analyze_actions()**: action distribution analysis\\\n- PyTorch NOT installed yet (`pip install torch`)\\\n\\\n### Data Collection Config (`configs/bots/gen5_collect.toml`)\\\n- Bot config that uses `gen5_rl` in teacher mode with transition recording\\\n- Outputs to `/tmp/gen5_transitions.jsonl`\\\n\\\n## Tests\\\n- 19 new Gen 5 unit tests, all pass\\\n- Full suite: 454 lib tests pass (+ 11 doc tests)\\\n\\\n## What's NOT Implemented Yet\\\n\\\n### 1. Opponent Model Features (60 dims)\\\nNeed to wire Gen 4 Observer data → `OpponentFeatures`. The Observer tracks per-player stats (vpip, pfr, af, etc.) that need to be extracted and converted to f32 features.\\\n\\\n**Blocker:** Need to understand how to access the Observer from within the RL strategy. The Observer is typically shared via `Arc<Mutex<Observer>>` in the Gen 4 adaptive strategy. The RL strategy would need a reference to the same Observer.\\\n\\\n### 2. Action History Features (160 dims)\\\nNeed to convert `ActionRecorder` data → `ActionFeatures`. The ActionRecorder tracks all actions in the current hand.\\\n\\\n**Blocker:** Need to iterate ActionRecorder entries and convert each to the 5-field tuple format.\\\n\\\n### 3. Neural Network Forward Pass\\\nThe `model_infer()` method is a stub. Need to:\\\n1. Choose ML framework: candle-rs (Rust-native) vs ONNX runtime vs tch-rs\\\n2. Implement MLP forward pass in Rust for inference\\\n3. Load PyTorch-trained weights into Rust model\\\n\\\n### 4. Training Pipeline\\\n1. Collect data: `gen5_collect.toml` bot config → JSONL transitions\\\n2. Train: `python scripts/train_gen5.py --data ...`\\\n3. Export: PyTorch → ONNX (for Rust inference)\\\n4. Deploy: Load ONNX in Rust, replace teacher with model\\\n\\\n## Open Questions for User\\\n\\\n1. **ML Framework for inference:** candle-rs (pure Rust, no deps) vs ONNX runtime (mature, GPU) vs tch-rs (PyTorch bindings)?\\\n   - **My recommendation:** ONNX runtime — train in PyTorch, export to ONNX, load in Rust. Clean separation.\\\n   - **Alternative:** candle-rs — single binary, no external runtime, but smaller community.\\\n\\\n2. **Observer access:** How should the RL strategy access Gen 4 Observer data?\\\n   - Option A: Pass `Arc<Mutex<Observer>>` as a config parameter to the RL strategy\\\n   - Option B: Store Observer in a global/thread-local and access via strategy\\\n   - Option C: The RL strategy creates its own Observer (duplicates tracking)\\\n   - **My recommendation:** Option A (explicit dependency injection)\\\n\\\n3. **Training approach:** Start with HU (5× faster convergence) or 6-max (target format)?\\\n   - **My recommendation:** Start HU, expand to 6-max after v1 converges\\\n\\\n4. **Reward signal:** Per-hand BB delta with all-in equity adjustment, or per-street reward shaping?\\\n   - **My recommendation:** Per-hand BB delta with all-in equity adjustment (cleaner, less noise)\\\n\\\n5. **Self-play opponent pool:** Mix of Gen 1-4 bots + earlier checkpoints of the policy, or pure self-play?\\\n   - **My recommendation:** Mixed pool (70% similar-strength, 20% weaker, 10% stronger)\\\n\\\n## Implementation Steps (clear, no questions needed)\\\n1. ✅ Feature extraction: card/equity, hand category, draws, board texture\\\n2. ⬜ Wire Gen 4 Observer → OpponentFeatures (after Q2 answered)\\\n3. ⬜ Wire ActionRecorder → ActionFeatures\\\n4. ⬜ Collect imitation learning data (run gen5_collect config)\\\n5. ⬜ Install PyTorch, train initial model\\\n6. ⬜ Implement ONNX inference in Rust (after Q1 answered)\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":6284}]"
metadata_diff: {"new":{"id":"486b52d6fe994be1ab6d74735bfe322f","parent_id":"e13f1845de9b4b6392ad866354fbd562","latitude":"0.00000000","longitude":"0.00000000","altitude":"0.0000","author":"","source_url":"","is_todo":0,"todo_due":0,"todo_completed":0,"source":"joplin-desktop","source_application":"net.cozic.joplin-desktop","application_data":"","order":1782505664599,"markup_language":1,"is_shared":0,"share_id":"","conflict_original_id":"","master_key_id":"","user_data":"","deleted_time":0},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-06-26T20:37:35.230Z
created_time: 2026-06-26T20:37:35.230Z
type_: 13