id: 243ce0c2b827464295e74f126a7d9c26
parent_id: 7575eb0ce9e74d4784488b89332fb312
item_type: 1
item_id: c7a39e16770542a08004fd73e971ca1f
item_updated_time: 1783068824217
title_diff: "[]"
body_diff: "[{\"diffs\":[[0,\"26-0\"],[-1,\"6-30. Training pipeline implemented. Four model versions trained (v1–v4), all degenerate. Root causes identified and fixed in code. Fresh data collection running; v5 retraining pending\"],[1,\"7-03. Range-weighted equity features added (280→290 dims). RL self-play loop running with frozen backbone + REINFORCE. Model at iteration ~110 (last successful update before FEATURE_DIM break). Need fresh data collection + retrain with new 290-dim features\"],[0,\".**\\\n\"]],\"start1\":55,\"start2\":55,\"length1\":192,\"length2\":264},{\"diffs\":[[0,\"ing \"],[-1,\"(DAgger). It does not replace the Gen 4 observer — i\"],[1,\"then RL fine-tuning (REINFORCE). I\"],[0,\"t us\"]],\"start1\":464,\"start2\":464,\"length1\":60,\"length2\":42},{\"diffs\":[[0,\"ses the \"],[1,\"Gen 4 \"],[0,\"observer\"]],\"start1\":505,\"start2\":505,\"length1\":16,\"length2\":22},{\"diffs\":[[0,\"re\\\n\\\n### \"],[-1,\"Network\"],[1,\"Multi-Head Network (current production)\"],[0,\"\\\n\\\n```\\\nIn\"]],\"start1\":678,\"start2\":678,\"length1\":23,\"length2\":55},{\"diffs\":[[0,\"\\\nInput(2\"],[-1,\"8\"],[1,\"9\"],[0,\"0) → Lin\"]],\"start1\":730,\"start2\":730,\"length1\":17,\"length2\":17},{\"diffs\":[[0,\"→ LayerNorm(512)\"],[1,\"     [backbone, frozen during RL]\"],[0,\"\\\n           → Li\"]],\"start1\":763,\"start2\":763,\"length1\":32,\"length2\":65},{\"diffs\":[[0,\"512)\"],[-1,\"\\\n           → Linear(256) → ReLU\"],[1,\"     [backbone, frozen during RL]\\\n           → Linear(256) → ReLU                       [backbone, frozen during RL]\"],[0,\"\\\n   \"]],\"start1\":857,\"start2\":857,\"length1\":40,\"length2\":124},{\"diffs\":[[0,\"   → Act\"],[-1,\"or\"],[1,\"ion\"],[0,\" head: L\"]],\"start1\":986,\"start2\":986,\"length1\":18,\"length2\":19},{\"diffs\":[[0,\"ear(\"],[-1,\"11)\"],[1,\"3) \"],[0,\" → m\"]],\"start1\":1007,\"start2\":1007,\"length1\":11,\"length2\":11},{\"diffs\":[[0,\" softmax\"],[1,\"  [passive/call/aggressive]\\\n           → Sizing head: Linear(5)  → softmax          [third/half/2third/pot/allin]\"],[0,\"\\\n       \"]],\"start1\":1023,\"start2\":1023,\"length1\":16,\"length2\":129},{\"diffs\":[[0,\" Value head:\"],[1,\" \"],[0,\" Linear(1)\\\n`\"]],\"start1\":1157,\"start2\":1157,\"length1\":24,\"length2\":25},{\"diffs\":[[0,\"inear(1)\"],[1,\"                     [not used in RL currently]\"],[0,\"\\\n```\\\n\\\n~1\"]],\"start1\":1172,\"start2\":1172,\"length1\":16,\"length2\":63},{\"diffs\":[[0,\"orm \"],[-1,\"added (was documented but missing — fixed 2026-06-29\"],[1,\"bug in Candle 0.8.4 fixed (manual layer_norm replacing broken `ops::layer_norm`\"],[0,\").\\\n\\\n\"]],\"start1\":1278,\"start2\":1278,\"length1\":60,\"length2\":87},{\"diffs\":[[0,\"ector (2\"],[-1,\"8\"],[1,\"9\"],[0,\"0 dims)\\\n\"]],\"start1\":1378,\"start2\":1378,\"length1\":17,\"length2\":17},{\"diffs\":[[0,\"s (6) |\\\n\"],[1,\"| **Range-weighted equity** | **5** | **hs_rw, ppot_rw, npot_rw, nutpot_rw, win_prob_rw** — exact enumeration vs tightest opponent, weighted by their HandRange |\\\n| **Multiway MC equity** | **5** | **hs_mc, ppot_mc, npot_mc, nutpot_mc, win_prob_mc** — 1000-sample MC vs all opponents |\\\n\"],[0,\"| Board \"]],\"start1\":1523,\"start2\":1523,\"length1\":16,\"length2\":301},{\"diffs\":[[0,\"tats\"],[-1,\" (bridged via `from_player_stats()`)\"],[0,\" |\\\n|\"]],\"start1\":2138,\"start2\":2138,\"length1\":44,\"length2\":8},{\"diffs\":[[0,\"** | **2\"],[-1,\"8\"],[1,\"9\"],[0,\"0** | |\\\n\"]],\"start1\":2258,\"start2\":2258,\"length1\":17,\"length2\":17},{\"diffs\":[[0,\" |\\\n\\\n\"],[-1,\"### Action Space (11 discrete)\\\n\\\n`Fold, Check, Call, BetThird(0.33), BetHalf(0.50), BetTwoThird(0.67), BetPot(1.0), Bet2x(2.0), Raise25x(2.5×), RaisePot(pot), AllIn`\\\n\\\nIllegal actions masked with -∞ logits before softmax. DAgger mode: teacher always computed first; model plays if `prob ≥ confidence_threshold` (0.3 default).\"],[1,\"**Range-weighted equity** (2026-07-03): `compute_vs_range()` from `HandPotential` — exact enumeration of all C(47,2) opponent pairs weighted by the tightest active opponent's `HandRange`. The `HandRange` is built from observer stats via `build_opponent_range()` + `effective_range_params()`. Observer lock is scoped to just the param extraction (~µs), dropped before expensive computation.\\\n\\\n**Multiway MC equity** (2026-07-03): `compute_vs_range_multiway_shared()` with 1000 samples using the same tightest-opponent range for all opponents. Deterministic seed (42) for reproducible features.\\\n\\\n### Action Space (11 discrete → 3+5 multi-head)\\\n\\\nUnderlying: `Fold, Check, Call, BetThird(0.33), BetHalf(0.50), BetTwoThird(0.67), BetPot(1.0), Bet2x(2.0), Raise25x(2.5×), RaisePot(pot), AllIn`\\\n\\\nMulti-head mapping: action_type (passive/call/aggressive) + sizing_class (third/half/2third/pot+/allin). `action_to_heads()` and `heads_to_action()` in `multi_head.rs`.\\\n\\\n## Training Pipeline\\\n\\\n### Phase 1: Imitation Learning (GPU, CUDA) — COMPLETE\\\n1. Collect transitions from Gen 4.5 teacher (5000-sample MC rollouts) in simulation\\\n2. Train multi-head MLP with class-weighted cross-entropy + sizing-weighted MSE\\\n3. Feature normalization (per-feature standardization saved to `.norm.json`)\\\n\\\n**Latest model**: `postflop_v10` — 97.9% action acc, 76.3% sizing acc. Trained on 4.68M postflop samples (9.25M total transitions from overnight collection). 30 epochs, ~12 min on GPU.\\\n\\\n### Phase 2: RL Fine-Tuning (REINFORCE) — IN PROGRESS\\\n- Frozen backbone + RL-adapted heads only (prevents entropy collapse)\\\n- Mixed table: 1 RL + 1 Gen4.5 + 1 Gen2 + 3 chump + 3 flock (9-max)\\\n- Advantage: mean reward baseline (not value head)\\\n- Entropy: 0.05 for exploration\\\n- Quick eval every 50 iterations (5k hands × 3 seeds vs Gen4.5 field)\\\n- Deep eval every 200 iterations (10k hands × 3 seeds × 3 fields)\\\n\\\n**Current RL results** (pre-290-dim, old 280-dim model):\\\n\\\n| Iter | Gen5RL avg | Gen4.5 avg | Gen2 avg |\\\n|------|-----------|------------|---------|\\\n| 50 | +88.8 | +104.3 | +47.3 |\\\n| 100 | +89.2 | +107.1 | +47.9 |\\\n| 150 | +23.9 | +116.5 | +50.7 |\\\n\\\nNote: iter 150 eval used the iter-110 model (training broken from iter 111+ due to FEATURE_DIM change). Gen5RL declining — needs fresh data with range-weighted features.\\\n\\\n### Phase 3: Live Fine-Tuning (future)\\\n- Collect Torn session data, fine-tune on real opponent distributions\"],[0,\"\\\n\\\n##\"]],\"start1\":2272,\"start2\":2272,\"length1\":331,\"length2\":2409},{\"diffs\":[[0,\"ction (2\"],[-1,\"8\"],[1,\"9\"],[0,\"0 dims) \"]],\"start1\":4761,\"start2\":4761,\"length1\":17,\"length2\":17},{\"diffs\":[[0,\"| ✅ \"],[-1,\"Comple\"],[1,\"+ range-weigh\"],[0,\"te\"],[1,\"d\"],[0,\" + \"],[-1,\"opponent bridge\"],[1,\"multiway MC equity\"],[0,\" |\\\n|\"]],\"start1\":4799,\"start2\":4799,\"length1\":34,\"length2\":45},{\"diffs\":[[0,\"|\\\n| \"],[-1,\"NN policy\"],[1,\"Multi-head\"],[0,\" net\"]],\"start1\":4981,\"start2\":4981,\"length1\":17,\"length2\":18},{\"diffs\":[[0,\"rm) | `gen5/\"],[-1,\"network\"],[1,\"multi_head\"],[0,\".rs` | ✅ Com\"]],\"start1\":5025,\"start2\":5025,\"length1\":31,\"length2\":34},{\"diffs\":[[0,\"| ✅ Complete\"],[1,\" (manual layer_norm fix)\"],[0,\" |\\\n| RL stra\"]],\"start1\":5052,\"start2\":5052,\"length1\":24,\"length2\":48},{\"diffs\":[[0,\"gy (\"],[-1,\"DAgger + \"],[0,\"teac\"]],\"start1\":5102,\"start2\":5102,\"length1\":17,\"length2\":8},{\"diffs\":[[0,\"fallback\"],[1,\" + inference + recording\"],[0,\") | `gen\"]],\"start1\":5114,\"start2\":5114,\"length1\":16,\"length2\":40},{\"diffs\":[[0,\"|\\\n| \"],[-1,\"Training CLI\"],[1,\"Imitation trainer\"],[0,\" (`t\"]],\"start1\":5185,\"start2\":5185,\"length1\":20,\"length2\":25},{\"diffs\":[[0,\"ain_gen5\"],[1,\"_multi\"],[0,\"`) | `bi\"]],\"start1\":5211,\"start2\":5211,\"length1\":16,\"length2\":22},{\"diffs\":[[0,\"n/train_gen5\"],[1,\"_multi\"],[0,\".rs` | ✅ Com\"]],\"start1\":5233,\"start2\":5233,\"length1\":24,\"length2\":30},{\"diffs\":[[0,\"| ✅ \"],[-1,\"Complete (GPU/CUDA) |\\\n| Data collection bot (`gen5_collect`) | registered | ✅ Complete |\\\n| Self-play loop script | `scripts/gen5_selfplay_loop.sh` | ✅ Complete |\\\n| Opponent\"],[1,\"GPU/CUDA, multi-head, class weights |\\\n| RL trainer (`train_gen5_rl`) | `bin/train_gen5_rl.rs` | ✅ REINFORCE, frozen backbone |\\\n| RL full-ring loop | `scripts/rl_fullring_loop.sh` | ✅ Persistent iteration count, deep eval |\\\n| Range-weighted equity in\"],[0,\" fea\"]],\"start1\":5256,\"start2\":5256,\"length1\":180,\"length2\":257},{\"diffs\":[[0,\"res \"],[-1,\"bridge \"],[0,\"| `gen5/\"],[-1,\"features\"],[1,\"rl_strategy\"],[0,\".rs`\"]],\"start1\":5515,\"start2\":5515,\"length1\":31,\"length2\":27},{\"diffs\":[[0,\"| ✅ \"],[-1,\"Complete (`extract_opponent_features\"],[1,\"`compute_range_equity\"],[0,\"()`\"],[-1,\")\"],[0,\" |\\\n\\\n##\"],[-1,\"# Training Features\\\n\\\n- **GPU support**: CUDA on RTX 5060 Ti (build with `CUDA_COMPUTE_CAP=89`, JIT-compiled at runtime)\\\n- **Class-weighted loss**: inverse-frequency weighting (capped at 10×) to prevent majority-class collapse\\\n- **Gradient clipping**: global norm clipping at 1.0\\\n- **Cosine LR schedule**: warmup (10% of epochs) + decay to 10% base LR, applied at epoch boundaries only\\\n- **Per-class accuracy analysis**: action distribution + weight reporting\\\n\\\n## What's NOT Implemented\\\n\\\n| Component | Status | Priority |\\\n|---|---|---|\\\n| Reward assignment (outcome → per-transition) | ❌ Not started | High — needed for PPO |\\\n| Showdown hand recording (revealed cards) | ❌ Not started | Medium — needed for range-prediction NN |\\\n| Model hot-reload for live inference | ❌ Not started | Medium |\\\n| PPO self-play training loop | ❌ Not started | Phase 2 |\\\n\\\n## Model History\\\n\\\n| Version | Training Data | Accuracy | Status |\\\n|---|---|---|---|\\\n| v1 | 818K transitions (buggy opponent features) | 50.2% | Degenerate (AllIn bias) |\\\n| v2 | 1.2M transitions (opponent features bridged) | 45.2% | Degenerate (majority-class collapse) |\\\n| v3 | — | 51.8% | Degenerate (AllIn policy) |\\\n| v4 | 1.08M transitions (LayerNorm added) | 66.3% | **Degenerate** — collapsed to Check/Fold. Loss 0.996 ≈ always-Check baseline 0.955 |\\\n\\\n### v4 Degeneracy — Root Causes (all fixed)\\\n\\\n1. **LayerNorm missing**: Network architecture documented LayerNorm but it wasn't implemented in code. Model had no normalization → unstable training.\\\n2. **`engine_action_to_discrete` broken**: All bets classified as BetHalf, all raises as Raise25x. Training labels were wrong → model learned degenerate policy.\\\n3. **Opponent features not bridged**: `compute_features()` passed `&[]` for opponent stats → 60/280 dims always zero.\\\n4. **Class imbalance**: Check/Fold dominate (60%+ of transitions). Plain cross-entropy loss → model collapses to majority class.\\\n5. **Per-batch optimizer recreation**: `set_lr()` called every batch (~47K times/epoch), destroying AdamW momentum state each time.\\\n\\\n**Fixes applied**: LayerNorm implemented, discretization fixed (classifies by pot fraction / bet multiple), opponent bridge added, class-weighted loss implemented, LR schedule moved to epoch boundaries.\\\n\\\n## Current Data Collection\\\n\\\nFresh self-play collection running (8 tables × 1M hands) with fixed discretization. New data shows **10/11 action classes active** (was 6/11 with buggy discretization). Action distribution:\\\n\\\n| Action | Share |\\\n|---|---|\\\n| Fold | 27.5% |\\\n| Check | 35.6% |\\\n| Call | 9.9% |\\\n| BetThird | 4.0% |\\\n| BetHalf | 4.5% |\\\n| BetTwoThird | 14.3% |\\\n| BetPot | 3.0% |\\\n| Raise25x | 0.3% |\\\n| RaisePot | 0.5% |\\\n| AllIn | 0.6% |\\\n\\\nRate: ~330 transitions/sec. Target: ~8M transitions.\\\n\\\n## Training Pipeline (planned)\\\n\\\n### Phase 1: DAgger Imitation Learning (current)\\\n1. Collect transitions from Gen 4 teacher in simulation\\\n2. Train MLP with class-weighted cross-entropy loss\\\n3. Deploy model as new teacher, collect again (DAgger iteration)\\\n4. Repeat until model matches or exceeds Gen 4\\\n\\\n### Phase 2: Self-Play RL via PPO (future)\\\n- Curriculum: Gen 1 → 2 → 3 → 4 (progressively harder opponents)\\\n- Pool: mix of all generations + earlier checkpoints\\\n- Reward: BB/hand with all-in equity adjustment\\\n- Requires reward assignment implementation\\\n\\\n### Phase 3: Live Fine-Tuning\\\n- Collect Torn session data\\\n- Fine-tune on real opponent distributions\\\n\\\n## Open Issues\\\n\\\n1. **v5 training pending**: All fixes are in code but need rebuild with CUDA + retrain on fresh data. Must verify per-class accuracy improves for minority classes (not just overall accuracy).\\\n\\\n2. **`gen5_pipeline.sh` bash arithmetic bug**: Line 164 has `+56.30: syntax error`. DAgger iterations never start. Needs fix for automation.\\\n\\\n3. **Small MLP capacity**: 280→512→512→256→11 may be too small for opponent-aware features. If v5 still underperforms, consider larger hidden layers or attention mechanism.\\\n\\\n4. **No validation gate**: Model deploys regardless of quality. Need automated check: if model loses to Gen 4 in simulation, reject and keep previous.\\\n\\\n5. **Action discretization is coarse** (11 actions): Sizing buckets may miss optimal bet sizes. Phase 2 consideration.\\\n\\\n## Key Files\\\n\\\n| File | Content |\\\n|---|---|\\\n| `holdem_bots/src/gen5/features.rs` | 280-dim feature extraction + OpponentFeatures bridge |\\\n| `holdem_bots/src/gen5/actions.rs` | 11-action space + legality masking |\\\n| `holdem_bots/src/gen5/recorder.rs` | JSONL transition recorder |\\\n| `holdem_bots/src/gen5/network.rs` | Candle MLP policy (LayerNorm + Trainer with class weights, grad clipping, cosine LR) |\\\n| `holdem_bots/src/gen5/rl_strategy.rs` | DAgger strategy (teacher fallback + model inference + recording) |\\\n| `holdem_bots/src/bin/train_gen5.rs` | CLI trainer (GPU/CUDA, class-weighted loss, per-class analysis) |\\\n| `configs/bots/gen5_collect.toml` | Data collection config (Gen 4 teacher + recording) |\\\n| `configs/bots/gen5_rl.toml` | Inference config (model-driven + teacher fallback) |\\\n| `scripts/gen5_selfplay_loop.sh` | DAgger self-play automation |\\\n| `scripts/gen5_train_collect.sh` | Parallel data collection\"],[1,\" Key Technical Details\\\n\\\n### Candle LayerNorm Bug (CRITICAL, FIXED)\\\nCandle-nn 0.8.4's `ops::layer_norm()` uses `apply_op3_no_bwd` with NO backward pass. Replaced with manual layer_norm using individual ops in `multi_head.rs`, `network.rs`, `range_net.rs`. 14/16 tensors now update during RL training. 2 remaining (`ln1_bias`, `ln2_bias`) use `broadcast_add` which doesn't compute gradients for broadcasted 1-D operand — impact negligible.\\\n\\\n### Equity Feature Sources\\\n- **Raw equity** (features 0-4): `HandPotential::compute()` — exact enumeration via Supersonic trie, uniform opponent distribution\\\n- **Range-weighted equity** (features 20-24): `compute_vs_range()` — exact enumeration weighted by tightest opponent's HandRange. Uses `build_opponent_range()` from Gen 4 range builder.\\\n- **Multiway MC equity** (features 25-29): `compute_vs_range_multiway_shared()` — 1000 samples, same range for all opponents\\\n\\\nGen 4.5 teacher uses 5000-sample MC rollouts with range-weighted sampling (`rollout_postflop_gen3.rs:run_multiway_mc()`). Gen 5 inference uses exact enumeration (~0.5ms). Teacher fallback triggers if model not loaded.\\\n\\\n## Model History\\\n\\\n| Version | Training Data | Action Acc | Sizing Acc | Status |\\\n|---|---|---|---|---|\\\n| v1-v4 | 818K-1.2M transitions | 45-66% | — | Degenerate (various bugs) |\\\n| v6 | 2.3M transitions | 85% | — | First working model |\\\n| v7-v8 | 2.3M transitions (full) | 90% | — | Full-street data |\\\n| v9 | 4.68M transitions | 97.9% | 82.0% | Positions 0-8, 9-max |\\\n| **v10** | **4.68M postflop** | **97.9%** | **76.3%** | **Initial RL base** |\\\n\\\n## RL Loop Status (2026-07-03 10:50)\\\n\\\n- **STOPPED** — Training broken since iteration 111 (FEATURE_DIM 280→290 mismatch)\\\n- Last successful model: iteration 110 (frozen since 09:29)\\\n- `rl_current.safetensors` = iter-110 model (280-dim, incompatible with new code)\\\n- Need: collect fresh 290-dim data → train postflop_v11 → restart RL\\\n\\\n## Open Issues\\\n\\\n1. **Fresh data collection needed**: 290-dim features require new transitions. Old data/models incompatible.\\\n2. **Gen5RL underperforms Gen4.5**: +89 vs +107 chips/hand at iter 100. Range-weighted features should help the model learn opponent-aware play.\\\n3. **Value head unused in RL**: Advantage uses mean reward baseline. Could add value-based advantage (A2C style).\\\n4. **`diag_rl_grad.rs` has stale API**: Missing 5th argument to `train_rl_step`. Diagnostic binary only.\\\n\\\n## Key Files\\\n\\\n| File | Content |\\\n|---|---|\\\n| `holdem_bots/src/gen5/features.rs` | **290-dim** feature extraction + range-weighted + multiway equity |\\\n| `holdem_bots/src/gen5/multi_head.rs` | Multi-head model + trainer (LayerNorm fix, REINFORCE, frozen backbone) |\\\n| `holdem_bots/src/gen5/rl_strategy.rs` | RL strategy + `compute_range_equity()` |\\\n| `holdem_bots/src/gen5/network.rs` | Candle MLP policy + `best_device()` GPU selection |\\\n| `holdem_bots/src/bin/train_gen5_multi.rs` | Imitation trainer (GPU/CUDA) |\\\n| `holdem_bots/src/bin/train_gen5_rl.rs` | RL trainer (REINFORCE) |\\\n| `scripts/rl_fullring_loop.sh` | RL loop (persistent count, deep eval, env-configurable) |\\\n| `scripts/gen5_collect_full.sh` | Full-street data collection |\\\n| `models/postflop_v10.safetensors` | Current imitation model (280-dim, needs v11 for 290-dim) |\\\n| `models/rl_current.safetensors` | RL model (iter 110, 280-dim, incompatible with new code)\"],[0,\" |\"]],\"start1\":5543,\"start2\":5543,\"length1\":5168,\"length2\":3362}]"
metadata_diff: {"new":{},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-07-03T08:57:57.859Z
created_time: 2026-07-03T08:57:57.859Z
type_: 13