id: 64972b502b8c482a9e953bdbd8a00f3d
parent_id: 39b1f30252c641b0852fd996e7b4ddd8
item_type: 1
item_id: c7a39e16770542a08004fd73e971ca1f
item_updated_time: 1784277726844
title_diff: "[]"
body_diff: "[{\"diffs\":[[0,\"arning\\\n\\\n\"],[1,\"> **Current state, 2026-07-17.** Imitation learning with G48 teacher. Data collection in progress.\\\n\\\n## Feature Vector (290 dimensions)\\\n\\\n| Category | Dims | Fields |\\\n|---\"],[0,\"|---|---\"]],\"start1\":26,\"start2\":26,\"length1\":16,\"length2\":185},{\"diffs\":[[0,\"(6) |\\\n| \"],[-1,\"**\"],[0,\"Range-we\"]],\"start1\":299,\"start2\":299,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\"d equity\"],[-1,\"**\"],[0,\" | \"],[-1,\"**5**\"],[1,\"5\"],[0,\" | \"],[-1,\"**\"],[0,\"hs_rw, p\"]],\"start1\":320,\"start2\":320,\"length1\":31,\"length2\":23},{\"diffs\":[[0,\"_prob_rw\"],[-1,\"**\"],[0,\" — exact\"]],\"start1\":374,\"start2\":374,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\" by \"],[-1,\"their \"],[0,\"Hand\"]],\"start1\":433,\"start2\":433,\"length1\":14,\"length2\":8},{\"diffs\":[[0,\"nge |\\\n| \"],[-1,\"**\"],[0,\"Multiway\"]],\"start1\":443,\"start2\":443,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\"uity\"],[-1,\"**\"],[0,\" | \"],[-1,\"**5**\"],[1,\"5\"],[0,\" | \"],[-1,\"**\"],[0,\"hs_m\"]],\"start1\":465,\"start2\":465,\"length1\":23,\"length2\":15},{\"diffs\":[[0,\"_prob_mc\"],[-1,\"**\"],[0,\" — 1000-\"]],\"start1\":515,\"start2\":515,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\" |\\\n\\\n\"],[-1,\"**Range-weighted equity** (2026-07-03): `compute_vs_range()` from `HandPotential` — exact enumeration of all C(47,2) opponent pairs weighted by the tightest active opponent's `HandRange`. The `HandRange` is built from observer stats via `build_opponent_range()` + `effective_range_params()`. Observer lock is scoped to just the param extraction (~µs), dropped before expensive computation.\\\n\\\n**Multiway MC equity** (2026-07-03): `compute_vs_range_multiway_shared()` with 1000 samples using the same tightest-opponent range for all opponents. Deterministic seed (42) for reproducible features.\\\n\\\n#\"],[0,\"## A\"]],\"start1\":1016,\"start2\":1016,\"length1\":602,\"length2\":8},{\"diffs\":[[0,\"in).\"],[-1,\" `action_to_heads()` and `heads_to_action()` in `multi_head.rs`.\"],[0,\"\\\n\\\n##\"]],\"start1\":1314,\"start2\":1314,\"length1\":72,\"length2\":8},{\"diffs\":[[0,\"UDA)\"],[-1,\" — COMPLETE\"],[0,\"\\\n1. \"]],\"start1\":1381,\"start2\":1381,\"length1\":19,\"length2\":8},{\"diffs\":[[0,\"rom \"],[-1,\"Gen 4.5 teacher (5000-sample MC rollouts)\"],[1,\"teacher bot\"],[0,\" in \"]],\"start1\":1410,\"start2\":1410,\"length1\":49,\"length2\":19},{\"diffs\":[[0,\"p_v1\"],[-1,\"0\"],[1,\"3\"],[0,\"` — \"],[-1,\"97.9% action acc, 76.3% sizing acc. Trained on 4.68M postflop samples (9.25M total transitions from overnight collection). 30 epochs, ~12 min on GPU.\"],[1,\"trained on G48 teacher data.\\\n\\\n### Data Collection Status (2026-07-17)\\\n- **Short-handed (3-6 handed)**: ✅ COMPLETE — 24.1M transitions from 10M hands (39GB)\\\n- **Full-ring (7-8 handed)**: 🔄 RUNNING — ~3.5M of ~20M transitions (7 opponent types: chump, flock, gen1, gen2, gen3, loose, tight)\\\n- **Teacher**: G48 (`teacher_postflop = \\\"formula\\\"`, 10K MC samples)\\\n- **Config**: `configs/bots/gen5_collect_g48.toml`\"],[0,\"\\\n\\\n##\"]],\"start1\":1624,\"start2\":1624,\"length1\":162,\"length2\":421},{\"diffs\":[[0,\") — \"],[-1,\"IN PROGRESS\"],[1,\"PLANNED\"],[0,\"\\\n- F\"]],\"start1\":2081,\"start2\":2081,\"length1\":19,\"length2\":15},{\"diffs\":[[0,\"only\"],[-1,\" (prevents entropy collapse)\\\n- Mixed table: 1 RL + 1 Gen4.5 + 1 Gen2 + 3 chump + 3 flock (9-max)\\\n- Advantage: mean reward baseline (not value head)\\\n- Entropy: 0.05 for exploration\\\n- Quick eval every 50 iterations (5k hands × 3 seeds vs Gen4.5 field)\\\n- Deep eval every 200 iterations (10k hands × 3 seeds × 3 fields)\\\n\\\n**Current RL results** (pre-290-dim, old 280-dim model):\\\n\\\n| Iter | Gen5RL avg | Gen4.5 avg | Gen2 avg |\\\n|------|-----------|------------|---------|\\\n| 50 | +88.8 | +104.3 | +47.3 |\\\n| 100 | +89.2 | +107.1 | +47.9 |\\\n| 150 | +23.9 | +116.5 | +50.7 |\\\n\\\nNote: iter 150 eval used the iter-110 model (training broken from iter 111+ due to FEATURE_DIM change). Gen5RL declining — needs fresh data with range-weighted features.\\\n\\\n### Phase 3: Live Fine-Tuning (future)\\\n- Collect Torn session data, fine-tune on real opponent distributions\\\n\\\n## What's Implemented\\\n\\\n| Component | File | Status |\\\n|---|---|---|\\\n| Feature extraction (290 dims) | `gen5/features.rs` | ✅ + range-weighted + multiway MC equity |\\\n| Action space (11 discrete + masking) | `gen5/actions.rs` | ✅ Complete |\\\n| Transition recorder (JSONL) | `gen5/recorder.rs` | ✅ Complete |\\\n| Multi-head network (Candle MLP + LayerNorm) | `gen5/multi_head.rs` | ✅ Complete (manual layer_norm fix) |\\\n| RL strategy (teacher fallback + inference + recording) | `gen5/rl_strategy.rs` | ✅ Complete |\\\n| Imitation trainer (`train_gen5_multi`) | `bin/train_gen5_multi.rs` | ✅ GPU/CUDA, multi-head, class weights |\\\n| RL trainer (`train_gen5_rl`) | `bin/train_gen5_rl.rs` | ✅ REINFORCE, frozen backbone |\\\n| RL full-ring loop | `scripts/rl_fullring_loop.sh` | ✅ Persistent iteration count, deep eval |\\\n| Range-weighted equity in features | `gen5/rl_strategy.rs` | ✅ `compute_range_equity()` |\"],[1,\"\\\n- Mixed table: 1 RL + mixed opponents\\\n- Quick eval every 50 iterations, deep eval every 200\\\n\\\n### Phase 3: Live Fine-Tuning (future)\\\n- Collect Torn session data, fine-tune on real opponent distributions\\\n\\\n## Next Steps\\\n1. Wait for full-ring collection → merge all transitions\\\n2. Train `postflop_v14` on combined ~44M transitions\\\n3. Evaluate v14 vs G48 in simulation\\\n4. PPO fine-tuning if imitation plateaus\\\n5. Deploy via `gen5_rl.toml`\"],[0,\"\\\n\\\n##\"]],\"start1\":2130,\"start2\":2130,\"length1\":1747,\"length2\":442},{\"diffs\":[[0,\"ug (\"],[-1,\"CRITICAL, \"],[0,\"FIXE\"]],\"start1\":2618,\"start2\":2618,\"length1\":18,\"length2\":8},{\"diffs\":[[0,\"()` \"],[-1,\"uses `apply_op3_no_bwd` with NO\"],[1,\"has no\"],[0,\" bac\"]],\"start1\":2663,\"start2\":2663,\"length1\":39,\"length2\":14},{\"diffs\":[[0,\"orm \"],[-1,\"using individual ops \"],[0,\"in `\"]],\"start1\":2717,\"start2\":2717,\"length1\":29,\"length2\":8},{\"diffs\":[[0,\"rs`.\"],[-1,\" 14/16 tensors now update during RL training. 2 remaining (`ln1_bias`, `ln2_bias`) use `broadcast_add` which doesn't compute gradients for broadcasted 1-D operand — impact negligible.\"],[0,\"\\\n\\\n##\"]],\"start1\":2766,\"start2\":2766,\"length1\":191,\"length2\":8},{\"diffs\":[[0,\"tion\"],[-1,\" via Supersonic trie\"],[0,\", un\"]],\"start1\":2874,\"start2\":2874,\"length1\":28,\"length2\":8},{\"diffs\":[[0,\"opponent\"],[-1,\" distribution\"],[0,\"\\\n- **Ran\"]],\"start1\":2888,\"start2\":2888,\"length1\":29,\"length2\":16},{\"diffs\":[[0,\"ange\"],[-1,\". Uses `build_opponent_range()` from Gen 4 range builder.\"],[0,\"\\\n- *\"]],\"start1\":3021,\"start2\":3021,\"length1\":65,\"length2\":8},{\"diffs\":[[0,\"ples\"],[-1,\", same range for all opponents\\\n\\\nGen 4.5 teacher uses 5000-sample MC rollouts with range-weighted sampling (`rollout_postflop_gen3.rs:run_multiway_mc()`). Gen 5 inference uses exact enumeration (~0.5ms). Teacher fallback triggers if model not loaded.\\\n\\\n## Model History\\\n\\\n| Version | Training Data | Action Acc | Sizing Acc | Status |\\\n|---|---|---|---|---|\\\n| v1-v4 | 818K-1.2M transitions | 45-66% | — | Degenerate (various bugs) |\\\n| v6 | 2.3M transitions | 85% | — | First working model |\\\n| v7-v8 | 2.3M transitions (full) | 90% | — | Full-street data |\\\n| v9 | 4.68M transitions | 97.9% | 82.0% | Positions 0-8, 9-max |\\\n| **v10** | **4.68M postflop** | **97.9%** | **76.3%** | **Initial RL base** |\\\n\\\n## RL Loop Status (2026-07-03 10:50)\\\n\\\n- **STOPPED** — Training broken since iteration 111 (FEATURE_DIM 280→290 mismatch)\\\n- Last successful model: iteration 110 (frozen since 09:29)\\\n- `rl_current.safetensors` = iter-110 model (280-dim, incompatible with new code)\\\n- Need: collect fresh 290-dim data → train postflop_v11 → restart RL\\\n\\\n## Open Issues\\\n\\\n1. **Fresh data collection needed**: 290-dim features require new transitions. Old data/models incompatible.\\\n2. **Gen5RL underperforms Gen4.5**: +89 vs +107 chips/hand at iter 100. Range-weighted features should help the model learn opponent-aware play.\\\n3. **Value head unused in RL**: Advantage uses mean reward baseline. Could add value-based advantage (A2C style).\\\n4. **`diag_rl_grad.rs` has stale API**: Missing 5th argument to `train_rl_step`. Diagnostic binary only.\"],[0,\"\\\n\\\n##\"]],\"start1\":3116,\"start2\":3116,\"length1\":1527,\"length2\":8},{\"diffs\":[[0,\"` | \"],[-1,\"**\"],[0,\"290-dim\"],[-1,\"**\"],[0,\" fea\"]],\"start1\":3200,\"start2\":3200,\"length1\":19,\"length2\":15},{\"diffs\":[[0,\"ORCE\"],[-1,\", frozen backbone\"],[0,\") |\\\n\"]],\"start1\":3357,\"start2\":3357,\"length1\":25,\"length2\":8},{\"diffs\":[[0,\"quity()`\"],[1,\" + `teacher_postflop` config\"],[0,\" |\\\n| `ho\"]],\"start1\":3437,\"start2\":3437,\"length1\":16,\"length2\":44},{\"diffs\":[[0,\"ction |\\\n\"],[1,\"| `holdem_bots/src/gen5/recorder.rs` | JSONL transition recorder |\\\n| `holdem_bots/src/gen5/actions.rs` | Action space + masking + `action_to_heads()` |\\\n\"],[0,\"| `holde\"]],\"start1\":3558,\"start2\":3558,\"length1\":16,\"length2\":168},{\"diffs\":[[0,\"pts/\"],[-1,\"rl_fullring_loop.sh` | RL loop (persistent count, deep eval, env-configurable\"],[1,\"gen5_collect_g48.sh` | Short-handed collection (3-6 handed\"],[0,\") |\\\n\"]],\"start1\":3870,\"start2\":3870,\"length1\":85,\"length2\":66},{\"diffs\":[[0,\"collect_\"],[-1,\"full\"],[1,\"g48_fullring\"],[0,\".sh` | F\"]],\"start1\":3952,\"start2\":3952,\"length1\":20,\"length2\":28},{\"diffs\":[[0,\"ull-\"],[-1,\"street data\"],[1,\"ring\"],[0,\" col\"]],\"start1\":3980,\"start2\":3980,\"length1\":19,\"length2\":12},{\"diffs\":[[0,\"g collection\"],[1,\" (7-8 handed)\"],[0,\" |\\\n| `models\"]],\"start1\":3987,\"start2\":3987,\"length1\":24,\"length2\":37},{\"diffs\":[[0,\"tflop_v1\"],[-1,\"0\"],[1,\"3\"],[0,\".safeten\"]],\"start1\":4028,\"start2\":4028,\"length1\":17,\"length2\":17},{\"diffs\":[[0,\"odel\"],[-1,\" (280-dim, needs v11 for 290-dim)\"],[0,\" |\\\n|\"]],\"start1\":4072,\"start2\":4072,\"length1\":41,\"length2\":8},{\"diffs\":[[0,\"el (\"],[-1,\"iter 110, 280-dim, incompatible with new code) |\"],[1,\"latest) |\\\n\\\n## Design Decisions\\\n\\\n- **NN takes equity + opponent stats as input** (not range prediction NN) — pragmatic, builds on Gen 4\\\n- **Unified network** for preflop + postflop (preflop zeroes board features)\\\n- **Opponent features as input** — cold start = all-zeros, network adapts as stats accumulate\\\n- **Hybrid training**: offline imitation → offline RL → live fine-tuning (live play too slow for weight updates)\\\n- **Curriculum plan**: Beat Gen 1 → 2 → 3 → 4 → self-play convergence (Nash approximation)\\\n\"]],\"start1\":4121,\"start2\":4121,\"length1\":52,\"length2\":514}]"
metadata_diff: {"new":{},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-07-17T08:46:48.348Z
created_time: 2026-07-17T08:46:48.348Z
type_: 13