id: 1eba0cf793af48d7bf4851784ed211a2
parent_id: e4dfb4fc77034575baecc406681365cc
item_type: 1
item_id: bac569872f9149acbfa4832eb553312c
item_updated_time: 1782797633789
title_diff: "[]"
body_diff: "[{\"diffs\":[[0,\"-06-\"],[-1,\"27\"],[1,\"30\"],[0,\"** — \"],[-1,\"Added answers to design questions, training curriculum, and implementation status\"],[1,\"Status sections revised. Design decisions retained as reference\"],[0,\".\\\n\\\n#\"]],\"start1\":63,\"start2\":63,\"length1\":96,\"length2\":78},{\"diffs\":[[0,\"\\\n###\"],[-1,\" Code\"],[0,\" Imp\"]],\"start1\":158,\"start2\":158,\"length1\":13,\"length2\":8},{\"diffs\":[[0,\"nted\"],[-1,\" (compiles, tests pass)\"],[0,\"\\\n- `\"]],\"start1\":170,\"start2\":170,\"length1\":31,\"length2\":8},{\"diffs\":[[0,\" vector \"],[-1,\"extraction \"],[0,\"(equity,\"]],\"start1\":213,\"start2\":213,\"length1\":27,\"length2\":16},{\"diffs\":[[0,\"cy (280→512→\"],[-1,\"512\"],[1,\"LN→512→LN\"],[0,\"→256→11+1), \"]],\"start1\":450,\"start2\":450,\"length1\":27,\"length2\":33},{\"diffs\":[[0,\"` — \"],[-1,\"`Strategy` trait impl\"],[1,\"DAgger strategy\"],[0,\" (te\"]],\"start1\":553,\"start2\":553,\"length1\":29,\"length2\":23},{\"diffs\":[[0,\"ck +\"],[-1,\" optional\"],[0,\" mod\"]],\"start1\":588,\"start2\":588,\"length1\":17,\"length2\":8},{\"diffs\":[[0,\"erence +\"],[-1,\" transition\"],[0,\" recordi\"]],\"start1\":602,\"start2\":602,\"length1\":27,\"length2\":16},{\"diffs\":[[0,\")\\\n- \"],[-1,\"Registered as `gen5_collect` bot type (collects transitions via Gen3 teacher)\\\n\\\n### What's NOT Done Yet\\\n- Training loop (generate → train → evaluate cycle)\\\n- Opponent features bridge (from Gen4 PlayerModel → OpponentFeatures struct)\\\n- Reward assignment (end-of-hand outcome → per-transition reward)\\\n- Model hot-reload for live inference\\\n- Hand database for live fine-tuning\\\n\\\n---\\\n\\\n## Design Decisions (Answers to Key Questions)\\\n\\\n### Q1: NN to Predict Ranges (Weight Tables) vs Current Heuristic?\\\n\\\n**Two viable approaches:**\\\n\\\n| Approach | A: NN predicts full range | B: NN takes range as input (current design) |\\\n|---|---|---|\\\n| **What it learns** | Maps (opponent stats, action sequence, board) → weighted 1326-hand range | Maps (equity vs predicted range, opponent stats) → best action |\\\n| **Training data** | Needs **showdown hands** (where opponent cards are revealed) — rare, ~1 per 5 hands | Needs **decision + outcome** pairs — every hand generates data |\\\n| **Complexity** | High — output space is 1326-dimensional | Low — output space is 11-dimensional |\\\n| **When it helps** | If range prediction is the bottleneck (Gen 4's heuristic ranges are wrong) | If strategy/sizing is the bottleneck (we don't exploit ranges well enough) |\\\n| **Verdict** | **Phase 2** — ambitious, powerful, but needs lots of showdown data | **Phase 1** — pragmatic, fast to implement, builds on Gen 4 |\\\n\\\n**Recommendation**: Start with Approach B (current design). If analysis shows\"],[1,\"`bin/train_gen5.rs` — CLI trainer with GPU/CUDA, class-weighted loss, gradient clipping, cosine LR\\\n- Registered as `gen5_collect` and `gen5_rl` bot types\\\n\\\n### NOT Implemented\\\n- Reward assignment (end-of-hand outcome → per-transition reward) — needed for PPO\\\n- Showdown hand recording (revealed cards for range training)\\\n- Model hot-reload for live inference\\\n- PPO training loop (self-play RL)\\\n\\\n---\\\n\\\n## Design Decisions\\\n\\\n### Q1: NN to Predict Ranges vs Current Design (NN takes range as input)?\\\n\\\n**Decision: Start with Approach B** (current design — NN takes equity + opponent stats as input, outputs action). Pragmatic, builds on Gen 4.\\\n\\\nIf\"],[0,\" ran\"]],\"start1\":620,\"start2\":620,\"length1\":1469,\"length2\":648},{\"diffs\":[[0,\"quality \"],[-1,\"is\"],[1,\"proves to be\"],[0,\" the bot\"]],\"start1\":1282,\"start2\":1282,\"length1\":18,\"length2\":28},{\"diffs\":[[0,\"e 2.\"],[-1,\"\\\n\\\nA **hybrid** is also possible: train a \"],[1,\" A hybrid (\"],[0,\"rang\"]],\"start1\":1404,\"start2\":1404,\"length1\":49,\"length2\":19},{\"diffs\":[[0,\" as \"],[-1,\"a *\"],[0,\"feat\"]],\"start1\":1438,\"start2\":1438,\"length1\":11,\"length2\":8},{\"diffs\":[[0,\"ctor\"],[-1,\"* whose output feeds into the policy NN. This is essentially multi-task learning.\\\n\\\n### Q2: Should We Start Recording All Hands?\\\n\\\n**YES — immediately.** Even without Gen 5, recorded hands are valuable for:\\\n\\\n1. **Imitation learning data** — every (state, action, outcome) tuple is a training sample\\\n2. **Opponent profiling** — track real opponent tendencies across sessions\\\n3. **Strategy validation** — replay hands with new configs to measure improvement\\\n4. **Range accuracy analysis** — compare predicted ranges to showdown cards\\\n5. **Bug detection** — replay problematic hands (hand_replay crate already does this)\\\n\\\n**Implementation plan:**\\\n- The `gen5_collect` bot type already records transitions to JSONL\\\n- Need to wire it into the **live server** as a secondary recorder (Gen4 strategy makes decisions, Gen5 recorder captures features alongside)\\\n- Also record **showdown hands** separately (hand_id, all hole cards, board, winner) for range training\\\n- Store in `data/hands/YYYY-MM-DD/` with daily rotation\\\n\\\n### Q3: Train Preflop AND Postflop?\\\n\\\n**Yes — both, with the same network.** The feature vector already covers both:\\\n\\\n- **Preflop**: board features are zeroed, equity computed vs random/range, action history is just the preflop betting round\\\n- **Postflop**: board features populated, equity computed vs board+ranges, action history includes all streets\\\n\\\n**Alternative**: Separate preflop/postflop networks\\\n- Pro: each converges faster (simpler input space)\\\n- Con: loses transfer learning (postflop play depends on preflop tendencies), more infrastructure\\\n- Verdict: **Unified network first**. Split only if convergence is slow.\\\n\\\n**For preflop specifically**: the existing NG preflop (169-hand tables) is already quite good. The NN's value-add is in:\\\n- Adapting to\"],[1,\" feeding policy NN) is also possible.\\\n\\\n### Q2: Train Preflop AND Postflop?\\\n\\\n**Decision: Unified network** for both. Feature vector already covers both — preflop zeroes board features, postflop populates them. Split only if convergence is slow.\\\n\\\nThe NN's value-add over NG preflop tables:\"],[0,\" opp\"]],\"start1\":1455,\"start2\":1455,\"length1\":1782,\"length2\":295},{\"diffs\":[[0,\"nges\"],[-1,\" (tight/loose players)\\\n-\"],[1,\",\"],[0,\" 3-b\"]],\"start1\":1775,\"start2\":1775,\"length1\":32,\"length2\":9},{\"diffs\":[[0,\"bet \"],[-1,\"pot \"],[0,\"play\"],[-1,\" (where static tables are weakest)\\\n- Stack-depth-dependent adjustments (current tables don't handle 40-60 BB well)\"],[1,\", stack-depth-dependent adjustments.\"],[0,\"\\\n\\\n### Q\"],[-1,\"4\"],[1,\"3\"],[0,\": NN\"]],\"start1\":1789,\"start2\":1789,\"length1\":138,\"length2\":56},{\"diffs\":[[0,\"NN with \"],[-1,\"Current \"],[0,\"Player M\"]],\"start1\":1843,\"start2\":1843,\"length1\":24,\"length2\":16},{\"diffs\":[[0,\"\\\n\\\n**\"],[-1,\"Yes — this is exactly the\"],[1,\"Decision: Yes —\"],[0,\" cur\"]],\"start1\":1873,\"start2\":1873,\"length1\":33,\"length2\":23},{\"diffs\":[[0,\"esign.**\"],[-1,\" The\"],[0,\" `Oppone\"]],\"start1\":1902,\"start2\":1902,\"length1\":20,\"length2\":16},{\"diffs\":[[0,\"eatures`\"],[-1,\" struct\"],[0,\" (20 fie\"]],\"start1\":1921,\"start2\":1921,\"length1\":23,\"length2\":16},{\"diffs\":[[0,\"mirrors \"],[-1,\"the \"],[0,\"Gen 4 `P\"]],\"start1\":1942,\"start2\":1942,\"length1\":20,\"length2\":16},{\"diffs\":[[0,\"ayer\"],[-1,\"Model`:\\\n\\\n```\\\nvpip, pfr, af, fold_to_bet, bet_freq, raise_freq, call_freq,\\\ncbet_freq, threebet, wtsd, hands_observed, stack_bb, wagered_bb,\\\nis_active, distance_from_hero, range_width, range_strength,\\\nis_aggressor, has_acted, confidence\\\n```\\\n\\\n**This is much easier than learning ranges from scratch** because:\\\n- We don't need showdown labels (unsupervised features work)\\\n- The model already condenses the action history into meaningful stats\\\n-\"],[1,\"Stats`.\"],[0,\" Col\"]],\"start1\":1959,\"start2\":1959,\"length1\":448,\"length2\":15},{\"diffs\":[[0,\"l-zeros \"],[-1,\"features \"],[0,\"→ networ\"]],\"start1\":2005,\"start2\":2005,\"length1\":25,\"length2\":16},{\"diffs\":[[0,\"line\"],[-1,\" play\\\n-\"],[1,\".\"],[0,\" As \"]],\"start1\":2039,\"start2\":2039,\"length1\":15,\"length2\":9},{\"diffs\":[[0,\"umulate,\"],[-1,\" the\"],[0,\" network\"]],\"start1\":2057,\"start2\":2057,\"length1\":20,\"length2\":16},{\"diffs\":[[0,\"network \"],[1,\"adapts \"],[0,\"automati\"]],\"start1\":2066,\"start2\":2066,\"length1\":16,\"length2\":23},{\"diffs\":[[0,\"ally\"],[-1,\" adapts\"],[1,\".\"],[0,\"\\\n\\\n### Q\"],[-1,\"5\"],[1,\"4\"],[0,\": RL\"]],\"start1\":2090,\"start2\":2090,\"length1\":23,\"length2\":17},{\"diffs\":[[0,\"lay \"],[-1,\"— Enough? How Fast?\\\n\\\n**RL in live play alone is NOT enough.** Here's why:\\\n\\\n| Factor | Live Play | Simulation |\\\n|---|---|---|\\\n| Hands/hour | ~100 | ~50,000 (32 cores) |\\\n| To reach 10M hands | ~3 years | ~8 days |\\\n| Opponent variety | 5-9 regulars | Unlimited bot pool |\\\n| Variance | High (real money at stake) | Controlled (equity-adjusted) |\\\n| Non-stationarity | Opponents adapt to us | We control the pool |\\\n\\\n**Recommended h\"],[1,\"Alone?\\\n\\\n**Decision: No.** H\"],[0,\"ybri\"]],\"start1\":2117,\"start2\":2117,\"length1\":433,\"length2\":35},{\"diffs\":[[0,\"ach:\"],[-1,\"**\"],[0,\"\\\n1. \"],[-1,\"**\"],[0,\"Phas\"]],\"start1\":2159,\"start2\":2159,\"length1\":16,\"length2\":12},{\"diffs\":[[0,\"offline)\"],[-1,\"**: I\"],[1,\": DAgger i\"],[0,\"mitation\"]],\"start1\":2176,\"start2\":2176,\"length1\":21,\"length2\":26},{\"diffs\":[[0,\"rom Gen \"],[-1,\"3/\"],[0,\"4 teache\"]],\"start1\":2213,\"start2\":2213,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\"cher\"],[-1,\" (~2M hands, ~8 hours)\"],[0,\"\\\n2. \"],[-1,\"**\"],[0,\"Phas\"]],\"start1\":2226,\"start2\":2226,\"length1\":36,\"length2\":12},{\"diffs\":[[0,\"offline)\"],[-1,\"**\"],[0,\": Self-p\"]],\"start1\":2243,\"start2\":2243,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\" PPO\"],[-1,\" (~50M hands, ~3 days)\"],[0,\"\\\n3. \"],[-1,\"**\"],[0,\"Phas\"]],\"start1\":2269,\"start2\":2269,\"length1\":36,\"length2\":12},{\"diffs\":[[0,\"3 (live)\"],[-1,\"**\"],[0,\": Fine-t\"]],\"start1\":2283,\"start2\":2283,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\" on \"],[-1,\"real \"],[0,\"Torn\"]],\"start1\":2302,\"start2\":2302,\"length1\":13,\"length2\":8},{\"diffs\":[[0,\"data\"],[-1,\" (~10k hands/week, lightweight updates)\\\n\\\n**Adaptation speed:**\\\n- **Per-opponent**: Instant via features (network sees updated VPIP/PFR/AF → adjusts)\\\n- **Weight updates**: S\"],[1,\"\\\n\\\nLive play is too s\"],[0,\"low (~1\"],[-1,\"k\"],[1,\"00\"],[0,\" hands\"],[-1,\" for meaningful gradient signal, impractical live)\\\n- **Cold start**: Network uses default features → plays baseline Gen 3/4 style until stats accumulate (~20 hands)\"],[1,\"/hour vs ~50k in simulation) for meaningful weight updates.\"],[0,\"\\\n\\\n### Q\"],[-1,\"6\"],[1,\"5\"],[0,\": Tr\"]],\"start1\":2311,\"start2\":2311,\"length1\":366,\"length2\":110},{\"diffs\":[[0,\"→ 4)\"],[1,\"?\"],[0,\"\\\n\\\n**\"],[-1,\"Excellent idea — standard curriculum learning.** Revised plan:\\\n\"],[1,\"Plan:**\"],[0,\"\\\n```\"]],\"start1\":2454,\"start2\":2454,\"length1\":75,\"length2\":20},{\"diffs\":[[0,\"les)\"],[-1,\"\\\n\"],[0,\"  \"],[-1,\"→ Network learns basic ABC poker\\\n  → Target:\"],[1,\"   →\"],[0,\" +30\"]],\"start1\":2513,\"start2\":2513,\"length1\":55,\"length2\":14},{\"diffs\":[[0,\"r 50k hands\\\n\"],[-1,\"  \\\n\"],[0,\"Stage 2: Bea\"]],\"start1\":2538,\"start2\":2538,\"length1\":27,\"length2\":24},{\"diffs\":[[0,\"postflop\"],[-1,\"_rollout\"],[0,\" + flock\"]],\"start1\":2574,\"start2\":2574,\"length1\":24,\"length2\":16},{\"diffs\":[[0,\"t)  \"],[-1,\"\\\n \"],[0,\" →\"],[-1,\" Network learns equity-based decisions\\\n  → Target:\"],[0,\" +10\"]],\"start1\":2593,\"start2\":2593,\"length1\":62,\"length2\":10},{\"diffs\":[[0,\"k hands\\\n\"],[-1,\"  \\\n\"],[0,\"Stage 3:\"]],\"start1\":2618,\"start2\":2618,\"length1\":19,\"length2\":16},{\"diffs\":[[0,\"flop\"],[-1,\"_rollout\"],[0,\" + m\"]],\"start1\":2654,\"start2\":2654,\"length1\":16,\"length2\":8},{\"diffs\":[[0,\"ool)\"],[-1,\"\\\n\"],[0,\"  \"],[1,\" \"],[0,\"→\"],[-1,\" Network learns range-aware play\\\n  → Target:\"],[0,\" +5 \"]],\"start1\":2668,\"start2\":2668,\"length1\":56,\"length2\":12},{\"diffs\":[[0,\"ands\"],[-1,\" (harder)\\\n  \"],[0,\"\\\nSta\"]],\"start1\":2698,\"start2\":2698,\"length1\":20,\"length2\":8},{\"diffs\":[[0,\"nts)\"],[-1,\"\\\n  → Network learns opponent exploitation\\\n  → Target:\"],[1,\"→\"],[0,\" +2 \"]],\"start1\":2748,\"start2\":2748,\"length1\":61,\"length2\":9},{\"diffs\":[[0,\"ands\"],[-1,\" (much harder)\\\n  \"],[0,\"\\\nSta\"]],\"start1\":2775,\"start2\":2775,\"length1\":25,\"length2\":8},{\"diffs\":[[0,\"ence\"],[-1,\"\\\n \"],[0,\" →\"],[-1,\" Network plays itself + all previous gens\\\n  → Target:\"],[0,\" Nas\"]],\"start1\":2806,\"start2\":2806,\"length1\":65,\"length2\":10},{\"diffs\":[[0,\"on\\\n```\\\n\\\n\"],[-1,\"**\"],[0,\"Advancem\"]],\"start1\":2841,\"start2\":2841,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\"rion\"],[-1,\"**: Move to next stage when win rate is\"],[1,\":\"],[0,\" sta\"]],\"start1\":2866,\"start2\":2866,\"length1\":47,\"length2\":9},{\"diffs\":[[0,\"ificant \"],[1,\"win rate \"],[0,\"(p < 0.0\"]],\"start1\":2890,\"start2\":2890,\"length1\":16,\"length2\":25},{\"diffs\":[[0,\"nds.\"],[-1,\" If losing, drop back a stage.\"],[0,\"\\\n\\\n### Q\"],[-1,\"7\"],[1,\"6\"],[0,\": Ta\"]],\"start1\":2930,\"start2\":2930,\"length1\":46,\"length2\":16},{\"diffs\":[[0,\"formance\"],[1,\"?\"],[0,\"\\\n\\\n**Phas\"]],\"start1\":2969,\"start2\":2969,\"length1\":16,\"length2\":17},{\"diffs\":[[0,\"**Phase \"],[-1,\"2\"],[1,\"3\"],[0,\" feature\"]],\"start1\":2980,\"start2\":2980,\"length1\":17,\"length2\":17},{\"diffs\":[[0,\".** \"],[-1,\"Implementation plan:\\\n\\\n```python\\\n# Meta-controller logic:\\\nif win_rate_last_500_hands > 10 BB/100:\\\n    move up in stakes (if table available)\\\nelif win_rate_last_500_hands < -15 BB/100:\\\n    move down in stakes\\\nif current_table_vpip < 0.20 for 100+ hands:\\\n    find looser table (table selection)\\\n```\\\n\\\nThis requires:\\\n- P\"],[1,\"Requires p\"],[0,\"erfo\"]],\"start1\":2997,\"start2\":2997,\"length1\":323,\"length2\":18},{\"diffs\":[[0,\"king in \"],[-1,\"the \"],[0,\"live ser\"]],\"start1\":3026,\"start2\":3026,\"length1\":20,\"length2\":16},{\"diffs\":[[0,\"ver \"],[-1,\"(rolling BB/100 window)\\\n- Table lobby scraping (available tables, their stats)\\\n- Blind level/stake detection\\\n- Userscript table-switching capability\\\n\\\n\"],[1,\"+ table lobby scraping. \"],[0,\"Defe\"]],\"start1\":3042,\"start2\":3042,\"length1\":158,\"length2\":32},{\"diffs\":[[0,\".\\\n\\\n### Q\"],[-1,\"8\"],[1,\"7\"],[0,\": Captur\"]],\"start1\":3112,\"start2\":3112,\"length1\":17,\"length2\":17},{\"diffs\":[[0,\" DOM\"],[-1,\" to Files\\\n\\\n**Yes —\"],[1,\"?\\\n\\\n**Planned\"],[0,\" for\"]],\"start1\":3139,\"start2\":3139,\"length1\":26,\"length2\":20},{\"diffs\":[[0,\"bugging \"],[-1,\"AND\"],[1,\"+\"],[0,\" trainin\"]],\"start1\":3162,\"start2\":3162,\"length1\":19,\"length2\":17},{\"diffs\":[[0,\".** \"],[-1,\"Plan:\\\n\\\n1. **\"],[0,\"User\"]],\"start1\":3185,\"start2\":3185,\"length1\":20,\"length2\":8},{\"diffs\":[[0,\"ode\\\"\"],[-1,\"**: At each `waiting_for_action`,\"],[0,\" save\"],[-1,\" the\"],[1,\"s\"],[0,\" ful\"]],\"start1\":3211,\"start2\":3211,\"length1\":50,\"length2\":14},{\"diffs\":[[0,\"TML \"],[-1,\"to a file\\\n2. **Storage**: `data/dom_snapshots/YYYY-MM-DD/hand_XXX.html`\\\n3. **Uses**:\\\n   - Debug parsing edge cases (new Torn UI changes)\\\n   - Extract features the scraper missed\\\n   - Build labeled dataset for scraper improvement\\\n   - Training data for DOM→features auto-extraction (future)\\\n\\\nAlso useful for the **standalone poker screen** — capture and replay full game states.\\\n\\\n---\\\n\\\n## Implementation Roadmap (Priority Order)\\\n\\\n### Immediate (next session)\\\n1. **Wire Gen5 recorder into live server** — record transitions alongside Gen4 decisions\\\n2. **Record showdown hands** — separate file with revealed cards for range training\\\n3. **Bridge PlayerModel → OpponentFeatures** — fill the 20-field structs from Gen4 observer\\\n4. **Add reward assignment** — end-of-hand outcome propagates to all transitions in that hand\\\n\\\n### Short-term (1-2 weeks)\\\n5. **Imitation learning Phase 1** — collect ~500k h\"],[1,\"at each decision. Useful for scraper debugging \"],[0,\"and\"],[-1,\"s\"],[0,\" f\"],[-1,\"rom Gen3/4 teacher, train MLP\\\n6. **Evaluate imitation model** — compare to Gen3/4 in simulation\\\n7. **DOM snapshot mode** — add to userscript for debugging\\\n\\\n### Medium-term (2-4 weeks)\\\n8. **Self-play RL (PPO)** — curriculum learning Gen 1→2→3→4\\\n9. **Model serving in live server** — load trained weights, forward pass for decisions\\\n10. **Table hopper prototype** — basic win-rate-based stake changes\\\n\\\n### Long-term (1-3 months)\\\n11. **Live fine-tuning** — adapt to Torn player pool\\\n12. **Range prediction NN** (if needed) — separate model for opponent ranges\\\n13. **Multi-site deployment** — generalize beyond Torn\"],[1,\"uture DOM→features auto-extraction.\"],[0,\"\\\n\\\n--\"]],\"start1\":3238,\"start2\":3238,\"length1\":1520,\"length2\":95},{\"diffs\":[[0,\"   │ │ (\"],[-1,\"hands)\"],[1,\"TODO) \"],[0,\"      │\\\n\"]],\"start1\":3838,\"start2\":3838,\"length1\":22,\"length2\":22},{\"diffs\":[[0,\"───┘ └──────\"],[-1,\"┬\"],[1,\"─\"],[0,\"───────┘\\\n   \"]],\"start1\":3891,\"start2\":3891,\"length1\":25,\"length2\":25},{\"diffs\":[[0,\"               │\"],[1,\"\\\n\"],[0,\"              │\\\n\"]],\"start1\":3926,\"start2\":3926,\"length1\":32,\"length2\":33},{\"diffs\":[[0,\"    \"],[-1,\"│\\\n                            ▼\"],[0,\"    \"]],\"start1\":3953,\"start2\":3953,\"length1\":39,\"length2\":8},{\"diffs\":[[0,\"   │  1.\"],[1,\" DAgger\"],[0,\" Imitati\"]],\"start1\":4130,\"start2\":4130,\"length1\":16,\"length2\":23},{\"diffs\":[[0,\"itation \"],[-1,\"Learning\"],[1,\" \"],[0,\"  │\\\n    \"]],\"start1\":4148,\"start2\":4148,\"length1\":24,\"length2\":17},{\"diffs\":[[0,\"    (Gen\"],[-1,\"3/\"],[0,\"4 teache\"]],\"start1\":4182,\"start2\":4182,\"length1\":18,\"length2\":16},{\"diffs\":[[0,\"teacher)\"],[1,\"  \"],[0,\"    │\\\n  \"]],\"start1\":4192,\"start2\":4192,\"length1\":16,\"length2\":18},{\"diffs\":[[0,\"512→\"],[1,\"LN→\"],[0,\"512→\"],[-1,\"256  \"],[1,\"LN\"],[0,\"  │\\\n\"]],\"start1\":4729,\"start2\":4729,\"length1\":17,\"length2\":17},{\"diffs\":[[0,\"     │  \"],[1,\"→256\"],[0,\"→11 acti\"]],\"start1\":4761,\"start2\":4761,\"length1\":16,\"length2\":20},{\"diffs\":[[0,\"ions\"],[-1,\" + 1 V \"],[1,\"+1V\"],[0,\" │\\\n \"]],\"start1\":4780,\"start2\":4780,\"length1\":15,\"length2\":11},{\"diffs\":[[0,\"\\\n## \"],[-1,\"Training Data Requirements\\\n\\\n| Data Type | Source | Volume Needed | Status |\\\n|---|---|---|---|\\\n| Imitation (state, action) | Gen3/4 teacher in sim | ~2M transitions | Recorder ready, needs wiring |\\\n| Self-play (state, action, reward) | NN vs bot pool | ~50M transitions | Needs training loop |\\\n| Showdown hands (cards revealed) | Torn live | ~10k hands | Needs showdown recorder |\\\n| Real opponent stats | Torn live (Gen4 observer) | Ongoing | Already saved to profiles/ |\\\n\\\n---\\\n\\\n## Feature Vector Layout (280 dims, implemented)\\\n\\\n| Section | Dims | Description |\\\n|---|---|---|\\\n| Card/Equity | 20 | hs, ppot, npot, nutpot, rpot + hand category (9 one-hot) + draws (6) |\\\n| Board texture | 10 | board_count, wetness, suit_count, connectivity, high_card + padding |\\\n| Betting/Situation | 25 | pot_bb, to_call_bb, spr, stack_bb + position (9) + street (4) + counts (3) + padding |\\\n| Hero state | 5 | wagered_bb, is_aggressor, is_pfr_aggressor, aggressor_present, facing_bet |\\\n| Opponent models (×3) | 60 | 20 fields each from Gen4 PlayerModel |\\\n| Action history (×8) | 160 | 20 fields each (player, street, action_type, amounts, padding) |\\\n| **Total** | **280** | |\\\n\\\n---\\\n\\\n## Network Architecture (implemented, Candle)\\\n\\\n```\\\nInput(280) → Linear(512) → ReLU\\\n           → Linear(512) → ReLU  \\\n           → Linear(256) → ReLU\\\n           → Actor head: Linear(11) → masked softmax\\\n           → Value head: Linear(1)\\\n```\\\n\\\n- **Actor-critic** (A2C/PPO compatible)\\\n- **Action masking**: illegal actions get -1e9 logits before softmax\\\n- **Inference**: <1ms on CPU (280→512→512→256→11 = ~1M params)\\\n- **Training**: AdamW optimizer, cross-entropy loss for imitation phase\\\n\\\n---\\\n\\\n## Open Questions (Updated)\\\n\\\n1. **Rust-native (candle) vs Python (PyTorch) for training?**\\\n   - Candle works for inference (already implemented)\\\n   - For PPO training loop, Python/PyTorch is more mature (stable-baselines3, etc.)\\\n   - **Recommendation**: Generate transitions in Rust (fast), train in Python (mature), deploy in Rust (candle for inference)\\\n\\\n2. **Equity MC in training loop?**\\\n   - Pre-compute and cache per decision point (current approach works)\\\n   - For self-play, each table computes MC independently (no sharing needed)\\\n\\\n3. **Reward design?**\\\n   - Phase 1 (imitation): no reward needed (supervised)\\\n   - Phase 2 (PPO): BB/hand with all-in equity adjustment (credit expected value, not runout luck)\\\n   - Consider per-street reward shaping: small positive for good folds, small negative for bad calls\\\n\\\n4. **Multi-table training?**\\\n   - Yes — run 100+ tables in parallel via rayon, collect transitions centrally\\\n   - This is how we reach 50k hands/hour on 32 cores\"],[1,\"Implementation Roadmap\\\n\\\n### Phase 1: DAgger Imitation Learning (in progress)\\\n1. ✅ Feature extraction, action space, recorder, network, strategy\\\n2. ✅ Opponent features bridge\\\n3. ✅ Training CLI with class-weighted loss + grad clipping + cosine LR\\\n4. ✅ GPU/CUDA support\\\n5. 🔄 Collect ~8M transitions from Gen 4 teacher (running)\\\n6. ⬜ Retrain v5 with all fixes, verify non-degeneracy\\\n7. ⬜ Evaluate v5 vs Gen 4 in simulation\\\n\\\n### Phase 2: Self-Play RL (future)\\\n8. ⬜ Implement reward assignment\\\n9. ⬜ PPO training loop with curriculum\\\n10. ⬜ Showdown hand recording (for range NN)\\\n\\\n### Phase 3: Live Deployment (future)\\\n11. ⬜ Model hot-reload in live server\\\n12. ⬜ Live fine-tuning on Torn data\\\n13. ⬜ Table hopper prototype\"]],\"start1\":5096,\"start2\":5096,\"length1\":2640,\"length2\":718}]"
metadata_diff: {"new":{},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-06-30T05:37:46.211Z
created_time: 2026-06-30T05:37:46.211Z
type_: 13