id: 9fb9a0008ad443869d1821812b346ce2
parent_id: 
item_type: 1
item_id: fae8ff26da434aaeab172970a45a6079
item_updated_time: 1782797680425
title_diff: "[{\"diffs\":[[1,\"Gen 5 — Session 2026-06-29: Fixes, Retraining, and Self-Play Infrastructure\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":75}]"
body_diff: "[{\"diffs\":[[1,\"# Gen 5 Session Summary — 2026-06-29\\\n\\\n## Fixes Applied Before Retraining\\\n\\\n### 1. VPIP Survivorship Bias (CODE FIX)\\\n- **Bug**: `self_observe` in `adaptive.rs:266` filtered players with `is_in_hand()` — excluded folded players before hero acts\\\n- **Impact**: Tight opponents (FlockBot etc.) only tracked in hands where they DIDN'T fold → VPIP inflated to ~0.90\\\n- **Fix**: Changed filter to only exclude `SittingOut` players\\\n\\\n### 2. VPIP `to_call` Computation (CODE FIX)\\\n- **Bug**: Preflop `current_bet` started at 0 instead of BB → SB completions had `to_call=0` → treated as blind postings → VPIP never recorded\\\n- **Fix**: Initialize `current_bet` to `gi.big_blind` for preflop\\\n\\\n### 3. Threshold Discount Removal (CODE FIX)\\\n- **Bug**: `range_tightness_scale` discount double-counted range narrowing (once in equity, once in threshold) → drove thresholds negative → 88-shove bug\\\n- **Fix**: Removed discount from `should_call`, `should_raise`, and `get_first_in_action`\\\n\\\n### 4. Post-Discount Threshold Sweep\\\n- **bet_turn_base**: 0.58 → **0.40** (+251 combined Δ across flock+chump fields)\\\n- **bet_flop_base**: 0.30 → **0.15** (+79 combined Δ)\\\n- Other params (call_base, raise_base, call_w_draw): minimal change, kept as-is\\\n\\\n### 5. G5-1: Opponent Features Bridge (CODE FIX)\\\n- **Bug**: `compute_features` passed `&[]` for opponent_stats → 60/280 feature dims always zero\\\n- **Fix**: Added `extract_opponent_features()` to `RlStrategy` — bridges observer stats via `OpponentFeatures::from_player_stats()` + situational fields (stack_bb, wagered_bb, distance, is_aggressor, has_acted)\\\n- Added `shared_observer()` accessor to `AdaptivePostflopGen4`\\\n- Stored `Option<Arc<Mutex<Observer>>>` in `RlStrategy` for feature extraction\\\n\\\n## Data Collection (v2)\\\n- **Tables**: 6 × 250k hands = 1.5M total\\\n- **Transitions**: 1,216,271 (up from 818,739 in v1)\\\n- **Profit/hand**: +63 (Gen 4 teacher vs mixed field)\\\n- **Profiles**: 42 opponent profile files\\\n- **File**: `/tmp/gen5_training/all_transitions.jsonl` (1.9 GB)\\\n\\\n## Model Training (v2)\\\n- **Epochs**: 20\\\n- **Samples**: 1,094,643 train / 121,628 eval\\\n- **Loss**: 2.03 → 0.90 (converged epoch 5)\\\n- **Eval accuracy**: 45.2% (11-class, random = 9%)\\\n- **Previous (v1)**: 50.2% accuracy — the drop is because opponent features are now non-zero, adding complexity the small MLP struggles with\\\n- **Model**: `models/gen5_v2.safetensors`\\\n\\\n## Accuracy Analysis\\\nThe 45.2% plateau (down from v1's 50.2%) is expected:\\\n- v1 had 60/280 feature dims zeroed (opponent stats) → easier to learn, but model was blind to opponent types\\\n- v2 has all 280 dims populated → harder to learn, but model can potentially exploit opponent tendencies\\\n- The MLP architecture (280 → hidden → 11) is likely the bottleneck\\\n- Future: larger network (280 → 256 → 128 → 11) or attention mechanism\\\n\\\n## Infrastructure Created\\\n\\\n### Inference Config (`gen5_rl.toml`)\\\n- `use_teacher = false` → model plays, Gen 4 catches illegal actions\\\n- `record_transitions = false` → no disk writes during live play\\\n- `model_path = \\\"models/gen5_v2.safetensors\\\"`\\\n- Registered in `configs/live_bots.toml`\\\n\\\n### Self-Play Loop (`scripts/gen5_selfplay_loop.sh`)\\\n- DAgger-style: collect → aggregate → train → validate\\\n- Supports `KEEP_OLD_DATA=true` for experience aggregation\\\n- Caps dataset at 3M transitions (FIFO trimming)\\\n- Per-iteration model saved as `models/gen5_iter{N}.safetensors`\\\n- Usage: `./scripts/gen5_selfplay_loop.sh 3` (3 iterations)\\\n\\\n## What Gen 5 Needs to Continue Improving\\\n\\\n### Near-Term (Iterative Imitation)\\\n1. **Run self-play loop** (`gen5_selfplay_loop.sh`) — automated collect→train→deploy\\\n2. **Larger network** — current MLP too small for 280-dim input with opponent features\\\n3. **Better teacher** — Gen 4 with fixed VPIP + new thresholds is already a better teacher\\\n\\\n### Medium-Term (Reward-Based Training)\\\n1. **Reward labeling** — add actual pot outcomes to transitions (currently only teacher action)\\\n2. **Policy gradient** — train with actual rewards instead of teacher labels\\\n3. **Experience replay** — sample from buffer, not just latest data\\\n4. **Exploration** — epsilon-greedy or temperature sampling during self-play\\\n\\\n### Open Issues\\\n- **G5-2**: `train_gen5.rs` requires `--features gen5_nn` — won't compile without it\\\n- **G5-3**: Only cross-entropy loss (behavioral cloning) — no reward signal yet\\\n- **G5-4**: No model validation gate (deploy regardless of quality)\\\n- **G5-5**: Small MLP architecture — needs capacity for opponent-aware features\\\n- **G5-6**: No online learning (can't update during live play)\\\n- **G5-7**: Action discretization is coarse (11 actions) — sizing buckets may miss optimal bet sizes\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":4617}]"
metadata_diff: {"new":{"id":"fae8ff26da434aaeab172970a45a6079","parent_id":"5a06903f7db44bfcb4c8c8a9cf0d2326","latitude":"0.00000000","longitude":"0.00000000","altitude":"0.0000","author":"","source_url":"","is_todo":0,"todo_due":0,"todo_completed":0,"source":"joplin-desktop","source_application":"net.cozic.joplin-desktop","application_data":"","order":1782710219472,"user_updated_time":1782710219472,"markup_language":1,"is_shared":0,"share_id":"","conflict_original_id":"","master_key_id":"","user_data":"","deleted_time":1782797680425},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2026-06-30T05:37:46.397Z
created_time: 2026-06-30T05:37:46.397Z
type_: 13