Gen 5 — Session 2026-06-29: Fixes, Retraining, and Self-Play Infrastructure

# Gen 5 Session Summary — 2026-06-29

## Fixes Applied Before Retraining

### 1. VPIP Survivorship Bias (CODE FIX)
- **Bug**: `self_observe` in `adaptive.rs:266` filtered players with `is_in_hand()` — excluded folded players before hero acts
- **Impact**: Tight opponents (FlockBot etc.) only tracked in hands where they DIDN'T fold → VPIP inflated to ~0.90
- **Fix**: Changed filter to only exclude `SittingOut` players

### 2. VPIP `to_call` Computation (CODE FIX)
- **Bug**: Preflop `current_bet` started at 0 instead of BB → SB completions had `to_call=0` → treated as blind postings → VPIP never recorded
- **Fix**: Initialize `current_bet` to `gi.big_blind` for preflop

### 3. Threshold Discount Removal (CODE FIX)
- **Bug**: `range_tightness_scale` discount double-counted range narrowing (once in equity, once in threshold) → drove thresholds negative → 88-shove bug
- **Fix**: Removed discount from `should_call`, `should_raise`, and `get_first_in_action`

### 4. Post-Discount Threshold Sweep
- **bet_turn_base**: 0.58 → **0.40** (+251 combined Δ across flock+chump fields)
- **bet_flop_base**: 0.30 → **0.15** (+79 combined Δ)
- Other params (call_base, raise_base, call_w_draw): minimal change, kept as-is

### 5. G5-1: Opponent Features Bridge (CODE FIX)
- **Bug**: `compute_features` passed `&[]` for opponent_stats → 60/280 feature dims always zero
- **Fix**: Added `extract_opponent_features()` to `RlStrategy` — bridges observer stats via `OpponentFeatures::from_player_stats()` + situational fields (stack_bb, wagered_bb, distance, is_aggressor, has_acted)
- Added `shared_observer()` accessor to `AdaptivePostflopGen4`
- Stored `Option<Arc<Mutex<Observer>>>` in `RlStrategy` for feature extraction

## Data Collection (v2)
- **Tables**: 6 × 250k hands = 1.5M total
- **Transitions**: 1,216,271 (up from 818,739 in v1)
- **Profit/hand**: +63 (Gen 4 teacher vs mixed field)
- **Profiles**: 42 opponent profile files
- **File**: `/tmp/gen5_training/all_transitions.jsonl` (1.9 GB)

## Model Training (v2)
- **Epochs**: 20
- **Samples**: 1,094,643 train / 121,628 eval
- **Loss**: 2.03 → 0.90 (converged epoch 5)
- **Eval accuracy**: 45.2% (11-class, random = 9%)
- **Previous (v1)**: 50.2% accuracy — the drop is because opponent features are now non-zero, adding complexity the small MLP struggles with
- **Model**: `models/gen5_v2.safetensors`

## Accuracy Analysis
The 45.2% plateau (down from v1's 50.2%) is expected:
- v1 had 60/280 feature dims zeroed (opponent stats) → easier to learn, but model was blind to opponent types
- v2 has all 280 dims populated → harder to learn, but model can potentially exploit opponent tendencies
- The MLP architecture (280 → hidden → 11) is likely the bottleneck
- Future: larger network (280 → 256 → 128 → 11) or attention mechanism

## Infrastructure Created

### Inference Config (`gen5_rl.toml`)
- `use_teacher = false` → model plays, Gen 4 catches illegal actions
- `record_transitions = false` → no disk writes during live play
- `model_path = "models/gen5_v2.safetensors"`
- Registered in `configs/live_bots.toml`

### Self-Play Loop (`scripts/gen5_selfplay_loop.sh`)
- DAgger-style: collect → aggregate → train → validate
- Supports `KEEP_OLD_DATA=true` for experience aggregation
- Caps dataset at 3M transitions (FIFO trimming)
- Per-iteration model saved as `models/gen5_iter{N}.safetensors`
- Usage: `./scripts/gen5_selfplay_loop.sh 3` (3 iterations)

## What Gen 5 Needs to Continue Improving

### Near-Term (Iterative Imitation)
1. **Run self-play loop** (`gen5_selfplay_loop.sh`) — automated collect→train→deploy
2. **Larger network** — current MLP too small for 280-dim input with opponent features
3. **Better teacher** — Gen 4 with fixed VPIP + new thresholds is already a better teacher

### Medium-Term (Reward-Based Training)
1. **Reward labeling** — add actual pot outcomes to transitions (currently only teacher action)
2. **Policy gradient** — train with actual rewards instead of teacher labels
3. **Experience replay** — sample from buffer, not just latest data
4. **Exploration** — epsilon-greedy or temperature sampling during self-play

### Open Issues
- **G5-2**: `train_gen5.rs` requires `--features gen5_nn` — won't compile without it
- **G5-3**: Only cross-entropy loss (behavioral cloning) — no reward signal yet
- **G5-4**: No model validation gate (deploy regardless of quality)
- **G5-5**: Small MLP architecture — needs capacity for opponent-aware features
- **G5-6**: No online learning (can't update during live play)
- **G5-7**: Action discretization is coarse (11 actions) — sizing buckets may miss optimal bet sizes

id: fae8ff26da434aaeab172970a45a6079
parent_id: 5a06903f7db44bfcb4c8c8a9cf0d2326
created_time: 2026-06-29T05:16:59.472Z
updated_time: 2026-06-30T05:34:40.425Z
is_conflict: 0
latitude: 0.00000000
longitude: 0.00000000
altitude: 0.0000
author: 
source_url: 
is_todo: 0
todo_due: 0
todo_completed: 0
source: joplin-desktop
source_application: net.cozic.joplin-desktop
application_data: 
order: 1782710219472
user_created_time: 2026-06-29T05:16:59.472Z
user_updated_time: 2026-06-29T05:16:59.472Z
encryption_cipher_text: 
encryption_applied: 0
markup_language: 1
is_shared: 0
share_id: 
conflict_original_id: 
master_key_id: 
user_data: 
deleted_time: 1782797680425
type_: 1